Image processing device, image processing method, presence display device, presence display method, and program

Abstract

To provide an image processing apparatus and method, a presence display apparatus and method, and a program that integrate an avatar animation function with a face authentication to reduce a cost, and can display a presence.SOLUTION: In a Web conference system 100, an image processing apparatus 10 comprises a face feature point information detection unit PKP that detects a plurality of feature points in a face portion each corresponding to a specific part of a face of a person in a photographed image in which the person is photographed by a camera CMR and generates one piece of feature point information indicating the plurality of feature points in the face portion, a feature amount extraction unit FCT that extracts a feature amount of the face portion based on the one piece of the feature point information, a face authentication unit FJP that authenticates an identity of the person based on the feature amount of the face portion, and an animation processing unit DCT that, when the authentication is performed by the face authentication unit, performs animation processing on an avatar image corresponding to the person based on the one piece of the feature point information and results of sequential detection of the plurality of feature points in the face portion after generating the one piece of the feature point information by the detection unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, a presence display device, a presence display method, and a program. [Background technology]

[0002] In recent years, the use of web conferencing (including video conferencing via the cloud or ASPs (Application Service Providers)) using browser applications on personal computers (PCs) has become widespread. People who do not want to show their private spaces at home are increasingly using virtual backgrounds. In addition, meeting the needs of women who want to participate in web conferences without makeup, a conferencing system has recently been released that allows participants to participate in web conferences using avatars. Web conferences are also now being held over the internet using smartphones.

[0003] Patent Document 1 describes a technology in which an image processing device extracts a portion corresponding to a specific part of the body or face in a person subject area in an image captured by an imaging device of a person, detects the person's posture or line of sight relative to the imaging device based on the feature amount of the extracted portion, determines a correction amount for the person subject area based on the detected posture or line of sight, and creates an image of the person with the posture or line of sight corrected based on the determined correction amount. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-201360 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even with the technology described in Patent Document 1, when a user joins a conference using an avatar, other participants cannot tell whether the person is actually participating, which poses a security problem. Therefore, it is necessary to add a personal authentication function to a web conference system with avatar functionality, but adding a new, independent personal authentication function not only increases processing load but also raises the issue of high costs.

[0006] With the spread of telecommuting, it is becoming increasingly difficult for businesses and companies to keep track of the work status of remote consultants and subordinates. Presence information provided by conventional presence display devices equipped with office communication tools such as chat tools, internal bulletin boards, internal social networking services (SNS), and groupware is displayed only as marks determined by the frequency of device operation, making it difficult to instantly identify the presence of a person. Another issue is that even if someone other than the person in question is operating the device, the presence of the person in question is displayed as the person in question.

[0007] The present invention has been made in consideration of the above-mentioned problems with the conventional technology, and aims to provide an image processing device, image processing method, presence display device, presence display method, and program that can display presence by combining with facial authentication, integrating an avatar animation function with facial authentication, thereby reducing costs without providing an independent personal authentication function. [Means for solving the problem]

[0008] The image processing device of the present invention is characterized by comprising: a detection unit that detects a plurality of feature points of a facial part corresponding to specific parts of the face of a person in an image captured by an imaging device, and generates a single piece of feature point information indicating the plurality of feature points of the facial part; an extraction unit that extracts feature amounts of the facial part based on the single feature point information; a face authentication unit that authenticates the identity of the person based on the feature amounts of the facial part; and an animation processing unit that, when authentication is performed by the face authentication unit, performs animation processing of an avatar image corresponding to the person based on the single feature point information and the results of sequential detection of the plurality of feature points of the facial part after the detection unit generates the single feature point information.

[0009] The image processing method of the present invention is characterized by including: a detection step of detecting a plurality of feature points of a facial portion corresponding to specific parts of the face of a person in an image captured by an imaging device, and generating a single piece of feature point information indicating the plurality of feature points of the facial portion; an extraction step of extracting feature amounts of the facial portion based on the single feature point information; a face authentication step of authenticating the person's identity based on the feature amounts of the facial portion; and an animation processing step of, when authentication is performed by the face authentication step, performing animation processing of an avatar image corresponding to the person based on the single feature point information and the results of sequential detection of the plurality of feature points of the facial portion after generation of the single feature point information by the detection step.

[0010] The program of the present invention causes a computer mounted on an image processing device to function as: a detection unit that detects multiple feature points of a facial part corresponding to specific parts of the face of a person in an image captured by an imaging device, and generates a single piece of feature point information indicating the multiple feature points of the facial part; an extraction unit that extracts feature amounts of the facial part based on the single feature point information; a face authentication unit that authenticates the person's identity based on the feature amounts of the facial part; and an animation processing unit that, when authentication is performed by the face authentication unit, performs animation processing of an avatar image corresponding to the person based on the single feature point information and the results of sequential detection of the multiple feature points of the facial part after the detection unit generates the single feature point information.

[0011] a detection unit that detects a plurality of feature points of a facial portion corresponding to specific parts of the face of a person in a captured image of the person captured by an imaging device, and generates a single piece of feature point information indicating the plurality of feature points of the facial portion; an extraction unit that extracts feature amounts of the facial portion based on the single feature point information; a face authentication unit that authenticates the identity of the person based on the feature amounts of the facial portion; an animation processing unit that, when authentication is performed by the face authentication unit, executes animation processing of an avatar image corresponding to the person based on the single feature point information and the result of sequential detection of the plurality of feature points of the facial portion after generation of the single feature point information by the detection unit; and a presence information providing unit that transmits presence information of the person, including the avatar image, to at least a terminal device other than the person that captured the image, based on the authentication result of the face authentication unit and the result of sequential detection of the plurality of feature points of the facial portion.

[0012] a presence display method for displaying presence information indicating the status of a person communicating via a terminal device, the presence display method comprising: a detection step for detecting a plurality of feature points of a facial portion corresponding to specific parts of the face of the person in a captured image of the person captured by an imaging device, and generating one piece of feature point information indicating the plurality of feature points of the facial portion; an extraction step for extracting feature amounts of the facial portion based on the one feature point information; a face authentication step for authenticating the person's identity based on the feature amounts of the facial portion; an animation processing step for, if authentication is performed in the face authentication step, performing animation processing of an avatar image corresponding to the person based on the one feature point information and results of sequential detection of the plurality of feature points of the facial portion after generation of the one feature point information in the detection step; and a presence information providing step for transmitting presence information of the person, including the avatar image, to at least a terminal device other than the person who captured the image, based on the authentication result of the face authentication step and the results of sequential detection of the plurality of feature points of the facial portion.

[0013] The program of the present invention causes a computer mounted on a presence display device to function as: a detection unit that detects a plurality of feature points in a facial portion of a person in an image captured by an imaging device, the feature points corresponding to specific parts of the person's face, and generates a piece of feature point information indicating the plurality of feature points in the facial portion; an extraction unit that extracts feature amounts of the facial portion based on the one feature point information; a face authentication unit that authenticates the person's identity based on the feature amounts of the facial portion; an animation processing unit that, when authentication is performed by the face authentication unit, animates an avatar image corresponding to the person based on the one feature point information and the result of sequential detection of the plurality of feature points in the facial portion after generation of the one feature point information by the detection unit; and a presence information providing unit that transmits presence information of the person, including the avatar image, to at least a terminal device other than the person that captured the image, based on the authentication result of the face authentication unit and the result of sequential detection of the plurality of feature points in the facial portion. [Effects of the Invention]

[0014] According to the present invention, an image processing device can be obtained that can reduce costs by not providing an independent personal authentication function. Also, according to the present invention, a presence display device can be obtained that can display the presence information of each person as visual and highly reliable information by displaying an avatar as the presence information if the person in question is present through face recognition, and displaying the person as away if the face cannot be detected or if someone other than the person in question is operating the device. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of an image processing apparatus according to a first embodiment of the present invention. [Figure 2] FIG. 10 is a schematic diagram illustrating an example of the configuration of a presence display device according to a second embodiment of the present invention. [Figure 3] 10 is a flowchart showing the operation procedure of the presence display device of the second embodiment. [Figure 4] FIG. 10 is a schematic diagram illustrating an example of an image displayed by the presence display device of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the following embodiments and examples, components having substantially the same functions and configurations are designated by the same reference numerals, and redundant description will be omitted.

[0017] (First Example) (Configuration explanation) 1 is a schematic block diagram showing an example of the configuration of a Web conference system 100 including an image processing device 10 of this embodiment. The system configuration includes the image processing device 10 that functions as a communication terminal 101 and is capable of communication.

[0018] The image processing device 10, i.e., a communication terminal 101, is connected to the Internet via a network NW so as to be able to communicate with the Internet. Examples of the communication terminal 101 include electronic devices such as desktop PCs, smartphones, tablet terminals, and notebook PCs.

[0019] The network NW may include a mobile phone network (LTE (Long Term Evolution) or 4G (fourth generation mobile communication standard) or 5G (fifth generation mobile communication standard)), an IP (Internet Protocol) network, a public switched telephone network, or, for example, a public network such as a satellite communication network, various WANs (Wide Area Networks), dedicated network such as VoPN (Voice over Packet Network), or an individual LAN (Local Area Network).

[0020] [Image processing device] 1, the image processing device 10 is a computer device that is connected to an imaging device such as a camera CMR and functions as an image processing device, and has a CPU (Central Processing Unit) 11, a storage device 12, an input unit 13 and an output unit 14 that are interfaces for a user who operates the device, and a communication unit 15, all of which are connected to each other via an internal bus (not shown). Input information and the like is acquired by the image processing device 10 via the input unit 13 and the communication unit 15.

[0021] The storage device 12 may include, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), an SSD (Solid State Drive), and a flash memory. Except for the ROM, these devices store various programs and data and serve as a work area for the CPU 11. The input unit 13 may include a camera CMR for inputting various data, and (not shown) a microphone, keyboard, mouse, touch panel, numeric keypad, and the like. The output unit 14 may include a display, touch panel, and the like for displaying the status to a user who inputs the data. The communication unit 15 communicates with the network NW and transmits and receives data. The input unit 13 and the output unit 14 may also include a USB (Universal Serial Bus) for connecting an external device, such as a recording medium.

[0022] The overall operation of the image processing device 10 is controlled by a CPU 11 that follows an operating system (OS) software program installed in the storage device 12. The CPU 11 executes image processing according to various application software programs stored in the storage device 12, for example, according to image processing application software.

[0023] The installed image processing application software is loaded into the storage device 12, and the CPU 11 operates the following functional units: (1) A facial feature point information detection unit PKP; (2) Feature extraction unit FCT; (3) Registration Unit TIP; (4) Facial recognition unit FJP, (5) It functions as an animation processing unit DCT.

[0024] Each functional unit of the image processing device 10 functions as follows.

[0025] [Facial feature point information detection unit PKP] The facial feature point information detection unit PKP detects multiple feature points in the facial area corresponding to specific parts of the face of a person, such as a user, in an image captured by an imaging device, and generates facial feature point information indicating the multiple feature points in the facial area.

[0026] The facial feature point information detection unit PKP first performs face detection processing to find faces in the captured image containing the captured face. When the captured image is read, the face detection processing searches the captured image for a pattern of light and dark arrangement like a light and dark template of a face, and detects and identifies the face as a rectangular area (frame) that surrounds the approximate position of the face.

[0027] The facial feature point information detection unit PKP then performs facial landmark detection processing, in other words, processing to find organ points (feature points) such as the eyes, eyebrows, nose, mouth, and contours of the face found by face detection. The facial feature point information detection unit PKP acquires a lot of information such as detailed face position, size, orientation, and facial expression.

[0028] The facial feature point information detection unit PKP, for example, identifies multiple feature points for each part of the face in a user image captured of the user, and identifies feature data based on the positions of these feature points—specifically, facial feature points calculated based on the position coordinates of the feature points—as feature data (e.g., coordinates, distances between coordinates, etc.). For example, the facial feature point information detection unit PKP identifies multiple feature points for parts corresponding to the left and right eyes (e.g., six points on the upper and lower eyelid margins and the inner and outer corners of each eye) and identifies eye features calculated based on the positions of these feature points as eye feature data. Similarly, for the left and right eyebrows, five points are identified between the inner and outer eyebrow ends for each eyebrow, and these are identified as eyebrow feature data. For the nose, nine points are identified on the bridge of the nose between the upper and lower roots, tip, and columella, and between the left and right nostril wings, and these are identified as nose feature data. For the upper and lower lips of the mouth, multiple feature points (e.g., eight points on each lip) are identified, and mouth features consisting of the upper and lower lips calculated based on the positions of these feature points are identified as mouth feature data. Similarly, for the facial contour of the cheeks and lower jaw, 17 points are identified on the contour, and these are identified as contour feature data. A collection of these feature data is taken as facial feature point information.

[0029] When the facial feature point information detection unit PKP acquires facial feature point information, it stores it in the feature table of the registration unit TIP together with the registration date and time and for each user ID. Note that the number of feature points identified by the facial feature point information detection unit PKP is not limited to the above example and may be changed depending on the terminal or environment used.

[0030] The facial feature point information detection unit PKP then performs face normalization processing on the facial image to convert the size and orientation of the face to meet certain conditions. For example, face rotation correction is performed based on the results of facial feature point detection, and normalization processing is performed to obtain a size of, for example, 1000 x 1000 pixels. Note that advanced normalization processing may also convert a diagonally facing face to a frontal orientation, or correct the color and contrast of the image.

[0031] After the facial feature point information is stored in the registration unit TIP, the facial feature point information detection unit PKP sequentially detects multiple feature points in the facial area and extracts the amount of variation in the feature points in order to have the animation processing unit DCT perform animation processing of the avatar image corresponding to the user.

[0032] [Feature Extraction Unit FCT] The feature extraction unit FCT performs face extraction processing to convert the facial feature point information obtained from the facial feature point information detection unit PKP into numerical data called features so that facial recognition processing can be performed, i.e., so that it is easier to determine who the face image represents.

[0033] For example, the feature extraction unit FCT converts facial feature point information into features of a string of numerical data, for example, about 1,000 pieces. Deep learning is used in most current facial recognition systems for feature extraction. Deep learning is a machine learning technique that teaches computers to perform tasks typically performed by humans, such as image identification, classification, and prediction. It is machine learning that uses deep neural networks, which prepare large amounts of data and allow the machine to automatically extract features from the data. Deep neural networks are algorithms that use algorithms modeled on the neural circuits of the human and animal brains, and are multi-layered neural networks designed for pattern recognition.

[0034] In this way, the feature extraction unit FCT extracts the feature amounts of the face part based on the facial feature point information, and stores the calculated feature amounts together with the facial feature point information in a feature table of the registration unit TIP.

[0035] [Registration Tips] The registration unit TIP can store data of predetermined settings of the device in addition to the feature table (feature amounts, facial feature point information, registration date and time, user ID). The registration unit TIP stores feature amounts obtained from a user image captured in advance of the user, which are referenced to determine (authenticate) the legitimacy of the user of the image processing device 10, as well as a basic avatar image (described later) of the user.

[0036] [Facial Recognition Department FJP] The face authentication unit FJP performs authentication processing to verify the identity of the user based on the feature amount of the face.

[0037] When new features calculated by the facial feature point information detection unit PKP and the feature extraction unit FCT from a new captured image of a user operating the camera CMR are given to the face authentication unit FJP, the face authentication unit FJP calculates a matching score as a numerical value for determining whether or not the new features and the features stored in the feature table of the registration unit TIP are the same person, and determines whether or not the same person is present based on whether or not the score exceeds a predetermined threshold. Here, the larger the matching score, the more likely it is that the person is the same.

[0038] That is, the face recognition unit FJP calculates a matching score from two feature amounts, and searches for and identifies the same person with feature amounts having a large matching score in the registration unit TIP. Note that when the face recognition unit FJP uses dissimilarity instead of the matching score, the larger the dissimilarity, the more likely it is that the person is a different person, so whether or not the person is the same is determined based on whether or not the dissimilarity is below a threshold value.

[0039] [Animation processing section DCT] The animation processor DCT animates an avatar image corresponding to the user based on the facial feature point information of the user who has undergone face authentication and the results of sequential detection of multiple facial feature points by the facial feature point information detector PKP. The form of the avatar image is not limited, and it may be, for example, a two-dimensional image or a three-dimensional image created from a two-dimensional image of the user.

[0040] For example, the animation processing unit DCT generates an avatar image of the person by matching the person's facial authentication information and facial feature point information sent from the face authentication unit FJP and the facial feature point information detection unit PKP with a three-dimensional model of a basic avatar image prepared in advance.

[0041] The 3D avatar model stores a shape key that moves the vertices of the 3D model's mouth to a specific position when, for example, the character opens their mouth. This allows the animation processor DCT to animate the avatar opening their mouth, closing their eyes, or shaking their head in response to the character opening their mouth, closing their eyes, or shaking their head. For example, morphing can be used to animate the avatar's mouth as it moves from an open to a closed state.

[0042] In other words, the animation processor DCT analyzes facial expressions from the amount of fluctuation of feature points extracted by the facial feature point information detector PKP (results of sequential detection of multiple feature points) and synthesizes the images as animation.The animation processor DCT then outputs the avatar image to the participating Web conference via the communication unit 15.As a result, the user's avatar image is displayed on the display screen of the communication terminal participating in the Web conference.

[0043] The personal authentication method according to this embodiment contributes to the integration of avatar creation and face authentication processing by utilizing facial feature point information extracted for avatar creation in face authentication.

[0044] (Explanation of effect) In this embodiment, the facial feature point information detection unit PKP extracts facial feature points using the avatar function, and the facial recognition unit FJP digitizes the facial feature points and uses them for face recognition. Therefore, even if a user participates in a conference using an avatar, it is possible to confirm their identity through face recognition. This makes it possible to authenticate a user without adding a new sensor or an independent identity recognition function, simplifying processing and reducing costs.

[0045] (Second Example) In the first embodiment, an example of an image processing device applied to a Web conference was described, but in the second embodiment, a presence display device 103 has the same configuration as the image processing device of the first embodiment, except for the addition of a presence information providing unit PIP. Therefore, only the configuration that differs from the first embodiment will be described. This embodiment can be applied to, for example, an office communication tool (a presence display device as a Web server) connected via a VPN (Virtual Private Network).

[0046] FIG. 2 is a schematic diagram illustrating an example of the configuration of the presence display device 103 of the second embodiment.

[0047] The presence information providing unit PIP of the presence display device 103 has the function of transmitting user presence information, including the user's avatar image, to at least one communication terminal 101 other than the communication terminal 101 that has captured an image of a certain user, via the communication unit 15, based on the authentication result of the face authentication unit FJP and the results of sequential detection of multiple feature points in the face by the face feature point information detection unit PKP.

[0048] In this embodiment, the presence information providing unit PIP can automatically determine the presence based on the facial feature point information detected by the facial feature point information detecting unit PKP.

[0049] (Explanation of operation) FIG. 3 is a flowchart showing the operation procedure of the presence display device of the second embodiment.

[0050] First, the CPU 11 (presence information providing unit PIP) determines whether the power of each communication terminal 101 participating in the conference is ON (OFF) (step S1). If the power is OFF (step S1: N), the presence is set to "business end" and the avatar is hidden (step S2).

[0051] If the power is ON (step S1: Y), it is determined whether the camera of each communication terminal is ON (OFF) (step S3). If the camera is OFF (step S3: N), the presence is unknown and the avatar is hidden (step S4).

[0052] If the camera is ON (step S3: Y), it is determined whether or not a face can be detected via the camera (step S5). If a face cannot be detected via the camera (step S5: N), the presence is set to "Away" and the avatar is hidden (step S6).

[0053] If a face can be detected via the camera (step S5: Y), it is determined whether the face detected through facial recognition matches the person who has logged in (step S7). If the face recognition result does not match the person who is logged in (step S7: N), it is determined whether anyone else has been identified (step S8). If no one other than the person has been identified (step S8: N), the presence is set to away, and a non-human character (such as an animal or a default image) is displayed as the avatar (step S9).

[0054] If the face recognition result does not match the person who is logged in, but there is another person who can be identified (step S8: Y), the presence is "Away" and the avatar of the identified person (whose facial expression is not controlled) is displayed (step S10).

[0055] As a result of face recognition, if the face matches that of the person who is logged in (face authentication) (step S7: Y), the feature points of the person's face are detected and it is determined whether or not there is a predetermined movement in the face within a predetermined time (step S11).If the result of detecting the feature points of the person's face (facial feature point information detection unit PKP) shows that there is no predetermined movement in the facial expression (step S11: N), the presence is displayed as "Available" (contactable) and the avatar shows the person (without predetermined facial expression control) (step S12).

[0056] If the result of detecting the facial feature points of the person is that there is a predetermined movement in the facial expression (step S11: Y), the presence is set to "present" (taking in), and the avatar displays the person (with facial expression controlled according to the facial movement) (step S13).

[0057] FIG. 4 is a schematic diagram illustrating an example of an image displayed on the communication terminal 101 by the presence display device 103 of the second embodiment.

[0058] (Explanation of effect) According to this embodiment, if the person is present, the presence information can be displayed as an avatar by facial recognition. When determining presence, whether the person is present or absent can be determined by whether the face is detected. Therefore, if the face cannot be detected or if someone other than the person is operating the device, the presence information of each person can be displayed as absent, making it possible to display the presence information of each person as visually reliable information.

[0059] According to this embodiment, some users may feel reluctant to turn on the camera while operating the communication terminal 101 from the viewpoint of privacy, etc. In this embodiment, the user's image is displayed as an avatar rather than as a presence indicator, making it easy to use for people who do not want to show their room when working from home, as well as their own face. Furthermore, facial feature point information makes it possible to determine whether each user is busy or available for contact. [Explanation of symbols]

[0060] 10 Image processing device 101 Communication terminal 103 Presence display device PKP Facial feature point information detection unit FCT feature extraction unit FJP Face Recognition Department DCT animation processing section PIP Presence Information Unit

Claims

1. a detection unit that detects a plurality of facial feature points corresponding to specific parts of a person's face in an image captured by an imaging device, generates facial feature point information for creating an avatar image indicating the plurality of facial feature points, sequentially detects the plurality of facial feature points for executing animation processing of the avatar image, and extracts amounts of variation of the feature points in the facial feature point information; an extraction unit that converts the facial feature point information into a feature amount of the facial part; a registration unit that stores, in advance, the facial feature point information and the facial feature amounts corresponding to the person, and an avatar image corresponding to the person, for each person; a face authentication unit that, when a new feature quantity digitized by the extraction unit is given from a newly captured image of the face of an operating person via the detection unit, performs face authentication of the operating person by a face authentication process that determines whether the operating person is the same person as any of the persons whose feature quantities of the face parts stored in the registration unit, based only on the new feature quantity and the feature quantities of the face parts stored in the registration unit; an animation processing unit that, only when the identity of the operating person is authenticated by the face authentication unit, executes animation processing of a plurality of feature points of a face portion of an avatar image corresponding to the operating person, based on the facial feature point information of the operating person stored in the registration unit and the result of the amount of variation in the sequential detection of a plurality of feature points of the face portion after the detection unit generates the facial feature point information of the operating person; 1. An image processing device comprising:

2. An image processing method performed by an image processing device, comprising: a detection step of detecting a plurality of facial feature points corresponding to specific parts of a person's face in a captured image of the person captured by an imaging device, generating facial feature point information for creating an avatar image indicating the plurality of facial feature points, sequentially detecting the plurality of facial feature points for executing animation processing of the avatar image, and extracting a variation amount of the feature points in the facial feature point information; an extraction step of converting the facial feature point information into a feature amount of the facial part; a registration step of storing, in advance, the facial feature point information and the facial feature amounts corresponding to the person, and an avatar image corresponding to the person, for each person; a face authentication step of authenticating the identity of a person who operates the device by performing face authentication processing to determine, when a new feature quantity digitized in the extraction step is given from a newly captured image of the face of the person who operates the device through the detection step, whether the person who operates the device is the same person as any of the persons whose feature quantities of the face parts stored in the registration step are based only on the new feature quantity and the feature quantities of the face parts stored in the registration step; an animation processing step of performing animation processing of a plurality of feature points of a face portion of an avatar image corresponding to the operating person, based on the facial feature point information of the operating person stored in the registration step and the result of the amount of variation in the sequential detection of a plurality of feature points of the face portion after the facial feature point information of the operating person is generated in the detection step, only if the identity of the operating person is authenticated in the face authentication step; An image processing method comprising:

3. The computer installed in the image processing device is a detection unit that detects a plurality of facial feature points corresponding to specific parts of a person's face in an image captured by an imaging device, generates facial feature point information for creating an avatar image indicating the plurality of facial feature points, sequentially detects the plurality of facial feature points for executing animation processing of the avatar image, and extracts amounts of variation of the feature points in the facial feature point information; an extraction unit that converts the facial feature point information into a feature amount of the facial part; a registration unit that stores, in advance, the facial feature point information and the facial feature amounts corresponding to the person, and an avatar image corresponding to the person, for each person; a face authentication unit that, when a new feature quantity digitized by the extraction unit is given from a newly captured image of the face of an operating person via the detection unit, performs face authentication of the operating person by a face authentication process that determines whether the operating person is the same person as any of the persons whose feature quantities of the face parts stored in the registration unit, based only on the new feature quantity and the feature quantities of the face parts stored in the registration unit; A program that functions as an animation processing unit for multiple feature points in the facial part of an avatar image corresponding to the operating person, based on the facial feature point information of the operating person stored in the registration unit and the result of the amount of variation in the sequential detection of multiple feature points in the facial part after the detection unit generates the facial feature point information of the operating person, only when the face authentication unit has authenticated the operating person.

4. A presence display device for displaying presence information indicating the status of a person communicating via a terminal device, comprising: a detection unit that detects a plurality of facial feature points corresponding to specific parts of a person's face in an image captured by an imaging device, generates facial feature point information for creating an avatar image indicating the plurality of facial feature points, sequentially detects the plurality of facial feature points for executing animation processing of the avatar image, and extracts amounts of variation of the feature points in the facial feature point information; an extraction unit that converts the facial feature point information into a feature amount of the facial part; a registration unit that stores, in advance, the facial feature point information and the facial feature amounts corresponding to the person, and an avatar image corresponding to the person, for each person; a face authentication unit that, when a new feature quantity digitized by the extraction unit is given from a newly captured image of the face of an operating person via the detection unit, performs face authentication of the operating person by a face authentication process that determines whether the operating person is the same person as any of the persons whose feature quantities of the face parts stored in the registration unit, based only on the new feature quantity and the feature quantities of the face parts stored in the registration unit; an animation processing unit that, only when the identity of the operating person is authenticated by the face authentication unit, executes animation processing of a plurality of feature points of a face portion of an avatar image corresponding to the operating person, based on the facial feature point information of the operating person stored in the registration unit and the result of the amount of variation in the sequential detection of a plurality of feature points of the face portion after the detection unit generates the facial feature point information of the operating person; a presence information providing unit that transmits presence information indicating whether the operating person is present or absent, including the avatar image, to at least a terminal device other than the operating person who captured the image, based on the identity authentication result of the face authentication unit and the result of the amount of variation in the sequential detection of a plurality of feature points of the face part; A presence display device comprising:

5. A presence display method performed by a presence display device for displaying presence information indicating the status of a person communicating via a terminal device, comprising: a detection step of detecting a plurality of facial feature points corresponding to specific parts of a person's face in a captured image of the person captured by an imaging device, generating facial feature point information for creating an avatar image indicating the plurality of facial feature points, sequentially detecting the plurality of facial feature points for executing animation processing of the avatar image, and extracting a variation amount of the feature points in the facial feature point information; an extraction step of converting the facial feature point information into a feature amount of the facial part; a registration step of storing, in advance, the facial feature point information and the facial feature amounts corresponding to the person, and an avatar image corresponding to the person, for each person; a face authentication step of authenticating the identity of a person who operates the device by performing face authentication processing to determine, when a new feature quantity digitized in the extraction step is given from a newly captured image of the face of the person who operates the device through the detection step, whether the person who operates the device is the same person as any of the persons whose feature quantities of the face parts stored in the registration step are based only on the new feature quantity and the feature quantities of the face parts stored in the registration step; only when the identity of the operating person is authenticated in the face authentication step, an animation processing step of a plurality of feature points of a face portion of an avatar image corresponding to the operating person, based on the facial feature point information of the operating person stored in the registration step and the result of the amount of variation in the sequential detection of a plurality of feature points of the face portion after the facial feature point information of the operating person is generated in the detection step; a presence information providing step of transmitting presence information indicating whether the operating person is present or absent, including the avatar image, to at least a terminal device other than the operating person who captured the image, based on the authentication result of the face authentication step and the result of the amount of variation in the sequential detection of a plurality of feature points of the face part; A presence display method comprising:

6. A computer installed in the presence display device, a detection unit that detects a plurality of facial feature points corresponding to specific parts of a person's face in an image captured by an imaging device, generates facial feature point information for creating an avatar image indicating the plurality of facial feature points, sequentially detects the plurality of facial feature points for executing animation processing of the avatar image, and extracts amounts of variation of the feature points in the facial feature point information; an extraction unit that converts the facial feature point information into a feature amount of the facial part; a registration unit that stores, in advance, the facial feature point information and the facial feature amounts corresponding to the person, and an avatar image corresponding to the person, for each person; a face authentication unit that, when a new feature quantity digitized by the extraction unit is given from a newly captured image of the face of an operating person via the detection unit, performs face authentication of the operating person by a face authentication process that determines whether the operating person is the same person as any of the persons whose feature quantities of the face parts stored in the registration unit, based only on the new feature quantity and the feature quantities of the face parts stored in the registration unit; an animation processing unit that, only when the identity of the operating person is authenticated by the face authentication unit, executes animation processing of a plurality of feature points of a face portion of an avatar image corresponding to the operating person, based on the facial feature point information of the operating person stored in the registration unit and the result of the amount of variation in the sequential detection of a plurality of feature points of the face portion after the detection unit generates the facial feature point information of the operating person; A program that functions as a presence information providing unit that transmits presence information, including the avatar image, of the operating person, whether the operating person is present or absent, to at least a terminal device other than the operating person who took the image, based on the result of the identity authentication by the face authentication unit and the result of the amount of variation in the sequential detection of multiple feature points in the face.

Images