Information processing device, online conference system, information processing method, and computer program

Abstract

An information processing device 1 comprises: an acquisition unit 11 that acquires information pertaining to a person, including at least an image of that person; a detection unit 12 that detects a facial region including the face of the person in the image; a region estimation unit 13 that, if the facial region is at least partially shielded, estimates a shielded region; a facial expression estimation unit 14 that estimates a facial expression of the person on the basis of the information pertaining to the person; an estimated facial expression image generation unit 15 that generates an estimated facial expression image of a region corresponding to the shielded region according to the facial expression estimated by the facial expression estimation unit 14; and a combined image generation unit 16 that generates a combined image on the basis of the image and the estimated facial expression image.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] Patent Literature 1 describes a technology for determining occluded areas in an input image representing a face, identifying the input image using areas other than those associated with an occlusion pattern based on the occluded areas, and further improving the recognition accuracy of facial images including occluded areas. Patent Literature 2 describes a technology for inputting a facial image, detecting areas in the facial image including features such as the eyes, nose, mouth, and cheeks, filling in the detected feature areas, and synthesizing pre-stored images of the features with the face image with the filled feature areas. Patent Literature 3 describes a technology for capturing a frontal image (moving image) of a user through the head-mounted display from a camera fixed to the head-mounted display, using the facial area not hidden by the head-mounted display in the moving image as is, and replacing the hidden area with an area cut out using a mask pattern of the head-mounted display from a still image captured in advance from the same viewpoint and stored in storage means when the head-mounted display is not being worn, and pasting the facial image synthesized from the moving image and still image onto the surface of an appropriate three-dimensional object such as a cube using a texture map technique, and outputting or displaying the resulting image as a person's head.

[0003] Japanese Patent Application Laid-Open No. 2021-103538 Japanese Patent Application Laid-Open No. 2002-352258 Japanese Patent Application Laid-Open No. 11-096366

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the techniques described in prior art documents.

[0005] One aspect of the information processing device disclosed herein comprises an acquisition means for acquiring information about a person including at least an image of the person, a detection means for detecting a facial area including the face of the person from the image, an estimation means for estimating an occluded area when at least a portion of the facial area is occluded, an expression estimation means for estimating an expression of the person based on information about the person, an estimated expression image generation means for generating an estimated expression image of an area corresponding to the occluded area according to the expression estimated by the expression estimation means, and a composite image generation means for generating a composite image based on the image and the estimated expression image.

[0006] One aspect of the information processing method disclosed herein includes obtaining information about a person including at least an image of the person, detecting a facial area including the person's face from the image, estimating the occluded area if at least a portion of the facial area is occluded, estimating the facial expression of the person based on the information about the person, generating an estimated facial expression image of the area corresponding to the occluded area according to the estimated facial expression, and generating a composite image based on the image and the estimated facial expression image.

[0007] One aspect of a recording medium of this disclosure is a recording medium having a computer program recorded thereon for causing a computer to execute an information processing method of acquiring information about a person including at least an image of the person, detecting a facial area including the person's face from the image, estimating an occluded area if at least a portion of the facial area is occluded, estimating a facial expression of the person based on the information about the person, generating an estimated facial expression image of an area corresponding to the occluded area according to the estimated facial expression, and generating a composite image based on the image and the estimated facial expression image.

[0008] FIG. 1 is a block diagram showing the configuration of an information processing device in a first embodiment. FIG. 2 is a block diagram showing the configuration of an information processing device in a second embodiment. FIG. 3 is a flowchart showing the flow of information processing operations performed by the information processing device in the second embodiment. FIG. 4 is a block diagram showing the configuration of an information processing device in a fourth embodiment. FIG. 5 is a flowchart showing the flow of learning operations performed by the information processing device in the fourth embodiment. FIG. 6 is a flowchart showing the flow of estimated facial expression image generation operations performed by the information processing device in the fifth embodiment. FIG. 7 is a block diagram showing the configuration of an information processing device in a sixth embodiment. FIG. 8 is a conceptual diagram showing a display example based on display control by the information processing device in the sixth embodiment. FIG. 9 is a conceptual diagram of an online conference system in a seventh embodiment. FIG. 10 is a block diagram showing the configuration of an online conference control device in the seventh embodiment. FIG. 11 is a flowchart showing the flow of online conference control operations performed by the online conference control device in the seventh embodiment.

[0009] Hereinafter, an information processing device, an information processing method, and a recording medium according to an embodiment will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 to which the first embodiment of the information processing device, the information processing method, and the recording medium is applied. [1-1: Configuration of Information Processing Device 1]

[0011] The configuration of an information processing device 1 in the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1 in the first embodiment.

[0012] As shown in FIG. 1, the information processing device 1 includes an acquisition unit 11, a detection unit 12, an area estimation unit 13, a facial expression estimation unit 14, an estimated facial expression image generation unit 15, and a composite image generation unit 16. The acquisition unit 11 acquires information about a person including at least an image of the person. The detection unit 12 detects a facial area including the person's face from the image. The area estimation unit 13 estimates an occluded area when at least a portion of the facial area is occluded. The facial expression estimation unit 14 estimates the facial expression of the person based on the information about the person. The estimated facial expression image generation unit 15 generates an estimated facial expression image of an area corresponding to the occluded area according to the facial expression estimated by the facial expression estimation unit 14. The composite image generation unit 16 generates a composite image based on the image and the estimated facial expression image. [1-2: Technical Effects of the Information Processing Device 1]

[0013] The information processing device 1 in the first embodiment generates a composite image based on an image and an image corresponding to the estimated facial expression of a person, so that even when at least a portion of the facial region of the person is occluded, an image corresponding to the facial expression of the person (i.e., a composite image) in which the facial region of the person is not occluded can be obtained. [2: Second Embodiment]

[0014] A second embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the second embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 2 to which the second embodiment of the information processing device, the information processing method, and the recording medium is applied. [2-1: Configuration of Information Processing Device 2]

[0015] The configuration of the information processing device 2 in the second embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the information processing device 2 in the second embodiment.

[0016] 2 , the information processing device 2 includes a calculation device 21 and a storage device 22. The information processing device 2 may further include a communication device 23, an input device 24, and an output device 25. However, the information processing device 2 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0017] The arithmetic device 21 includes, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 21 reads a computer program. For example, the arithmetic device 21 may read a computer program stored in the storage device 22. For example, the arithmetic device 21 may read a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (e.g., an input device 24, described later) not shown in the drawings that is included in the information processing device 2. The arithmetic device 21 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the information processing device 2 via the communication device 23 (or another communication device). The arithmetic device 21 executes the read computer program. As a result, logical functional blocks for executing the operations to be performed by the information processing device 2 are realized within the arithmetic device 21. In other words, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing the operations (in other words, processing) to be performed by the information processing device 2.

[0018] 2 shows an example of logical functional blocks implemented within the arithmetic device 21 for executing information processing operations. As shown in FIG. 2, the arithmetic device 21 includes an acquisition unit 211, which is a specific example of "acquisition means" described in the appendix, a detection unit 212, which is a specific example of "detection means" described in the appendix, a region estimation unit 213, which is a specific example of "estimation means" described in the appendix, a facial expression estimation unit 214, which is a specific example of "facial expression estimation means," an estimated facial expression image generation unit 215, which is a specific example of "estimated facial expression image generation means" described in the appendix, and a composite image generation unit 216, which is a specific example of "synthetic image generation means" described in the appendix. The operations of the acquisition unit 211, the detection unit 212, the region estimation unit 213, the facial expression estimation unit 214, the estimated facial expression image generation unit 215, and the composite image generation unit 216 will be described below with reference to FIG. 3.

[0019] The storage device 22 can store desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that the arithmetic device 21 temporarily uses when the arithmetic device 21 is executing a computer program. The storage device 22 may store data that the information processing device 2 stores long-term. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-temporary recording medium.

[0020] The communication device 23 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown).

[0021] The input device 24 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 24 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0022] The output device 25 is a device that outputs information to the outside of the information processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (a so-called speaker) that can output sound. For example, the output device 25 may output information on paper. That is, the output device 25 may include a printing device (a so-called printer) that can print desired information on paper. [2-2: Information Processing Operation Performed by the Information Processing Device 2]

[0023] The flow of information processing operations performed by the information processing device 2 in the second embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of information processing operations performed by the information processing device 2 in the second embodiment.

[0024] 3, the acquisition unit 211 acquires information about a person including at least an image of the person (step S20). In addition to the image of the person, the acquisition unit 211 may also acquire, as the information about the person, audio information acquired when the image of the person is generated, for example.

[0025] The detection unit 212 detects a face region including a person's face from the image (step S21). The detection unit 212 may detect the face region by applying a known face detection process to the image. The detection unit 212 may detect a region having facial features as the face region. The region having facial features may be a region including characteristic parts that make up the face, such as the eyes, nose, and mouth. There are no particular limitations on the method used by the detection unit 212 to detect the face region. For example, the detection unit 212 may detect the face region based on extraction of edges or patterns that are characteristic of the face region.

[0026] The detection unit 212 may detect a face region using a neural network that has undergone machine learning to detect a face region, and may be configured with a convolutional neural network (hereinafter also referred to as "CNN").

[0027] When at least a portion of the face region is occluded, the region estimation unit 213 estimates the occluded region (step S22). In the second embodiment, the occluded region where at least a portion of the face region is occluded may be a mask region occluded by a mask worn by a person. When at least a portion of the face region is occluded by a mask worn by a person, the region estimation unit 213 may estimate the occluded mask region. For example, when feature points such as the wings of the nose and the corners of the mouth are not detected from the face region, the region estimation unit 213 may determine that the face region includes a mask region. The mask region hidden by the mask may be a predetermined region including the wings of the nose, the corners of the mouth, etc.

[0028] The facial expression estimation unit 214 estimates the facial expression of the person based on information about the person (step S23). When at least a portion of the face region is obscured by a mask worn by the person, the facial expression estimation unit 214 may use information obtainable from other areas of the face region as information about the person. In this case, the facial expression estimation unit 214 may estimate the facial expression of the person based on, for example, information obtainable from other areas of the face region than the mask region. Furthermore, the facial expression estimation unit 214 may estimate the facial expression of the person based on at least one of the angle of the face, the pose of the person, and the gesture performed by the person, in addition to or instead of the information obtainable from other areas of the face region than the mask region. Furthermore, the facial expression estimation unit 214 may estimate the facial expression of the person based on audio information obtained when the image of the person was generated, in addition to or instead of information obtainable from the image of the person. The audio information may include at least one of information indicating the state of speech and information indicating the content of speech. The state of speech may include at least one of the tone and tempo of speech. Furthermore, the facial expression estimation unit 214 may estimate a person's facial expression based on information indicating the surrounding circumstances when the image of the person was generated, in addition to or instead of information about the person himself / herself. The facial expression estimation unit 214 may adopt, as the information about the person, information that improves the estimation accuracy of the person's facial expression.

[0029] The facial expression estimation unit 214 may estimate the facial expression of a person based on, for example, a predetermined rule. For example, the facial expression of a person may be estimated based on the state of facial muscle movement. The state of facial muscle movement may include at least one of a state of eyebrow raising, a state of eyebrow lowering, and a state of cheek raising. The facial expression estimation unit 214 may estimate the facial expression of a person by combining the states of multiple facial muscle movement. The facial expression estimation unit 214 may estimate the facial expression of a person to be at least one of a joyful expression, a surprise expression, a fearful expression, a disgust expression, an anger expression, a sadness expression, and a neutral expression. For example, the facial expression estimation unit 214 may estimate that the facial expression is a joyful expression when the cheeks of a person are raised more than a predetermined level.

[0030] In the second embodiment, the occluded region, in which at least a portion of the face region is occluded, is a mask region occluded by a mask worn by the face. However, the occluded region may be, for example, a region occluded by sunglasses. In this case, the facial expression estimation unit 214 may estimate the facial expression of the person from the state of the mouth. The state of the mouth may include, for example, at least one of a state in which the upper lip is raised, a state in which the corners of the mouth are raised, a state in which dimples are formed, and a state in which the chin is raised.

[0031] The estimated facial expression image generating unit 215 generates an estimated facial expression image of the area corresponding to the occluded area, according to the facial expression estimated by the facial expression estimating unit 214 (step S24).

[0032] The composite image generation unit 216 generates a composite image based on the image and the estimated facial expression image. The composite image generation unit 216 may generate the composite image so that at least the occluded area is hidden by the estimated facial expression image. In other words, the composite image generation unit 216 may complement the occluded area of ​​the person's face area with an image corresponding to the estimated facial expression of the person. [2-3: Technical Effects of Information Processing Device 2]

[0033] The information processing device 2 in the second embodiment generates a composite image based on an image and an image of a mask area corresponding to the estimated facial expression of a person, so that even if the person is wearing a mask, an image corresponding to the person's facial expression can be obtained in which the person's mouth is not obscured.

[0034] Recently, due to changes in hygiene awareness, wearing a mask is recommended, especially in crowded places. However, when taking commemorative photos in crowded places, such as tourist spots, the photos tend to be dull because only people wearing masks are captured. In other words, there is a demand for recording natural facial images without masks, even in crowded places where people are hesitant to remove their masks.

[0035] In contrast, the information processing device 2 in the second embodiment generates a composite image of a person without a mask based on an image of an area corresponding to the mask area according to the estimated facial expression of the person when the person is wearing a mask, thereby providing a natural-looking face image without a mask. Therefore, a photo taken in a crowded place will include a natural-looking face image without a mask, making it possible to record an attractive photo. [3: Third Embodiment]

[0036] An information processing device, an information processing method, and a recording medium according to a third embodiment will be described below. The information processing device, the information processing method, and the recording medium according to the third embodiment will be described below using an information processing device 3 to which the information processing device, the information processing method, and the recording medium according to the third embodiment are applied.

[0037] In the third embodiment, when at least a part of the facial region is covered by a mask worn on the face, the facial expression estimation unit 214 may estimate the facial expression of the person based on the region around the person's eyes in the facial region as a region other than the mask region included in the facial region. The facial expression estimation unit 214 may estimate the facial expression of the person based on information obtainable from the region around the eyes included in the facial region.

[0038] The facial expression estimation unit 214 may extract the area around the eyes from the face area based on, for example, the distance between the eyes included in the face. The facial expression estimation unit 214 may also extract the area around the eyes from the face area based on both sides of the lower part of the nose bridge included in the face.

[0039] Furthermore, the facial expression estimation unit 214 may estimate the facial expression of a person based on, for example, the angle of the face or the pose / gesture of the person, in addition to the area information around the eyes included in the face area. Furthermore, the facial expression estimation unit 214 may estimate the facial expression of a person based on, for example, audio information acquired when an image of the person is generated, in addition to the area information around the eyes included in the face area. Furthermore, the facial expression estimation unit 214 may estimate the facial expression of a person based on information indicating the surrounding situation when an image of the person is generated, in addition to the area information around the eyes included in the face area. As in the second embodiment, the facial expression estimation unit 214 may adopt, as information about a person, information that improves the estimation accuracy of the facial expression of the person. [Technical effect of information processing device 3]

[0040] The information processing device 3 in the third embodiment can estimate the facial expression under the mask from image information around the eyes and synthesize an image of a face without a mask with an appropriate expression. [4: Fourth Embodiment]

[0041] A fourth embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the fourth embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 4 to which the fourth embodiment of the information processing device, the information processing method, and the recording medium is applied. [4-1: Configuration of Information Processing Device 4]

[0042] The configuration of the information processing device 4 in the fourth embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 4 in the fourth embodiment.

[0043] As shown in FIG. 4 , the information processing device 4 in the fourth embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment. Furthermore, the information processing device 4 may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment. However, the information processing device 4 does not need to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 4 in the fourth embodiment differs from the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment in that the calculation device 21 includes a learning unit 417 and performs a learning operation. Other features of the information processing device 4 may be the same as other features of at least one of the information processing device 2 in the second embodiment and the information processing device 3 in the third embodiment. [4-2: Learning Operation Performed by the Information Processing Device 4]

[0044] The flow of information processing operations performed by the information processing device 4 in the fourth embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of information processing operations performed by the information processing device 4 in the fourth embodiment.

[0045] 5, the acquisition unit 211 acquires learning information including sample information on a sample person with a predetermined facial expression and an expression label indicating the predetermined facial expression (step S40). The predetermined facial expressions may include at least one of an expression of joy, an expression of surprise, an expression of fear, an expression of disgust, an expression of anger, an expression of sadness, and a neutral expression. The expression label may be a label indicating each of these facial expressions. Furthermore, labels may be provided for each of the multiple levels of intensity of each facial expression.

[0046] The acquisition unit 211 may acquire, from the storage device 22, the learning information stored in the storage device 22. The acquisition unit 211 may acquire, via the communication device 23, the learning information from an external device.

[0047] The detection unit 212 detects a face area including a person's face from the image (step S21). The facial expression estimation unit 214 estimates the facial expression of the sample person based on the sample information (step S41).

[0048] The learning unit 417 causes the facial expression estimation unit 214 to learn a method for estimating a person's facial expression based on the facial expression label and the estimation result of the facial expression of the sample person by the facial expression estimation unit 214 (step S42). The learning unit 417 may construct a facial expression estimation model capable of estimating the facial expression of a person whose facial region is at least partially occluded. The facial expression estimation unit 214 may use the facial expression estimation model to estimate the facial expression of a person whose facial region is at least partially occluded based on information about the person. By using the trained facial expression estimation model, the facial expression estimation unit 214 can accurately estimate the facial expression of a person whose facial region is at least partially occluded.

[0049] The parameters that define the operation of the facial expression estimation model may be stored in the storage device 22. The parameters that define the operation of the facial expression estimation model may be parameters that are updated by a learning operation, such as weights and biases of a neural network.

[0050] The images used to learn the facial expressions of the face hidden by the mask area only need to show the state of the person outside the mask area. In other words, learning may be performed using an area other than the mask area. In other words, the images used for learning may be images of a person wearing a mask or images of a person not wearing a mask. [4-3: Technical Effects of Information Processing Device 4]

[0051] The information processing device 4 according to the fourth embodiment can realize highly accurate estimation of a person's facial expression through machine learning. [5: Fifth Embodiment]

[0052] A fifth embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the fifth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 5 to which the fifth embodiment of the information processing device, the information processing method, and the recording medium is applied.

[0053] An information processing device 5 according to the fifth embodiment will be described with reference to FIG. 6. The fifth embodiment describes a specific example of the operation of generating an estimated facial expression image in the second to fourth embodiments described above (i.e., the operation corresponding to step S24 in FIG. 3). In the fifth embodiment, images of people with various facial expressions, at least images of people whose occluded areas are not occluded, may be pre-registered in the storage device 22. Other parts of the operation of generating an estimated facial expression image may be the same as at least one of the second to fourth embodiments. Therefore, the following will describe in detail the parts that differ from the embodiments already described, and will omit explanations of other overlapping parts as appropriate. [5-1: Estimated Facial Expression Image Generation Operation Performed by Information Processing Device 5]

[0054] The flow of estimated facial expression image generation (i.e., operation when generating an estimated facial expression image) by the information processing device 5 according to the fifth embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of the estimated facial expression image generation operation by the information processing device 5 according to the fifth embodiment.

[0055] 6, the estimated facial expression image generation unit 215 estimates who the person is to be processed (step S50). The estimated facial expression image generation unit 215 may perform face authentication using the face area detected by the detection unit 212, and estimate who the person is to be processed.

[0056] The estimated facial expression image generation unit 215 searches for and acquires an image estimated to be an image of the person to be processed (hereinafter, sometimes referred to as the "person") from among pre-registered images of people whose occluded areas are at least not occluded (step S51). The estimated facial expression image generation unit 215 determines whether or not an image of the person has been acquired in step S51 (step S52).

[0057] If an image of the person can be acquired in step S51 (step S52: Yes), estimated facial expression image generation unit 215 determines whether or not there is an image of the person with a facial expression corresponding to the facial expression estimated in step S23 (step S53). The facial expression corresponding to the estimated facial expression may include a facial expression that matches or is similar to the estimated facial expression.

[0058] If there is an image of the person having a facial expression corresponding to the facial expression estimated in step S23 (step S53: Yes), the estimated facial expression image generation unit 215 generates an estimated facial expression image (step S54) based on a pre-registered image of the person having a facial expression corresponding to the facial expression estimated by the facial expression estimation unit 214. The estimated facial expression image generation unit 215 may select a pre-registered image of the person having a facial expression corresponding to the facial expression estimated by the facial expression estimation unit 214, and correct the brightness of the image, the person's posture, etc. to generate an estimated facial expression image.

[0059] If there is no image of the person having a facial expression corresponding to the facial expression estimated in step S23 (step S53: No), the estimated facial expression image generation unit 215 generates an estimated facial expression image based on a pre-registered image of the person in which at least the occluded area is not occluded (step S55). If there is no pre-registered image of the person having a facial expression corresponding to the facial expression estimated by the facial expression estimation unit 214, the estimated facial expression image generation unit 215 may select an arbitrary image of the person and convert the facial expression in the image into a facial expression corresponding to the facial expression estimated by the facial expression estimation unit 214 to generate an estimated facial expression image. The estimated facial expression image generation unit 215 may apply a deep learning technique such as a generative adversarial network (GAN) to generate an image of a facial expression corresponding to the facial expression estimated by the facial expression estimation unit 214 from the facial expression of the image as an estimated facial expression image.

[0060] In step S51, if an image of the person cannot be obtained (step S52: No), the estimated facial expression image generation unit 215 may apply a deep learning technique such as GAN to generate an image of a facial expression corresponding to the facial expression estimated by the facial expression estimation unit 214 from the facial expression in the image as an estimated facial expression image (step S56).

[0061] Note that only one image of the person may be registered per person. That is, the facial expression image generation unit 215 may omit the operation of step S53 and perform the operation of step S55. Furthermore, the estimated facial expression image generation unit 215 may generate an estimated facial expression image by applying a deep learning technique such as GAN, regardless of whether or not a person's image is present. That is, the facial expression image generation unit 215 may omit the operations of steps S50 to S52 and perform the operation of step S56.

[0062] Furthermore, the image generated in this embodiment does not have to be intended for use in person authentication. Therefore, the estimated facial expression image generation unit 215 may generate a facial image with an expression that matches the situation of the person when the image was generated, rather than the individuality. [5-2: Technical Effects of the Information Processing Device 5]

[0063] The information processing device 5 in the fifth embodiment generates the estimated facial expression image based on a pre-registered image of a person in which at least the occluded area is not occluded, thereby obtaining an image that looks like the person. Furthermore, if an image of a person in which at least the occluded area of ​​the facial expression corresponding to the estimated facial expression is not occluded has been pre-registered, the information processing device 5 generates the estimated facial expression image based on the pre-registered image, thereby obtaining an image that looks more like the person. [6: Sixth Embodiment]

[0064] A sixth embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the sixth embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 6 to which the sixth embodiment of the information processing device, the information processing method, and the recording medium is applied. [6-1: Configuration of Information Processing Device 6]

[0065] The configuration of the information processing device 6 in the sixth embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the configuration of the information processing device 6 in the sixth embodiment.

[0066] As shown in FIG. 7 , the information processing device 6 in the sixth embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 2 in the second embodiment to the information processing device 5 in the fifth embodiment. Furthermore, the information processing device 6 may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment to the information processing device 5 in the fifth embodiment. However, the information processing device 6 does not need to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 6 in the sixth embodiment differs from the information processing device 2 in the second embodiment to the information processing device 5 in the sixth embodiment in that the calculation device 21 includes a display control unit 618. Other features of the information processing device 6 may be the same as at least one other feature of the information processing device 2 in the second embodiment to the information processing device 5 in the fifth embodiment. [6-2: Information Processing Operation Performed by the Information Processing Device 6]

[0067] When the composite image generating unit 216 generates a composite image, the display control unit 618 displays the composite image in place of the image, and displays information indicating that the image is generated by the composite image generating unit 216 by superimposing it on the composite image. For example, as illustrated in FIG. 8( a), when the composite image generating unit 216 generates a composite image, the display control unit 618 may display text such as "mask area complement image" in the lower right corner of the display mechanism D. Alternatively, as illustrated in FIG. 8( b), when the composite image generating unit 216 generates a composite image, the display control unit 618 may display a semi-transparent mask by superimposing it on an area corresponding to the mask area in the uncombined image. [6-3: Technical Effects of Information Processing Device 6]

[0068] When displaying a composite image, the information processing device 6 according to the sixth embodiment displays information indicating that the image is a composite image by superimposing it on the composite image, so that the user can easily tell whether the image is a composite image or not. [7: Seventh Embodiment]

[0069] A seventh embodiment of an online conference system will be described. Hereinafter, the seventh embodiment of the online conference system will be described using an online conference system 700 to which the seventh embodiment of the online conference system is applied. [7-1: Configuration of the online conference system 700] As illustrated in FIG. 9, the online conference system 700 in the seventh embodiment may include an online conference control device 7 and multiple terminals 70 that hold a conference (in FIG. 9, terminals 70-1, 70-2, 70-3, ..., terminal 70-N are illustrated). The online conference control device 7 is capable of communicating with the multiple terminals 70. The multiple terminals 70 may hold an online conference. The multiple terminals 70 may hold a web conference. [7-2: Configuration of the online conference control device 7]

[0070] The configuration of the online conference control device 7 will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the configuration of the online conference control device 7 in the seventh embodiment.

[0071] 10 , the online conference control device 7 includes a calculation device 71 and a storage device 72. The online conference control device 7 may further include a communication device 73, an input device 74, and an output device 75. However, the online conference control device 7 does not have to include at least one of the communication device 73, the input device 74, and the output device 75. The calculation device 71, the storage device 72, the communication device 73, the input device 74, and the output device 75 may be connected via a data bus 76.

[0072] The arithmetic device 71 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The arithmetic device 71 loads a computer program. For example, the arithmetic device 71 may load a computer program stored in the storage device 72. For example, the arithmetic device 71 may load a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (e.g., an input device 74, described later) not shown in the figure that is included in the online conference control device 7. The arithmetic device 71 may acquire (i.e., download or load) the computer program from a device (not shown) located outside the online conference control device 7 via the communication device 73 (or another communication device). The arithmetic device 71 executes the loaded computer program. As a result, a logical function block for executing the operation to be performed by the online conference control device 7 is realized within the arithmetic device 71. In other words, the arithmetic device 71 can function as a controller for realizing the logical function block for executing the operation (in other words, processing) to be performed by the online conference control device 7.

[0073] 10 shows an example of logical functional blocks implemented within the computing device 71 for executing online conference control operations. As shown in FIG. 10 , the computing device 71 includes an acquisition unit 711, which is a specific example of an "acquisition means" described in the appendix, a detection unit 712, which is a specific example of a "detection means" described in the appendix, an area estimation unit 713, which is a specific example of an "estimation means" described in the appendix, an expression estimation unit 714, which is a specific example of an "expression estimation means," an estimated expression image generation unit 715, which is a specific example of an "estimated expression image generation means" described in the appendix, a composite image generation unit 716, which is a specific example of a "synthetic image generation means" described in the appendix, and an output control unit 719, which is a specific example of an "output control means." The operations of the acquisition unit 711, the detection unit 712, the area estimation unit 713, the expression estimation unit 714, the estimated expression image generation unit 715, the composite image generation unit 716, and the output control unit 719 will be described later with reference to FIG. 11 .

[0074] The storage device 72 can store desired data. For example, the storage device 72 may temporarily store a computer program executed by the arithmetic device 71. The storage device 72 may temporarily store data that the arithmetic device 71 temporarily uses when the arithmetic device 71 is executing a computer program. The storage device 72 may also store data that the online conference control device 7 stores long-term. The storage device 72 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 72 may include a non-temporary recording medium.

[0075] The communication device 73 is capable of communicating with devices external to the online conference control device 7 via a communication network (not shown). The online conference control device 7 may be capable of communicating with each of the multiple terminals 70 via the communication device 73.

[0076] The input device 74 is a device that accepts information input to the online conference control device 7 from outside the online conference control device 7. For example, the input device 74 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the online conference control device 7. For example, the input device 74 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the online conference control device 7.

[0077] The output device 75 is a device that outputs information to the outside of the online conference control device 7. For example, the output device 75 may output information as an image. That is, the output device 75 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 75 may output information as sound. That is, the output device 75 may include an audio device (a so-called speaker) that can output sound. For example, the output device 75 may output information on paper. That is, the output device 75 may include a printing device (a so-called printer) that can print desired information on paper. [7-3: Online Conference Control Operation Performed by the Online Conference Control Device 7]

[0078] The flow of the online conference control operation performed by the online conference control device 7 in the seventh embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the flow of the online conference control operation performed by the online conference control device 7 in the seventh embodiment.

[0079] 11 , the acquisition unit 711 acquires information about a person, including at least an image of the person, from at least one terminal 70 among the multiple terminals 70 participating in the conference (step S70). The acquisition unit 711 may acquire information about the person, including at least an image of the person operating the terminal 70. The acquisition unit 711 may acquire information about the person, including a video of the person operating the terminal 70.

[0080] The detection unit 712 detects a face region including a person's face from the image (step S71). The region estimation unit 713 estimates the occluded region when at least a portion of the face region is occluded (step S72). The facial expression estimation unit 714 estimates the person's facial expression based on information about the person (step S73). The estimated facial expression image generation unit 715 generates an estimated facial expression image of the region corresponding to the occluded region according to the facial expression estimated by the facial expression estimation unit 714 (step S74). The composite image generation unit 716 generates a composite image based on the image and the estimated facial expression image (step S75).

[0081] Note that the operation performed by the detection unit 712 may be similar to the operation performed by at least one of the detection units 212 in the second to sixth embodiments. Furthermore, the operation performed by the region estimation unit 713 may be similar to the operation performed by at least one of the region estimation units 213 in the second to sixth embodiments. Furthermore, the operation performed by the facial expression estimation unit 714 may be similar to the operation performed by at least one of the facial expression estimation units 214 in the second to sixth embodiments. Furthermore, the operation performed by the estimated facial expression image generation unit 715 may be similar to the operation performed by at least one of the estimated facial expression image generation units 215 in the second to sixth embodiments. Furthermore, the operation performed by the composite image generation unit 716 may be similar to the operation performed by at least one of the composite image generation units 216 in the second to sixth embodiments.

[0082] When the composite image generation unit 716 generates a composite image, the output control unit 719 outputs the composite image to the multiple terminals 70 in place of the image (step S76). When the acquisition unit 711 acquires a video of a person operating the terminal 70, the output control unit 719 may output the image or the composite image to the multiple terminals 70 in real time. Alternatively, when outputting the composite image to the multiple terminals 70, the output control unit 719 may output the image at a later time than when outputting an image to the multiple terminals 70. When outputting the composite image to the multiple terminals 70, the output control unit 719 may output the image with a delay of, for example, several seconds, compared to when outputting an image to the multiple terminals 70.

[0083] Note that at least one of the information processing devices 2 in the second embodiment to the information processing device 6 in the sixth embodiment may perform the synthetic image generation operation in real time. Alternatively, at least one of the information processing devices 2 in the second embodiment to the information processing device 6 in the sixth embodiment may have a time lag of, for example, several seconds.

[0084] In addition, when the acquisition unit 711 acquires a still image of a person operating the terminal 70, the learning unit 717 may generate a composite image offline, and the output control unit 719 may output the composite image generated offline to multiple terminals 70.

[0085] Note that, when the acquisition unit 711 acquires information about a person including a video of the person, the region estimation unit 713 does not need to perform the estimation process for each frame. That is, the region estimation unit 713 may perform the estimation process for each predetermined number of frames. That is, the facial expression estimation unit 714 may generate estimated facial expression images corresponding to the same facial expression for the predetermined number of frames.

[0086] Furthermore, in the online conference control device 7 in the seventh embodiment, the arithmetic device 71 may include a learning unit 717. That is, similar to the learning unit 417 in the fourth embodiment, the learning unit 717 may cause the facial expression estimation unit 714 to learn a method for estimating a person's facial expression based on the facial expression label and the estimation result of the facial expression of the sample person by the facial expression estimation unit 714.

[0087] Furthermore, in the online conference control device 7 of the seventh embodiment, the arithmetic device 71 may include a display control unit 718. That is, similar to the display control unit 618 of the sixth embodiment, when the composite image generation unit 716 generates a composite image, the display control unit 718 may display the composite image in place of the image, and may superimpose information indicating that the image was generated by the composite image generation unit 716 on the composite image. [7-4: Technical Effects of the Online Conference Control Device 7]

[0088] The online conference control device 7 in the seventh embodiment generates a composite image based on an image and an image of a mask area corresponding to the estimated facial expression of a person, so that even if the person is wearing a mask, an image corresponding to the person's facial expression can be obtained in which the person's mouth is not obscured.

[0089] Recently, due to changes in hygiene awareness, wearing a mask is recommended, especially in crowded places. While many people would prefer not to wear a mask when participating in online communication, wearing a mask is recommended when participating from a shared space such as a satellite office. In other words, there is a demand for delivering natural-looking facial images without a mask, even in crowded places where people are hesitant to remove their masks.

[0090] In contrast, the online conference control device 7 in the seventh embodiment generates a composite image of a person without a mask based on an image of an area corresponding to the mask area according to the estimated facial expression of the person when the person is wearing a mask, thereby providing a natural-looking face image without a mask. Therefore, even when a person participates from a shared location such as a satellite office, a natural-looking face image without a mask can be delivered. [8: Note]

[0091] The following supplementary notes are further disclosed with respect to the above-described embodiments. [Supplementary Note 1] An information processing device comprising: an acquisition means for acquiring information about a person including at least an image of the person; a detection means for detecting a face region including the face of the person from the image; an estimation means for estimating an occluded region when at least a portion of the face region is occluded; an expression estimation means for estimating an occluded region based on information about the person; an estimated expression image generation means for generating an estimated expression image of an area corresponding to the occluded region according to the expression estimated by the expression estimation means; and a composite image generation means for generating a composite image based on the image and the estimated expression image. [Supplementary Note 2] The information processing device according to Supplementary Note 1, wherein the occluded region in which at least a portion of the face region is occluded is a mask region occluded by a mask worn by the person. [Supplementary Note 3] The information processing device according to Supplementary Note 2, wherein the expression estimation means estimates the expression of the person based on areas around the person's eyes in the face region. [Supplementary Note 4] The information processing device according to any one of Supplements 1 to 3, wherein the acquisition means acquires learning information including sample information on a sample person with a predetermined facial expression and a facial expression label indicating the predetermined facial expression, the facial expression estimation means estimates a facial expression of the sample person based on the sample information, and learning means causes the facial expression estimation means to learn a method of estimating the facial expression of the person based on the facial expression label and a result of estimation of the facial expression of the sample person by the facial expression estimation means. [Supplementary Note 5] The information processing device according to any one of Supplements 1 to 3, wherein the estimated facial expression image generation means generates the estimated facial expression image based on a pre-registered image of the person in which at least the occluded region is not occluded. [Supplementary Note 6] The information processing device according to Supplementary Note 5, wherein the estimated facial expression image generation means generates the estimated facial expression image based on a pre-registered image of the person with a facial expression corresponding to the facial expression estimated by the facial expression estimation means.[Supplementary Note 7] The information processing device according to any one of Supplementary Notes 1 to 3, further comprising: a display control means, when the composite image generation means has generated the composite image, causing the composite image to be displayed in place of the image, and causing information indicating that the image is generated by the composite image generation means to be superimposed on the composite image. [Supplementary Note 8] An online conference system comprising: an acquisition means for acquiring information about a person including at least an image of the person from at least one terminal of a plurality of terminals participating in a conference; a detection means for detecting a face region including the face of the person from the image; an estimation means for estimating an obstructed region when at least a portion of the face region is obstructed; an expression estimation means for estimating an expression of the person based on the information about the person; an estimated expression image generation means for generating an estimated expression image of an area corresponding to the obstructed region in accordance with the expression estimated by the expression estimation means; a composite image generation means for generating a composite image based on the image and the estimated expression image; and an output control means, when the composite image generation means has generated the composite image, outputting the composite image to the plurality of terminals in place of the image. [Supplementary Note 9] An information processing method comprising: acquiring information about a person including at least an image of the person, detecting a face region including the person's face from the image, estimating an occluded region if at least a portion of the face region is occluded, estimating a facial expression of the person based on the information about the person, generating an estimated facial expression image of the region corresponding to the occluded region according to the estimated facial expression, and generating a composite image based on the image and the estimated facial expression image. [Supplementary Note 10] A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method comprising: acquiring information about a person including at least an image of the person, detecting a face region including the person's face from the image, estimating an occluded region if at least a portion of the face region is occluded, estimating a facial expression of the person based on the information about the person, generating an estimated facial expression image of the region corresponding to the occluded region according to the estimated facial expression, and generating a composite image based on the image and the estimated facial expression image.

[0092] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0093] 1, 2, 3, 4, 5, 6 Information processing device 11, 211, 711 Acquisition unit 12, 212, 712 Detection unit 13, 213, 713 Area estimation unit 14, 214, 714 Facial expression estimation unit 15, 215, 715 Estimated facial expression image generation unit 16, 216, 716 Composite image generation unit 417, 717 Learning unit 618, 718 Display control unit 700 Online conference system 7 Online conference control device 70 Terminal 719 Output control unit

Claims

1. an acquisition means for acquiring information about a person including at least an image of the person; a detection means for detecting a face area including the face of the person from the image; an estimating means for estimating an occluded region when at least a part of the face region is occluded; Facial expression estimation means for estimating a facial expression of the person based on information about the person; an estimated facial expression image generating means for generating an estimated facial expression image of an area corresponding to the occluded area in accordance with the facial expression estimated by the facial expression estimating means; a synthetic image generating means for generating a synthetic image based on the image and the estimated facial expression image; An information processing device comprising:

2. The occluded region in which at least a part of the face region is occluded is a mask region that is occluded by a mask worn by the person. The information processing device according to claim 1 .

3. The facial expression estimation means estimates the facial expression of the person based on a region around the person's eyes in the face region. The information processing device according to claim 2 .

4. The acquiring means acquires learning information including sample information on a sample person having a predetermined facial expression and a facial expression label indicating the predetermined facial expression; The facial expression estimation means estimates a facial expression of the sample person based on the sample information, The method further includes a learning unit that causes the facial expression estimation unit to learn a method of estimating the facial expression of the sample person based on the facial expression label and a result of the estimation of the facial expression of the sample person by the facial expression estimation unit.

3. The information processing device according to claim 1 or 2.

5. The estimated facial expression image generating means generates the estimated facial expression image based on a pre-registered image of the person in which at least the occluded area is not occluded.

3. The information processing device according to claim 1 or 2.

6. The estimated facial expression image generating means generates the estimated facial expression image based on an image of the person having a facial expression corresponding to the facial expression estimated by the facial expression estimating means and registered in advance. The information processing device according to claim 5 .

7. The display control means further includes a display control means for, when the composite image generating means generates the composite image, displaying the composite image in place of the image and superimposing information indicating that the image is generated by the composite image generating means on the composite image.

3. The information processing device according to claim 1 or 2.

8. An acquisition means for acquiring information about a person, including at least an image of the person, from at least one terminal among a plurality of terminals that are participating in a conference; a detection means for detecting a face area including the face of the person from the image; an estimating means for estimating an occluded region when at least a part of the face region is occluded; Facial expression estimation means for estimating a facial expression of the person based on information about the person; an estimated facial expression image generating means for generating an estimated facial expression image of an area corresponding to the occluded area in accordance with the facial expression estimated by the facial expression estimating means; a synthetic image generating means for generating a synthetic image based on the image and the estimated facial expression image; an output control means for outputting the composite image to the plurality of terminals in place of the image when the composite image generation means generates the composite image; An online conference system comprising:

9. obtaining information about the person including at least an image of the person; Detecting a face area including the face of the person from the image; If at least a portion of the face region is occluded, estimating an occluded region; estimating a facial expression of the person based on information about the person; generating an estimated facial expression image of an area corresponding to the occlusion area according to the estimated facial expression; A synthetic image is generated based on the image and the estimated facial expression image. A computer implemented information processing method.

10. On the computer, obtaining information about the person including at least an image of the person; Detecting a face area including the face of the person from the image; If at least a portion of the face region is occluded, estimating an occluded region; estimating a facial expression of the person based on information about the person; generating an estimated facial expression image of an area corresponding to the occlusion area according to the estimated facial expression; A synthetic image is generated based on the image and the estimated facial expression image. A computer program for executing an information processing method.