Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises: an acquisition means for acquiring a face image; a generation means for generating a processed face image obtained by processing at least a part of a face region of the face image; a first output means for outputting the processed face image; a reception means for receiving input of feature information for the processed face image; and a second output means for outputting the feature information and the face image in association with each other.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] It is known to detect feature points from a face image that has been subjected to image processing. For example, Japanese Patent Application Laid-Open No. 2007-203499 discloses a process of rotating a registered face so that the face pose is the same as an estimated face pose.

[0003] Japanese Patent Application Laid-Open No. 2022-028090

[0004] This disclosure aims to improve upon the related art discussed above.

[0005] One aspect of the information processing device disclosed herein comprises an acquisition means for acquiring a facial image, a generation means for generating a processed facial image by processing at least a portion of the facial area of ​​the facial image, a first output means for outputting the processed facial image, a reception means for receiving input of feature information for the processed facial image, and a second output means for outputting the feature information in association with the facial image.

[0006] One aspect of the information processing method disclosed herein is an information processing method executed by a computer, which acquires a facial image, generates a processed facial image by processing at least a portion of the facial area of ​​the facial image, outputs the processed facial image, accepts input of feature information for the processed facial image, and outputs the feature information in association with the facial image.

[0007] One aspect of the recording medium of this disclosure is a computer program recorded thereon that causes a computer to execute an information processing method that acquires a facial image, generates a processed facial image by processing at least a portion of the facial area of ​​the facial image, outputs the processed facial image, accepts input of feature information for the processed facial image, and associates the feature information with the facial image and outputs it.

[0008] FIG. 1 is a block diagram showing the configuration of an information processing device according to the present disclosure. FIG. 2 is a flowchart showing the flow of information processing operation in the information processing device according to the present disclosure. FIG. 3 is a diagram showing an example of facial feature points. FIG. 4 is a block diagram showing the configuration of an information processing device according to the present disclosure. FIG. 5 is a flowchart showing the flow of information processing operation in the information processing device according to the present disclosure. FIG. 6 is a schematic diagram showing an information processing method in the information processing device according to the present disclosure. FIG. 7 is a block diagram showing the configuration of an information processing device according to the present disclosure. FIG. 8 is a schematic diagram showing the information processing method in the information processing device according to the present disclosure. FIG. 9 is a block diagram showing the configuration of an information processing device according to the present disclosure. FIG. 10 is a schematic diagram showing the information processing method in the information processing device according to the present disclosure.

[0009] Hereinafter, an information processing device, an information processing method, and a recording medium according to an embodiment will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 according to this disclosure. [1-1: Configuration of Information Processing Device 1]

[0011] The configuration of an information processing device 1 according to this disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an information processing device 1 according to this disclosure.

[0012] 1, the information processing device 1 includes an image acquisition unit 111, an image generation unit 112, a first output unit 113, an input reception unit 114, and a second output unit 115. The processes performed by the image acquisition unit 111, the image generation unit 112, the first output unit 113, the input reception unit 114, and the second output unit 115 will be described with reference to FIG. 2. [1-2: Information Processing Method Executed by the Information Processing Device 1]

[0013] An information processing method executed by the information processing device 1 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of the flow of the information processing method executed by the information processing device 1. The information processing device 1 may be realized by a computer executing a computer program for causing a computer to execute the information processing method.

[0014] 2, the image acquisition unit 111 acquires a face image (step S11). The image generation unit 112 generates an edited face image by editing at least a part of the face area of ​​the face image (step S12). The first output unit 113 outputs the edited face image (step S13).

[0015] The input receiving unit 114 receives input of feature information of the processed facial image (step S14). The feature information is information indicating the features of the facial image. The second output unit 115 outputs the feature information input in step S14 in association with the facial image acquired in step S11 (step S15). [1-3: Technical Effects of Information Processing Device 1]

[0016] The information processing device 1 according to this disclosure can assist in inputting accurate feature information. [2: Second embodiment]

[0017] A second embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the second embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 2 according to this disclosure. [2-1: Input of feature points]

[0018] In this embodiment, the facial image feature information is information indicating facial feature points. When constructing a facial feature point database, input of the facial feature points is required. The facial feature point database includes a data set of facial images and facial feature points in the facial images. The facial feature point database may be used in processing that uses facial feature points, such as face matching (referred to as "feature point processing").

[0019] When generating a dataset of face images and facial feature points, it is required to input defined feature points. The defined feature points are, for example, feature points that are specified to be used in feature point processing. Therefore, when generating a dataset, it is required to input all defined feature points.

[0020] 3 shows an example of defined feature points. For example, as shown in FIG. 3, the inner corners of the left and right eyebrows, the arch of the eyebrows, the outer corners of the eyebrows, the pupils, the outer corners of the eyes, the wings of the nose, the nostrils, the corners of the mouth, the protuberance of the nose, and the center of the lips may be defined as feature points to be used in the feature point processing.

[0021] There are cases where it is easy to identify facial feature points from a facial image, and cases where it is difficult. For example, when the positions of facial feature points in a facial image are not visible, it corresponds to a case where it is difficult to identify facial feature points from a facial image.

[0022] The position of the facial feature points in the face image cannot be seen when a part of the face area in the face image is in a blind spot. When a part of the face is occluded, the part of the face area in the face image is in a blind spot. Also, when the face is tilted relative to the plane of the face image, the part of the face area in the face image is in a blind spot.

[0023] Furthermore, when the quality of a facial image is low, it is often difficult to accurately identify facial feature points from the facial image. For example, a facial image with low quality corresponds to a facial image with low resolution. Furthermore, for example, the quality of an image taken in a dark environment, such as at night, is often not suitable for inputting feature points.

[0024] However, as described above, even when it is difficult to identify facial feature points from a facial image, it is necessary to input the feature points with high accuracy. Facial feature points that are difficult to identify from a facial image are sometimes referred to as "difficult-to-identify feature points." Furthermore, facial feature points that are easy to identify from a facial image are sometimes referred to as "easy-to-identify feature points."

[0025] Furthermore, the feature point processing process requires the input of facial feature points. The feature point processing process corresponds to, for example, the preprocessing process in face matching. The more feature points input in the preprocessing, the more accurate the feature point processing can often be. In other words, easily identifiable feature points alone provide little information, which may have a significant impact on feature point processing.

[0026] Furthermore, depending on the system that performs feature point processing, it may be necessary to input all defined feature points, including hard-to-identify feature points. In this case, it is not possible to skip inputting hard-to-identify feature points.

[0027] As described above, there are cases where it is necessary to input difficult-to-identify feature points. This embodiment relates to inputting difficult-to-identify feature points. [2-2: Configuration of Information Processing Device 2]

[0028] The configuration of the information processing device 2 according to this disclosure will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 2 according to this disclosure.

[0029] 4 , the information processing device 2 includes a calculation device 11, a storage device 12, and a communication device 13. In addition to the calculation device 11, the storage device 12, and the communication device 13, the information processing device 2 may further include an input device 14 and an output device 15. However, the information processing device 2 does not necessarily have to include at least one of the input device 14 and the output device 15. The calculation device 11, the storage device 12, the communication device 13, the input device 14, and the output device 15 may be connected via a data bus 16.

[0030] The arithmetic device 11 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit).

[0031] The arithmetic device 11 reads a computer program 121 including at least one of computer program code and computer program instructions. For example, the arithmetic device 11 may read the computer program 121 stored in the storage device 12. For example, the arithmetic device 11 may read the computer program 121 stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the information processing device 2. The computer program 121 read from the recording medium may be stored in the storage device 12. The arithmetic device 11 may acquire (i.e., download or read) the computer program 121 from a device (not shown) located outside the information processing device 1 via the communication device 13 (or another communication device). The downloaded computer program 121 may be stored in the storage device 12.

[0032] The arithmetic device 11 executes the loaded computer program 121. As a result, logical functional blocks for executing processing to be performed by the information processing device 2 (e.g., information processing described below) are realized within the arithmetic device 11. In other words, the arithmetic device 11, together with the storage device 12, etc. in which the computer program 121 is recorded (in other words, together with the storage device 12 and the computer program 121 recorded in the storage device 12, etc.), can function as a controller or computer for realizing the logical functional blocks for executing processing to be performed by the information processing device 2. In other words, together with at least one processor included in the arithmetic device 11, the memory (recording medium) included in the storage device 12, etc., and the computer program 121 are configured so that the information processing device 2 performs processing to be performed by the information processing device 2 (e.g., information processing described below). The arithmetic device 11 may output information to another computer, cloud server, or other device (not shown) provided outside the information processing device 2 via the communication device 13 (or other communication device).

[0033] The recording medium for recording the computer program 121 executed by the arithmetic device 11 may be at least one of a CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, and Blu-ray (registered trademark) optical disk, magnetic medium such as magnetic tape, magneto-optical disk, semiconductor memory such as USB memory, and any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 121 (for example, a general-purpose device or a dedicated device in which the computer program 121 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 121 may be realized by a logical processing block realized within the arithmetic device 11 when the arithmetic device 11 (i.e., processor) executes the computer program 121, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic device 11, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0034] Fig. 4 shows an example of logical functional blocks realized in the arithmetic device 11 to execute information processing. As shown in Fig. 4, an image acquisition unit 211, an image generation unit 212, an output unit 213, a feature point reception unit 214, and a registration unit 215 are realized in the arithmetic device 11. Note that the processes performed by the image acquisition unit 211, the image generation unit 212, the output unit 213, the feature point reception unit 214, and the registration unit 215 will be described with reference to Figs. 5 and 6.

[0035] The storage device 12 includes at least one memory capable of storing desired data. In other words, the storage device 12 includes at least one memory containing desired data. For example, the storage device 12 may store a computer program 121 executed by the arithmetic device 11. In this case, the storage device 12 (memory) may be used as the above-mentioned recording medium for recording the computer program 121 executed by the arithmetic device 11. The storage device 12 may temporarily store data used by the arithmetic device 11 when the arithmetic device 11 is executing the computer program 121. The storage device 12 may also store data to be stored long-term by the information processing device 2. The storage device 12 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 12 may include a non-transitory recording medium.

[0036] A registered image database DB may be realized within the storage device 12. The registered image database DB may also be realized in a storage device outside the information processing device 2.

[0037] The communication device 13 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown). The communication device 13 may be a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), Bluetooth (registered trademark), or USB (Universal Serial Bus).

[0038] The input device 14 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 14 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 14 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0039] The output device 15 is a device that outputs information to the outside of the information processing device 2. For example, the output device 15 may output information as an image. That is, the output device 15 may include a display device (a so-called display) that can display an image showing the information to be output. The display device may be a touch panel with an input function. For example, the output device 15 may output information as sound. That is, the output device 15 may include an audio device (a so-called speaker) that can output sound. For example, the output device 15 may output information on paper. That is, the output device 15 may include a printing device (a so-called printer) that can print desired information on paper.

[0040] Incidentally, in situations where highly accurate feature points are required, a final human check is often performed on the input of feature points by a human or feature points detected by a non-human entity (for example, artificial intelligence (AI)). Examples of situations where highly accurate feature points are required include database construction and face matching. In this embodiment, a case where an input of feature points of a face image is received from a user who uses the information processing device 2 will be described. [2-3: Information processing method executed by information processing device 2]

[0041] 5 and 6, the information processing operation in the information processing device 2 will be described. Fig. 5 is a flowchart showing the flow of the information processing operation executed by the information processing device 2. Fig. 6 is a conceptual diagram of the information processing operation executed by the information processing device 2.

[0042] As shown in Fig. 5, the image acquisition unit 211 acquires a face image (step S21). The image acquisition unit 211 acquires a face image for which facial feature points are to be input. Fig. 6(a) illustrates an example of a face image acquired by the image acquisition unit 211. As illustrated in Fig. 6(a), the face image acquired by the image acquisition unit 211 may have a blind spot in part of the face area. In the example illustrated in Fig. 6(a), part of the face is occluded, so that part of the face area in the face image is a blind spot.

[0043] The image generation unit 212 generates an edited face image by editing at least a portion of the face region of the face image (step S22). Fig. 6(b) shows an example of an edited face image in which at least a portion of the face region of the face image has been edited by the image generation unit 212. As shown in Fig. 6(b), the edited face image edited by the image generation unit 212 is a face image in which blind spots in the face image shown in Fig. 6(a) have been corrected so that they are visible.

[0044] The output unit 213 outputs the processed facial image (step S23). The output unit 213 may display the processed facial image illustrated in Fig. 6(b) on a display device. The output unit 213 may display the processed facial image on, for example, a touch panel serving as the input device 14 and the output device 15.

[0045] The feature point receiving unit 214 receives input of feature points of the processed facial image from the user (step S24). Specifically, the feature point receiving unit 214 receives input of feature points on the processed facial image designated by the user from the user. The feature point receiving unit 214 may receive, for example, an input operation by the user on the processed facial image displayed on a touch panel serving as the input device 14. The user's input operation is an operation for designating facial feature points in the processed facial image. Fig. 6(c) illustrates an example in which facial feature points in the processed facial image are designated.

[0046] The output unit 213 associates the feature points input in step S24 with the facial image acquired in step S11 and outputs them (step S25). Specifically, the output unit 213 may superimpose the feature points on the facial image based on the positions of the feature points on the processed facial image designated by the user. The output unit 213 may display the facial image on which the feature points are superimposed on, for example, a touch panel as the output device 15. Fig. 6(d) illustrates an example in which feature points are superimposed on the facial image based on the positions of the feature points on the processed facial image designated by the user.

[0047] The registration unit 215 associates the feature points input in step S24 with the face image acquired in step S11 and registers them in the registered image database DB (step S26). The registration unit 215 may also register the unprocessed face image on which the feature points are superimposed in the registered image database DB.

[0048] That is, in this embodiment, the processed face image is used when inputting feature points, and the face image to be registered is not the processed face image but the unprocessed face image (the face image acquired in step S21). [2-4: Technical Effects of Information Processing Device 2]

[0049] Feature point detection using AI is widely used, and AI can detect feature points with a certain degree of accuracy even from low-quality images. However, as described above, when a human performs a final check on the input of feature points detected by AI, if the image quality is low, it is often difficult for the human to determine whether the feature points detected by AI should be corrected. Furthermore, because the image quality is low, even if the feature points are corrected, there is a limit to the accuracy. Furthermore, accuracy is likely to vary depending on the person performing the final check. As such, low image quality can cause problems such as difficulty in identifying the exact positions of feature points and individual variability in the identified positions.

[0050] The information processing device 2 according to this disclosure can improve the accuracy of inputting facial feature points. Feature points can be input with high accuracy even for invisible facial parts. Furthermore, the information processing device 2 can reduce the number of steps required for inputting and correcting facial images.

[0051] In other embodiments, feature information may be input from sources other than a user of the information processing device. For example, feature information may be input from a trained network (referred to as "feature point detection AI"). By combining with feature point detection AI and increasing the resolution, it is expected that the detection accuracy of the feature point detection AI will also improve. By using it in combination with feature point detection AI, further load reduction can be achieved. For example, the user may modify the initial feature point position estimation by the feature point detection AI model as needed.

[0052] This embodiment may also be applied to images other than "face" images. This embodiment can be applied to images that undergo processing to extract feature points. [3: Third Embodiment]

[0053] A third embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the third embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 3 according to this disclosure. [3-1: Configuration of Information Processing Device 3]

[0054] The information processing device 3 is configured as a device for accurately inputting feature points, similar to the information processing device 1 and the information processing device 2. Furthermore, the information processing device 3 may be configured as a device for determining whether or not to generate a processed face image.

[0055] 7, the calculation device 11 in the third embodiment further includes a determination unit 316. The determination unit 316 determines whether or not the image generation unit 212 is to generate a processed face image.

[0056] Furthermore, in the third embodiment, a judgment model that can be constructed by machine learning is implemented within the arithmetic device 11 by the arithmetic device 11 executing the computer program 121. An example of a judgment model that can be constructed by machine learning is a judgment model including a neural network (so-called artificial intelligence (AI)). In this case, learning the judgment model may include learning parameters of the neural network (e.g., at least one of a weight and a bias). The arithmetic device 11 executes at least a judgment process using the judgment model. A judgment model that has been constructed by machine learning may be implemented in the arithmetic device 11. A judgment model that has been constructed by offline machine learning using teacher data may be implemented in the arithmetic device 11. Furthermore, the judgment model implemented in the arithmetic device 11 may be updated by online machine learning on the arithmetic device 11. Alternatively, the arithmetic device 11 may execute information processing using a judgment model implemented in a device external to the arithmetic device 11 (i.e., a device external to the information processing device 3) in addition to or instead of the judgment model implemented in the arithmetic device 11.

[0057] The determination unit 316 uses the determination model to determine whether or not to generate an edited face image. When a face image is input, the determination model outputs information indicating whether or not it is necessary for the image generation unit 212 to generate an edited face image. [3-2: Information Processing Method Executed by Information Processing Device 3]

[0058] 8, the image acquisition unit 211 acquires a face image (step S21). The determination unit 316 uses the determination model to determine whether or not to generate a processed face image (step S30).

[0059] The determination model outputs information indicating that an edited face image needs to be generated when a face image in which part of the face area is blind spot is input. Therefore, the determination model outputs information indicating that an edited face image needs to be generated when a face image in which part of the face is occluded is input. Furthermore, the determination model outputs information indicating that an edited face image needs to be generated when a face image in which the face is tilted relative to the plane of the face image is input. In other words, the determination model outputs information indicating that an edited face image needs to be generated when a face image that does not include part of the frontal face area that would be included if the face were photographed from the front is input.

[0060] Alternatively, the determination model may output information indicating that a processed face image needs to be generated when a face image that does not include some parts corresponding to facial feature points is input. A face image that does not include some parts corresponding to facial feature points is a face image in which some feature points cannot be detected.

[0061] Alternatively, the determination model may be a mechanism that, when a face image in which a part of a face region is in a blind spot is input, outputs information indicating the blind spot.Furthermore, the determination model may be a mechanism that, when a face image in which a part of a region corresponding to a facial feature point is not included is input, outputs information indicating the region corresponding to the missing feature point.

[0062] If it is determined that an edited face image is to be generated (step S30: Yes), the image generating unit 212 generates an edited face image (step S22), and the output unit 213 outputs the edited face image (step S23).

[0063] The feature point receiving unit 314 receives input of facial feature points from the user (step S34). If it is determined that generation of an edited face image is necessary (step S30: Yes), the feature point receiving unit 314 receives input of feature points of the edited face image from the user. On the other hand, if it is determined that generation of an edited face image is not necessary (step S30: No), the feature point receiving unit 314 receives input of feature points of the face image acquired in step S21 from the user.

[0064] The output unit 313 associates the feature points input in step S34 with the face image acquired in step S21 and outputs them (step S35). The registration unit 315 associates the feature points input in step S24 with the face image acquired in step S11 and registers them in the registered image database DB (step S36). [3-3: Technical Effects of Information Processing Device 3]

[0065] The information processing device 3 according to this disclosure generates an edited face image when an edited face image is required for the face image. Therefore, compared to generating edited face images for all face images, the input of facial feature points can be completed in a relatively short time. [4: Fourth Embodiment]

[0066] Fourth Embodiment of Information Processing Apparatus, Information Processing Method, and Recording Medium Will Be Described Hereinafter, a fourth embodiment of an information processing apparatus, an information processing method, and a recording medium will be described using an information processing apparatus 4 according to this disclosure.

[0067] The information processing device 4 is configured as a device for accurately inputting feature points, similar to the information processing device 1 to the information processing device 3. Furthermore, the information processing device 4 may be configured as a device for determining whether or not to generate an edited face image, similar to the information processing device 3. The fourth embodiment differs from the first to third embodiments in the operation of the image generation unit 412. The image generation unit 412 generates an edited face image by superimposing an image for editing on at least a part of a face region that is a blind spot in the face image. [4-1: Information Processing Method Executed by Information Processing Device 4]

[0068] 10A and 10B are conceptual diagrams of information processing operations executed by the information processing device 4. Fig. 10A illustrates an example of an acquired face image. [Superimposition Example 1: Superimposing a processed face image over the entire face region]

[0069] The image generating unit 412 may generate an edited face image by superimposing an image for editing on the entire face region in the face image. Fig. 10(a) illustrates an example in which a part of the face region (the lower left region in Fig. 10) is occluded. As illustrated in Fig. 10(b), the image generating unit 412 may generate an edited face image by superimposing an image for editing SI1 on the entire face region in the face image. Fig. 10(b) illustrates an example of an edited face image generated by superimposing the image for editing SI1. [Superimposition Example 2: Superimposing an edited face image SI on a part of the face region]

[0070] The image generating unit 412 may generate an edited face image by superimposing an image for editing on a part of a face region that is a blind spot in the face image. The image generating unit 412 may determine which region is a blind spot using a determination model. The image generating unit 412 may generate an edited face image by superimposing an image for editing SI2 on a part of a face region that is occluded in the face image, as exemplified in FIG. 10(c). FIG. 10(c) illustrates an example of an edited face image generated by superimposing an image for editing SI2. [4-2: Technical Effects of Information Processing Device 4]

[0071] The information processing device 4 according to this disclosure can compensate for the invisible parts of the face by superimposing an image for processing, thereby assisting in input of facial feature points. [5: Fifth Embodiment]

[0072] A fifth embodiment of an information processing device, an information processing method, and a recording medium will be described. Hereinafter, a fifth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 5 according to this disclosure.

[0073] The information processing device 5 is configured as a device for accurately inputting feature points, similar to the information processing devices 1 to 4. Furthermore, the information processing device 5 may be configured as a device for determining whether or not to generate a processed face image, similar to the information processing devices 3 and 4. The fifth embodiment differs from the first to fourth embodiments in the operation of the image generation unit 512. [5-1: Information Processing Method Executed by Information Processing Device 5]

[0074] The image generation unit 512 generates an edited face image based on three-dimensional information about the face shape. The image generation unit 512 generates an edited face image using a face shape sample. The face shape sample includes three-dimensional information about the face shape. [a. Example 1 of Acquiring a Face Shape Sample: Selecting from Already Created Face Shape Samples]

[0075] In acquisition example 1, the image generation unit 512 uses facial shape samples that have already been created. In acquisition example 1, a plurality of facial shape samples have already been created before the start of a series of operations that begins with acquiring a facial image. The plurality of facial shape samples that have already been created may be stored in, for example, a facial shape storage unit 522 that is realized in the storage device 12. Note that the storage unit that stores the plurality of facial shape samples that have already been created may be realized in a storage device other than the storage device 12.

[0076] Each of the plurality of face shape samples may be characterized by the positional relationship of facial features. In other words, each of the plurality of face shape samples is characterized by the positional relationship of facial feature points. The plurality of face shape samples may be created so that any human face resembles one of the plurality of face shape samples.

[0077] The image generation unit 512 obtains positional relationship information indicating the positional relationship of facial features from the face region of the face image. Based on the positional relationship information, the image generation unit 512 selects a face shape sample corresponding to the positional relationship information from the face shape storage unit 522. [b. Generation of processed face image]

[0078] The image generating unit 512 generates a processed face image using the selected face shape sample. The image generating unit 512 aligns the three-dimensional face shape sample so that the sample matches the face image.

[0079] The alignment operation performed by the image generation unit 512 starts with superimposing a two-dimensional image capturing a face shape sample from the front on the face image, and gradually changes the capture angle. The alignment operation may be performed manually. The alignment operation includes at least one of rotation, translation, and scaling. For example, the image generation unit 512 may enlarge the face shape to make the face appear fuller, or may shrink the face shape to make the face appear slimmer.

[0080] The image generation unit 512 may also align the facial shape samples using information on the visible parts of the facial image. The image generation unit 512 may also align the facial shape samples based on the visible feature points of the facial image. The image generation unit 512 may also achieve alignment by geometric calculation.

[0081] The image generation unit 512 may use AI that outputs information regarding the alignment of a face shape sample when visible feature points of a face image are input. Alternatively, the image generation unit 512 may use AI that outputs information regarding the alignment of a face shape sample when the face image itself is input. [c. Facial Shape Sample Acquisition Example 2: Generating a Facial Shape Sample According to an Acquired Facial Image]

[0082] The image generation unit 512 may estimate a three-dimensional face shape from a two-dimensional face image (i.e., a face image acquired by the image acquisition unit 211) and generate information indicating the three-dimensional face shape. For example, techniques such as "Face Alignment" and "Face Reconstruction" may be used to estimate a three-dimensional face shape from a two-dimensional face image. When estimating a three-dimensional face shape, the image generation unit 512 may also generate a processed face image in the same way as when selecting from already created face shape samples. [d. Use of Texture Information]

[0083] When estimating a three-dimensional face shape from a two-dimensional face image, information representing the texture of the face (referred to as "texture information") may be estimated in addition to the three-dimensional face shape. The texture information is information indicating the skin type of the face, etc. The texture information includes information that can identify facial feature points, i.e., information indicating the contours of facial features, etc. In this embodiment, the texture information may be used to assist in inputting the feature points.

[0084] The output unit 513 displays a facial image based on the texture information on the input device 14 and a touch panel serving as the output device 15. The user inputs facial feature points into the facial image based on the texture information. The feature point receiving unit 514 receives the input of facial feature points from the user. The output unit 513 may determine the positions of the feature points on the two-dimensional facial image based on the input feature point positions, and output a facial image on which the feature points are superimposed. The output unit 513 may calculate and output the coordinates of the feature points on the two-dimensional facial image based on the input feature point positions. [5-2: Technical Effects of Information Processing Device 5]

[0085] The information processing device 5 according to this disclosure estimates and generates a three-dimensional face shape based on visible facial parts, and compensates for invisible parts of a two-dimensional facial image based on the three-dimensional face shape, thereby assisting in the input of feature points. It is difficult to identify the position of feature points that are hidden by a profile view, etc., using only a two-dimensional facial image. The information processing device 5 uses three-dimensional facial information, thereby enabling highly accurate input of feature points. [6: Sixth Embodiment]

[0086] An information processing device, an information processing method, and a recording medium according to a sixth embodiment will be described. Hereinafter, an information processing device, an information processing method, and a recording medium according to a sixth embodiment will be described using an information processing device 6 according to this disclosure.

[0087] The information processing device 6 is configured as a device for accurately inputting feature points, similar to the information processing devices 1 to 5. Furthermore, the information processing device 6 may be configured as a device for determining whether or not to generate a processed face image, similar to the information processing devices 3 to 5. The sixth embodiment differs from the first to fifth embodiments in the operation of the output unit 613. [6-1: Information Processing Method Executed by Information Processing Device 6]

[0088] 13A and 13B are conceptual diagrams of information processing operations executed by the information processing device 6. Fig. 13A illustrates examples of face shape samples corresponding to face images.

[0089] The image generation unit 612 rotates a face shape sample corresponding to the face image to generate a processed face image. This processed face image may be referred to as a rotated face image. The image generation unit 612 may generate a rotated face image in which the face shape sample corresponding to the face image is captured from the front. The output unit 613 outputs the rotated face image. Fig. 13(b) illustrates an example of a rotated face image in which the face shape sample corresponding to the face image is captured from the front.

[0090] The feature point receiving unit 614 receives input of the positions of the feature points on the rotated face image designated by the user. Fig. 13C illustrates an example in which a facial feature point on the rotated face image is designated.

[0091] The image generating unit 612 rotates the rotated face image on which the feature points are superimposed to make the face face oriented in the same way as the acquired face image. The output unit 613 outputs the acquired face image in association with the positions of the rotated feature points. Fig. 13(d) illustrates an example in which the positions of the rotated feature points are superimposed on the acquired face image. [6-2: Technical Effects of the Information Processing Device 6]

[0092] The information processing device 6 according to this disclosure generates a rotated face image capturing a face shape sample corresponding to a face image from the front, so that the positions of invisible feature points can be accurately input. Furthermore, the information processing device 6 makes the posture of the three-dimensional face shape with input feature points the same as the posture of the acquired face image, so that the positions of invisible feature points can be accurately reflected in the acquired face image. [7: Seventh Embodiment]

[0093] Seventh Embodiment of an Information Processing Device, an Information Processing Method, and a Recording Medium Will Be Described Hereinafter, a seventh embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 7 according to this disclosure.

[0094] The information processing device 7 is configured as a device for accurately inputting feature points, similar to the information processing devices 1 to 6. Furthermore, the information processing device 7 may be configured as a device for determining whether or not to generate a processed face image, similar to the information processing devices 3 to 6. The seventh embodiment may further provide a user interface. The seventh embodiment differs from the first to sixth embodiments in the operation of the output unit 713. [7-1: Information Processing Method Executed by the Information Processing Device 7]

[0095] Fig. 15 is a conceptual diagram of an information processing operation executed by the information processing device 7. As illustrated in Fig. 15(a), the output unit 713 displays a face image before processing and a processed face image after processing on the display unit Da. Fig. 15(a) illustrates a case in which the processed face image after processing is displayed on the left side of the display unit Da, and the face image before processing is displayed on the right side of the display unit Da. As illustrated on the right side of the display unit Da in Fig. 15(a), the unprocessed face image has part of the face hidden, making it difficult to input feature points.

[0096] The feature point receiving unit 714 receives input of the positions of feature points designated by the user on the processed face image. For example, the feature point receiving unit 714 receives input of the positions of feature points designated by the user on the processed face image shown on the left side of Figures 15(b) and 15(c). Note that Figure 15(b) illustrates an example of the display unit Db when feature points are being input, and Figure 15(c) illustrates an example of the display unit Dc when feature point input is complete.

[0097] The output unit 713 displays the feature points superimposed on the face image in response to input of the positions of the feature points designated by the user on the processed face image. The output unit 713 displays the feature points superimposed on the face image, for example, as shown on the right side of Figures 15(b) and (c), in response to input of the positions of the feature points designated by the user on the processed face image shown on the left side of Figures 15(b) and (c). In this way, the output unit 713 can reflect and display the feature points on the face image in response to input of the feature points on the processed face image. As exemplified on the right side of Figure 15(c), the positions of invisible feature points can also be accurately reflected. [7-2: Technical Effects of Information Processing Device 7]

[0098] The information processing device 7 according to this disclosure provides a user interface that assists in inputting accurate feature points. [8: Supplementary Note]

[0099] The above-described embodiment can be further described as in the following supplementary notes, but is not limited to them. [Supplementary Note 1] An information processing device comprising: an acquisition means for acquiring a facial image; a generation means for generating an edited facial image by editing at least a part of the facial area of ​​the facial image; a first output means for outputting the edited facial image; a reception means for receiving input of feature information for the edited facial image; and a second output means for outputting the feature information in association with the facial image. [Supplementary Note 2] The information processing device according to Supplementary Note 1, wherein the reception means receives input of the feature information from a user who uses the information processing device. [Supplementary Note 3] The information processing device according to Supplementary Note 1, comprising: a determination means for determining whether or not the generation means should generate the edited facial image. [Supplementary Note 4] The information processing device according to Supplementary Note 3, wherein the determination means uses a determination model to determine whether or not the generation means should generate the edited facial image, and the determination model, when the facial image is input, outputs information indicating whether or not the generation means needs to generate the edited facial image. [Supplementary Note 5] The information processing device according to Supplementary Note 4, wherein the determination model outputs information indicating that the generation means needs to generate the processed face image when a face image in which a portion of a face region is a blind spot is input. [Supplementary Note 6] The information processing device according to Supplementary Note 4, wherein the determination model outputs information indicating that the generation means needs to generate the processed face image when a face image in which a portion of a region corresponding to a facial feature point is input. [Supplementary Note 7] The information processing device according to Supplementary Note 1, wherein the generation means generates the processed face image by superimposing an image for processing on at least a portion of the face region in the blind spot in the face image. [Supplementary Note 8] The information processing device according to Supplementary Note 1, wherein the generation means obtains positional relationship information indicating a positional relationship between facial features from the face region of the face image, obtains three-dimensional information of a face shape corresponding to the positional relationship information from a face shape database, and generates the processed face image based on the three-dimensional information of the face shape.[Supplementary Note 9] The information processing device according to Supplementary Note 2, wherein the feature information is information indicating facial feature points, the receiving means receives input of positions of the feature points on the processed face image designated by the user, and the second output means superimposes the feature points on the face image based on the input of the positions of the feature points. [Supplementary Note 10] The information processing device according to Supplementary Note 9, further comprising third output means for displaying a rotated face image obtained by rotating a face region of the processed face image on a display unit, the receiving means receiving input of the positions of the feature points designated by the user on the rotated face image. [Supplementary Note 11] The information processing device according to Supplementary Note 9, further comprising output means including the first output means and the second output means, the output means displays the face image and the processed face image on a display unit, and displays the feature points superimposed on the face image in accordance with input of the positions of the feature points designated by the user on the processed face image. [Supplementary Note 12] An information processing method executed by a computer, comprising: acquiring a facial image, generating an edited facial image by editing at least a part of the facial area of ​​the facial image, outputting the edited facial image, accepting input of feature information for the edited facial image, and outputting the feature information and the facial image in association with each other. [Supplementary Note 13] A recording medium having recorded thereon a computer program that causes a computer to execute an information processing method, comprising: acquiring a facial image, generating an edited facial image by editing at least a part of the facial area of ​​the facial image, outputting the edited facial image, accepting input of feature information for the edited facial image, and outputting the feature information and the facial image in association with each other.

[0100] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0101] 1, 2, 3, 4, 5, 6, 7 Information processing device 111, 211 Image acquisition unit 112, 212, 412, 512 Image generation unit 113 First output unit 114 Input reception unit 115 Second output unit 213, 313, 513, 613, 713 Output unit 214, 314, 514, 614, 714 Feature point reception unit 215, 315 Registration unit 316 Determination unit

Claims

1. An information processing device comprising: an acquisition means for acquiring a facial image; a generation means for generating an edited facial image by editing at least a portion of the facial area of ​​the facial image; a first output means for outputting the edited facial image; a reception means for receiving input of characteristic information for the edited facial image; and a second output means for outputting the characteristic information in association with the facial image.

2. The information processing device according to claim 1, wherein the accepting means accepts input of the characteristic information from a user who uses the information processing device.

3. The information processing device according to claim 1, further comprising a determination means for determining whether or not the processed facial image is to be generated by the generation means.

4. The information processing device according to claim 3, wherein the determination means uses a determination model to determine whether or not the processed facial image should be generated by the generation means, and the determination model outputs, when the facial image is input, information indicating whether or not the processed facial image should be generated by the generation means.

5. The information processing device according to claim 4, wherein the determination model outputs information indicating that the processed face image needs to be generated by the generation means when a face image in which part of the face area is blind is input.

6. The information processing device according to claim 4, wherein when a facial image that does not include part of an area corresponding to a facial feature point is input, the determination model outputs information indicating that the processed facial image needs to be generated by the generation means.

7. The information processing device according to claim 1, wherein said generating means generates said processed face image by superimposing an image for processing on at least a part of a face region in a blind spot in said face image.

8. The information processing device according to claim 1, wherein the generating means obtains positional relationship information indicating the positional relationship of facial features from the face area of ​​the face image, obtains three-dimensional information of the face shape corresponding to the positional relationship information from a face shape database, and generates the processed face image based on the three-dimensional information of the face shape.

9. The information processing device according to claim 2, wherein the feature information is information indicating facial feature points, the receiving means receives input of the positions of the feature points specified by the user on the processed face image, and the second output means superimposes the feature points on the face image based on the input of the positions of the feature points.

10. An information processing device according to claim 9, further comprising a third output means for displaying a rotated face image obtained by rotating the face area of ​​the processed face image on a display unit, wherein the receiving means receives input of the position of the feature point specified by the user on the rotated face image.

11. An information processing device according to claim 9, comprising an output means including the first output means and the second output means, wherein the output means displays the facial image and the processed facial image on a display unit, and displays the feature points superimposed on the facial image in accordance with input of the positions of the feature points specified by the user on the processed facial image.

12. An information processing method executed by a computer, comprising: acquiring a facial image; generating an edited facial image by editing at least a portion of the facial area of ​​the facial image; outputting the edited facial image; accepting input of feature information for the edited facial image; and outputting the feature information in association with the facial image.

13. A recording medium having recorded thereon a computer program that causes a computer to execute an information processing method that includes acquiring a facial image, processing at least a portion of the facial area of ​​the facial image to generate a processed facial image, outputting the processed facial image, accepting input of feature information for the processed facial image, and outputting the feature information in association with the facial image.

Images