Information processing device, information processing method, and recording medium

Abstract

This information processing device includes: an acquisition means for acquiring a face image; an image processing means for applying, to the face image, image quality enhancement processing for enhancing the quality of the face image; a first output means for outputting the processed face image subjected to the image quality enhancement processing; an input reception means for receiving input of feature information for the processed face image; and a second output means for outputting the feature information and the face image in association with each other.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] It is known to detect feature points from a face image that has been subjected to image processing. For example, Patent Document 1 discloses a process of correcting a face area and detecting feature points of the face using the image that has been subjected to the face area correction process.

[0003] JP 2013-065119 A

[0004] This disclosure aims to improve upon the related art discussed above.

[0005] One aspect of the information processing device disclosed herein comprises an acquisition means for acquiring a facial image, an image processing means for applying a quality improvement process to the facial image to increase the quality of the facial image, a first output means for outputting a processed facial image that has been subjected to the quality improvement process, an input accepting means for accepting input of feature information for the processed facial image, and a second output means for outputting the feature information in association with the facial image.

[0006] One aspect of the information processing method disclosed herein is an information processing method executed by a computer, which acquires a facial image, applies a quality improvement process to the facial image to increase the quality of the facial image, outputs a processed facial image that has been subjected to the quality improvement process, accepts input of feature information for the processed facial image, and outputs the feature information in association with the facial image.

[0007] One aspect of the recording medium of this disclosure is a recording medium having recorded thereon a computer program that causes a computer to execute an information processing method that acquires a facial image, applies a quality improvement process to the facial image to increase the quality of the facial image, outputs a processed facial image that has been subjected to the quality improvement process, accepts input of feature information for the processed facial image, and outputs the feature information in association with the facial image.

[0008] FIG. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a flowchart showing a flow of information processing operation in an information processing device according to the present disclosure. FIG. 3 is a diagram showing an example of facial feature points. FIG. 4 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 5 is a flowchart showing a flow of information processing operation in an information processing device according to the present disclosure. FIG. 6 is a schematic diagram showing an information processing method in an information processing device according to the present disclosure. FIG. 7 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 8 is a schematic diagram showing an information processing method in an information processing device according to the present disclosure. FIG. 9 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 10 is a schematic diagram showing an information processing method in an information processing device according to the present disclosure. FIG. 11 is a flowchart showing a flow of information processing operation in an information processing device according to the present disclosure. FIG. 12 is a schematic diagram showing an information processing method in an information processing device according to the present disclosure.

[0009] Hereinafter, an information processing device, an information processing method, and a recording medium according to an embodiment will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 according to this disclosure. [1-1: Configuration of Information Processing Device 1]

[0011] The configuration of an information processing device 1 according to this disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an information processing device 1 according to this disclosure.

[0012] 1, the information processing device 1 includes an image acquisition unit 111, an image processing unit 112, a first output unit 113, an input reception unit 114, and a second output unit 115. The processes performed by the image acquisition unit 111, the image processing unit 112, the first output unit 113, the input reception unit 114, and the second output unit 115 will be described with reference to FIG. 2. [1-2: Information Processing Method Executed by the Information Processing Device 1]

[0013] An information processing method executed by the information processing device 1 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of the flow of the information processing method executed by the information processing device 1. The information processing device 1 may be realized by a computer executing a computer program for causing a computer to execute the information processing method.

[0014] 2, the image acquisition unit 111 acquires a facial image (step S11). The image processing unit 112 performs quality improvement processing on the facial image to improve the quality of the facial image (step S12). The first output unit 113 outputs the facial image that has been subjected to the quality improvement processing (referred to as a "processed facial image") (step S13).

[0015] The input receiving unit 114 receives input of feature information of the processed face image (step S14). The feature information is information indicating the features of the face image. The second output unit 115 outputs the feature information input in step S14 in association with the face image acquired in step S11 (step S15). [1-3: Technical Effects of Information Processing Device 1]

[0016] The information processing device 1 according to this disclosure can assist in inputting accurate feature information. [2: Second embodiment]

[0017] A second embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the second embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 2 according to this disclosure. [2-1: Input of feature points]

[0018] In this embodiment, the facial image feature information is information indicating facial feature points. When constructing a facial feature point database, input of the facial feature points is required. The facial feature point database includes a data set of facial images and facial feature points in the facial images. The facial feature point database may be used in processing that uses facial feature points, such as face matching (referred to as "feature point processing").

[0019] When generating a dataset of face images and facial feature points, it is required to input defined feature points. The defined feature points are, for example, feature points that are specified to be used in feature point processing. Therefore, when generating a dataset, it is required to input all defined feature points.

[0020] 3 shows an example of defined feature points. For example, as shown in FIG. 3, the inner corners of the left and right eyebrows, the arch of the eyebrows, the outer corners of the eyebrows, the pupils, the outer corners of the eyes, the wings of the nose, the nostrils, the corners of the mouth, the protuberance of the nose, and the center of the lips may be defined as feature points to be used in the feature point processing.

[0021] There are cases where it is easy to identify facial feature points from a facial image, and cases where it is difficult. For example, when the positions of facial feature points in a facial image are not visible, it corresponds to a case where it is difficult to identify facial feature points from a facial image.

[0022] The position of the facial feature points in the face image cannot be seen when a part of the face area in the face image is in a blind spot. When a part of the face is occluded, the part of the face area in the face image is in a blind spot. Also, when the face is tilted relative to the plane of the face image, the part of the face area in the face image is in a blind spot.

[0023] Furthermore, when the quality of a facial image is low, it is often difficult to accurately identify facial feature points from the facial image. For example, a facial image with low quality corresponds to a facial image with low resolution. Furthermore, for example, the quality of an image taken in a dark environment, such as at night, is often not suitable for inputting feature points.

[0024] However, as described above, even when it is difficult to identify facial feature points from a facial image, it is necessary to input the feature points with high accuracy. Facial feature points that are difficult to identify from a facial image are sometimes referred to as "difficult-to-identify feature points." Furthermore, facial feature points that are easy to identify from a facial image are sometimes referred to as "easy-to-identify feature points."

[0025] Furthermore, the feature point processing process requires the input of facial feature points. The feature point processing process corresponds to, for example, the preprocessing process in face matching. The more feature points input in the preprocessing, the more accurate the feature point processing can often be. In other words, easily identifiable feature points alone provide little information, which may have a significant impact on feature point processing.

[0026] Furthermore, depending on the system that performs feature point processing, it may be necessary to input all defined feature points, including hard-to-identify feature points. In this case, it is not possible to skip inputting hard-to-identify feature points.

[0027] As described above, there are cases where it is necessary to input difficult-to-identify feature points. This embodiment relates to inputting difficult-to-identify feature points. [2-2: Configuration of Information Processing Device 2]

[0028] The configuration of the information processing device 2 according to this disclosure will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 2 according to this disclosure.

[0029] 4 , the information processing device 2 includes a calculation device 11, a storage device 12, and a communication device 13. In addition to the calculation device 11, the storage device 12, and the communication device 13, the information processing device 2 may further include an input device 14 and an output device 15. However, the information processing device 2 does not necessarily have to include at least one of the input device 14 and the output device 15. The calculation device 11, the storage device 12, the communication device 13, the input device 14, and the output device 15 may be connected via a data bus 16.

[0030] The arithmetic device 11 includes at least one processor (i.e., one processor or multiple processors) as hardware. The processor may include, for example, a processor conforming to a von Neumann computer architecture. The processor conforming to the von Neumann computer architecture may include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor may include, for example, a processor conforming to a non-von Neumann computer architecture. The processor conforming to the non-von Neumann computer architecture may include at least one of an FPGA (Field Programmable Gate Array) and an ASIC (Application Specific Circuit).

[0031] The arithmetic device 11 reads a computer program 121 including at least one of computer program code and computer program instructions. For example, the arithmetic device 11 may read the computer program 121 stored in the storage device 12. For example, the arithmetic device 11 may read the computer program 121 stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the information processing device 2. The computer program 121 read from the recording medium may be stored in the storage device 12. The arithmetic device 11 may acquire (i.e., download or read) the computer program 121 from a device (not shown) located outside the information processing device 1 via the communication device 13 (or another communication device). The downloaded computer program 121 may be stored in the storage device 12.

[0032] The arithmetic device 11 executes the loaded computer program 121. As a result, logical functional blocks for executing processing to be performed by the information processing device 2 (e.g., information processing described below) are realized within the arithmetic device 11. In other words, the arithmetic device 11, together with the storage device 12, etc. in which the computer program 121 is recorded (in other words, together with the storage device 12 and the computer program 121 recorded in the storage device 12, etc.), can function as a controller or computer for realizing the logical functional blocks for executing processing to be performed by the information processing device 2. In other words, together with at least one processor included in the arithmetic device 11, the memory (recording medium) included in the storage device 12, etc., and the computer program 121 are configured so that the information processing device 2 performs processing to be performed by the information processing device 2 (e.g., information processing described below). The arithmetic device 11 may output information to another computer, cloud server, or other device (not shown) provided outside the information processing device 2 via the communication device 13 (or other communication device).

[0033] The recording medium for recording the computer program 121 executed by the arithmetic device 11 may be at least one of a CD-ROM, CD-R, CD-RW, flexible disk, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, and Blu-ray (registered trademark) optical disk, magnetic medium such as magnetic tape, magneto-optical disk, semiconductor memory such as USB memory, and any other medium capable of storing a program. The recording medium may include a device capable of recording the computer program 121 (for example, a general-purpose device or a dedicated device in which the computer program 121 is implemented in a state in which it can be executed in at least one of the forms of software and firmware). Furthermore, each process or function included in the computer program 121 may be realized by a logical processing block realized within the arithmetic device 11 when the arithmetic device 11 (i.e., processor) executes the computer program 121, or may be realized by hardware such as a predetermined gate array (FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) included in the arithmetic device 11, or may be realized in a form that mixes logical processing blocks and partial hardware modules that realize some elements of the hardware.

[0034] A high-quality model that can be constructed by machine learning is implemented within the arithmetic device 11 by the arithmetic device 11 executing the computer program 121. An example of a high-quality model that can be constructed by machine learning is a high-quality model including a neural network (so-called artificial intelligence (AI)). In this case, learning of the high-quality model may include learning of parameters of the neural network (e.g., at least one of a weight and a bias). The arithmetic device 11 executes at least a quality enhancement process using the high-quality model. A high-quality model that has been constructed by machine learning may be implemented in the arithmetic device 11. A high-quality model that has been constructed by offline machine learning using training data may be implemented in the arithmetic device 11. Furthermore, the high-quality model implemented in the arithmetic device 11 may be updated by online machine learning on the arithmetic device 11. Alternatively, the calculation device 11 may perform information processing using a high-quality model implemented in a device external to the calculation device 11 (i.e., a device provided outside the information processing device 2) in addition to or instead of the high-quality model implemented in the calculation device 11.

[0035] Fig. 4 shows an example of logical functional blocks implemented within the arithmetic device 11 to execute information processing. As shown in Fig. 4, an image acquisition unit 211, an image processing unit 212, an output unit 213, a feature point reception unit 214, and a registration unit 215 are implemented within the arithmetic device 11. The image processing unit 212 executes quality improvement processing using the above-described quality improvement model. Note that the processing performed by each of the image acquisition unit 211, the image processing unit 212, the output unit 213, the feature point reception unit 214, and the registration unit 215 will be described with reference to Figs. 5 and 6.

[0036] The storage device 12 includes at least one memory capable of storing desired data. In other words, the storage device 12 includes at least one memory containing desired data. For example, the storage device 12 may store a computer program 121 executed by the arithmetic device 11. In this case, the storage device 12 (memory) may be used as the above-mentioned recording medium for recording the computer program 121 executed by the arithmetic device 11. The storage device 12 may temporarily store data used by the arithmetic device 11 when the arithmetic device 11 is executing the computer program 121. The storage device 12 may also store data to be stored long-term by the information processing device 2. The storage device 12 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 12 may include a non-transitory recording medium.

[0037] A registered image database DB may be realized in the storage device 12. The registered image database DB may be realized in a storage device outside the information processing device 2.

[0038] The communication device 13 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown). The communication device 13 may be a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), Bluetooth (registered trademark), or USB (Universal Serial Bus).

[0039] The input device 14 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 14 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 14 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0040] The output device 15 is a device that outputs information to the outside of the information processing device 2. For example, the output device 15 may output information as an image. That is, the output device 15 may include a display device (a so-called display) that can display an image showing the information to be output. The display device may be a touch panel with an input function. For example, the output device 15 may output information as sound. That is, the output device 15 may include an audio device (a so-called speaker) that can output sound. For example, the output device 15 may output information on paper. That is, the output device 15 may include a printing device (a so-called printer) that can print desired information on paper.

[0041] Incidentally, in situations where highly accurate feature points are required, it is often the case that a human inputs feature points, or a human performs a final check on the input of feature points detected by a non-human (e.g., AI). Situations where highly accurate feature points are required include situations where databases are constructed, situations where face matching is performed, etc. In this embodiment, a case where input of feature points of a face image is received from a user who uses the information processing device 2 will be described. [2-3: Information processing method executed by the information processing device 2]

[0042] 5 and 6, the information processing operation in the information processing device 2 will be described. Fig. 5 is a flowchart showing the flow of the information processing operation executed by the information processing device 2. Fig. 6 is a conceptual diagram of the information processing operation executed by the information processing device 2.

[0043] As shown in Fig. 5, the image acquisition unit 211 acquires a face image (step S21). The image acquisition unit 211 acquires a face image for which facial feature points are to be input. Fig. 6(a) shows an example of a face image acquired by the image acquisition unit 211. As shown in Fig. 6(a), the face image acquired by the image acquisition unit 211 may be a blurry, low-quality face image.

[0044] The image processing unit 212 performs quality enhancement processing on the facial image to improve the quality of the facial image (step S22). The image processing unit 212 may perform quality enhancement processing on the facial image using a quality enhancement model. The quality enhancement model is a mechanism that, when a facial image is input, outputs a processed facial image that has been subjected to quality enhancement processing. FIG. 6B shows an example of a facial image whose quality has been enhanced by the image processing unit 212. As shown in FIG. 6B, the facial image whose quality has been enhanced by the image processing unit 212 is a higher-quality facial image than the low-quality facial image shown in FIG. 6A.

[0045] In this embodiment, the quality improvement model may be a mechanism for increasing the resolution of an image, which may be called "super-resolution," "upscaling," or the like.

[0046] Note that the AI ​​can achieve higher resolution with greater accuracy by focusing on specific images. The quality improvement model used in this embodiment is an AI specialized for increasing the resolution of facial images. In other words, an AI specialized for increasing the resolution of fingerprint images may be constructed depending on the requirements.

[0047] The output unit 213 outputs the processed face image (step S23). The output unit 213 may display the output of the high-quality model on, for example, a touch panel serving as the input device 14 and the output device 15. The output unit 213 may display the processed face image illustrated in Fig. 6(b) on a display device.

[0048] Even when high-resolution images are obtained using AI, such as a high-quality model, the positions of facial features tend to appear in statistically likely positions. Therefore, inputting feature points into a face image with high resolution using AI can be performed more accurately and quickly than inputting feature points into a face image that has not been high-resolution. Note that the tendency for the positions of facial features to appear in statistically likely positions becomes more accurate and stable as more data is used for learning.

[0049] The feature point receiving unit 214 receives input of feature points of the processed face image from the user (step S24). Specifically, the feature point receiving unit 214 receives input of feature points on the processed face image designated by the user from the user. The feature point receiving unit 214 may receive, for example, an input operation by the user on the processed face image displayed on a touch panel serving as the input device 14. The user's input operation is an operation of designating facial feature points in the processed face image. Fig. 6(c) illustrates an example in which facial feature points in the processed face image are designated.

[0050] However, when high-resolution images are created using AI, identity is often lost. For a facial image of a human face, high-resolution images created using AI may result in a facial image of a different person. In other words, high-resolution images created using AI can change identity, so it is often inappropriate to use processed facial images as registration images.

[0051] Therefore, in this embodiment, a high-resolution face image (processed face image) is used when inputting feature points, and the face image to be registered is not the processed face image but the face image that has not been high-resolution (the face image acquired in step S21).

[0052] The output unit 213 associates the feature points input in step S24 with the facial image acquired in step S21 and outputs them (step S25). Specifically, the output unit 213 may superimpose the feature points on the facial image based on the positions of the feature points on the processed facial image designated by the user. The output unit 213 may display the facial image on which the feature points are superimposed on, for example, a touch panel serving as the output device 15. Figure 6(d) illustrates an example in which feature points are superimposed on a facial image based on the positions of the feature points on the processed facial image designated by the user.

[0053] The registration unit 215 associates the feature points input in step S24 with the face image acquired in step S21 and registers them in the registered image database DB (step S26). The registration unit 215 may also register a face image with superimposed feature points that has not been made high-resolution in the registered image database DB. [2-4: Comparison Results]

[0054] For low-resolution facial images, it is particularly difficult to input feature points corresponding to the edges of the face. By increasing the resolution of a low-resolution facial image, feature points can be input accurately and quickly.

[0055] For example, prepare feature point A input to a face image with a resolution suitable for inputting feature points, feature point B input to a low-resolution face image with a low resolution unsuitable for inputting feature points, and feature point C input to a processed face image obtained by increasing the resolution of the low-resolution image. In this case, when feature point A, feature point B, and feature point C are compared, the similarity between feature point A and feature point C is higher than the similarity between feature point A and feature point B. [2-5: Technical Effects of Information Processing Device 2]

[0056] Feature point detection using AI is widely used, and AI can detect feature points with a certain degree of accuracy even from low-quality images. However, as described above, when a human performs a final check on the input of feature points detected by AI, if the image quality is low, it is often difficult for the human to determine whether the feature points detected by AI should be corrected. Furthermore, because the image quality is low, even if the feature points are corrected, there is a limit to the accuracy. Furthermore, accuracy is likely to vary depending on the person performing the final check. As such, low image quality can cause problems such as difficulty in identifying the exact positions of feature points and individual variability in the identified positions.

[0057] The information processing device 2 according to this disclosure can improve the accuracy of inputting feature points. Compared to inputting feature points to a low-quality face image, it is possible to realize input of feature points with high accuracy. Since the quality of the face image is improved, it is possible to reduce individual differences (variation in coordinates, etc.) between individuals who input feature points.

[0058] Furthermore, input and correction work for low-quality images requires more man-hours than work for high-quality images. The information processing device 2 can reduce the man-hours required for input and correction work for face images.

[0059] In other embodiments, feature information may be input from sources other than a user of the information processing device. For example, feature information may be input from a trained network (referred to as "feature point detection AI"). By combining with feature point detection AI and increasing the resolution, it is expected that the detection accuracy of the feature point detection AI will also improve. By using it in combination with feature point detection AI, further load reduction can be achieved. For example, the user may modify the initial feature point position estimation by the feature point detection AI model as needed.

[0060] This embodiment may also be applied to images other than "face" images. This embodiment can be applied to images that undergo processing to extract feature points. [3: Third Embodiment]

[0061] A third embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the third embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 3 according to this disclosure. [3-1: Configuration of Information Processing Device 3]

[0062] The information processing device 3 is configured as a device for accurately inputting feature points, similar to the information processing device 1 and the information processing device 2. Furthermore, the information processing device 3 may be configured as a device for determining whether or not to perform quality improvement processing.

[0063] 7, the arithmetic device 11 in the third embodiment further includes a determination unit 316. The determination unit 316 determines whether or not to perform quality improvement processing on the face image. [3-2: Information Processing Method Executed by Information Processing Device 3]

[0064] As shown in FIG. 8 , the image acquisition unit 211 acquires a facial image (step S21). The determination unit 316 determines whether or not to apply quality improvement processing to the facial image (step S30). The determination unit 316 may determine the quality based on a user's visual judgment, or may automatically determine the quality. That is, the determination unit 316 may receive input of a user's determination result regarding whether or not to apply quality improvement processing to the facial image, and determine whether or not to apply quality improvement processing to the facial image. Alternatively, the determination unit 316 may use a determination model to determine whether or not to apply quality improvement processing to the facial image. The determination model may be constructed by machine learning and implemented in the computing device 11. The determination model may be, for example, AI including a neural network. Alternatively, the determination unit 316 may perform automatic determination without using a model constructed by machine learning. The automatic determination may be performed without receiving input of the determination result from the user. For example, the determination unit 316 may determine whether to perform quality improvement processing on a face image based on information calculated directly from pixel values ​​of the image. The information calculated directly from pixel values ​​of the image may be image statistics. Examples of image statistics include contrast, sharpness, histogram, noise intensity, etc. The determination unit 316 may determine that an image with low sharpness (blurred) or high noise intensity is of low quality, for example.

[0065] If it is determined that the facial image should be subjected to quality improvement processing (step S30: Yes), the image processing unit 212 performs quality improvement processing on the facial image to improve the quality of the facial image (step S22). The output unit 213 outputs the processed facial image (step S23).

[0066] The feature point receiving unit 314 receives input of facial feature points from the user (step S34). If it is determined that high-quality processing is necessary (step S30: Yes), the feature point receiving unit 314 receives input of feature points of the processed facial image from the user. On the other hand, if it is determined that high-quality processing is not necessary (step S30: No), the feature point receiving unit 314 receives input of feature points of the facial image acquired in step S21 from the user.

[0067] The output unit 313 associates the feature points input in step S34 with the facial image acquired in step S21 and outputs them (step S35). The registration unit 315 associates the feature points input in step S34 with the facial image acquired in step S21 and registers them in the registered image database DB (step S36). [3-3: Technical Effects of Information Processing Device 3]

[0068] The information processing device 3 according to this disclosure performs quality improvement processing on facial images that require quality improvement processing. Therefore, compared to performing quality improvement processing on all facial images, the input of facial feature points can be completed in a relatively short time. [4: Fourth Embodiment]

[0069] Fourth Embodiment of Information Processing Apparatus, Information Processing Method, and Recording Medium Will Be Described Hereinafter, a fourth embodiment of an information processing apparatus, an information processing method, and a recording medium will be described using an information processing apparatus 4 according to this disclosure.

[0070] The information processing device 4 is configured as a device for accurately inputting feature points, similar to the information processing device 1 to the information processing device 3. Furthermore, the information processing device 4 may be configured as a device for determining whether or not to perform quality improvement processing, similar to the information processing device 3. The fourth embodiment differs from the first to third embodiments in the operation of the output unit 413. [4-1: Information Processing Method Executed by the Information Processing Device 4]

[0071] 10A and 10B are conceptual diagrams of information processing operations executed by the information processing device 4. As illustrated in Fig. 10A, the output unit 413 displays a face image before quality improvement processing and a processed face image after quality improvement processing on the display unit Da. Fig. 10A illustrates a case where the processed face image after quality improvement processing is displayed on the left side of the display unit Da, and the face image before quality improvement processing is displayed on the right side of the display unit Da.

[0072] The feature point receiving unit 414 receives input of the positions of feature points designated by the user on the processed face image. For example, the feature point receiving unit 414 receives input of the positions of feature points designated by the user on the processed face image shown on the left side of Figures 10(b) and 10(c). Note that Figure 10(b) illustrates an example of the display unit Db when feature points are being input, and Figure 10(c) illustrates an example of the display unit Dc when feature point input is complete.

[0073] The output unit 413 displays the feature points superimposed on the facial image in response to input of the positions of the feature points designated by the user on the processed facial image. The output unit 413 displays the feature points superimposed on the facial image, for example, as shown on the right side of Figures 10(b) and (c), in response to input of the positions of the feature points designated by the user on the processed facial image shown on the left side of Figures 10(b) and (c). In this way, the output unit 413 can display the feature points reflected on the facial image in response to input of the feature points for the processed facial image. [4-2: Technical Effects of Information Processing Device 4]

[0074] The information processing device 4 according to this disclosure provides a user interface that assists in accurate input of feature points. [5: Fifth Embodiment]

[0075] A fifth embodiment of an information processing device, an information processing method, and a recording medium will be described. Hereinafter, a fifth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 5 according to this disclosure.

[0076] The information processing device 5 is configured as a device for accurately inputting feature points, similar to the information processing device 1 to the information processing device 4. Furthermore, the information processing device 5 may be configured as a device for determining whether or not to perform quality improvement processing, similar to the information processing device 3 and the information processing device 4. The fifth embodiment may further be configured as a device for learning a determination model. [5-1: Configuration of the information processing device 5]

[0077] As shown in Fig. 11, a learning unit 517 is further implemented in the arithmetic device 11 in the fifth embodiment. The learning unit 517 causes the determination unit 516 to learn a determination method. In other words, the learning unit 517 causes the determination model used by the determination unit 516 to learn the determination method. [5-2: Information Processing Method Executed by Information Processing Device 5]

[0078] The learning unit 517 uses learning data in which facial images are associated with information indicating whether or not high-resolution processing of the facial images is required (referred to as "necessity information"). Whether or not high-resolution processing of the facial images is required may be determined by a user. The learning data may be constructed by accumulating cases in which the user requested high-resolution processing and cases in which the user did not request high-resolution processing.

[0079] The learning unit 517 causes the determination model to learn a determination method so that the determination model to which the face image is input outputs the same as the associated necessity information. The learning of the determination model by the learning unit 517 may include learning of parameters of the neural network (for example, at least one of weights and biases).

[0080] Furthermore, when the determination model determines that high resolution is not necessary, cases in which the user requests high resolution may be accumulated to construct learning data for improving the accuracy of the determination of the determination model.While the user ultimately decides whether or not high resolution processing of a facial image is necessary, the determination model is constructed so that it can make a determination similar to the user's decision by learning the user's decision.[5-3: Technical Effects of the Information Processing Device 5]

[0081] The information processing device 5 according to this disclosure can provide a mechanism that automatically determines whether or not high resolution is required and can achieve high resolution of a low resolution image without the user being aware of it. [6: Sixth Embodiment]

[0082] An information processing device, an information processing method, and a recording medium according to a sixth embodiment will be described. Hereinafter, an information processing device, an information processing method, and a recording medium according to a sixth embodiment will be described using an information processing device 6 according to this disclosure.

[0083] The information processing device 6 is configured as a device for accurately inputting feature points, similar to the information processing device 1 to the information processing device 4. Furthermore, the information processing device 6 may be configured as a device for determining whether or not to perform quality improvement processing, similar to the information processing device 3 and the information processing device 4. The sixth embodiment may further be configured as a device for improving the quality of image portions for which the user desires quality improvement. [6-1: Configuration of the information processing device 6]

[0084] 12, the calculation device 11 in the sixth embodiment further includes a part receiving unit 618. The part receiving unit 618 receives a user's designation of a part of a face image to be subjected to quality improvement processing. [6-2: Information Processing Method Executed by Information Processing Device 6]

[0085] 13A and 13B are conceptual diagrams of information processing operations executed by the information processing device 6. As illustrated in Fig. 13A, the user may specify that quality improvement processing be performed on the eye region based on the position of the mouse cursor C. In this case, the partial receiving unit 618 receives a request for quality improvement processing on the eye region corresponding to the position of the mouse cursor C.

[0086] The image processing unit 612 performs quality improvement processing on a portion of the face image designated by the user. The image processing unit 612 may also perform real-time high resolution processing on the area around the mouse cursor. As illustrated in FIG. 13( b), the image processing unit 612 may also perform quality improvement processing on an eye area A. Furthermore, the output unit 613 may enlarge the eye area A on the display to support more accurate feature point input.

[0087] Furthermore, statistics of areas where high resolution is frequently required may be collected to improve the performance of the high quality model. The performance of the high quality model may be improved so that partial high resolution is automatically performed based on the tendency of the area specified by the user. [6-3: Technical Effects of the Information Processing Device 6]

[0088] The information processing device 6 according to this disclosure increases the resolution of not the entire image but only a portion designated by the user. This reduces the processing time and the processing load. [7: Seventh Embodiment]

[0089] Seventh Embodiment of an Information Processing Device, an Information Processing Method, and a Recording Medium Will Be Described Hereinafter, a seventh embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 7 according to this disclosure.

[0090] The information processing device 7 is configured as a device for accurately inputting feature points, similar to the information processing device 1 to the information processing device 6. Furthermore, similar to the information processing device 6, the information processing device 7 may be configured as a device for improving the quality of an image portion for which the user desires improvement. [7-1: Information Processing Method Executed by the Information Processing Device 7]

[0091] 15 and 16, the information processing operation in the information processing device 7 will be described. Fig. 15 is a flowchart showing the flow of the information processing operation executed by the information processing device 7. Fig. 16 is a conceptual diagram of the information processing operation executed by the information processing device 7.

[0092] As shown in FIG. 15 , the image acquisition unit 211 acquires a facial image (step S21). The portion acceptance unit 718 accepts, from the user, a designation of a portion of the facial image to be subjected to quality improvement processing. FIG. 16A illustrates a facial image acquired by the image acquisition unit 211. As illustrated in FIG. 16A , the facial image acquired by the image acquisition unit 211 may be a blurred, low-quality facial image. Also, as illustrated in FIG. 16A , the user may specify that quality improvement processing be performed on the eye area A based on the position of the mouse cursor C. In this case, the portion acceptance unit 718 accepts a request for quality improvement processing on the eye area A corresponding to the position of the mouse cursor C. The portion acceptance unit 718 determines whether or not designation of a portion of the facial image to be subjected to quality improvement processing has been accepted from the user (step S70).

[0093] When the designation of the portion of the facial image to be subjected to the quality improvement process is accepted (step S70: Yes), the image processing unit 712 performs the quality improvement process to improve the image quality of the designated portion of the facial image (step S72). FIG. 16(b) shows an example of a facial image whose quality has been improved by the image processing unit 712. As shown in FIG. 16(b), the image processing unit 712 may perform the quality improvement process on the eye area A. When the entire face is designated, the image processing unit 712 may perform the quality improvement process on the entire facial image. The output unit 713 outputs the processed facial image in which the quality improvement process has been performed on the designated portion of the facial image (step S73).

[0094] The feature point receiving unit 714 receives input of facial feature points from the user (step S74). If designation of a portion of the facial image to be subjected to quality improvement processing of the facial image is received (step S70: Yes), the feature point receiving unit 714 receives input of feature points from the user for a processed facial image in which quality improvement processing has been performed on the designated portion of the facial image. Figure 16(c) illustrates a case in which facial feature points in the processed facial image are designated. On the other hand, if designation of a portion of the facial image to be subjected to quality improvement processing of the facial image is not received (step S70: No), the feature point receiving unit 714 receives input of feature points for the facial image acquired in step S21 from the user.

[0095] The output unit 713 associates the feature points input in step S74 with the facial image acquired in step S21 and outputs them (step S75). The registration unit 315 associates the feature points input in step S74 with the facial image acquired in step S21 and registers them in the registered image database DB (step S76). Fig. 16(d) illustrates an example in which feature points are superimposed on a facial image based on the positions of the feature points on the processed facial image designated by the user. [7-2: Technical Effects of Information Processing Device 3]

[0096] The information processing device 7 according to this disclosure can reduce the processing required for increasing the resolution. Specifically, the information processing device 7 can reduce the processing time and the required hardware specifications. Furthermore, since the information processing device 7 can partially increase the resolution, it becomes easier for the user to focus on the high-resolution area. [8: Supplementary Note]

[0097] The above-described embodiment can be further described as, but is not limited to, the following supplementary notes. [Supplementary Note 1] An information processing device comprising: an acquisition means for acquiring a facial image; an image processing means for applying quality improvement processing to the facial image to improve the quality of the facial image; a first output means for outputting a processed facial image that has been subjected to the quality improvement processing; an input accepting means for accepting input of feature information for the processed facial image; and a second output means for outputting the feature information in association with the facial image. [Supplementary Note 2] The information processing device according to Supplementary Note 1, wherein the input accepting means accepts input of the feature information from a user using the information processing device. [Supplementary Note 3] The information processing device according to Supplementary Note 2, wherein the feature information is information indicating facial feature points; the input accepting means accepts input of positions of the feature points on the processed facial image designated by the user; and the second output means superimposes the feature points on the facial image based on the input of the positions of the feature points. [Supplementary Note 4] The information processing device according to Supplementary Note 1, comprising: a determination means for determining whether or not to apply the quality improvement processing to the facial image. [Supplementary Note 5] The information processing device according to Supplementary Note 3, comprising output means including the first output means and the second output means, wherein the output means displays the facial image and the processed facial image on a display unit, and displays the feature point superimposed on the facial image in accordance with input of the position of the feature point specified by the user on the processed facial image. [Supplementary Note 6] The information processing device according to Supplementary Note 4, comprising learning means for causing the determination means to learn a determination method using learning data that associates the facial image with information indicating whether the quality improvement process is required for the facial image. [Supplementary Note 7] The information processing device according to Supplementary Note 1, comprising part receiving means that receives, from a user using the information processing device, designation of a part of the facial image to be subjected to the quality improvement process, wherein the image processing means applies the quality improvement process to a part of the facial image. [Supplementary Note 8] The information processing device according to Supplementary Note 7, wherein the first output means outputs the processed facial image in which the quality improvement process has been performed on a part of the facial image.[Supplementary Note 9] An information processing method executed by a computer, comprising: acquiring a facial image, performing quality improvement processing on the facial image to improve the quality of the facial image, outputting the processed facial image after the quality improvement processing, accepting input of feature information for the processed facial image, and outputting the feature information and the facial image in association with each other. [Supplementary Note 10] A recording medium having recorded thereon a computer program that causes a computer to execute an information processing method, comprising: acquiring a facial image, performing quality improvement processing on the facial image to improve the quality of the facial image, outputting the processed facial image after the quality improvement processing, accepting input of feature information for the processed facial image, and outputting the feature information and the facial image in association with each other.

[0098] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0099] 1, 2, 3, 4, 5, 6 Information processing device 111, 211 Image acquisition unit 112, 212, 612 Image processing unit 113 First output unit 114 Input reception unit 115 Second output unit 213, 313, 413, 613 Output unit 214, 314, 414 Feature point reception unit 215, 315 Registration unit 316, 516 Determination unit 517 Learning unit 618 Partial reception unit

Claims

1. An information processing device comprising: an acquisition means for acquiring a facial image; an image processing means for applying a quality improvement process to the facial image to increase the quality of the facial image; a first output means for outputting a processed facial image that has been subjected to the quality improvement process; an input receiving means for receiving input of feature information for the processed facial image; and a second output means for outputting the feature information in association with the facial image.

2. The information processing device according to claim 1, wherein the input receiving means receives input of the characteristic information from a user who uses the information processing device.

3. The information processing device according to claim 2, wherein the feature information is information indicating facial feature points, the input receiving means receives input of the positions of the feature points specified by the user on the processed face image, and the second output means superimposes the feature points on the face image based on the input of the positions of the feature points.

4. The information processing device according to claim 1, further comprising: a determining means for determining whether or not to apply the quality improvement processing to the face image.

5. An information processing device according to claim 3, comprising an output means including the first output means and the second output means, wherein the output means displays the facial image and the processed facial image on a display unit, and displays the feature points superimposed on the facial image in accordance with input of the positions of the feature points specified by the user on the processed facial image.

6. An information processing device according to claim 4, further comprising: a learning means for causing said determining means to learn a determination method using learning data that associates said face image with information indicating whether said face image needs to be subjected to said quality improvement processing.

7. An information processing device according to claim 1, further comprising a part receiving means for receiving, from a user using the information processing device, a designation of a part of the facial image to be subjected to the quality improvement processing, and wherein the image processing means applies the quality improvement processing to the part of the facial image.

8. The information processing device according to claim 7, wherein the first output means outputs the processed face image in which the quality improvement processing has been performed on the face image portion.

9. An information processing method executed by a computer, comprising: acquiring a facial image; performing a quality improvement process on the facial image to improve the quality of the facial image; outputting a processed facial image that has been subjected to the quality improvement process; accepting input of feature information for the processed facial image; and outputting the feature information in association with the facial image.

10. A recording medium having recorded thereon a computer program that causes a computer to execute an information processing method that includes acquiring a facial image, performing a quality improvement process on the facial image to improve the quality of the facial image, outputting a processed facial image that has undergone the quality improvement process, accepting input of feature information for the processed facial image, and outputting the feature information in association with the facial image.

Images