Information processing device, information processing method, and computer program

Abstract

Visualize the differences between multiple facial images. [Solution] The information processing device includes: a difference calculation means that calculates the difference in the three-dimensional shapes of multiple faces using the three-dimensional information of multiple faces; a texture generation means that generates a difference texture, which is texture data based on the difference; an identification means that identifies the orientation and size of a face in a 2D face image, which is a two-dimensional image including the face of a target person; a 3D image generation means that generates a 3D face image, which is a three-dimensional image of a face corresponding to the identified orientation and size of the face; a difference image generation means that generates a difference 3D image by pasting the difference texture onto the 3D face image; and a display image generation means that generates a difference display image that visualizes the difference in the 2D face image by overlaying the difference 3D image onto the 2D face image.

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of an information processing apparatus, an information processing method, and a computer program. [Background Art]

[0002] As an apparatus of this type, an apparatus that outputs information related to the surface and shape of a face is known. For example, Patent Document 1 discloses obtaining a three-dimensional difference face image by three-dimensionally imaging difference information of facial surface textures before and after temporal change of a subject, and outputting the difference face image and the underlying face image side by side so that both can be compared. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2022-024565 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] An object of the present disclosure is to provide an information processing apparatus, an information processing method, and a computer program that aim to improve the technology disclosed in prior art documents. [Means for Solving the Problem]

[0005] One aspect of the information processing device disclosed herein comprises: a difference calculation means for calculating the difference in the three-dimensional shapes of a plurality of faces using three-dimensional information of a plurality of faces; a texture generation means for generating a difference texture, which is texture data based on the difference; a identification means for identifying the orientation and size of a face in a two-dimensional face 2D image, which is a two-dimensional image including the face of a target person; a 3D image generation means for generating a face 3D image, which is a three-dimensional image of a face corresponding to the identified orientation and size of the face; a difference image generation means for generating a difference 3D image in which the difference texture corresponds to the identified orientation and size of the face by pasting the difference texture onto the face 3D image; and a display image generation means for generating a difference display image in which the difference is visualized in the face 2D image by superimposing the difference 3D image onto the face 2D image.

[0006] One aspect of the information processing method disclosed herein involves at least one computer using three-dimensional information of multiple faces to calculate the difference in the three-dimensional shapes of the multiple faces, generating a difference texture which is texture data based on the difference, identifying the orientation and size of the face in a two-dimensional image of a target person, a 2D image of the face, generating a 3D image of the face which is a three-dimensional image of the face corresponding to the identified orientation and size of the face, pasting the difference texture onto the 3D image of the face to generate a difference 3D image in which the difference texture corresponds to the identified orientation and size of the face, and superimposing the difference 3D image onto the 2D image of the face to generate a difference display image which visualizes the difference in the 2D image of the face.

[0007] One aspect of the computer program of this disclosure is for at least one computer, This information processing method involves using the 3D information of multiple faces to calculate the difference in the 3D shapes of the multiple faces, generating a difference texture which is texture data based on the difference, identifying the orientation and size of the face in a 2D face image which is a 2D image containing the face of the target person, generating a 3D face image which is a 3D image of the face corresponding to the identified orientation and size of the face, applying the difference texture to the 3D face image to generate a difference 3D image in which the difference texture corresponds to the identified orientation and size of the face, and overlaying the difference 3D image onto the 2D face image to generate a difference display image which visualizes the difference in the 2D face image. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the hardware configuration of the first information processing device. [Figure 2] This is a block diagram showing the functional configuration of the first information processing device. [Figure 3] This is a flowchart showing the operation flow of the first information processing device. [Figure 4] This is a schematic diagram illustrating an example of the operation of the first information processing device. [Figure 5] This is a flowchart illustrating the flow of 3D image generation by the second information processing device. [Figure 6] This is a flowchart illustrating the flow of 3D image generation by the third information processing device. [Figure 7] This is a block diagram showing the functional configuration of the fourth information processing device. [Figure 8] This is a plan view showing an example of highlighting using color coding in the fourth information processing device. [Figure 9] This is a plan view showing an example of highlighting using a predetermined mark in the fourth information processing device. [Modes for carrying out the invention]

[0009] The following describes embodiments of the information processing device, information processing method, and computer program with reference to the drawings.

[0010] <First Embodiment> The first information processing device will be described with reference to Figures 1 to 4.

[0011] (Hardware configuration) First, the hardware configuration of the first information processing device will be described with reference to Figure 1. Figure 1 is a block diagram showing the hardware configuration of the first information processing device.

[0012] As shown in Figure 1, the first information processing device 10 includes a processor 11, RAM (Random Access Memory) 12, ROM (Read Only Memory) 13, a storage device 14, an input device 15, and an output device 16. The processor 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are all connected via a data bus 17. Note that the data bus 17 may be an interface other than a data bus (for example, LAN or USB).

[0013] The processor 11 reads a computer program. For example, the processor 11 is configured to read a computer program stored in at least one of the RAM 12, ROM 13, and storage device 14. Alternatively, the processor 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 11 may also obtain (i.e., read) a computer program from a device (not shown) located outside the first information processing device 10 via a network interface. The processor 11 performs various processes by executing the read computer program. When the processor 11 executes the read computer program, a functional block related to the process performed by the first information processing device 10 is realized within the processor 11. That is, the processor 11 may function as a controller that performs various controls in the first information processing device 10.

[0014] The processor 11 may be configured as, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a quantum processor. The processor 11 may consist of one of these, or it may be configured to use multiple of them in parallel.

[0015] RAM12 temporarily stores computer programs executed by processor 11. RAM12 also temporarily stores data that processor 11 uses temporarily while executing computer programs. RAM12 may be, for example, D-RAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). Alternatively, other types of volatile memory may be used instead of RAM12.

[0016] The ROM 13 stores computer programs executed by the processor 11. The ROM 13 may also store other fixed data. The ROM 13 may be, for example, P-ROM (Programmable Read Only Memory) or EPROM (Erasable Read Only Memory). Also, another type of non-volatile memory may be used instead of the ROM 13.

[0017] The storage device 14 stores data to be stored for a long term by the first information processing apparatus 10. The storage device 14 may operate as a temporary storage device for the processor 11. The storage device 14 may store computer programs executed by the processor 11. The storage device 14 may include, for example, at least one of a hard disk drive, a magneto-optical disk drive, an SSD (Solid State Drive), and a disk array device.

[0018] The input device 15 is a device that receives input instructions from a user of the first information processing apparatus 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, a touch panel, and a touch pen. The input device 15 may be a device capable of voice input including, for example, a microphone.

[0019] The output device 16 is a device that outputs information related to the first information processing apparatus 10 to the outside. For example, the output device 16 may be a display device capable of displaying information related to the first information processing apparatus 10 (e.g., a display, a monitor, etc.). Also, the output device 16 may be a speaker or the like capable of outputting information related to the information processing apparatus 10 by voice.

[0020] The first information processing device 10 may be configured to include some of the components described in Figure 1. For example, the first information processing device 10 may be configured to include only the processor 11, RAM 12, and ROM 13 from the components described above. In this case, the storage device 14, input device 15, and output device 16 may each be provided as external devices to the first information processing device 10. Furthermore, some of the arithmetic functions of the first information processing device 10 may be implemented by an external server or cloud service.

[0021] (Functional configuration) Next, the functional configuration of the first information processing device 10 will be described with reference to Figure 2. Figure 2 is a block diagram showing the functional configuration of the first information processing device.

[0022] In Figure 2, the first information processing device 10 is configured as a device that generates an image visualizing the differences between multiple face images. The first information processing device 10 includes, as components for realizing its function, a difference calculation unit 110, a texture generation unit 120, a face information identification unit 130, a 3D image generation unit 140, a difference image generation unit 150, and a display image generation unit 160. Note that each of the difference calculation unit 110, texture generation unit 120, face information identification unit 130, 3D image generation unit 140, difference image generation unit 150, and display image generation unit 160 may be processing blocks realized by the processor 11 (see Figure 1) described above.

[0023] The difference calculation unit 110 is configured to acquire 3D information (hereinafter referred to as "3D data") of multiple faces. The 3D data includes information about the 3D shape of the face. For example, the 3D data may include the 3D shape of the face and an unfolded face image. The difference calculation unit 110 acquires 3D data of at least two faces. The difference calculation unit 110 may be configured to acquire pre-stored 3D data from, for example, a storage device 14 (see Figure 1, for example) or external storage. The difference calculation unit 110 is then configured to calculate the difference in the 3D shapes of multiple faces using the acquired 3D data of multiple faces. The method for calculating the difference in 3D shapes is not particularly limited, and existing technologies may be applied as appropriate.

[0024] The texture generation unit 120 is configured to generate a difference texture based on the difference in the 3D shapes of multiple faces calculated by the difference calculation unit 110. The difference texture is obtained by converting the difference in the 3D shapes of multiple faces into texture data. The difference texture may, for example, have a color and density that changes according to the size of the difference. For example, the difference texture may be texture data in which the density is lighter in areas with a large difference and darker in areas with a small difference.

[0025] The face information identification unit 130 is configured to acquire a 2D face image, which is a two-dimensional image containing the face of a target person. Here, the "target person" is the person whose 3D shape difference is determined by the first information processing device 10. The 2D face image may be acquired, for example, from a camera that photographs the target person. Alternatively, the 2D face image may be acquired from a storage device 14 (see Figure 1) that stores previously captured images. The face information identification unit 130 is configured to analyze the acquired 2D face image and identify the orientation and size of the face contained in the 2D face image. For example, the face information identification unit 130 may identify the orientation and size of the face by detecting the facial feature points contained in the 2D face image. The method for identifying the orientation and size of the face is not particularly limited, and existing technologies may be applied as appropriate.

[0026] The 3D image generation unit 140 is configured to generate a 3D face image, which is a three-dimensional image of a face. The 3D face image is generated as a three-dimensional image of a face corresponding to the orientation and size of the face in the 2D face image identified by the face information identification unit 130. The 3D image generation unit 140 may generate a 3D face image using, for example, pre-stored 3D data. Alternatively, the 3D image generation unit 140 may generate new 3D data and generate a 3D image from the generated 3D data. An example of 3D data is a 3D model that mimics an average face shape. The 3D image generation unit 140 generates a 3D face image by matching the orientation and size of the face included in the 3D data with the orientation and size of the face included in the 2D face image. The method of generating a 3D face image by the 3D image generation unit 140 will be explained in more detail in other embodiments described later.

[0027] The difference image generation unit 150 is configured to generate a difference 3D image by applying the difference texture generated by the texture generation unit 120 to the 3D face image generated by the 3D image generation unit 140. The difference 3D image is obtained by deforming the difference texture to conform to the three-dimensional shape of the face included in the 3D face image. Therefore, the shape of the difference 3D image corresponds to the orientation and size of the face identified by the face information identification unit 130. In other words, the shape of the difference 3D image corresponds to the orientation and size of the face of the target person in the 2D face image.

[0028] The display image generation unit 160 is configured to generate a difference display image by overlaying the difference 3D image generated by the difference image generation unit 150 onto a 2D face image whose face orientation and size have been identified by the face information identification unit 130. As a result, the difference display image is generated in the 2D face image, The difference calculated by the difference calculation unit 110 (i.e., the difference in the 3D shapes of multiple faces) is visualized in the image. The display image generation unit 160 may output the generated difference display image to a display device such as a display. In this case, the difference display image will be displayed on the output device. In this case, the display device may be one of the output devices 16 provided by the first information processing device 10. In this case, the first information processing device 10 will have the function of displaying the difference display image to the user.

[0029] (Flow of operations) Next, the operation flow of the first information processing device 10 will be explained with reference to Figure 3. Figure 3 is a flowchart showing the operation flow of the first information processing device.

[0030] As shown in Figure 3, when the operation of the first information processing device 10 starts, the difference calculation unit 110 first acquires 3D information of multiple faces (step S101). Then, the difference calculation unit 110 uses the acquired 3D information of multiple faces to calculate the difference in the 3D shapes of the multiple faces (step S102). After that, the texture generation unit 120 generates a difference texture based on the difference calculated by the difference calculation unit 110 (step S103).

[0031] Next, the face information identification unit 130 acquires a 2D image of the target person's face (step S104). The face information identification unit 130 then identifies the orientation and size of the face included in the acquired 2D image (step S105). Subsequently, the 3D image generation unit 140 generates a 3D image of the face corresponding to the orientation and size of the face identified by the face information identification unit 130 (step S106).

[0032] Next, the difference image generation unit 150 generates a difference 3D image by pasting the difference texture generated by the texture generation unit 120 onto the face 3D image generated by the 3D image generation unit 140 (step S107). Then, the display image generation unit 160 generates a difference display image by superimposing the difference 3D image generated by the difference image generation unit 150 onto the face 2D image whose orientation and size have been identified by the face information identification unit 130.

[0033] (Specific examples of actions) Next, with reference to Figure 4, a specific example of the operation of the first information processing device 10 will be described. Figure 4 is a schematic diagram showing an example of the operation of the first information processing device.

[0034] In the example shown in Figure 4, the 3D data of person A's face and the 3D data of person B's face are acquired by the difference calculation unit 110. Therefore, the difference calculation unit 110 calculates the difference between the 3D shape of person A's face and the 3D shape of person B's face. Then, the texture generation unit 120 generates a difference texture based on the difference between the 3D shape of person A's face and the 3D shape of person B's face.

[0035] On the other hand, a 2D image of the target person's face is acquired by the face information identification unit 130. Therefore, the face information identification unit 130 identifies the orientation and size of the target person's face included in the 2D image. Then, the 3D image generation unit 140 generates a 3D image of the face so as to match the orientation and size of the target person's face identified by the face information identification unit 130.

[0036] Subsequently, the difference texture generated by the texture generation unit 120 is applied to the 3D face image generated by the 3D image generation unit 140. As a result, the difference image generation unit 150 generates a difference 3D image based on the difference between the face of person A and the face of person B. Then, the difference 3D image generated by the difference image generation unit 150 is superimposed on the 2D face image of the target person (i.e., the image in which the orientation and size of the face have been identified by the face information identification unit 130). Consequently, the difference display image generated by the display image generation unit 160 is an image that visualizes the difference between the face of person A and the face of person B on the 2D face image of the target person.

[0037] (Technical effects) Next, we will explain the technical effects obtained by the first information processing device 10.

[0038] As explained in Figures 1 to 4, the first information processing device 10 generates a difference 3D image based on the difference calculated from the 3D information of multiple faces. Then, the generated difference 3D image is superimposed on the 2D image of the target person's face to generate a difference display image. In this way, it is possible to visualize and display the difference in the 3D shape of multiple faces on the 2D image of the target person's face. In other words, it becomes possible to highlight and present to the user the differences between multiple faces (for example, areas with large individual differences).

[0039] The first information processing device 10 can be used, for example, to determine which of several people whose 3D data is already stored matches the face of a newly acquired subject. Specifically, by visualizing the differences in the 3D shape of the face, it becomes easy to determine which parts of the subject have distinctive features (i.e., parts that are significantly different from others). Therefore, by focusing on such distinctive areas, it becomes possible to accurately determine which person the subject matches or does not match anyone.

[0040] It should be noted that differences in the 3D shape of a face occur even when two face images are simply superimposed. However, it is not easy to visually detect where these differences in the 3D shape of a face lie. For example, if facial features with significant variations in shading (such as the eyes or mouth) overlap, the faces of different people may appear to be the same. However, in this embodiment, as described above, the differences are visualized on the 2D image of the target person's face, making it easy to visually grasp differences that would be difficult to detect by simply superimposing the images.

[0041] <Second Embodiment> The second information processing device 10 will be described with reference to Figure 5. Note that the second information processing device 10 differs from the first information processing device 10 described above in some operations, but other parts may be the same as those of the first information processing device 10. Therefore, the parts that differ from the first embodiment will be explained in detail below, while other overlapping parts will be omitted as appropriate.

[0042] (3D image generation process) First, referring to Figure 5, we will explain the flow of the 3D image generation operation by the second information processing device 10 (i.e., the operation when the 3D image generation unit 140 generates a 3D face image). Figure 5 is a flowchart showing the flow of the 3D image generation operation by the second information processing device.

[0043] As shown in Figure 5, when the 3D image generation operation by the second information processing device 10 is started, the 3D image generation unit 140 first acquires information regarding the orientation and size of the face identified by the face information identification unit 130 (i.e., information regarding the orientation and size of the face of the target person in the 2D face image) (step S201).

[0044] Next, the 3D image generation unit 140 acquires the 3D data used by the difference calculation unit 110 to calculate the difference in the three-dimensional shape of the face (step S202). The 3D data acquired here may be just one of the multiple 3D data used to calculate the difference (for example, in the example shown in Figure 4, it is sufficient to acquire either the 3D data of person A's face or the 3D data of person B's face). In other words, the 3D image generation unit 140 does not need to acquire all of the multiple 3D data used to calculate the difference.

[0045] Next, the 3D image generation unit 140 generates a 3D face image using the 3D data acquired in step S202 (step S203). More specifically, the 3D image generation unit 140 generates a 3D face image that includes a face matching the orientation and size of the face identified by the face information identification unit 130, using the 3D data acquired in step S202.

[0046] (Technical effects) Next, we will explain the technical effects obtained by the second information processing device 10.

[0047] As explained in Figure 5, the second information processing device 10 generates a 3D face image using the 3D data used to calculate the difference. In this way, the 3D face image can be generated while at least partially reflecting the shape of the face used to calculate the difference. Therefore, the shape of the difference 3D image generated using the 3D face image can be made to closely resemble the 3D shape of the face used to calculate the difference. As a result, it becomes easier to compare the difference with the face used to calculate the difference, for example.

[0048] <Third Embodiment> The third information processing device 10 will be described with reference to Figure 6. Note that the third information processing device 10 differs in some operations from the first and second information processing devices 10 described above, but other parts may be the same as those of the first and second information processing devices 10. Therefore, the following will explain in detail the parts that differ from the embodiments already described, and will omit explanations of other overlapping parts as appropriate.

[0049] (3D image generation process) First, the flow of the 3D image generation operation by the third information processing device 10 will be explained with reference to Figure 6. Figure 6 is a flowchart showing the flow of the 3D image generation operation by the third information processing device.

[0050] As shown in Figure 6, when the 3D image generation operation by the third information processing device 10 is started, the 3D image generation unit 140 first acquires information regarding the orientation and size of the face identified by the face information identification unit 130 (i.e., information regarding the orientation and size of the face of the target person in the 2D face image) (step S301).

[0051] Next, the 3D image generation unit 140 acquires 3D data of multiple faces that were not used in the difference calculation unit 110 to calculate the difference in the three-dimensional shape of the faces (step S302). The 3D data acquired here may be a large amount of face data (for example, several hundred to several thousand images). Then, the 3D image generation unit 140 generates 3D data showing the average shape of the faces from the acquired 3D data of multiple faces (step S303).

[0052] Next, the 3D image generation unit 140 generates a 3D face image using the average shape 3D data acquired in step S303 (step S304). More specifically, the 3D image generation unit 140 generates a 3D face image that includes a face matching the orientation and size of the face identified by the face information identification unit 130, using the average shape 3D data acquired in step S303.

[0053] (Technical effects) Next, we will explain the technical effects obtained by the third information processing device 10.

[0054] As explained in Figure 6, the third information processing device 10 generates a 3D face image using the average shape calculated from the 3D data of multiple faces. In this way, it is possible to generate a 3D face image based on an average face (in other words, a face without prominent features). Therefore, it is possible to suppress the significant influence of a particular person's features on the difference 3D image generated using the 3D face image, and the difference display image generated using that difference 3D image. As a result, the influence of individual features on judgments using the difference display image (for example, determining which person it matches) can be reduced.

[0055] <Fourth Embodiment> The fourth information processing device 10 will be described with reference to Figures 7 to 9. Note that the fourth information processing device 10 differs in some configurations and operations from the first to third information processing devices 10 described above, while other parts may be the same as those of the first to third information processing devices 10. Therefore, the following will explain in detail the parts that differ from the embodiments already described, and will omit explanations of other overlapping parts as appropriate.

[0056] (Functional configuration) First, the functional configuration of the fourth information processing device 10 will be explained with reference to Figure 7. Figure 7 is a block diagram showing the functional configuration of the fourth information processing device. Note that in Figure 7, the same reference numerals are used for elements as in Figure 2.

[0057] In Figure 7, the fourth information processing device 10 includes, as components for realizing its functions, a difference calculation unit 110, a texture generation unit 120, a face information identification unit 130, a 3D image generation unit 140, a difference image generation unit 150, and a display image generation unit 160. In particular, the display image generation unit 160 in the fourth information processing device 10 includes an enhancement processing unit 165.

[0058] The enhancement processing unit 165 is configured to perform a process (hereinafter referred to as "enhancement processing") that visually enhances the differences visualized in the difference display image. Specifically, the enhancement processing unit 165 performs enhancement processing so that areas where the difference is above a predetermined threshold are displayed with more emphasis than areas where the difference is below a predetermined value. The enhancement processing unit 165 may, for example, perform enhancement processing on the generated difference display image. Alternatively, the enhancement processing unit 165 may perform enhancement processing on the difference 3D image used to generate the difference display image. In this case, the display image generation unit 160 can generate the difference display image by overlaying the difference 3D image on which the enhancement processing has been performed with the 2D image of the target person's face. Specific examples of the enhancement processing performed by the enhancement processing unit 165 will be described in detail below.

[0059] (Highlighting using color coding) Referring to Figure 8, we will now explain the use of color coding for highlighting. Figure 8 is a plan view showing an example of color coding for highlighting in the fourth information processing device.

[0060] As shown in Figure 8, the enhancement processing unit 165 may perform enhancement by color-coding the difference display image. Specifically, the enhancement processing unit 165 may perform processing such that each region in the difference display image is colored differently according to the size of the difference. For example, the enhancement processing unit 165 may perform processing such that regions with relatively large differences are displayed in a conspicuous color (e.g., red), and regions with relatively small differences are displayed in a dark color (e.g., black). In addition, the enhancement processing unit 165 may perform processing such that each region in the difference display image has a different density according to the size of the difference. For example, the enhancement processing unit 165 may display regions with relatively large differences lightly, and regions with relatively small differences darkly.

[0061] In the example shown in Figure 8, for the sake of explanation, only a portion of the difference display image (for example, the area where the difference exceeds a predetermined threshold) is color-coded. However, color coding according to the size of the difference may be applied to the entire difference display image (or the entire face area). In this case, the difference display image may be an image to which a color gradient is applied, where the color and density change in steps according to the size of the difference.

[0062] (Highlighting using designated marks) Next, we will explain the use of predetermined marks for highlighting, referring to Figure 9. Figure 9 is a plan view showing an example of use of predetermined marks for highlighting in the fourth information processing device.

[0063] As shown in Figure 9, the enhancement processing unit 165 may perform enhancement by superimposing a predetermined mark (in this case, an "arrow") onto the difference display image. Specifically, the enhancement processing unit 165 may superimpose a predetermined mark onto areas where the difference is greater than or equal to a predetermined threshold. If there are multiple areas in the difference display image where the difference is greater than or equal to a predetermined threshold, the predetermined mark may be superimposed onto each of them. In the example shown in Figure 9, the predetermined mark, an arrow, is superimposed onto the area around the eye, the area above the ear, and the area at the tip of the nose.

[0064] The display of a predetermined mark may be changed according to the magnitude of the difference in each area. More specifically, the emphasis processing unit 165 may change at least one of the length, thickness, size, color, color intensity, and movement (i.e., various animations) of the predetermined mark according to the magnitude of the difference. The emphasis processing unit 165 may change the display so that the larger the difference, the more the movement area is emphasized.

[0065] In the example shown in Figure 9, the difference is largest in the area around the eyes, second largest in the area above the ears, and third largest in the area at the tip of the nose. Therefore, the enhancement processing unit 165 superimposes the largest arrow on the area around the eyes, the second largest arrow on the area above the ears, and the third largest arrow on the area at the tip of the nose.

[0066] The arrows described above are merely examples of predetermined marks, and marks other than the arrows may be superimposed on the motion area. Furthermore, elements other than size may be changed depending on the magnitude of the motion. For example, the emphasis processing unit 165 may make the color of the arrow corresponding to one motion area with a relatively large difference a conspicuous color (e.g., red), while making the color of the arrows corresponding to other motion areas with relatively small differences a dark color (e.g., black). Alternatively, the emphasis processing unit 165 may make the predetermined mark corresponding to one motion area with a relatively large movement blink, while not blinking the predetermined marks corresponding to other motion areas with relatively small movement.

[0067] (Technical effects) Next, we will explain the technical effects obtained by the fourth information processing device 10.

[0068] As explained in Figures 7 to 9, the fourth information processing device 10 performs emphasis processing according to the size of the difference. In this way, it is possible to highlight the differences visualized in the difference display image. For example, as explained in Figure 8, if emphasis is displayed using color coding, it is possible to easily grasp the size of the difference by the difference in color and density. Also, as explained in Figure 10, if emphasis is displayed using predetermined marks, it is possible to easily grasp areas where the difference is larger than others. In this case, if the display manner of the marks is changed according to the size of the difference, it is also possible to grasp the magnitude of the difference between the areas in which the marks are displayed.

[0069] Note that the display methods described in Figures 8 and 9 are merely examples, and the highlighting process may be performed using other methods as well. In other words, differences may be highlighted using methods other than color coding or predetermined marks.

[0070] Furthermore, if the system can generate highlighting in multiple ways, the user may be able to select which way the highlighting is performed based on their actions. For example, if the user chooses to perform highlighting using color coding, a difference display image like the one shown in Figure 8 may be generated, and if the user chooses to perform highlighting using a predetermined mark, a difference display image like the one shown in Figure 9 may be generated.

[0071] The processing method of recording a program that operates the configuration of each embodiment in order to realize the functions of each embodiment described above on a recording medium, reading the program recorded on the recording medium as code, and executing it on a computer is also included in the scope of each embodiment. In other words, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, not only the recording medium on which the above-mentioned program is recorded, but also the program itself is included in each embodiment.

[0072] As recording media, for example, floppy disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs can be used. Furthermore, the scope of each embodiment is not limited to programs that perform processing on the recording media alone, but also includes programs that operate on the OS and perform processing in cooperation with other software and the functions of expansion boards. In addition, the program itself may be stored on a server, and part or all of the program may be made available for download from the server to the user terminal. The program may be provided to the user in, for example, SaaS (Software as a Service) format.

[0073] <Note> The embodiments described above may also be described in the following appendix, but are not limited to these.

[0074] (Note 1) The information processing device described in Appendix 1 is an information processing device comprising: a difference calculation means that calculates the difference in the three-dimensional shapes of a plurality of faces using the three-dimensional information of a plurality of faces; a texture generation means that generates a difference texture, which is texture data based on the difference; a identification means that identifies the orientation and size of a face in a two-dimensional image of a face, which is a two-dimensional image of a target person; a 3D image generation means that generates a three-dimensional image of a face, which is a three-dimensional image of a face, corresponding to the identified orientation and size of the face; a difference image generation means that generates a difference 3D image in which the difference texture corresponds to the identified orientation and size of the face by pasting the difference texture onto the face 3D image; and a display image generation means that generates a difference display image in which the difference is visualized in the face 2D image by superimposing the difference 3D image onto the face 2D image.

[0075] (Note 2) The information processing device described in Appendix 2 is the information processing device described in Appendix 1, wherein the 3D image generation means generates the face 3D image based on at least one of the three-dimensional information of the plurality of faces used when calculating the difference.

[0076] (Note 3) The information processing device described in Appendix 3 is the information processing device described in Appendix 1, wherein the 3D image generation means generates the 3D face image based on the average shape calculated from 3D information of other faces acquired separately from the 3D information of the multiple faces used when calculating the difference.

[0077] (Note 4) The information processing device described in Appendix 4 is the information processing device described in any one of Appendix 1 to 3, wherein the display image generation means generates the difference display image such that the region where the difference is greater than or equal to a predetermined threshold is highlighted more than the region where the difference is less than the predetermined value.

[0078] (Note 5) The information processing device described in Appendix 5 is the information processing device described in Appendix 4, wherein the display image generation means performs color coding according to the size of the difference and generates the difference display image.

[0079] (Note 6) The information processing device described in Appendix 6 is the information processing device described in Appendix 4 or 5, wherein the display image generation means generates the difference display image such that a predetermined mark is superimposed on the region where the difference is greater than or equal to the predetermined threshold.

[0080] (Note 7) The information processing device described in Appendix 7 is the information processing device described in Appendix 6, wherein the display image generation means changes at least one of the length, thickness, size, color, color intensity, and movement of the predetermined mark according to the size of the difference.

[0081] (Note 8) The information processing method described in Appendix 8 is an information processing method in which at least one computer calculates the difference in the three-dimensional shapes of multiple faces using the three-dimensional information of multiple faces, generates a difference texture which is texture data based on the difference, identifies the orientation and size of the face in a two-dimensional face 2D image which is a two-dimensional image including the face of the target person, generates a three-dimensional face 3D image which is a three-dimensional image of the face corresponding to the identified orientation and size of the face, applies the difference texture to the face 3D image to generate a difference 3D image in which the difference is made to correspond to the identified orientation and size of the face, and overlays the difference 3D image onto the face 2D image to generate a difference display image which visualizes the difference in the face 2D image.

[0082] (Note 9) The computer program described in Appendix 9 is a computer program that causes at least one computer to execute an information processing method which involves using the three-dimensional information of multiple faces to calculate the difference in the three-dimensional shapes of multiple faces, generating a difference texture which is texture data based on the difference, identifying the orientation and size of the face in a two-dimensional image of a target person, a 2D image of a face, a 2D image of a face, a 3D image of a face corresponding to the identified orientation and size of the face, pasting the difference texture onto the 3D image of the face to generate a difference 3D image in which the difference texture corresponds to the identified orientation and size of the face, and overlaying the difference 3D image onto the 2D image of the face to generate a difference display image in which the difference is visualized in the 2D image of the face.

[0083] (Note 10) The recording medium described in Appendix 10 is a recording medium on which a computer program is recorded that causes at least one computer to execute an information processing method, which involves calculating the difference in the three-dimensional shapes of multiple faces using the three-dimensional information of multiple faces, generating a difference texture which is texture data based on the difference, identifying the orientation and size of the face in a two-dimensional face 2D image which is a two-dimensional image including the face of a target person, generating a three-dimensional face 3D image which is a three-dimensional image of the face corresponding to the identified orientation and size of the face, pasting the difference texture onto the face 3D image to generate a difference 3D image in which the difference texture corresponds to the identified orientation and size of the face, and superimposing the difference 3D image onto the face 2D image to generate a difference display image which visualizes the difference in the face 2D image.

[0084] This disclosure may be modified as appropriate, insofar as it does not contradict the gist or idea of ​​the invention as can be inferred from the claims and the specification as a whole, and information processing devices, information processing methods, and computer programs with such modifications are also included in the technical idea of ​​this disclosure. [Explanation of Symbols]

[0085] 10 Information Processing Devices 11 processors 12 RAM 13 ROM 14 Storage device 15 Input device 16 Output device 17 Data bus 110 Difference calculation part 120 Texture generation unit 130 Facial Information Identification Unit 140 3D Image Generation Unit 150 Difference Image Generation Unit 160 Display Image Generation Unit 165 Enhancement Processing Unit

Claims

1. A difference calculation means that uses the three-dimensional information of multiple faces to calculate the difference in the three-dimensional shapes of the multiple faces, A texture generation means that generates a difference texture, which is texture data based on the aforementioned difference, A means for identifying the orientation and size of a face in a 2D image of a face, which is a two-dimensional image including the face of a target person, A 3D image generation means for generating a 3D face image, which is a three-dimensional image of a face corresponding to the orientation and size of the face identified above, A difference image generation means that generates a difference 3D image in which the difference texture corresponds to the orientation and size of the identified face by applying the difference texture to the 3D face image, A display image generation means generates a difference display image that visualizes the difference in the face 2D image by overlaying the difference 3D image onto the face 2D image, An information processing device equipped with the following features.

2. The 3D image generation means generates the face 3D image based on at least one of the 3D information of the plurality of faces used when calculating the difference. The information processing apparatus according to claim 1.

3. The 3D image generation means generates the 3D face image based on the average shape calculated from the 3D information of other faces acquired separately from the 3D information of the multiple faces used when calculating the difference. The information processing apparatus according to claim 1.

4. The display image generation means generates the difference display image such that the region where the difference is greater than or equal to a predetermined threshold is highlighted more than the region where the difference is less than the predetermined value. The information processing apparatus according to any one of claims 1 to 3.

5. The display image generation means generates the difference display image by color-coding according to the size of the difference. The information processing apparatus according to claim 4.

6. The display image generation means generates the difference display image such that a predetermined mark is superimposed on the region where the difference is greater than or equal to the predetermined threshold. The information processing apparatus according to claim 4.

7. The display image generation means changes at least one of the length, thickness, size, color, color intensity, and movement of the predetermined mark according to the size of the difference. The information processing apparatus according to claim 6.

8. At least one computer, Using the 3D information of multiple faces, the difference in the 3D shapes of the multiple faces is calculated. A difference texture, which is texture data based on the aforementioned difference, is generated. In a 2D image of a face, which is a two-dimensional image including the face of the subject, the orientation and size of the face are identified. A 3D image of the face, which is a three-dimensional image of the face corresponding to the orientation and size of the face identified above, is generated. By applying the difference texture to the 3D face image, a difference 3D image is generated in which the difference texture corresponds to the specified face orientation and size. By overlaying the aforementioned difference 3D image onto the face 2D image, a difference display image is generated in which the difference is visualized in the face 2D image. Information processing methods.

9. On at least one computer, Using the 3D information of multiple faces, the difference in the 3D shapes of the multiple faces is calculated. A difference texture, which is texture data based on the aforementioned difference, is generated. In a 2D image of a face, which is a two-dimensional image including the face of the subject, the orientation and size of the face are identified. A 3D image of the face, which is a three-dimensional image of the face corresponding to the orientation and size of the face identified above, is generated. By applying the difference texture to the 3D face image, a difference 3D image is generated in which the difference texture corresponds to the specified face orientation and size. By overlaying the aforementioned difference 3D image onto the face 2D image, a difference display image is generated in which the difference is visualized in the face 2D image. A computer program that executes information processing methods.

Images