Information processing device, information processing method, and computer program

Abstract

Visualize and display the differences between two facial images. [Solution] The information processing device includes a morphing image generation means that generates a morphing image by superimposing a 2D face image, which is a two-dimensional image including a face, and a 3D face image, which is a three-dimensional image including a face; a motion identification means that identifies a motion region in which the magnitude of movement in the morphing image exceeds a predetermined standard; a display information generation means that generates display information that visualizes the motion region; and an output means that outputs the display information to a display unit.

Description

[[Technical Field]]

[0001] The present disclosure relates to the technical field of information processing apparatuses, information processing methods, and computer programs. [[Background Art]]

[0002] A technique for generating a composite image by compositing a plurality of face images is known. For example, Patent Document 1 discloses creating a three-dimensional face model by mapping a plurality of face images of the same person onto three-dimensional face shape data, and creating a composite face image by mapping a complementary face image having information corresponding to a missing portion when a partial missing portion exists in a reference face image used for mapping. [[Prior Art Literature]] [[Patent Literature]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2011-186816 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] An object of the present disclosure is to provide an information processing apparatus, an information processing method, and a computer program that aim to improve the technology disclosed in prior art documents. [[Means for Solving the Problem]]

[0005] According to one aspect of the information processing apparatus of the present disclosure, the apparatus comprises: morphing image generation means for generating a morphing image by superimposing a two-dimensional face image that is a two-dimensional image including a face and a three-dimensional face image that is a three-dimensional image including a face; motion specifying means for specifying a motion region where a magnitude of motion in the morphing image is equal to or greater than a predetermined criterion; display information generation means for generating display information that visualizes the motion region; and output means for outputting the display information to a display unit.

[0006] One aspect of the information processing method disclosed herein involves at least one computer generating a morphing image by superimposing a 2D face image, which is a two-dimensional image including a face, and a 3D face image, which is a three-dimensional image including a face, identifying a motion region in the morphing image where the magnitude of movement exceeds a predetermined standard, generating display information that visualizes the motion region, and outputting the display information to a display unit.

[0007] One aspect of the computer program of this disclosure involves causing at least one computer to execute an information processing method that generates a morphing image by superimposing a 2D face image, which is a two-dimensional image including a face, and a 3D face image, which is a three-dimensional image including a face, identifies a motion region in the morphing image where the magnitude of movement exceeds a predetermined standard, generates display information that visualizes the motion region, and outputs the display information to a display unit. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the hardware configuration of the first information processing device. [Figure 2] This is a block diagram showing the functional configuration of the first information processing device. [Figure 3] This is a schematic diagram illustrating an example of the process of generating a morphing image. [Figure 4] This is a flowchart showing the operation flow of the first information processing device. [Figure 5] This is a block diagram showing the functional configuration of the second information processing device. [Figure 6] This is a flowchart showing the flow of motion area identification operation by the second information processing device. [Figure 7] This is a block diagram showing the functional configuration of the third information processing device. [Figure 8] This flowchart shows the flow of the morphing image generation operation by the third information processing device. [Figure 9]This is a plan view illustrating an example of a method for visualizing the motion area using predetermined marks. [Figure 10] This is a plan view illustrating an example of a method for visualizing motion regions using color gradients. [Modes for carrying out the invention]

[0009] The following describes embodiments of the information processing device, information processing method, and computer program with reference to the drawings.

[0010] <First Embodiment> The first information processing device will be described with reference to Figures 1 to 4.

[0011] (Hardware configuration) First, the hardware configuration of the first information processing device will be described with reference to Figure 1. Figure 1 is a block diagram showing the hardware configuration of the first information processing device.

[0012] As shown in Figure 1, the first information processing device 10 includes a processor 11, RAM (Random Access Memory) 12, ROM (Read Only Memory) 13, a storage device 14, an input device 15, and an output device 16. The processor 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are all connected via a data bus 17. Note that the data bus 17 may be an interface other than a data bus (for example, LAN or USB).

[0013] The processor 11 reads a computer program. For example, the processor 11 is configured to read a computer program stored in at least one of the RAM 12, ROM 13, and storage device 14. Alternatively, the processor 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 11 may also obtain (i.e., read) a computer program from a device (not shown) located outside the first information processing device 10 via a network interface. The processor 11 performs various processes by executing the read computer program. When the processor 11 executes the read computer program, a functional block related to the process performed by the first information processing device 10 is realized within the processor 11. That is, the processor 11 may function as a controller that performs various controls in the first information processing device 10.

[0014] The processor 11 may be configured as, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a quantum processor. The processor 11 may consist of one of these, or it may be configured to use multiple of them in parallel.

[0015] RAM12 temporarily stores computer programs executed by processor 11. RAM12 also temporarily stores data that processor 11 uses temporarily while executing computer programs. RAM12 may be, for example, D-RAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). Alternatively, other types of volatile memory may be used instead of RAM12.

[0016] The ROM 13 stores computer programs executed by the processor 11. The ROM 13 may also store other fixed data. The ROM 13 may be, for example, a P-ROM (Programmable Read Only Memory) or an EPROM (Erasable Read Only Memory). Alternatively, another type of non-volatile memory may be used in place of the ROM 13.

[0017] The storage device 14 stores data that the first information processing apparatus 10 stores for a long term. The storage device 14 may operate as a temporary storage device for the processor 11. The storage device 14 may store computer programs executed by the processor 11. The storage device 14 may include, for example, at least one of a hard disk drive, a magneto-optical disk drive, an SSD (Solid State Drive), and a disk array device.

[0018] The input device 15 is a device that receives input instructions from a user of the first information processing apparatus 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, a touch panel, and a touch pen. The input device 15 may also be a device capable of voice input that includes, for example, a microphone.

[0019] The output device 16 is a device that outputs information related to the first information processing apparatus 10 to the outside. For example, the output device 16 may be a display device capable of displaying information related to the first information processing apparatus 10 (e.g., a display, a monitor, etc.). The output device 16 may also be a speaker or the like capable of outputting information related to the information processing apparatus 10 as voice.

[0020] The first information processing device 10 may be configured to include some of the components described in Figure 1. For example, the first information processing device 10 may be configured to include only the processor 11, RAM 12, and ROM 13 from the components described above. In this case, the storage device 14, input device 15, and output device 16 may each be provided as external devices to the first information processing device 10. Furthermore, some of the arithmetic functions of the first information processing device 10 may be implemented by an external server or cloud service.

[0021] (Functional configuration) Next, the functional configuration of the first information processing device 10 will be described with reference to Figure 2. Figure 2 is a block diagram showing the functional configuration of the first information processing device.

[0022] In Figure 2, the first information processing device 10 is configured as a device that outputs information visualizing the difference between a 2D image and a 3D image. The first information processing device 10 includes, as components for realizing its function, a morphing image generation unit 110, a motion region identification unit 120, a display information generation unit 130, and a display information output unit 140. Note that each of the morphing image generation unit 110, the motion region identification unit 120, the display information generation unit 130, and the display information output unit 140 may be a processing block realized by the processor 11 (see Figure 1) described above.

[0023] The morphing image generation unit 110 is configured to acquire a 2D face image, which is a two-dimensional image including a face, and a 3D face image, which is a three-dimensional image including a face. For example, the morphing image generation unit 110 may acquire a 2D face image and a 3D face image by capturing a person's face using a camera or the like. Alternatively, the morphing image generation unit 110 may acquire a 2D face image and a 3D face image that have been pre-stored in storage or the like. Furthermore, the morphing image generation unit 110 may generate and acquire a 3D face image from 3D face data such as a three-dimensional shape (3D model) of the face or an unfolded face image. The 2D face image and the 3D face image may include parts of the entire face, such as the eyes, nose, mouth, forehead, chin, cheeks, and ears, or they may include only parts of these parts. In addition, the 2D face image and the 3D face image may include parts other than the face, such as the neck.

[0024] The morphing image generation unit 110 generates a morphing image (for example, an image in which the part of the face captured in the 2D face image gradually changes into the 3D face image) by superimposing the acquired 2D face image and the 3D face image. The morphing image generation unit 110 may generate the morphing image by superimposing the 2D face image and the 3D face image, for example. The morphing image generation unit 110 may generate the morphing image by performing a process to match the size and orientation of the face included in the 2D face image with the face included in the 3D face image. Specifically, when the morphing image generation unit 110 generates a 3D face image from 3D face data, it should generate the 3D face image so that it matches the orientation and size of the face in the 2D face image. In this case, for example, the morphing image generation unit 110 can match the orientation and size of the face in each image based on the feature points such as the eyes, nose, and mouth and the contour of the face included in the 2D face image and the feature points such as the eyes, nose, and mouth and the contour of the face in the 3D face image. The morphing image generation unit 110 may also generate morphing images by incorporating the skin texture and fine irregularity information of the face into the 3D model.

[0025] The motion region identification unit 120 is configured to identify regions with significant motion (hereinafter referred to as "motion regions") in the morphing image generated by the morphing image generation unit 110. A motion region is a region where the difference between the 2D face image and the 3D face image is relatively large. Specifically, the motion region identification unit 120 identifies regions where the magnitude of motion in the morphing image exceeds a predetermined standard as motion regions. The predetermined standard may be a threshold value set in advance depending on how large a difference (i.e., the difference between the 2D face image and the 3D face image) is to be visualized. For example, the absolute value of the difference may be set as the threshold value. Also, "above a predetermined standard" may be defined as, for example, "the top 5 locations when the areas with motion (parts of the face such as the nose and ears) are arranged in descending order of the difference." The motion region identification unit 120 may identify motion regions by, for example, calculating the optical flow in the morphing image. However, the motion region identification unit 120 may identify motion regions using means other than optical flow.

[0026] The display information generation unit 130 is configured to generate display information that visualizes the motion region identified by the motion region identification unit 120. The display information only needs to be information that visualizes (in other words, makes visually easy to understand) which part of the morphing image is the motion region. The display information may be, for example, information that adds visual effects or predetermined marks to the morphing image. The display information generation unit 130 may, for example, generate display information that visually emphasizes the motion region in the morphing image more than the parts other than the motion region. Specific examples of display information will be explained in detail in other embodiments described later.

[0027] The display information output unit 140 is configured to output the display information generated by the display information generation unit 130 to the display unit. When the display information output unit 140 outputs the display information, the display unit displays the motion area in a visualized manner. The display unit here may be configured as an external device of the first information processing device 10. Alternatively, the display unit may be a device provided by the first information processing device 10 (for example, a display included in the output device 16 (see Figure 1) described above). Furthermore, there may be multiple display units to which the display information output unit 140 outputs the display information. In this case, the display visualizing the motion area may be provided to multiple users.

[0028] (Example of generating a morphing image) Next, with reference to Figure 3, an example of morphing image generation by the first information processing device 10 (i.e., a specific example of the operation of the morphing image generation unit 110) will be described. Figure 3 is a schematic diagram showing an example of the operation of generating a morphing image.

[0029] In Figure 3, the morphing image generation unit 110 first acquires a 2D face image. Then, the morphing image generation unit 110 analyzes the orientation and size of the faces included in the 2D face image.

[0030] Next, the morphing image generation unit 110 generates a 3D face image from pre-stored 3D face data. Specifically, the morphing image generation unit 110 generates a 3D face image that matches the orientation and size of the face included in the acquired 2D face image. The 3D face data may be generated using face images taken from multiple angles.

[0031] Subsequently, the morphing image generation unit 110 generates a morphing image by superimposing the 2D face image and the 3D face image. The morphing image generation unit 110 superimposes each image based on, for example, the feature points of the eyes, nose, and mouth of the face contained in the 2D face image and the feature points of the eyes, nose, and mouth of the face in the 3D face image. Furthermore, as described above, the orientation and size of the faces contained in the 2D face image and the 3D face image match. Therefore, in the morphing image, the faces contained in the 2D face image and the faces contained in the 3D face image overlap, and the difference between the 2D face image and the 3D face image is expressed as movement. For example, if there is a large difference around the eyes between the 2D face image and the 3D face image, the movement of the area around the eyes in the morphing image will be large.

[0032] (Flow of operations) Next, the operation flow of the first information processing device 10 will be explained with reference to Figure 4. Figure 4 is a flowchart showing the operation flow of the first information processing device.

[0033] As shown in Figure 4, when the operation of the first information processing device 10 begins, the morphing image generation unit 110 first acquires a 2D face image (step S101). The morphing image generation unit 110 also acquires a 3D face image (step S102). Note that the processes in steps S101 and S102 may be executed one after the other, or they may be executed simultaneously in parallel.

[0034] Next, the morphing image generation unit 110 generates a morphing image by superimposing the 2D face image and the 3D face image (step S103). After that, the motion region identification unit 120 identifies the regions with large movements (i.e., regions with a large difference between the 2D face image and the 3D face image) from the morphing image generated by the morphing image generation unit 110 as motion regions (step S104).

[0035] Next, the display information generation unit 130 generates display information that visualizes the motion region identified by the motion region identification unit 120 (step S105). Then, the display information output unit 140 outputs the display information generated by the display information generation unit 130 to the display unit (step S106). Through this series of operations, the difference between the 2D face image and the 3D face image is visualized and displayed on the display unit.

[0036] (Technical effects) Next, we will explain the technical effects obtained by the first information processing device 10.

[0037] As explained in Figures 1 to 4, the first information processing device 10 generates a morphing image by superimposing a 2D face image and a 3D face image. Then, the motion region in the morphing image is identified, and display information visualizing the motion region is output. In this way, the difference between the 2D face image and the 3D face image can be visualized and communicated to the user in an easy-to-understand manner. For example, a user looking at the display unit can easily determine which parts of the 2D face image and the 3D face image have significant differences.

[0038] The first information processing device 10 can be used, for example, to determine whether a person in a 2D face image and a person in a 3D face image are the same person. Specifically, by visualizing the motion region, it becomes easier to identify areas where there is a large difference between the 2D face image and the 3D face image. By focusing on such motion regions, it becomes possible to accurately determine whether a person in a 2D face image and a person in a 3D face image are the same person or different people.

[0039] Note that differences between a 2D face image and a 3D face image occur even when they are simply superimposed. However, it is not easy to visually detect where the differences lie between a 2D face image and a 3D face image. For example, if facial features with significant changes in shading (e.g., eyes or mouth) overlap, the faces of different people may appear to be the same. However, in this embodiment, as described above, the motion area (i.e., the area where the difference between the 2D face image and the 3D face image is large) is displayed as gradually changing, making it easy to visually grasp differences that are difficult to detect by simply superimposing them.

[0040] <Second Embodiment> The second information processing device 10 will be described with reference to Figures 5 and 6. Note that the second information processing device 10 differs from the first information processing device 10 described above in some configurations and operations, while other parts may be the same as those of the first information processing device 10. Therefore, the parts that differ from the first embodiment will be described in detail below, while other overlapping parts will be omitted as appropriate.

[0041] (Functional configuration) First, the functional configuration of the second information processing device 10 will be explained with reference to Figure 5. Figure 5 is a block diagram showing the functional configuration of the second information processing device. In Figure 5, the same reference numerals are used for elements as in Figure 2.

[0042] In Figure 5, the second information processing device 10 includes, as components for realizing its function, a morphing image generation unit 110, a motion region identification unit 120, a display information generation unit 130, and a display information output unit 140. In particular, the motion region identification unit 120 in the second information processing device 10 includes a feature region detection model 125.

[0043] The feature region detection model 125 is a model that detects feature regions from the image generated by the morphing image generation unit 110. Here, the feature region is the region corresponding to the feature points of the face in the morphing image (e.g., eyes, nose, mouth, etc.). The feature points may be pre-set based on, for example, the parts used for facial recognition. The feature region detection model 125 may be a model that takes the morphing image as input and outputs information about the feature region. The feature region detection model 125 may be a model composed of a neural network.

[0044] The motion region identification unit 120 in the second information processing device 10 identifies motion regions using the feature regions detected by the feature region detection model 125 described above. Specifically, the motion region identification unit 120 identifies motion regions by prioritizing the tracking of feature regions in the morphing image over regions other than feature regions. In other words, the motion region identification unit 120 identifies motion regions by tracking feature regions in the morphing image with greater emphasis than other regions. For example, the motion region identification unit 120 may increase the density of locations where motion is detected in feature regions compared to the density of locations where motion is detected outside of feature regions. Specifically, for example, motion may be detected at intervals of 20 pixels in feature regions and at intervals of 60 pixels in regions other than feature regions.

[0045] (Motion area identification operation) Next, with reference to Figure 6, the flow of motion region identification operation by the second information processing device 10 (i.e., the operation in which the motion region identification unit 120 identifies the motion region using the feature region detection model 125) will be explained. Figure 6 is a flowchart showing the flow of motion region identification operation by the second information processing device.

[0046] As shown in Figure 6, when the motion region identification operation by the second information processing device 10 is started, the motion region identification unit 120 first detects feature regions in the morphing image using the feature region detection model 125 (step S201). At this time, the feature region detection model 125 may detect multiple feature regions.

[0047] Next, the motion region identification unit 120 tracks the feature region detected by the feature region detection model 125 with priority over other regions (step S202). Then, the motion region identification unit 120 identifies the motion region based on the tracking results (step S203). The information regarding the motion region identified in this way will be used by the display information generation unit 130 when generating display information.

[0048] (Technical effects) Next, we will explain the technical effects obtained by the second information processing device 10.

[0049] As explained in Figures 5 and 6, the second information processing device 10 identifies the motion region by prioritizing the tracking of feature regions corresponding to facial feature points over other regions. In this way, since the distinctive parts are prioritized for tracking, it is possible to identify the motion region with greater accuracy compared to tracking all regions of the face on average. Furthermore, the feature regions to be prioritized for tracking are detected by a pre-trained feature region detection model 125. In this way, it is possible to automatically detect feature regions with high accuracy.

[0050] <Third Embodiment> The third information processing device 10 will be described with reference to Figures 7 and 8. Note that the third information processing device 10 differs from the first and second information processing devices 10 described above in some configurations and operations, while other parts may be the same as those of the first and second information processing devices 10. Therefore, the following will provide a detailed explanation of the parts that differ from the embodiments already described, while omitting explanations of other overlapping parts as appropriate.

[0051] (Functional configuration) First, the functional configuration of the third information processing device 10 will be explained with reference to Figure 7. Figure 7 is a block diagram showing the functional configuration of the third information processing device. In Figure 7, the same reference numerals are used for elements as in Figures 2 and 5.

[0052] In Figure 7, the fourth information processing device 10 includes a morphing image generation unit 110, a motion region identification unit 120, a display information generation unit 130, and a display information output unit 140 as components for realizing its function. In particular, the morphing image generation unit 110 in the third information processing device 10 includes a missing part completion model 115.

[0053] The missing image completion model 115 is configured as a model that completes information about missing parts in at least one of a 2D face image and a 3D face image. The missing part may be, for example, a part of the face that is hidden and not visible in the image. Alternatively, the missing part may be a part of the image where blurring or other issues have occurred. The missing image completion model 115 may be, for example, a model that takes a 2D face image or a 3D face image as input and outputs a 2D face image or a 3D face image with the missing parts completed. Alternatively, the missing image completion model 115 may be a model that completes the missing parts of the 2D face image and the 3D face image while simultaneously performing an overlay process of the two images. The missing image completion model 115 may be a model configured by a neural network.

[0054] (Morphing image generation process) Next, with reference to Figure 8, the flow of the morphing image generation operation by the third information processing device 10 (i.e., the operation when the morphing image generation unit 110 generates a morphing image using the missing part completion model 115) will be explained. Figure 6 is a flowchart showing the flow of the morphing image generation operation by the third information processing device.

[0055] As shown in Figure 8, when the morphing image generation operation by the third information processing device 10 is started, the morphing image generation unit 110 first determines whether or not there are missing parts in the 2D face image or the 3D face image (step S201).

[0056] If there are missing parts in the 2D face image or 3D face image (step S201: YES), the morphing image generation unit 110 performs a process to fill in the missing parts using the missing part completion model 115 (step S202). Then, the morphing image generation unit 110 generates a morphing image using the 2D face image and 3D face image from which the missing parts have been completed (step S203).

[0057] On the other hand, if there are no missing parts in either the 2D face image or the 3D face image (step S201: NO), step S202 described above is omitted. That is, the process of filling in the missing parts using the missing part completion model 115 is not performed. In this case, the morphing image generation unit 110 generates a morphing image using the acquired 2D face image and 3D face image as they are (step S203).

[0058] (Technical effects) Next, we will explain the technical effects obtained by the third information processing device 10.

[0059] As explained in Figures 7 and 8, the third information processing device 10 generates a morphed image while filling in the missing parts. In this way, it is possible to appropriately generate a morphed image even if there are missing parts in the 2D or 3D face image. Furthermore, the processing for filling in the missing parts is performed by a pre-trained missing part completion model 115. This makes it possible to automatically perform appropriate completion processing. For example, by using the missing part completion model 115, missing parts can be completed without preparing images taken from a different angle.

[0060] <Fourth Embodiment> The fourth information processing device 10 will be described with reference to Figures 9 and 10. Note that the fourth information processing device 10 differs in some configurations and operations from the first to third information processing devices 10 described above, while other parts may be the same as those of the first to third information processing devices 10. Therefore, the following will explain in detail the parts that differ from the embodiments already described, and will omit explanations of other overlapping parts as appropriate.

[0061] (Display information using designated marks) The display information generation unit 130 in the fourth information processing device 10 is configured to generate display information using predetermined marks. Below, the display information generated using predetermined marks will be specifically described with reference to Figure 9. Figure 9 is a plan view showing an example of a method for visualizing a motion area using predetermined marks.

[0062] As shown in Figure 9, the display information generation unit 130 generates display information by superimposing a predetermined mark (in this case, an "arrow") onto the morphing image. If there are multiple motion regions in the morphing image, a predetermined mark may be superimposed on each of them. In the example shown in Figure 9, the region around the eyes, the region above the ears, and the region at the tip of the nose are identified as motion regions, and therefore, a predetermined mark, an arrow, is superimposed on each of these regions.

[0063] The display manner of a predetermined mark may be changed according to the magnitude of movement in each movement area. More specifically, the display information generation unit 130 may change at least one of the length, thickness, size, color, color intensity, and movement (i.e., various animations) of the predetermined mark according to the magnitude of movement in the movement area. The display information generation unit 130 may change the display manner so that the movement area is emphasized as the movement is greater. Specifically, the display information generation unit 130 may lengthen the mark, increase its thickness, bring the color closer to a specific color such as red, increase the color intensity, or change the speed of movement or display manner as the movement is greater.

[0064] In the example shown in Figure 9, the movement in the area around the eyes is the largest, followed by the area above the ears, and then the area at the tip of the nose. Therefore, the display information generation unit 130 superimposes the largest arrow over the area around the eyes, the second largest arrow over the area above the ears, and the third largest arrow over the area at the tip of the nose.

[0065] The arrows described above are merely examples of predetermined marks, and marks other than the arrows may be superimposed on the motion area. Furthermore, elements other than size may be changed depending on the magnitude of the motion. For example, the display information generation unit 130 may make the color of the arrow corresponding to one motion area with relatively large motion a conspicuous color (e.g., red), while making the color of the arrows corresponding to other motion areas with relatively small motion a dark color (e.g., black). Alternatively, the display information generation unit 130 may make the predetermined mark corresponding to one motion area with relatively large motion blink, while not blinking the predetermined marks corresponding to other motion areas with relatively small motion.

[0066] (Display information using color gradients) The display information generation unit 130 in the fourth information processing device 10 is configured to generate display information using a color gradient. Below, the display information generated using a color gradient will be described in detail with reference to Figure 10. Figure 10 is a plan view showing an example of a method for visualizing a motion area using a color gradient.

[0067] As shown in Figure 10, the display information generation unit 130 generates display information by applying a color gradient to the morphing image. Specifically, the display information generation unit 130 generates display information such that the color and density of each region of the morphing image change in steps according to the magnitude of the movement. In the example shown in Figure 10, the region around the eyes, the region above the ears, and the region at the tip of the nose are identified as movement regions, and each of these regions is darker in color than the other regions.

[0068] Furthermore, color gradients may be applied to areas other than the motion region. In this case, even in areas with small movements that are not specifically designated as motion regions (i.e., areas where the magnitude of movement is below a predetermined standard), the color and density should be changed according to the magnitude of the movement.

[0069] (Technical effects) Next, we will explain the technical effects obtained by the fourth information processing device 10.

[0070] As explained in Figures 9 and 10, the fourth information processing device 10 can generate display information in various ways, allowing for more appropriate visualization of motion regions. For example, as explained in Figure 9, by superimposing a predetermined mark onto the morphing image, it is possible to visually display which parts are motion regions in an easy-to-understand manner. In this case, by changing the display manner of the mark according to the magnitude of the motion in the motion region, it is possible to emphasize motion regions with greater movement compared to other motion regions. Also, as explained in Figure 10, by applying a color gradient to the morphing image, it is possible to visually display the distribution of motion magnitudes in an easy-to-understand manner.

[0071] The display information described in Figures 9 and 10 is merely an example, and the display information may be generated in other ways. That is, motion areas may be visualized using methods other than the predetermined marks and color gradients.

[0072] Furthermore, if display information can be generated in multiple ways, the user may be able to select which way the display information is generated through user operation or other means. For example, if the user chooses to generate display information using marks, the display information shown in Figure 9 may be generated, and if the user chooses to generate display information using color gradients, the display information shown in Figure 10 may be generated.

[0073] The processing method of recording a program that operates the configuration of each embodiment in order to realize the functions of each embodiment described above on a recording medium, reading the program recorded on the recording medium as code, and executing it on a computer is also included in the scope of each embodiment. In other words, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, not only the recording medium on which the above-mentioned program is recorded, but also the program itself is included in each embodiment.

[0074] As recording media, for example, floppy disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs can be used. Furthermore, the scope of each embodiment is not limited to programs that perform processing on the recording media alone, but also includes programs that operate on the OS and perform processing in cooperation with other software and the functions of expansion boards. In addition, the program itself may be stored on a server, and part or all of the program may be made available for download from the server to the user terminal. The program may be provided to the user in, for example, SaaS (Software as a Service) format.

[0075] <Note> The embodiments described above may also be described in the following appendix, but are not limited to these.

[0076] (Note 1) The information processing device described in Appendix 1 is an information processing device comprising: a morphing image generation means that generates a morphing image by superimposing a face 2D image, which is a two-dimensional image including a face, and a face 3D image, which is a three-dimensional image including a face; a motion identification means that identifies a motion region in which the magnitude of motion in the morphing image exceeds a predetermined standard; a display information generation means that generates display information that visualizes the motion region; and an output means that outputs the display information to a display unit.

[0077] (Note 2) The information processing device described in Appendix 2 is the information processing device described in Appendix 1, wherein the motion identification means identifies the motion region by prioritizing the tracking of the feature region corresponding to the facial feature points in the morphing image over regions other than the feature region.

[0078] (Note 3) The information processing device described in Appendix 3 is the information processing device described in Appendix 2, wherein the motion identification means detects the feature region from the morphing image using a machine learning model.

[0079] (Note 4) The information processing device described in Appendix 4 is the information processing device described in any one of Appendix 1 to 3, wherein the morphing image generation means generates the morphing image by supplementing the missing portion in at least one of the 2D face image and the 3D face image.

[0080] (Note 5) The information processing device described in Appendix 5 is the information processing device described in Appendix 4, wherein the morphing image generation means uses a machine learning model to fill in the missing parts.

[0081] (Note 6) The information processing device described in Appendix 6 is an information processing device described in any one of Appendix 1 to 5, wherein the display information generation means generates the display information by superimposing a predetermined mark indicating the motion region onto the morphing image.

[0082] (Note 7) The information processing device described in Appendix 7 is the information processing device described in Appendix 6, wherein the display information generation means changes at least one of the length, thickness, size, color, color intensity, and movement of the predetermined mark according to the magnitude of the movement of the movement area.

[0083] (Note 8) The information processing device described in Appendix 8 is an information processing device described in any one of Appendix 1 to 7, wherein the display information generation means generates the display information by applying a color gradient to the morphing image according to the magnitude of the motion of the motion region.

[0084] (Note 9) The information processing method described in Appendix 9 is an information processing method in which at least one computer generates a morphing image by superimposing a 2D face image, which is a two-dimensional image including a face, and a 3D face image, which is a three-dimensional image including a face, identifies a motion region in the morphing image where the magnitude of movement exceeds a predetermined standard, generates display information that visualizes the motion region, and outputs the display information to a display unit.

[0085] (Note 10) The computer program described in Appendix 10 is a computer program that causes at least one computer to execute an information processing method which involves generating a morphing image by superimposing a 2D face image, which is a two-dimensional image including a face, and a 3D face image, which is a three-dimensional image including a face, identifying a motion region in the morphing image where the magnitude of movement exceeds a predetermined standard, generating display information that visualizes the motion region, and outputting the display information to a display unit.

[0086] (Note 11) The recording medium described in Appendix 11 is a recording medium on which a computer program is recorded that causes at least one computer to execute an information processing method which involves generating a morphing image by superimposing a 2D face image, which is a two-dimensional image including a face, and a 3D face image, which is a three-dimensional image including a face, identifying a motion region in the morphing image where the magnitude of movement exceeds a predetermined standard, generating display information that visualizes the motion region, and outputting the display information to a display unit.

[0087] This disclosure may be modified as appropriate, insofar as it does not contradict the gist or idea of ​​the invention as can be inferred from the claims and the specification as a whole, and information processing devices, information processing methods, and computer programs with such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0088] 10 Information Processing Devices 11 processors 12 RAM 13 ROM 14 Storage device 15 Input device 16 Output device 17 Data bus 110 Morphing Image Generation Unit 115 Missing Part Interpolation Model 120 Motion area identification unit 125 Specific Area Detection Models 130 Display information generation section 140 Display Information Output Unit

Claims

1. A morphing image generation means that generates a morphing image by superimposing a 2D face image, which is a two-dimensional image including a face, and a 3D face image, which is a three-dimensional image including a face. Motion identification means for identifying a motion region in the morphing image where the magnitude of motion exceeds a predetermined standard, A display information generation means that generates display information that visualizes the aforementioned motion region, Output means for outputting the aforementioned display information to the display unit, An information processing device equipped with the following features.

2. The motion identification means identifies the motion region by tracking the feature region corresponding to the facial feature point in the morphing image with priority over regions other than the feature region. The information processing apparatus according to claim 1.

3. The motion identification means detects the feature region from the morphing image using a machine learning model. The information processing apparatus according to claim 2.

4. The morphing image generation means generates the morphing image by filling in the missing portion in at least one of the 2D face image and the 3D face image. The information processing apparatus according to any one of claims 1 to 3.

5. The morphing image generation means uses a machine learning model to fill in the missing parts. The information processing apparatus according to claim 4.

6. The display information generation means generates the display information by superimposing a predetermined mark indicating the motion region onto the morphing image. The information processing apparatus according to any one of claims 1 to 3.

7. The display information generation means changes at least one of the length, thickness, size, color, color intensity, and movement of the predetermined mark according to the magnitude of the movement of the movement area. The information processing apparatus according to claim 6.

8. The display information generation means generates the display information by applying a color gradient to the morphing image that corresponds to the magnitude of the motion in the motion region. The information processing apparatus according to any one of claims 1 to 3.

9. At least one computer, A morphing image is generated by superimposing a 2D face image (a two-dimensional image including a face) and a 3D face image (a three-dimensional image including a face). Identify the motion region in the morphing image where the magnitude of the motion exceeds a predetermined standard. The aforementioned motion region is visualized and display information is generated. The display information is output to the display unit. Information processing methods.

10. On at least one computer, A morphing image is generated by superimposing a 2D face image (a two-dimensional image including a face) and a 3D face image (a three-dimensional image including a face). Identify the motion region in the morphing image where the magnitude of the motion exceeds a predetermined standard. The aforementioned motion region is visualized and display information is generated. The display information is output to the display unit. A computer program that executes information processing methods.

Images