Face change output method

Abstract

To provide a technology that presents a face surface texture or a specific state change of a face shape in an easy-to-understand manner.SOLUTION: A face change output method comprises the steps of: acquiring original face information indicating at least one or both of face surface texture information and three-dimensional face shape information; acquiring predicted change face information by applying one or both of the face surface texture information and the face shape information indicated by the acquired original face information to the learned model; acquiring, based on the acquired change face information and the acquired original face information, a difference information of the face surface texture or the face shape, which indicates a difference between the face surface texture information of the changed face information and the original face information, or a difference between the face shape information of them; and outputting a three-dimensional difference face image obtained by converting the acquired difference information into a three-dimensional image using at least the face shape information indicated by the original face information or the face shape information indicated by the changed face information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technology for predicting and outputting specific state changes in a human face using a trained model obtained by machine learning and facial image data. [Background technology]

[0002] The following Patent Document 1 discloses a technology for analyzing evaluation indices for the entire face with high precision by extracting a hemoglobin pigment image, a melanin pigment image, and a shadow image from a facial image using independent component analysis, and then applying frequency component separation and principal component analysis to these extracted images to obtain evaluation indices for melanin pigment factors and hemoglobin pigment factors that cause color unevenness, as well as for each factor of unevenness such as wrinkles and sagging. Patent Document 2 below discloses a method for predicting aging using a trained model obtained by machine learning. The trained model includes a shape aging model for predicting changes in face shape due to aging, a texture aging model for predicting changes in face surface texture due to aging, and a three-dimensional prediction model for predicting three-dimensional data from a two-dimensional image. This method extracts feature points of a target image, estimates a face direction in the target image using the extracted feature points, generates first three-dimensional data based on the three-dimensional prediction model and the estimated face direction, generates second three-dimensional data from the first three-dimensional data using the shape aging model, applies a texture aging model to the two-dimensional image generated based on the first three-dimensional data to generate an aging texture, and synthesizes the aging texture with the second three-dimensional data to generate an aging face model. The following non-patent document 1 discloses a method for removing facial shadows by pigment vector analysis, and for extracting hemoglobin and melanin components by applying independent component analysis to a facial image from which the shadows have been removed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-5281 A [Patent Document 2] JP 2016-194892 A [Non-patent literature]

[0004] [Non-Patent Document 1] Norimichi Tsumura etc., “Image-based skin color and texture analysis / synthesis by extracting hemoglobin and melanin information in the skin,” acm Transactions on Graphics, Vol. 22, No. 3.pp. 770-779(2003). (Proceedings of ACM SIGGRAPH 2003) Summary of the Invention [Problem to be solved by the invention]

[0005] The above-mentioned aging prediction method can predict aging texture from a two-dimensional face image and output an aging face model, while other analysis methods can process the face image to change the facial surface texture or obtain a facial evaluation index. However, current methods have room for improvement in terms of presenting information about facial surface texture in a more understandable way. For example, the predicted aging model may show texture conditions that are undesirable for the subject, which may lead the subject to distrust the model and not try to grasp the necessary information.

[0006] The present invention relates to a technique for presenting specific changes in facial surface texture or facial shape in an easily comprehensible manner. In this specification, the term "facial surface texture" refers to a spatial distribution feature derived from the surface or interior of facial skin. Examples of "facial surface texture" include features that generate a spatial distribution of the facial surface, such as uneven skin color, such as redness or blemishes, skin gloss, and uneven or rough skin surface, such as wrinkles, pores, and scales. Other examples include features that generate a spatial distribution of the facial surface, such as the state of adhesion of a coating film on skin to which a makeup cosmetic has been applied, uneven skin color, gloss, and uneven surface unevenness caused by sebum secretion. In addition, a "specific state change" is a specific state change predicted by a trained model obtained by machine learning, and examples include changes over time, which are state changes that occur over time, and state changes that occur with the application of cosmetics, beauty treatments (excluding medical procedures), or food and beverages for beauty or health purposes. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the aspects of the present invention each adopt the following configurations. An aspect of the present invention relates to a face change output method executed by one or more processors that can use a trained model obtained by machine learning, the trained model including at least one or both of a shape change model predicting a specific state change of a three-dimensional face shape or a texture change model predicting a specific state change of a texture of a face surface. The face change output method includes an acquisition step of acquiring original face information indicating at least one or both of face surface texture information or three-dimensional face shape information, a prediction step of acquiring predicted changed face information by applying one or both of the face surface texture information or the face shape information indicated by the acquired original face information to the trained model, a difference acquisition step of acquiring difference information of face surface texture or face shape indicating a difference between the face surface texture information or face shape information between the changed face information and the original face information based on the acquired changed face information and the acquired original face information, and an output step of outputting a three-dimensional difference face image obtained by three-dimensionally imaging the acquired difference information using at least the face shape information indicated by the original face information or the face shape information indicated by the changed face information. Further, another aspect of the present invention relates to, for example, a face change output device (information processing device, computer) capable of executing the face change output method according to the above aspect, a program for causing a computer to execute the face change output method according to the above aspect, and a computer-readable storage medium having such a program recorded thereon. This storage medium includes a non-transitory tangible medium. Effect of the Invention

[0008] According to the above aspect, it is possible to provide a technique for presenting a specific state change in facial surface texture or facial shape in an easily comprehensible manner. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device capable of executing a facial change output method according to an embodiment of the present invention. [Diagram 2] 5 is a diagram showing a processing flow of a face change output method according to the first embodiment. FIG. [Diagram 3] FIG. 11 is a diagram showing a process flow of a face change output method according to a first modified example. [Figure 4] FIG. 11 is a diagram showing a processing flow of a face change output method according to the second embodiment. [Diagram 5] FIG. 11 is a diagram showing a process flow of a face change output method according to a second modified example. [Figure 6] FIG. 13 is a diagram showing a processing flow of a face change output method according to the third embodiment. [Figure 7] FIG. 13 is a diagram showing a process flow of a face change output method according to a third modified example. [Figure 8] FIG. 13 is a diagram showing a process flow of a face change output method according to a fourth modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A preferred embodiment of the present invention (hereinafter, referred to as the present embodiment) will be described below. Note that the present embodiment described below is merely an example, and the present invention is not limited to the configuration described below.

[0011] The facial change output method according to this embodiment (hereinafter referred to as the present method) is executed by one or more processors included in one or more information processing devices. FIG. 1 is a diagram conceptually illustrating an example of the hardware configuration of an information processing device 10 capable of executing the present method. The information processing device 10 is a so-called computer, and includes a CPU 11, a memory 12, an input / output interface (I / F) 13, a communication unit 14, etc. The information processing device 10 may be a stationary PC (Personal Computer), or may be a portable terminal such as a portable PC, a smartphone, or a tablet.

[0012] The CPU 11 is a so-called processor, and may include not only a general CPU (Central Processing Unit), but also an application specific integrated circuit (ASIC), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), etc. The memory 12 is a RAM (Random Access Memory), a ROM (Read Only Memory), and an auxiliary storage device (such as a hard disk). The input / output I / F 13 can be connected to user interface devices such as a display device 15 and an input device 16. The display device 15 is a device that displays a screen corresponding to drawing data processed by the CPU 11 or the like, such as an LCD (Liquid Crystal Display) or CRT (Cathode Ray Tube) display. The input device 16 is a device that accepts input of user operations, such as a keyboard, a mouse, etc. The display device 15 and the input device 16 may be integrated and realized as a touch panel. The communication unit 14 communicates with other computers via a communication network and exchanges signals with other devices such as a printer, etc. A portable recording medium, etc. can also be connected to the communication unit 14.

[0013] The hardware configuration of the information processing device 10 is not limited to the example in Fig. 1. The information processing device 10 may include other hardware elements not shown. Furthermore, the number of each hardware element is not limited to the example in Fig. 1. For example, the information processing device 10 may have multiple CPUs 11. Furthermore, the information processing device 10 may be realized by multiple computers consisting of multiple housings.

[0014] The information processing device 10 can execute the present method by the CPU 11 executing a computer program stored in the memory 12. This computer program is installed from a portable recording medium such as a CD (Compact Disc) or a memory card, or from another computer on a network, via the input / output I / F 13 or the communication unit 14, and stored in the memory 12.

[0015] The information processing device 10 (CPU 11) can use a trained model obtained by machine learning using training data. The "trained model" here is a model obtained by machine learning using training data, i.e., supervised learning, and can be expressed as an AI (Artificial Intelligence) model, Machine Learning (ML) model, etc. The trained model used in this embodiment may be a regression equation obtained by regression analysis, or may be a neural network model obtained by principal component analysis, deep learning, or the like, and the data structure and learning algorithm of the model are not limited. For example, the trained model is realized by a combination of a computer program and a parameter, or a combination of a plurality of functions and parameters. The face image is subjected to dimension reduction and coding processing, for example, using principal component analysis. When the trained model is constructed with a neural network, and the input layer, intermediate layer, and output layer are regarded as a unit of one neural network, the trained model may refer to one neural network or a combination of multiple neural networks. In addition, the trained model may be composed of a combination of multiple multiple regression equations, or may be composed of one multiple regression equation. The trained model may be stored in memory 12 within the information processing device 10, or may be stored in the memory of another computer that the information processing device 10 can access via communication.

[0016] In this way, the information processing device 10 can be described as a face change output device that is capable of using a trained model and executing a face change output method. Hereinafter, the trained model used in this method will be referred to as an AI model. In addition, in the following description, this method will be described as being executed by the CPU 11.

[0017] [First embodiment] A face change output method according to a first embodiment (hereinafter, referred to as a first method) will be described below with reference to Fig. 2. Fig. 2 is a diagram showing a process flow of the face change output method according to the first embodiment.

[0018] The AI ​​model used in the first embodiment is obtained by machine learning using training data including multiple combinations of two-dimensional facial image data sets before and after the passage of time and three-dimensional facial shape information before the passage of time, and is capable of predicting three-dimensional facial shape information and corresponding facial surface texture information, as well as facial surface texture information after a specific state change, from two-dimensional facial image data. Here, "the AI ​​model is capable of predicting three-dimensional facial shape information and facial surface texture information" means that three-dimensional facial shape information and facial surface texture information can be obtained based on information output from the AI ​​model.

[0019] Specifically, the AI ​​model in the first embodiment includes at least a three-dimensional prediction model and a texture change model. The three-dimensional prediction model can be obtained by machine learning using a pair of two-dimensional face image data and corresponding three-dimensional face shape information as training data, and can predict three-dimensional face shape information from two-dimensional face image data. Various well-known methods can be adopted for the data structure, generation method, and learning method of this three-dimensional prediction model, and for example, a three-dimensional prediction model generated by the method disclosed in the above-mentioned Patent Document 2 can be used. The three-dimensional face shape information used in the first embodiment may be information indicating the three-dimensional shape of the face, and the format thereof is not limited. However, the three-dimensional face shape information is preferably modeled information so that data of each individual is expressed by the same number of data points of the same topology in order to enable statistical processing. Furthermore, it is more preferable that the three-dimensional face shape information is expressed by a homology model defined so that each data point has the same anatomical meaning. For this reason, the three-dimensional face shape information in the first embodiment is a homology model that standardizes and expresses the face shape of each individual using polygons with the same number of vertices and the same topological geometric structure that are anatomically associated, and is shown as three-dimensional coordinate data. In the following description, the three-dimensional face shape information may also be referred to as a homology model.

[0020] The texture change model is obtained by machine learning using facial surface texture information before and after a specific state change as teacher data, and can predict facial surface texture information after a specific state change from the facial surface texture information before the specific state change. Various well-known techniques can be used for the data structure and learning method of the texture change model. When the texture change model is a texture time-varying model that predicts time-varying changes, for example, the generation method and learning method of the texture aging model disclosed in the above-mentioned Patent Document 2 can be used. For example, both the texture aging model using principal component analysis and the texture aging model using WAVELET transformation may be used, or either one of them may be used. In addition, the facial surface texture information applied to the teacher data and the AI ​​model may be image information such as a cylindrical coordinate system other than a rectangular coordinate system.

[0021] The facial surface texture information may be image data or data in a format other than image data as long as it indicates the texture of the facial surface. In this embodiment, the facial surface texture information will be described as image data. Furthermore, when the specific state change predicted by the AI ​​model is a change over time, the change over time is a change that occurs with age, season, day, morning and evening, etc., and the length of time that has passed is not limited. The AI ​​model may predict aging changes five or ten years from now, changes one week or one month from now, or changes after a certain period of time has passed since the application of makeup cosmetics. The change over time predicted by such an AI model can be set by a facial image data set before and after the passage of time that is used as training data. The AI ​​model may also predict changes in condition before and after the application of cosmetics, beauty treatments (excluding medical procedures), or beauty or health foods and beverages. The two-dimensional face image data included in the training data may be image data of an uncovered face or image data of a made-up face. For example, the training data may be a data set of image data of a made-up face immediately after applying makeup and image data of the made-up face after makeup has come off over time. In addition, the training data is preferably formed only from facial image datasets of the same person before and after a period of time, but may also include facial image datasets of two different people who may correspond to before and after a specific state change, for example, people with similar facial features or facial texture features, or a dataset may be formed from facial image data of either one of the people before or after a specific state change, and facial image data of the other person generated by image processing of that facial image data.

[0022] It is desirable that the facial image data used as training data for the texture variation model is image data in which the orientation of the face in the image is normalized, and further, the positions of specific facial features (eyes, nose, mouth, etc.) of each individual in the population are normalized. In this embodiment, facial surface texture information that indicates the facial surface texture of each individual and is normalized to fit the homology model of that face is used as training data. However, the facial image data used in the training data, or the facial image data that is used as the training data, may not only be a visible light image, but may also be an image captured using wavelength bands other than visible light, such as infrared light.

[0023] Fig. 2 illustrates a texture time-varying model that predicts a time-varying change as a specific state change, and in the following description, the first method will be described using the example in which a time-varying change is predicted as a specific state change as shown in Fig. 2. However, the first method is not limited to a time-varying change.

[0024] The first method includes steps (S21) to (S27) as shown in FIG. Step (S21) is a data acquisition step for acquiring two-dimensional face image data of a subject (hereinafter, may be referred to as an original face image BP). In the following description, the face of the subject shown in the face image data acquired in step (S21) may be referred to as an original face or a face before change over time. The data of the original face image BP is acquired as an image file in, for example, a Joint Photographic Experts Group (JPEG) format, a Bitmap image (BMP) format, a Tagged Image File Format (TIFF) format, a Graphic Interchange Format (GIF) format, or the like. However, the data format of the image is not limited, and may be a format in which brightness information of each pixel is listed. The acquired image may be a color image, or a grayscale image obtained by grayscaling a color image. The CPU 11 may acquire the original face image BP from a camera that captured the face image, or may acquire the original face image BP from another computer or a portable recording medium.

[0025] In the original face image BP acquired in step (S21), it is preferable that the state of the subject of the image is consistent with that of the face image used in the training data of the AI ​​model. For example, if the face image of the training data shows a face surface texture viewed from the front, it is preferable that the original face image BP also shows the face surface texture viewed from the front of the subject. Therefore, in step (S21), the acquired face image data may be subjected to a normalization process to match the acquired face image data with the face image data of the AI ​​model, thereby acquiring the original face image BP. In this normalization process, for example, the size and position of the face in the image data, the position of specific parts of the face (eyes, nose, mouth, etc.) are normalized, and the background is removed. When normalizing the positions of specific facial features (eyes, nose, mouth, etc.), in step (S21), a predetermined image recognition process may be performed on the acquired facial image data to automatically recognize the specific features such as the eyes, nose, mouth, eyebrows, etc. In this case, in step (S21), it is also possible to check whether the acquired facial image data is usable or not based on the presence or absence and positional relationship of the specific features indicated by the result of the automatic recognition.

[0026] Step (S22) is a step of acquiring original face information indicating at least the facial shape information and facial surface texture information of the subject's original face by applying the two-dimensional facial image data (original facial image BP) acquired in step (S21) to a three-dimensional predictive model of the AI ​​model. In this embodiment, in step (S22), an original face texture image BT and an original face homology model BF are generated as original face information. Specifically, first, an original face homology model BF is generated from the original face image BP using a three-dimensional prediction model. The original face homology model BF is three-dimensional face shape information related to the original face of the subject, and is shown as mesh data (three-dimensional coordinate data). Next, an original face texture image BT is generated from the original face image BP using the generated original face homology model BF. The original face texture image BT is face surface texture information related to the original face of the subject, and is image information normalized to fit the original face homology model BF. In this normalization, for example, the position of each feature point (specific part) of the original face image BP is matched to the original face homology model BF, texture information not included in the original face image BP is complemented, and conversion to a cylindrical coordinate system is performed.

[0027] Step (S23) is a prediction step for acquiring predicted changed face information by applying the face surface texture information indicated by the original face information acquired in step (S22) to the trained model. In this embodiment, in step (S23), the original face texture image BT acquired in step (S22) is applied to the texture time-varying model of the AI ​​model to generate a predicted texture image AT. Therefore, the predicted texture image AT indicates a face surface texture obtained by changing the face surface texture of the original face of the subject appearing in the original face image BP over time based on the prediction by the AI ​​model.

[0028] Step (S24) is a difference acquisition step of acquiring facial surface texture difference information indicating a difference between the facial surface texture information of the modified facial information and the original facial information based on the modified facial information acquired in step (S23) and the original facial information acquired in step (S22). In this embodiment, in step (S24), difference information between the original facial texture image BT acquired in step (S22) and the predicted texture image AT acquired in step (S23) is acquired. The difference information acquired in step (S24) may be any information indicating the difference between the facial surface textures, and in this embodiment, the difference image data is acquired as the difference information. Hereinafter, the difference information acquired in step (S24) may be referred to as a difference texture image DT.

[0029] Any known method can be used to obtain the difference information between the facial surface textures. For example, the difference information can be obtained by extracting difference data, extracting necessary information from the difference data, encoding the necessary information, and visualizing the encoded necessary information. For example, images are compared on a pixel-by-pixel or block-by-block basis to calculate the difference value (signed or absolute value) or ratio of the gray values, or to extract the difference data by comparing and discriminating the gray values. The extracted difference data may contain unnecessary information, such as electronic noise or uneven skin information caused by lighting different from the original facial surface texture. Thus, unnecessary information can be removed by applying a frequency filter or a spatial filter to the extracted difference data. In addition, in order to extract only the necessary information from the difference data, extraction via morphological transformation may be performed. The necessary information extracted in this way is coded by binarization, multi-value conversion (grading), etc., and the coded necessary information can be visualized by imaging or labeling it as differences in color or brightness.

[0030] Step (S25) is a step of converting the difference information of the facial surface texture before and after the change over time of the subject acquired in step (S24) into a three-dimensional image using the face shape information indicated by the original face information acquired in step (S22) to acquire a three-dimensional difference face image. In this embodiment, in step (S25), the difference texture image DT acquired in step (S24) is combined with the original face homology model BF acquired in step (S22) to acquire three-dimensional face image data. In this embodiment, the three-dimensional face image data generated in step (S25) indicates a three-dimensional face image having a three-dimensional face shape of the original face (before the change over time) and having difference information of the facial surface texture before and after the change over time of the subject as texture information, and this face image (three-dimensional difference face image) is expressed as a three-dimensional difference image DBF.

[0031] Step (S26) is a step of acquiring a three-dimensional original face image based on the original face information acquired in step (S22). In this embodiment, in step (S26), the original face texture image BT acquired in step (S22) and the original face homology model BF are synthesized to acquire three-dimensional face image data. The three-dimensional face image data generated in step (S26) indicates a three-dimensional face image having a three-dimensional face shape of the original face (before change over time) and having a face surface texture of the original face (before change over time). The face image (three-dimensional original face image) indicated by this face image data is expressed as a three-dimensional face image BTF.

[0032] Step (S27) is an output step of outputting the three-dimensional difference face image acquired in step (S25) and the three-dimensional original face image based on the original face information acquired in step (S26) so as to be comparable. The output in step (S27) may be realized by displaying the differential face image and the original face image on the display device 15, by printing them on a printer device, or by sending them to a portable recording medium or to another computer via the communication unit 14. In addition, as an output format, the CPU 11 may display (output) the differential face image and the original face image side by side so that they can be compared, or may display either one in a switchable manner, or may display them superimposed. However, in step (S27), only the three-dimensional difference face image (three-dimensional difference image DBF) may be output. In this case, step (S26) is not necessary.

[0033] In this way, by outputting the difference information in the facial surface texture before and after a specific state change of the subject as a three-dimensional difference image DBF that shows the three-dimensional facial shape, the specific state change in the facial surface texture can be presented in a way that makes it easy to understand. Furthermore, by outputting the three-dimensional difference image DBF so that it can be compared with the three-dimensional face image BTF, the difference in face surface texture can be more easily grasped by comparing the two face images. By outputting the facial surface texture information after a specific state change predicted as the texture of a three-dimensional facial image (predicted texture image AT) as difference information before and after the specific state change rather than outputting it as is, it is possible to reduce the possibility of causing discomfort to the subject.

[0034] Here, the CPU 11 can output the three-dimensional difference image DBF acquired in step (S25) and the three-dimensional face image BTF acquired in step (S26) as a multi-viewpoint image. Examples of the multi-viewpoint image include a free viewpoint image in which the viewpoint can be freely changed by a user operation using the input device 16, a three-dimensional display, virtual reality (VR), augmented reality (AR), and the like. In this way, by outputting the difference information as a multi-viewpoint video, the differences in facial surface texture can be confirmed from various viewpoints.

[0035] [First modified example] The above-described first embodiment can also be modified as shown in Fig. 3. Fig. 3 is a diagram showing the flow of processing of a face change output method according to the first modified example. In the first modified example, a step (S31) is further executed between the step (S24) and the step (S25) in the flow of the above-described first embodiment. A texture time-varying model for predicting a time-varying change as a specific state change is also illustrated in Fig. 3. However, the first modified example is not limited to the time-varying change.

[0036] Step (S31) is a superimposing step for generating image information by superimposing the difference information acquired in step (S24) and the facial surface texture information indicated by the original face information acquired in step (S22). In the example of Fig. 3, in step (S31), a composite texture image is generated by combining the difference texture image DT acquired in step (S24) with the original face texture image BT acquired in step (S22). In step (S25), the synthetic texture image generated in step (S31) is synthesized with the original face homology model BF acquired in step (S22) to acquire three-dimensional face image data. Therefore, the generated three-dimensional face image data shows a three-dimensional face image having a three-dimensional face shape of the original face (before time-dependent change) and a texture in which difference information of the face surface texture before and after the subject's time-dependent change is superimposed on the face surface texture of the original face (before time-dependent change). This face image (three-dimensional difference face image) is expressed as a three-dimensional difference image DTBF.

[0037] According to the first variant, the facial surface texture of the original face (before a specific state change) and the difference information between the facial surface texture of the subject before and after a specific state change are displayed superimposed, making it easier to grasp the specific state change in the facial surface texture.

[0038] [Second embodiment] In the above-described first embodiment and first modified example, a specific state change prediction of facial surface texture information and a difference calculation between pieces of facial surface texture information are performed. In the second embodiment, in addition to the facial surface texture information, the three-dimensional facial shape information of the subject is processed. The following describes the facial change output method according to the second embodiment (hereinafter, referred to as the second method), focusing on the differences from the first embodiment. In the following description, the same contents as those in the first embodiment will be omitted as appropriate.

[0039] FIG. 4 is a diagram showing a process flow of the face change output method according to the second embodiment. The AI ​​model used in the second embodiment is obtained by machine learning using training data including multiple sets of two-dimensional facial image data and three-dimensional facial shape information before and after a specific state change, and is capable of predicting three-dimensional facial shape information after a specific state change and facial surface texture information that matches that facial shape information from the two-dimensional facial image data. The three-dimensional face shape information and face surface texture information used in the second embodiment are as described in the first embodiment.

[0040] The AI ​​model in the second embodiment includes at least a three-dimensional prediction model, a texture change model, and a shape change model. The three-dimensional prediction model and the texture change model are the same as those in the first embodiment. The shape change model can be obtained by machine learning using training data including multiple sets of three-dimensional face shape information (homology models) before and after a specific state change, and can predict three-dimensional face shape information (homology models) after a specific state change from the three-dimensional face shape information (homology models) predicted by the three-dimensional prediction model. Various well-known methods can be adopted for the data structure, generation method, and learning method of this shape change model. When the shape change model predicts changes over time, for example, a shape aging model generated by the method disclosed in the above-mentioned Patent Document 2 can be used for the shape change model.

[0041] Fig. 4 illustrates a texture time-varying model and a shape time-varying model that predict a time-varying change as a specific state change, and in the following explanation, the second method will be explained using the example shown in Fig. 4 in which a time-varying change is predicted as a specific state change. However, the second method is not limited to a time-varying change.

[0042] The second method includes steps (S41) to (S48) as shown in FIG. Step (S41) and step (S42) are the same as step (S21) and step (S22) in the first embodiment. That is, step (S41) is a data acquisition step of acquiring two-dimensional face image data (original face image BP) of a subject, and step (S42) is a step of acquiring original face information indicating at least face shape information and face surface texture information of the original face of the subject by applying the two-dimensional face image data (original face image BP) acquired in step (S41) to a three-dimensional prediction model of an AI model. Furthermore, step (S43) is the same as step (S23) in the first embodiment, that is, in step (S43), the original face texture image BT acquired in step (S42) is applied to the texture time-varying model of the AI ​​model to generate a predicted texture image AT.

[0043] Step (S44) is a prediction step for acquiring changed face information including face shape information after predicted time-dependent changes by applying face shape information indicated by the original face information acquired in step (S42) to the trained model. In this embodiment, in step (S44), the original face homology model BF acquired in step (S42) is applied to the AI ​​model shape time-dependent change model to generate a predicted homology model AF. Therefore, the predicted homology model AF indicates a face shape obtained by changing the face shape of the original face of the subject over time based on the prediction by the AI ​​model.

[0044] In this way, steps (S43) and (S44) can be described as prediction steps for acquiring predicted changed face information by applying both the face surface texture information (original face texture image BT) and face shape information (original face homology model BF) indicated by the original face information acquired in step (S42) to the trained model (texture time-varying model and shape time-varying model). The acquired changed face information indicates at least the face surface texture information and face shape information of the subject after time-varying predicted by the AI ​​model.

[0045] Step (S45) is the same as step (S24) in the first embodiment. That is, in step (S45), facial surface texture difference information indicating the difference between the facial surface texture information of the modified facial information and the original facial information is acquired. Here, the difference information acquired is the difference information between the original facial texture image BT acquired in step (S42) and the predicted texture image AT acquired in step (S43).

[0046] Step (S46) is a step of converting the difference information of the facial surface texture before and after the change over time of the subject acquired in step (S45) into a three-dimensional image using the facial shape information indicated by the changed face information acquired in step (S44) to acquire a three-dimensional difference face image. In this embodiment, in step (S46), the difference texture image DT acquired in step (S45) is combined with the predicted homology model AF acquired in step (S44) to acquire three-dimensional face image data. In this embodiment, the three-dimensional face image data generated in step (S46) indicates a three-dimensional face image having a three-dimensional face shape after the change over time of the subject and having difference information of the facial surface texture before and after the change over time of the subject as texture information, and this face image (three-dimensional difference face image) is represented as a three-dimensional difference image DAF.

[0047] Step (S47) is the same as step (S26) in the first embodiment. That is, in step (S47), the original face texture image BT and the original face homology model BF acquired in step (S42) are synthesized to acquire a three-dimensional original face image (three-dimensional face image BTF).

[0048] Step (S48) is similar to step (S25) in the first embodiment, and the three-dimensional difference face image obtained in step (S46) and the three-dimensional original face image based on the original face information obtained in step (S47) are output so as to be comparable. However, in this embodiment, the difference face image that is output so as to be comparable with the original face image is, as described above, a three-dimensional difference image DAF that has a three-dimensional face shape of the subject after the subject changes over time and has difference information of the face surface texture before and after the subject changes over time as texture information. The output form in step (S48) and the like are as described in the first embodiment. In this embodiment, the CPU 11 displays on the display device 15 the three-dimensional difference image DAF acquired in step (S46) and the three-dimensional face image BTF acquired in step (S47) in a comparative manner as a free viewpoint image in which the viewpoint can be freely changed by user operation using the input device 16. However, in step (S48), only a three-dimensional difference face image (three-dimensional difference image DAF) may be output. Also, this difference face image may be displayed as a free viewpoint image. In such a case, step (S47) is not necessary.

[0049] In this way, in the second embodiment, the difference information of the facial surface texture before and after a specific state change of the subject is output as a three-dimensional difference image DAF that shows the three-dimensional face shape of the subject after the specific state change, so that not only the change in the facial surface texture accompanying the specific state change but also the face shape after the specific state change can be grasped. As for other effects, the second embodiment can achieve the same effects as the first embodiment.

[0050] Although not shown in FIG. 4, a three-dimensional face image of the subject after a specific state change can be generated by combining the predicted texture image AT and the predicted homology model AF. The CPU 11 may be configured to output, in addition to the three-dimensional difference image DAF and the three-dimensional face image BTF, a three-dimensional face image of the subject after a specific state change so as to be able to be compared.

[0051] [Second modified example] The above-described second embodiment can also be modified as shown in Fig. 5. Fig. 5 is a diagram showing the flow of processing of a face change output method according to the second modified example. In the second modified example, a step (S51) is further executed between the step (S45) and the step (S46) in the flow of the above-described second embodiment. 5 also illustrates a texture time-varying model and a shape time-varying model for predicting a time-varying change as a specific state change, although the second modified example is not limited to a time-varying change.

[0052] Step (S51) is a superimposing step for generating image information by superimposing the difference information acquired in step (S45) and the facial surface texture information indicated by the original face information acquired in step (S42). In the example of Fig. 5, in step (S51), a composite texture image is generated by combining the difference texture image DT acquired in step (S45) with the original face texture image BT acquired in step (S42). In step (S46), the synthetic texture image generated in step (S51) is synthesized with the predicted homology model AF acquired in step (S44) to acquire three-dimensional facial image data. Therefore, the generated three-dimensional facial image data represents a three-dimensional facial image having a three-dimensional facial shape of the subject after the change over time, and a texture in which difference information of the facial surface texture before and after the change over time of the subject is superimposed on the facial surface texture of the original face (before the change over time). This facial image (three-dimensional difference facial image) is referred to as a three-dimensional difference image DTAF.

[0053] According to the second variant, the facial surface texture of the original face (before a specific state change) and the difference information between the facial surface texture of the subject before and after a specific state change are displayed superimposed, making it easier to grasp the specific state change in the facial surface texture.

[0054] [Third embodiment] In the above-described first embodiment, second embodiment, and each modified example, a specific state change prediction of facial surface texture information and facial shape information, and a difference calculation between pieces of facial surface texture information are performed. In the third embodiment, prediction of a specific state change of facial surface texture information is omitted, and a difference calculation between facial shape information is performed. The following describes a facial change output method according to the third embodiment (hereinafter, referred to as the third method), focusing on differences from the above-mentioned embodiments and modified examples. In the following description, the same contents as those described above are omitted as appropriate.

[0055] FIG. 6 is a diagram showing a process flow of the face change output method according to the third embodiment. The AI ​​model used in the third embodiment is obtained by machine learning using training data including multiple sets of three-dimensional facial shape information before and after a specific state change and two-dimensional facial image data before the specific state change, and is capable of predicting at least three-dimensional facial shape information after a specific state change from the two-dimensional facial image data. The three-dimensional face shape information and face surface texture information used in the third embodiment are as described above.

[0056] The AI ​​model in the third embodiment includes at least a three-dimensional prediction model and a shape change model. The three-dimensional prediction model and the shape change model are the same as those in the first or second embodiment. Fig. 6 illustrates a shape time-varying model that predicts a time-varying change as a specific state change, and in the following description, the third method will be described using the example in which a time-varying change is predicted as a specific state change as shown in Fig. 6. However, the third method is not limited to a time-varying change.

[0057] The third method includes steps (S61) to (S67) as shown in FIG. Step (S61) and step (S62) are the same as step (S21) and step (S22) in the first embodiment and step (S41) and step (S42) in the second embodiment. That is, step (S61) is a data acquisition step of acquiring two-dimensional face image data (original face image BP) of the subject, and step (S62) is a step of acquiring original face information indicating at least face shape information and face surface texture information of the original face of the subject by applying the two-dimensional face image data (original face image BP) acquired in step (S61) to a three-dimensional prediction model of the AI ​​model.

[0058] Step (S63) is a prediction step of acquiring changed face information including face shape information after predicted time-dependent change by applying face shape information indicated by the original face information acquired in step (S62) to the trained model. In this embodiment, in step (S63), the original face homology model BF acquired in step (S62) is applied to the shape time-dependent change model of the AI ​​model to generate a predicted homology model AF. Therefore, the predicted homology model AF indicates a face shape obtained by changing the face shape of the original face of the subject over time based on the prediction by the AI ​​model.

[0059] Step (S64) is a difference acquisition step of acquiring face shape difference information indicating a difference between the face shape information of the modified face information and the original face information based on the modified face information acquired in step (S63) and the original face information acquired in step (S62). In this embodiment, in step (S64), difference information between the original face homology model BF acquired in step (S62) and the predicted homology model AF acquired in step (S63) is acquired. The difference information acquired in step (S64) may be any information that indicates the difference between facial shapes, and any known method may be used to acquire the difference information between facial shapes.

[0060] The difference between homologous models can be calculated using, for example, a method using normal distance, a method using shortest distance, etc. In the method using normal distance, the intersection point between the average normal from the polygon vertices of one homologous model and the face (triangle) of the other homologous model is calculated, and the distance to the shortest intersection point is calculated. In the method using shortest distance, the distance from the polygon vertex of one homologous model to the polygon vertex of the other homologous model is calculated, and the vertex with the shortest distance is searched for. However, the method of acquiring the difference between homologous models is not limited to such an example, and any well-known method can be used. In this embodiment, the facial shape difference information is generated as image information.

[0061] Step (S65) is a step of converting the facial shape difference information of the subject before and after the change over time acquired in step (S64) into a three-dimensional image using the facial shape information indicated by the original facial information acquired in step (S62) to acquire a three-dimensional difference facial image. In this embodiment, in step (S65), the facial shape difference information (image) acquired in step (S64) is combined with the original facial homology model BF acquired in step (S62) to acquire three-dimensional facial image data. In this embodiment, the three-dimensional facial image data generated in step (S65) indicates a three-dimensional facial image having a three-dimensional facial shape of the original face (before the change over time) and having difference information of the facial shape of the subject before and after the change over time as texture information, and this facial image (three-dimensional difference facial image) is represented as a three-dimensional difference image DFF.

[0062] Step (S66) is the same as step (S26) in the first embodiment and step (S47) in the second embodiment. That is, in step (S66), the original face texture image BT and the original face homology model BF acquired in step (S62) are synthesized to acquire a three-dimensional original face image (three-dimensional face image BTF).

[0063] Step (S67) is similar to step (S25) in the first embodiment and step (S48) in the second embodiment, and the three-dimensional difference face image acquired in step (S65) and the three-dimensional original face image based on the original face information acquired in step (S66) are output so as to be comparable. The output form in step (S67) is as described in the first embodiment. In this embodiment, the CPU 11 displays on the display device 15 the three-dimensional difference image DFF acquired in step (S65) and the three-dimensional face image BTF acquired in step (S66) in a comparative manner as a free viewpoint image in which the viewpoint can be freely changed by user operation using the input device 16. However, in step (S67), only a three-dimensional difference face image (three-dimensional difference image DFF) may be output. Also, this difference face image may be displayed as a free viewpoint image. In such a case, step (S66) is not necessary.

[0064] In this way, in the third embodiment, the difference information in the facial shape of the subject before and after a specific state change is output as a three-dimensional difference image DFF that shows the three-dimensional facial shape of the subject's original face (before the specific state change), making it easy to understand the change in facial shape that accompanies the specific state change.

[0065] [Third Modification] The above-described third embodiment can also be modified as shown in Fig. 7. Fig. 7 is a diagram showing the flow of processing of a face change output method according to the third modified example. In the third modified example, a step (S71) is further executed between the step (S64) and the step (S65) in the flow of the above-described third embodiment. 7 also illustrates a shape change over time model that predicts a change over time as a specific state change, although the third modified example is not limited to a change over time.

[0066] Step (S71) is a superimposing step for generating image information by superimposing the difference information acquired in step (S64) and the face surface texture information indicated by the original face information acquired in step (S62). In the example of Fig. 7, in step (S71), a composite texture image is generated by combining the difference image acquired in step (S64) with the original face texture image BT acquired in step (S62). In step (S65), the synthetic texture image generated in step (S71) is synthesized with the original face homology model BF acquired in step (S62) to acquire three-dimensional face image data. Therefore, the generated three-dimensional face image data represents a three-dimensional face image having a three-dimensional face shape of the subject before the change over time and a texture in which difference information of the face shape before and after the change over time of the subject is superimposed on the face surface texture of the original face (before the change over time). This face image (three-dimensional difference face image) is expressed as a three-dimensional difference image DTFF.

[0067] According to the third variant, the facial surface texture of the original face (before a specific state change) and the difference information between the subject's facial shape before and after a specific state change are displayed superimposed, making it easier to understand the specific state change in facial shape.

[0068] [Fourth modified example] In each of the above-mentioned embodiments and variants, the difference information of the facial surface texture or facial shape before and after a specific state change is used as a texture, and a three-dimensional difference face image (three-dimensional difference image DBF, DTBF, DAF, DTAF, DFF, or DTFF) having the three-dimensional facial shape before or after a specific state change is output so as to be comparable with the three-dimensional original face image (three-dimensional face image BTF). However, as shown in FIG. 8, two or more three-dimensional difference face images may be output so as to be comparable with the three-dimensional original face image.

[0069] 8 is a diagram showing the flow of processing of a face change output method according to a fourth modified example, which is a modified example of the second embodiment shown in FIG. As shown in FIG. 8, in the fourth modified example, in addition to the steps of the second embodiment, a step (S81) and a step (S82) are further executed. A texture time-varying model and a shape time-varying model that predict a time-varying change as a specific state change are also illustrated in Fig. 8. However, the fourth modified example is not limited to a time-varying change.

[0070] Step (S81) is the same as step (S64) in the third embodiment, and is a difference acquisition step for acquiring difference information of face shape. That is, in step (S81), difference information between the original face homology model BF acquired in step (S42) and the predicted homology model AF acquired in step (S44) is acquired. Step (S82) is the same as step (S65) in the third embodiment, and is a step of converting the difference information of the face shape before and after the change over time into a three-dimensional image using the face shape information indicated by the original face information to obtain a three-dimensional difference face image. By step (S82), a three-dimensional difference face image (difference image DFF) is obtained, which has the three-dimensional face shape of the original face (before the change over time) and has the difference information of the face shape before and after the change over time of the subject as texture information.

[0071] In step (S48) in the fourth variant, a three-dimensional difference image DAF showing the difference in facial surface texture before and after the subject's change over time, a three-dimensional difference image DFF showing the difference in the subject's facial shape before and after the subject's change over time, and the three-dimensional original face image (three-dimensional face image BTF) are output so as to be comparable. 8, a three-dimensional difference image DBF, DTBF or DTAF may be output instead of the three-dimensional difference image DAF, or a three-dimensional difference image DTFF may be output instead of the three-dimensional difference image DFF. Also, a three-dimensional difference image showing the difference in the facial shape before and after the change over time of the subject and a three-dimensional difference image showing the difference in the facial surface texture before and after the change over time of the same subject may be output so as to be comparable, and the three-dimensional original face image may not be output. That is, in the step (S48) (output step) in the fourth modified example, at least a first differential face image obtained by converting the difference information of the facial surface texture into a three-dimensional image and a second differential face image obtained by converting the difference information of the facial shape into a three-dimensional image are output so as to be comparable. This allows the user to compare specific state changes in the facial surface texture with specific state changes in the facial shape, making it easy for the user to understand the specific state changes in the facial surface texture and facial shape.

[0072] In the output form in the fourth modified example, all of the 3D difference images DAF and DFF and the 3D face image BTF may be output in an arranged manner, or all or some combinations of the two pairs may be output in an arranged manner so as to be directly compared. Also, each image may be displayed on the display device 15 in a sequential switching manner. Furthermore, it is also possible to allow a user to specify which image is to be output by a user operation using the input device 16. For example, the CPU 11 can cause the display device 15 to display an input screen that allows a user to specify whether to output only the three-dimensional difference image DAF, only the three-dimensional difference image DFF, or both, obtain output type information specified by a user operation on the input screen, and display only the three-dimensional difference image DAF, only the three-dimensional difference image DFF, or both, specified by the output type information, on the display device 15. Of course, the three-dimensional face image BTF may be included as a candidate in such output image specification. That is, the facial change output method further includes a step of allowing a user to specify by operation whether to output only the first difference face image obtained by three-dimensionally imaging the difference information of the facial surface texture, to output only the second difference face image obtained by three-dimensionally imaging the difference information of the facial shape, or to output both the first difference face image and the second difference face image, and a step of acquiring output type information designated by the user operation, and in the output step, it is also possible to output one or both of the first difference face image or the second difference face image designated by the acquired output type information. In this way, the user can specify the image he or she wants to see, improving the comprehension of specific state changes in facial surface texture and facial shape.

[0073] [Other variations] In each of the above-mentioned embodiments and each of the modified examples, the original face texture image BT and the original face homology model BF are generated from the two-dimensional face image data showing the original face of the subject by using an AI model (three-dimensional prediction model), but the CPU 11 may acquire the original face texture image BT and the original face homology model BF from another computer, a portable recording medium, or the like. In this case, the two-dimensional face image data acquisition step (S21, etc.) and the original face information acquisition step (S22, etc.) are not necessary, and the three-dimensional prediction model is also not necessary. In this case, the two-dimensional face image data acquisition step (S21, etc.) and the original face information acquisition step (S22, etc.) may be executed by another computer, and the CPU 11 may acquire the original face texture image BT and the original face homology model BF from that computer.

[0074] In addition, the difference information of the facial surface texture or facial shape before and after a specific state change of the subject is converted into a three-dimensional image and output using the facial shape information indicated by the original facial information or the facial shape information after a specific state change, but the facial surface texture difference information and facial shape difference information may also be superimposed and converted into a three-dimensional image using the facial shape information indicated by the original facial information or the facial shape information after a specific state change. In this way, each of the specific state changes of the facial surface texture and the specific state changes of the facial shape can be confirmed on a single three-dimensional image. Some or all of the above contents may be specified as follows. However, the above contents are not limited to the following descriptions.

[0075] <1> One or more processors capable of using a trained model obtained by machine learning, the trained model including at least one or both of a shape change model for predicting a specific state change of a three-dimensional face shape and a texture change model for predicting a specific state change of a texture of a face surface, acquiring original face information indicating at least one or both of face surface texture information and three-dimensional face shape information; a prediction step of acquiring predicted changed face information by applying one or both of the facial surface texture information and the facial shape information indicated by the acquired original face information to the trained model; a difference acquiring step of acquiring difference information of a face surface texture or a face shape indicating a difference between face surface texture information or face shape information of the changed face information and the original face information based on the acquired changed face information and the acquired original face information; an output step of outputting a three-dimensional difference face image obtained by converting the acquired difference information into a three-dimensional image using at least the face shape information indicated by the original face information or the face shape information indicated by the changed face information; Implement the face changing output method.

[0076] <2> The trained model includes at least the shape change model and the texture change model, In the acquiring step, original face information indicating at least face surface texture information and three-dimensional face shape information is acquired; In the prediction step, the facial surface texture information and the facial shape information indicated by the acquired original facial information are applied to the trained model to acquire predicted changed facial information, In the difference acquisition step, difference information of facial surface texture is acquired, the difference information indicating a difference between facial surface texture information of the changed face information and the original face information; In the output step, a three-dimensional difference face image is output, the difference information being obtained by three-dimensionally imaging the difference information using at least face shape information indicated by the changed face information. <1> The face change output method according to claim 1. <3> The trained model includes at least the texture change model, In the acquiring step, original face information indicating at least face surface texture information and three-dimensional face shape information is acquired; In the prediction step, predicted changed face information is obtained by applying the face surface texture information indicated by the acquired original face information to the trained model, In the difference acquisition step, difference information of facial surface texture indicating a difference between facial surface texture information of the changed face information and the original face information is acquired, In the output step, a three-dimensional difference face image is output, the difference information of the acquired face surface texture is three-dimensionally imaged using at least the face shape information indicated by the original face information. <1> or <2> The face change output method according to claim 1. <4> The trained model includes at least the shape change model, In the acquiring step, original face information indicating at least three-dimensional face shape information is acquired, In the prediction step, the face shape information indicated by the acquired original face information is applied to the trained model to acquire predicted changed face information, In the difference acquisition step, difference information on face shape indicating a difference between face shape information of the modified face information and the original face information is acquired, In the output step, a three-dimensional difference face image is output, the difference face image being obtained by converting the acquired difference information of face shape into a three-dimensional image using at least the face shape information indicated by the original face information. <1> from <3> 13. A facial change output method according to claim 12, <5> The output step includes: a superimposing step of generating image information by superimposing the acquired difference information and the facial surface texture information indicated by the original face information, outputting the three-dimensional difference face image obtained by converting the generated image information into a three-dimensional image using the face shape information indicated by the original face information or the face shape information indicated by the changed face information; <1> from <4> 13. A facial change output method according to claim 12, <6> In the output step, a first differential face image obtained by converting the difference information of the facial surface texture into a three-dimensional image and a second differential face image obtained by converting the difference information of the facial shape into a three-dimensional image are output so as to be comparable. <1> from <5> 13. A facial change output method according to claim 12, <7> a step of allowing a user to specify whether to output only the first difference face image, only the second difference face image, or both the first difference face image and the second difference face image; acquiring output type information designated by the user operation; Further execute In the output step, one or both of the first difference face image and the second difference face image designated by the acquired output type information are output. <6> The face change output method according to claim 1. <8> In the output step, a three-dimensional original face image based on the acquired original face information and the three-dimensional difference face image are displayed on a display device so that the three-dimensional original face image based on the acquired original face information can be compared with each other and a viewpoint can be freely changed by a user operation. <1> from <7> 13. A facial change output method according to claim 12, <9> An apparatus capable of using the trained model, <1> from <8> 2. A face change output device capable of executing the face change output method according to claim 1.

[0077] In the process flows shown in the figures, each step is assigned a symbol, but the order of steps in each embodiment is not limited to the order of the symbols. In each embodiment, the order of steps can be appropriately determined as long as it does not cause any problems. In addition, the above-mentioned embodiments and modifications can be combined as long as the contents are not contradictory. [Explanation of symbols]

[0078] 10 Information processing device (face change output device) 11 CPU 12. Memory 13 Input / Output Interface 14 Communication unit 15 Display device 16 Input Devices

Claims

1. One or more processors capable of using trained models obtained by machine learning, including at least a shape change model for predicting a specific state change of a three-dimensional face shape and a texture change model for predicting a specific state change of a texture of a face surface; acquiring original face information indicating at least face surface texture information and three-dimensional face shape information; A prediction step of acquiring predicted changed face information by applying the facial surface texture information and the facial shape information indicated by the acquired original face information to the trained model; a difference obtaining step of obtaining facial surface texture difference information indicating a difference between facial surface texture information of the obtained changed facial information and the obtained original facial information; an output step of outputting a three-dimensional difference face image obtained by converting the acquired difference information into a three-dimensional image using at least face shape information indicated by the changed face information; Implement the face changing output method.

2. One or more processors capable of using a trained model obtained by machine learning, the trained model including at least a texture change model for predicting a specific state change of a texture of a facial surface; acquiring original face information indicating at least face surface texture information and three-dimensional face shape information; A prediction step of acquiring predicted changed face information by applying the face surface texture information indicated by the acquired original face information to the trained model; a difference acquiring step of acquiring facial surface texture difference information indicating a difference between facial surface texture information of the acquired changed face information and the acquired original face information; an output step of outputting a three-dimensional differential face image obtained by converting the acquired difference information of the facial surface texture into a three-dimensional image using at least the face shape information indicated by the original face information; Implement the face changing output method.

3. One or more processors capable of using a trained model obtained by machine learning, the trained model including at least one or both of a shape change model for predicting a specific state change of a three-dimensional face shape and a texture change model for predicting a specific state change of a texture of a face surface, acquiring original face information indicating at least face surface texture information and three-dimensional face shape information; a prediction step of acquiring predicted changed face information by applying one or both of the facial surface texture information and the facial shape information indicated by the acquired original face information to the trained model; a difference acquiring step of acquiring difference information of a face surface texture or a face shape indicating a difference between face surface texture information or face shape information of the changed face information and the original face information based on the acquired changed face information and the acquired original face information; an output process of generating image information by superimposing the acquired difference information and the facial surface texture information indicated by the original face information, and outputting a three-dimensional difference face image obtained by three-dimensionally imaging the generated image information using the facial shape information indicated by the original face information or the facial shape information indicated by the changed face information; Implement the face changing output method.

4. One or more processors capable of using trained models obtained by machine learning, including at least a shape change model for predicting a specific state change of a three-dimensional face shape and a texture change model for predicting a specific state change of a texture of a face surface; acquiring original face information indicating at least face surface texture information and three-dimensional face shape information; A prediction step of acquiring predicted changed face information by applying the facial surface texture information and the facial shape information indicated by the acquired original face information to the trained model; a difference acquiring step of acquiring facial surface texture difference information indicating a difference between facial surface texture information between the modified face information and the original face information and facial shape difference information indicating a difference between facial shape information between the modified face information and the original face information based on the acquired modified face information and the acquired original face information; an output process for outputting, in a manner that allows comparison between a first three-dimensional differential face image obtained by three-dimensionally imaging the acquired difference information of face surface texture using at least the face shape information indicated by the original face information or the face shape information indicated by the changed face information, and a second three-dimensional differential face image obtained by three-dimensionally imaging the acquired difference information of face shape using at least the face shape information indicated by the original face information or the face shape information indicated by the changed face information; Implement the face changing output method.

5. a step of allowing a user to specify whether to output only the first difference face image, only the second difference face image, or both the first difference face image and the second difference face image; acquiring output type information designated by the user operation; Further execute In the output step, one or both of the first difference face image and the second difference face image designated by the acquired output type information are output. The facial change output method according to claim 4.

6. One or more processors capable of using a trained model obtained by machine learning, the trained model including at least one or both of a shape change model for predicting a specific state change of a three-dimensional face shape and a texture change model for predicting a specific state change of a texture of a face surface, acquiring original face information indicating at least one or both of face surface texture information and three-dimensional face shape information; a prediction step of acquiring predicted changed face information by applying one or both of the facial surface texture information and the facial shape information indicated by the acquired original face information to the trained model; a difference acquiring step of acquiring difference information of a face surface texture or a face shape indicating a difference between face surface texture information or face shape information of the changed face information and the original face information based on the acquired changed face information and the acquired original face information; an output step of outputting a three-dimensional difference face image obtained by converting the acquired difference information into a three-dimensional image using at least the face shape information indicated by the original face information or the face shape information indicated by the changed face information; Run In the output step, a three-dimensional original face image based on the acquired original face information and the three-dimensional difference face image are displayed on a display device so that the three-dimensional original face image based on the acquired original face information can be compared with each other and a viewpoint can be freely changed by a user operation. Facial change output method.

7. A face change output device that can use the trained model and executes the face change output method according to claim 1 .

Images