Data construction device, data construction method, and data construction program

Abstract

A data construction device (10) comprises: a data acquisition unit (11) that acquires speech image data pertaining to speech by a first speaker and facial image data pertaining to the face of a second speaker; a data extension unit (121) that generates, on the basis of the speech image data and the facial image data acquired by the data acquisition unit (11), extended speech image data pertaining to speech by the second speaker; and a data construction unit (122) that constructs, as training data for training a learning model, a set of the extended speech image data generated by the data extension unit (121) and an impression label pertaining to an impression of the speech.

Description

Data construction device, data construction method, and data construction program

[0001] The present invention relates to a data construction device, a data construction method, and a data construction program.

[0002] A technique is known in which speech video data is converted into speech image data relating to facial expression utterances that correspond to the individuality of the facial image data, mouth movements corresponding to the speech audio data, and the type and intensity of emotions by inputting facial image data of neutral (average) expressions, speech audio data, and the type and intensity of emotions. For example, such a technique includes a converter that uses speech video data of multiple actors to train a learning model for estimating speech video data with converted faces (see Non-Patent Document 1).

[0003] Sanjana Sinha, Sandika Biswas, Ravindra Yadav, Brojeshwar Bhowmick, “Emotion-Controllable Generalized Talking Face Generation”, arXiv:2205.01155v1 [cs.CV] 2 May 2022

[0004] However, conventional technologies may not be able to convert speech data into natural-looking image data at low cost. For example, conventional technologies using a speech video database containing speeches by multiple speakers, each labeled with an emotion label indicating the speaker's emotion, may require high database construction costs when training a learning model. For example, conventional technologies require high construction costs for converting speech video data into speech data based on information other than emotion, such as speech impressions (e.g., "friendly," "considerate," etc.). However, constructing such a database is costly. Furthermore, to reduce costs, a conventional technology may construct a database of impression-labeled, multiple-speaker speech video data containing impression-labeled, multiple-speaker speech video data containing impression-labeled, multiple-speaker speech video data containing impression-labeled, single-speaker speech video data containing impression-labeled, single-speaker speech video data containing impression-labeled, single-speaker speech video data containing impression-labeled, single-speaker speech video data containing impression-labeled, single-speaker speech video data containing impression-labeled, single-speaker speech video data containing impression-labeled, single-speaker speech video data, and training a learning model using the conventional technology. However, the training data may not contain data that matches the individuality of the speaker's face, resulting in speech video data with unnatural facial expressions being output.

[0005] In order to solve the above-mentioned problems, the present invention comprises a data acquisition unit that acquires speech image data related to an utterance by a first speaker and facial image data related to the face of a second speaker, a data extension unit that generates extended speech image data related to the utterance by the second speaker based on the speech image data and the facial image data acquired by the data acquisition unit, and a data construction unit that constructs a pair of the extended speech image data generated by the data extension unit and an impression label related to the impression of the utterance as training data for training a learning model.

[0006] According to the present invention, conversion to natural speech image data can be performed at low cost.

[0007] FIG. 1 is a diagram illustrating an example of an overview of a data construction system according to a first embodiment. FIG. 2 is a diagram illustrating an example of a configuration of a data construction system according to the first embodiment. FIG. 3 is a diagram illustrating an example of data extension. FIG. 4 is a diagram illustrating an example of data extension. FIG. 5 is a diagram illustrating an example of learning. FIG. 6 is a flowchart illustrating an example of a flow of processing executed by the data construction system according to the first embodiment. FIG. 7 is a diagram illustrating an example of a configuration of a data construction system according to a second embodiment. FIG. 8 is a diagram illustrating an example of calculation. FIG. 9 is a diagram illustrating an example of calculation. FIG. 10 is a diagram illustrating an example of calculation. FIG. 11 is a diagram illustrating an example of learning. FIG. 12 is a flowchart illustrating an example of a flow of processing executed by the data construction system according to the second embodiment. FIG. 13 is a diagram illustrating an example of a configuration of a computer that executes a data construction program. FIG. 14 is a diagram illustrating an example of an overview of a data construction system according to a reference technology. FIG. 15 is a diagram illustrating an example of an overview of a data construction system according to a reference technology.

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments. In addition, in the description of the drawings, the same parts are given the same reference numerals, and duplicated explanations are omitted, and explanations of members having similar functions and similar processes are also omitted.

[0009] (0. Reference Technology) (Example of Overview) Before describing this embodiment, an example of an overview of a data construction system 1# according to the reference technology will be described using Figures 14 and 15. Figures 14 and 15 are diagrams for explaining an example of an overview of a data construction system according to the reference technology. The data construction system 1# according to the reference technology combines speech video data and facial image data, and trains a learning model that outputs speech image data related to facial expressions of speech according to the individuality of the facial image data, etc. Here, the speech image data refers to image data related to a speech by a first speaker and may include still image data. The facial image data refers to image data related to a second speaker.

[0010] 14, the data construction system 1# generates output speech video data for which the speaker and speech content are the same and in which the speaker's facial expression is converted according to the emotion label, based on the emotion label related to the speaker's emotion and the input speech video data. In this case, the data construction system 1# uses, for example, the input speech video data and the emotion label as input, and trains a learning model by supervised learning using the feature quantities of the speaker's facial expression in the output speech video data as output ground truth data.

[0011] In the example shown in Figure 15, the data construction system 1# generates output speech video data from input speech video data with emotion labels assigned in an emotion-labeled multi-speaker speech video database, which is a database of speech video data by multiple speakers with emotion labels, where the speakers and speech content are the same but the speakers' facial expressions are converted according to different emotion labels. Here, "multiple" is not particularly limited as long as it is a predetermined number or more, but can refer to, for example, hundreds or thousands of people or more. Also, "few" is not particularly limited as long as it is less than the predetermined number, but can refer to, for example, two or three people.

[0012] For example, the data construction system 1# uses input speech video data and emotion labels as input, and a facial expression feature sequence relating to the time series of features of the speaker's facial expression in the output speech video data as output ground truth data, and trains a learning model through supervised learning. As an example, a converter in the data construction system 1# (which converts the speaker's facial expression in the speech video data) first extracts video features from the video data of the input speech video data and extracts audio features from the audio data of the input speech video data. Next, the converter trains a learning model so that it can estimate the speaker's facial expression feature sequence in the output speech video data based on, for example, the video features, audio features, and emotion labels.

[0013] (Example of Estimation) In the example shown in FIG. 14 , the data construction system 1# uses a learning model to estimate features of a speaker's facial expression in output speech video data based on emotion labels and input speech video data, and estimates output speech video data based on these features. In the example shown in FIG. 15 , the converter extracts video features from the video data of the input speech video data and audio features from the audio data, as in training the learning model. For example, the converter estimates a facial expression feature sequence based on video features, audio features, and emotion labels. For example, the converter estimates output speech video data by rendering based on input speaker image data and a facial expression feature sequence related to a speaker input to the learning model. As an example, the converter estimates output speech video data in which the speaker's facial expression in the input speaker image data is converted according to the facial expression feature sequence to an expression corresponding to the speaker's expression in the output speech video data. The converter uses, for example, the rendering technology described in the following reference document. References: Aliaksandr Siarohin, Stephane Lathuiliere, Sergey Tulyakov, Elisa Ricci and Nicu Sebe, “First Order Motion Model for Image Animation”, NeurIPS 2019

[0014] (Example of Problem) However, with the above-mentioned reference technology, there are cases where conversion to natural speech image data cannot be performed at low cost. For example, with the reference technology, in order to perform conversion to speech video data according to impression labels relating to the impression of the speech that is different from the emotion label, it is necessary to newly construct a database of speech videos from multiple speakers with impression labels, which is costly to construct. This embodiment has been made to solve this problem, and makes it possible to perform conversion to natural speech image data at low cost. Details of this embodiment will be described later.

[0015] (1. First Embodiment) (Example of Overview) An example of an overview of a data construction system 1 according to a first embodiment will be described using FIG. 1. FIG. 1 is a diagram for explaining an example of an overview of a data construction system according to a first embodiment. In the example shown in FIG. 1, the data construction system 1 acquires input speech image data (speech image data) and face image data, generates extended speech image data based on the input speech image data and the face image data, and constructs pairs of the extended speech image data and impression labels as learning data.

[0016] Speech image data refers to image data related to a speech by a first speaker and may include still image data. Facial image data refers to image data related to a second speaker. Augmented speech image data refers to broadly defined speech image data related to a speech in which the facial expressions of the second speaker have been converted and may include still image data. Training data refers to data used for training a learning model or the like that uses impression labels, speech image data, and facial image data as inputs and outputs facial features of the second speaker in the augmented speech image data.

[0017] For example, the data construction system 1 generates extended speech video data containing extended speech image data in which the individual facial features have been converted so that the face of the first speaker becomes the face of the second speaker in the facial image data, while conforming to the facial expression of the first speaker in the speech video data with impression labels that includes the input speech image data, and constructs a pair of the extended speech video data and the impression label as learning data.

[0018] (Example of configuration of data construction system) An example of the configuration of the data construction system 1 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the data construction system according to the first embodiment. In the example shown in Fig. 2, the data construction system 1 includes a data construction device 10.

[0019] (Data Construction Device) The data construction device 10 acquires speech image data and face image data, generates extended speech image data based on the speech image data and the face image data, and constructs pairs of the extended speech image data and impression labels as learning data. In the example shown in Figure 2, the data construction device 10 has a data acquisition unit 11, a control unit 12, an output unit 13, and a storage unit 14.

[0020] (Data Acquisition Unit) The data acquisition unit 11 acquires speech image data related to speech by a first speaker and facial image data related to the face of a second speaker. For example, the data acquisition unit 11 may acquire speech video data and facial image data including speech image data. As an example, the data acquisition unit 11 may acquire speech image data from a minority speaker speech image database related to speech by each of a predetermined number of minority speakers, and acquire facial image data from a multiple facial image database related to each facial image of a predetermined number or more of multiple speakers. When the minority speaker speech image database includes speech video data, it is also referred to as a minority speaker speech video database. For example, the data acquisition unit 11 acquires speech video data (i) from the minority speaker speech video database and facial image data (k) from the multiple facial image database. Because the minority speaker speech image database and the multiple facial image data can be constructed at low cost, the data acquisition unit 11 can reduce costs by acquiring image data from these databases.

[0021] For example, the data acquisition unit 11 may further acquire facial features of a second speaker in the extended speech image data. As an example, the data acquisition unit 11 acquires a facial landmark of the speaker of the extended speech video data (i, k) by extracting facial features (landmarks) of the speaker of the extended speech video data (i, k) from the extended speech video data (i, k). i is a number corresponding to the speaker number of the speech video data. k is a number corresponding to the speaker number of the face image data.

[0022] (Control Unit) The control unit 12 controls the entire data construction device 10. For example, the control unit 12 is configured with one or more processors having internal memory storing programs that define various processing procedures and control information, and the processor executes each process using the programs and internal memory. The control unit 12 is realized, for example, by electronic circuits such as a central processing unit (CPU), a micro processing unit (MPU), or a graphics processing unit (GPU), or by integrated circuits such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In the example shown in FIG. 2, the control unit 12 includes a data expansion unit 121, a data construction unit 122, a learning unit 123, and an estimation unit 124.

[0023] (Data Expansion Unit) The data expansion unit 121 generates expanded speech image data related to the speech by the second speaker based on the speech image data and facial image data acquired by the data acquisition unit 11. For example, the data expansion unit 121 may generate expanded speech video data including expanded speech image data based on the speech video data and facial image data acquired by the data acquisition unit 11. The data expansion unit 121 is realized, for example, by a converter that converts the features of the speaker's facial expression. An example of data expansion by the data expansion unit 121 will be described below with reference to Figures 3 and 4. Figures 3 and 4 are diagrams for explaining an example of data expansion.

[0024] 3, the data expansion unit 121 generates extended speech video data (i, k) based on impression-labeled speech video data (i) and facial image data (k) in the impression-labeled speech video database. As an example, the data expansion unit 121 generates extended speech video data (i, k) related to speech by a speaker in facial image data (k) in which facial expression features have been converted so that the speaker's face in the speech video data (i) has an expression converted to the expression of the speaker's face in the facial image data (k).

[0025] In the example shown in Figure 4, the data expansion unit 121 performs data expansion of the speech video data using a few-speaker speech video database containing speech video data (i) by a few speakers and a multiple-face image database containing a large number of face image data (k). For example, the data expansion unit 121 generates expanded speech video data (i, k) having the facial expressions of the speakers in the speech video data (i) and the individual characteristics of the speakers in the facial image data (k) based on the speech video data (i) in the few-speaker speech video database and the facial image data (k) in the multiple-person face image database. As an example, the data expansion unit 121 generates each frame image of the expanded speech video data (i, k) corresponding to all combinations of each frame image of the speech video data (i) in the impression-labeled few-speaker speech video database and each face image data (k) in the multiple-face image database, and generates expanded speech video data (i, k) having the facial expressions of the speakers in the speech video data (i) and the individual characteristics of the speakers in the facial image data (k). For example, the data augmentation unit 121 generates the augmented speech video data (i, k) using a facial expression conversion technique such as that described in Reference 2 below. The data augmentation unit 121 may also perform the above-described processing for each i and k. Reference 2: Sirui Zhao, Shukang Yin, Huaying Tang, Rijin Jin, Yifan Xu, Tong Xu, Enhong Chen, "Fine-grained Micro-Expression Generation based on Thin-Plate Spline and Relative AU Constraint," MM '22: Proceedings of the 30th ACM International Conference on Multimedia, October 2022, Pages 7150-7154

[0026] (Data Construction Unit) The data construction unit 122 constructs pairs of the extended speech image data generated by the data expansion unit 121 and impression labels related to the impressions of the speech as training data for training a learning model. The data construction unit 122 can construct speech video data of natural facial expressions from a large number of speakers as training data at low cost, thereby enabling conversion to natural speech image data at low cost. For example, the data construction unit 122 may construct pairs of the extended speech video data generated by the data expansion unit 121 and impression labels as training data.

[0027] For example, as shown in FIG. 4 , the data constructing unit 122 constructs an impression-labeled extended speech video database containing impression-labeled extended speech video data (i, k) as a database for training data. As an example, the data constructing unit 122 adds pairs of extended speech video data (i, k) and impression labels related to the impressions of the speech (impression-labeled extended speech video data) to the impression-labeled extended speech video database. As a result, the database includes speech video data of natural facial expressions from a large number of speakers. By training a learning model using data including a large number of speakers, the data constructing unit 122 can convert the facial expressions of speakers in the extended speech video data to natural facial expressions that match the individuality of the input speaker's face. Because a small-speaker speech video database and large-face image data can be constructed at low cost, the data constructing unit 122 can convert the data into natural speech image data at low cost.

[0028] (Learning Unit) The learning unit 123 uses the learning data, speech image data, and face image data constructed by the data construction unit 122 to train a learning model that takes impression labels, speech image data, and face image data as input and outputs facial features of the second speaker in the extended speech image data. In this way, the learning unit 123 trains the learning model in the same manner as the reference technology, except that the learning model is trained using extended utterance video data with impression labels as training data. As a result, the learning unit 123 realizes conversion of the speaker's facial expression in the extended speech image data to a natural expression that matches the individuality of the face of the speaker (input speaker) in the face image data input to the learning model, and can convert to natural speech image data at low cost.

[0029] An example of learning by the learning unit 123 will be described below with reference to Fig. 5. Fig. 5 is a diagram showing an example of learning. For example, as shown in Fig. 5, the learning unit 123 may train a learning model that receives impression labels, speech video data, and face image data from the extended speech video data with impression labels, speech video data, and face image data constructed by the data construction unit 122, and outputs facial features of the second speaker in the extended speech video data.

[0030] As an example, the learning unit 123 uses the impression-labeled extended speech video data (i, k), the speech video data (i), and the facial image data (k) to train a learning model that takes the impression-labeled speech video data (i) and the facial image data (k) as input and outputs a landmark of the speaker's face in the extended speech video data (i, k). This allows the learning unit 123 to train the learning model so that conversion into natural speech video data in which the speaker's facial expression (facial expression feature) is converted into a natural expression (impression conversion facial expression feature) that matches the individuality of the input speaker's face is possible at low cost.

[0031] (Estimation unit 124) The estimation unit 124 estimates facial features of the second speaker in the extended utterance image data based on the learning model trained by the learning unit 123. For example, as shown in FIG. 5 , the estimation unit 124 estimates the coordinates (post-conversion facial expression features) of the speaker's facial landmark at each time in the impression-labeled extended utterance video data (i, k) based on the learning model, and estimates the speaker's facial landmark at each time (impression conversion facial expression features). Furthermore, for example, the estimation unit 124 estimates output utterance video data corresponding to the impression-labeled extended utterance video data (i, k) in which the impression conversion facial expression features are reflected by rendering. Thus, in the example shown in FIG. 5 , the estimation unit 124 performs estimation in the same manner as the reference technology, except that the estimation targets are the facial features of the second speaker in the extended utterance video data and the output utterance video data corresponding to the extended utterance video data. The estimation unit 124 is realized by, for example, a converter that converts the feature amount of the speaker's facial expression.

[0032] (Output Unit) The output unit 13 outputs various data. For example, the output unit 13 outputs output utterance video data corresponding to the impression-labeled extended utterance video data (i, k) estimated by the estimation unit 124. The output unit 13 is realized by, for example, a display unit such as a display.

[0033] (Storage Unit) The storage unit 14 stores various data. For example, the storage unit 14 stores a database of speech videos from a few speakers with impression labels, a database of multiple face images, a learning model, an operating system (OS) and various programs executed by the data construction device 10. The storage unit 14 is realized by, for example, a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk, or a rewritable semiconductor memory such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM).

[0034] (Example of processing flow) An example of processing flow executed by the data construction device 10 of the data construction system 1 according to the first embodiment will be described below with reference to Fig. 6. Fig. 6 is a flowchart showing an example of processing flow executed by the data construction system according to the first embodiment.

[0035] In step S11, the data acquisition unit 11 acquires speech image data relating to a speech by a first speaker and face image data relating to a face of a second speaker. For example, when training a learning model, the data acquisition unit 11 acquires speech video data (i) from the few-speaker speech video database and face image data (k) from the multiple-face image database.

[0036] In step S12, the data expansion unit 121 generates expanded speech image data related to speech by the second speaker based on the speech image data and face image data acquired by the data acquisition unit 11. For example, the data expansion unit 121 generates expanded speech video data (i, k) related to speech by the speaker of the face image data, in which the facial expression of the speaker in the speech video data (i) is converted into the facial expression of the speaker in the face image data (k).

[0037] In step S13, the data construction unit 122 constructs, as training data for training the learning model, pairs of the extended utterance image data and impression labels related to the impressions of the utterances generated by the data expansion unit 121. For example, the data construction unit 122 constructs, as a database for training data, an extended utterance video database with impression labels, which includes extended utterance video data (i, k) with impression labels.

[0038] In step S14, the learning unit 123 trains a learning model that takes the impression label, speech image data, and face image data as input and outputs the facial features of the second speaker in the extended speech image data, using the training data, speech image data, and face image data constructed by the data construction unit 122. For example, the learning unit 123 trains a learning model that takes the impression-labeled speech video data (i, k), speech video data (i), and face image data (k) as input and outputs the speaker's facial landmark in the extended speech video data (i, k).

[0039] In step S15, the estimation unit 124 estimates facial features of the second speaker in the extended utterance image data based on the learning model learned by the learning unit 123. For example, the estimation unit 124 estimates the coordinates of the speaker's facial landmark at each time in the impression-labeled extended utterance video data (i, k) based on the learning model, and estimates output utterance video data corresponding to the impression-labeled extended utterance video data (i, k) in which the speaker's facial landmark at each time is reflected.

[0040] (2. Second Embodiment) (Example of Overview) The data construction system 1, for example, estimates the coordinates of a speaker's facial landmark at each time based on a learning model, and estimates output speech video data corresponding to impression-labeled augmented speech video data (i, k) reflecting the speaker's facial landmark at each time. The position of this landmark includes two types of information: facial individuality (average landmark position) and changes in facial expression (difference from the average landmark position). Therefore, in the above example, the data construction system 1 simultaneously estimates two different types of information.

[0041] In contrast, the data construction system 1X according to the second embodiment trains a learning model using training data, impression labels, speech image data, and facial image data, taking the impression labels, speech image data, and facial image data as inputs and outputting feature quantities of facial expression changes of a second speaker in the extended speech image data. For example, during training of the learning model, the data construction system 1X normalizes the speaker's facial landmarks to a mean value of 0 and a variance of 1, thereby eliminating facial individuality information and creating vectors that represent only facial expression changes. In the above example, the data construction system 1X estimates feature quantities of facial expression changes of the second speaker in the extended speech image data, such as vectors that represent only facial expression changes. Therefore, compared to the data construction system 1, the data construction system 1X simplifies the input / output relationship with the learning model, facilitating training of the learning model, improving conversion accuracy, and enabling the synthesis of speech video data and facial image data to achieve facial expressions that match individual characteristics with less training data.

[0042] (Example of configuration of data construction system) An example of the configuration of a data construction system 1X according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the configuration of a data construction system according to the second embodiment. In the example shown in Fig. 7, the data construction system 1X includes a data construction device 10X.

[0043] (Data Construction Device) The data construction device 10X uses learning data, impression labels, speech image data, and face image data to train a learning model that takes the impression labels, speech image data, and face image data as inputs and outputs feature quantities of facial expression changes of the second speaker in the extended speech image data. The data construction device 10X has a control unit 12X instead of the control unit 12 in the first embodiment. Except for this point, the data construction device 10X is similar to the data construction device 10 according to the first embodiment.

[0044] (Control Unit) The control unit 12X further includes a calculation unit 125, and includes a learning unit 123X and an estimation unit 124X instead of the learning unit 123 and the estimation unit 124 in the first embodiment. Except for this, the control unit 12X is similar to the control unit 12 in the first embodiment.

[0045] (Calculation Unit) The calculation unit 125 calculates feature amounts of changes in facial expressions of the second speaker in the extended utterance image data based on the learning data constructed by the data construction unit 122. An example of calculation by the calculation unit 125 will be described below with reference to Figures 8 to 10. Figures 8 to 10 are diagrams for explaining an example of calculation.

[0046] 8, the calculation unit 125 extracts a facial landmark of each speaker from the extended speech video data of each speaker included in the impression-labeled extended speech video database. This impression-labeled extended speech video database includes an impression-labeled speech video database of each speaker, and the calculation unit 125 extracts, for example, a facial landmark of each speaker from the impression-labeled extended speech video data of each speaker in the impression-labeled extended speech video database of each speaker.

[0047] 8, the calculation unit 125 calculates the facial landmark statistics of each speaker, which mainly contain information about the individuality of each speaker, based on the average value and variance of the coordinates of the facial landmark of each speaker, based on all data corresponding to each speaker. For example, the calculation unit 125 calculates the facial landmark statistics of each speaker based on data including various facial expressions of each speaker, and therefore, among the information included in the facial landmark sequence of each speaker related to the time series of the facial features of each speaker in the extended speech video data of each speaker, facial expression information is averaged, and it is possible to calculate the facial landmark of each speaker, which mainly contains information about the individuality of each speaker.

[0048] In the example shown in FIG. 9 , the calculation unit 125 extracts Landmark(t=1), Landmark(t=2), ..., Landmark(t=T-1), and Landmark(t=T) of the kth speaker from t=1 to t=T from the impression-labeled extended utterance video data (i, k) as facial features of the kth speaker. t is a frame number corresponding to each time in the impression-labeled extended utterance video data (i, k). T is the last frame number of the impression-labeled extended utterance video data (i, k). Next, in the example shown in FIG. 9 , the calculation unit 125 calculates the average value and variance of the coordinates of the landmark of the kth speaker's face based on the landmark feature of the frame number corresponding to each time, which is all data corresponding to each speaker that includes the landmark of the kth speaker's face. Next, in the example shown in FIG. 9 , the calculation unit 125 calculates the average value and variance of the landmark of the kth speaker's face as the landmark statistics of the kth speaker's face. For example, the calculation unit 125 calculates the average value and variance of the coordinates of the landmark of the face of the kth speaker as the landmark statistics of the face of the kth speaker. The calculation unit 125 calculates the landmark statistics of the face of each speaker, for example, in the same way as in the above example.

[0049] 10, the calculation unit 125 normalizes the facial landmark of each speaker based on the average value and variance of the facial landmark of each speaker. For example, the calculation unit 125 standard-normalizes the landmark statistics of each speaker so that the average value of the facial landmark of each speaker is 0 and the variance is 1, and calculates the normalized value as the facial expression change landmark of each speaker.

[0050] (Learning Unit) The learning unit 123X uses the learning data, impression labels, speech image data, and facial image data constructed by the data construction unit 122 to train a learning model that takes the impression labels, speech image data, and facial image data as input and outputs feature quantities of facial expression changes of the second speaker in the extended speech image data. In this way, by using the facial expression change landmark as an estimation target, the learning unit 123X can simplify the input / output relationship for the learning model, easily train the learning model, and improve conversion accuracy. Furthermore, the learning unit 123X can synthesize speech video data and facial image data to create facial expressions that match the individual with a small amount of training data.

[0051] For example, the learning unit 123X may train a learning model that outputs the difference between the facial feature of the second speaker and the average facial feature of the second speaker as the feature of the facial change of the second speaker. As an example, the learning unit 123X trains the learning model using the average value and variance of the facial landmark of each speaker that has been standard-normalized by the calculation unit 125 as the facial change landmark sequence of each speaker. In this way, the learning unit 123X can train the learning model so that the facial feature from which the individual information of the face has been deleted and only the facial change is expressed can be estimated as the feature of the facial change of the second speaker in the augmented speech image data.

[0052] An example of learning by the learning unit 123X will be described below with reference to Fig. 11. Fig. 11 is a diagram for explaining one example of learning. In the example shown in Fig. 11, the learning unit 123X trains a learning model in the same manner as the learning unit 123 in the first embodiment, except that the learning unit 123X trains a learning model that estimates an expression change landmark related to the facial expression feature amount of each speaker and an expression change landmark sequence related to the time series of the facial expression feature amount of each speaker.

[0053] 11 , the learning unit 123X trains a learning model that receives the impression label, the input speech video data, and the input face image data as input and outputs a speaker's facial expression change Landmark sequence in the output speech video data, using the facial expression change Landmark sequence, impression label, input speech video data, input face image data, and output speech video data calculated by the calculation unit 125. In this way, the learning unit 123X trains the learning model so that it can estimate each speaker's facial expression change Landmark sequence, which mainly contains facial expression information, out of the information contained in each speaker's facial Landmark sequence.

[0054] (Estimation unit 124X) The estimation unit 124X estimates feature quantities of changes in facial expressions of the second speaker in the extended speech image data based on the learning model learned by the learning unit 123X. An example of estimation by the estimation unit 124X will be described below with reference to FIG. 11 .

[0055] 11 , the estimation unit 124X performs estimation in the same manner as the data construction system 1 according to the first embodiment, except that the estimation targets are facial expression change landmarks and facial expression change landmark sequences. For example, the estimation unit 124X uses a learning model to estimate facial expression change landmarks and facial expression change landmark sequences based on impression labels, video features extracted from video data of the input utterance video data, and audio features extracted from audio data of the input utterance video data.

[0056] 11 , the estimation unit 124X extracts landmarks of the input speaker's face in the video data of the input speech video data and calculates landmark statistics of the input speaker's face to obtain the landmark statistics of the input speaker's face. Next, the estimation unit 124X performs decoding (reflecting individuality) based on, for example, the facial expression change landmark sequence and the landmark statistics of the input speaker's face in the input speaker image data to obtain a landmark sequence in which the facial expression change landmark sequence is added to the landmark statistics of the input speaker's face. Next, the estimation unit 124X performs rendering based on, for example, the landmark sequence and the input speaker image to obtain output speech video data that reflects the individuality of the speaker's face in the input speaker image (average landmark position) and facial expression changes (difference from the average landmark position).

[0057] (Example of processing flow) An example of processing flow executed by the data construction device 10X of the data construction system 1X according to the second embodiment will be described below with reference to Fig. 12. Fig. 12 is a flowchart showing an example of processing flow executed by the data construction system according to the second embodiment. Steps S21 to S23 are the same as steps S11 to S13 in the first embodiment.

[0058] In step S24, the calculation unit 125 calculates features of facial expression changes of the second speaker in the extended utterance image data based on the learning data constructed by the data construction unit 122. For example, the calculation unit 125 calculates the average value and variance of the coordinates of the facial landmarks of each speaker in the extended utterance video data with each impression label. Next, the calculation unit 125 calculates facial landmark statistics of each speaker and an facial expression change landmark sequence of each speaker based on, for example, the average value and variance of the coordinates of the facial landmarks of each speaker.

[0059] In step S25, the learning unit 123X trains a learning model that takes the impression label, speech image data, and facial image data as input and outputs feature quantities of facial expression changes of the second speaker in the extended speech image data, using the training data, impression label, speech image data, and facial image data constructed by the data construction unit 122. For example, the learning unit 123X trains a learning model that takes the impression label, input speech video data, and input facial image data as input and outputs a speaker's facial expression change Landmark sequence in the output speech video data, using the facial expression change Landmark sequence, impression label, input speech video data, input facial image data, and output speech video data calculated by the calculation unit 125.

[0060] In step S26, the estimation unit 124X estimates features of facial expression changes of the second speaker in the extended speech image data based on the learning model trained by the learning unit 123X. For example, the estimation unit 124X uses the learning model to estimate a facial expression change landmark of the input speaker based on the impression label, video features extracted from video data in the input speech video data, and audio features extracted from audio data in the input speech video data.

[0061] (3. Program) The data construction device 10 and the data construction device 10X can be implemented by installing a data construction program on a computer as package software or online software. For example, the computer can function as the data construction device 10 and the data construction device 10X by executing the data construction program.

[0062] Fig. 13 is a diagram showing an example of the configuration of a computer that executes a data construction program. In the example shown in Fig. 13, the computer 1000 has a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0063] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0064] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the data construction device 10 and the data construction device 10X are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, on the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configurations of the data construction device 10 and the data construction device 10X are stored on the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD.

[0065] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0066] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0067] (4. Modifications) The components of the illustrated units are conceptual functional units and do not necessarily have to be physically configured as illustrated. In other words, the specific form of distribution and integration of the devices is not limited to that shown in the drawings. All or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by the devices can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic. For example, the learning unit 123 and the estimation unit 124 shown in FIG. 2 can be realized as a learning device and an estimation device, respectively, independent of the data construction device 10.

[0068] Furthermore, among the above processes, all or part of the processes described as being performed automatically can be performed manually, and all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the processing procedures, specific names, various information, parameters, etc. shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0069] REFERENCE SIGNS LIST 1 Data construction system 10 Data construction device 11 Data acquisition unit 12 Control unit 13 Output unit 14 Storage unit 121 Data expansion unit 122 Data construction unit 123 Learning unit 124 Estimation unit

Claims

1. A data construction device comprising: a data acquisition unit that acquires speech image data relating to an utterance by a first speaker and facial image data relating to the face of a second speaker; a data expansion unit that generates extended speech image data relating to the utterance by the second speaker based on the speech image data and the facial image data acquired by the data acquisition unit; and a data construction unit that constructs a pair of the extended speech image data generated by the data expansion unit and an impression label relating to an impression of the utterance as learning data for training a learning model.

2. The data construction device described in claim 1, characterized in that the data acquisition unit acquires the speech image data from a minority speaker speech image database relating to speech by each of a minority of speakers less than a predetermined number, and acquires the facial image data from a multiple facial image database relating to facial image data of each of a multiple number of speakers equal to or greater than a predetermined number.

3. The data construction device described in claim 1, characterized in that the data acquisition unit acquires speech video data including the speech image data and the facial image data, the data expansion unit generates extended speech video data including the extended speech image data based on the speech video data and the facial image data acquired by the data acquisition unit, and the data construction unit constructs a pair of the extended speech video data and the impression label generated by the data expansion unit as the learning data.

4. The data construction device according to any one of claims 1 to 3, further comprising a learning unit that trains the learning model using the learning data, the speech image data, and the face image data constructed by the data construction unit, and that inputs the impression label, the speech image data, and the face image data, and outputs the facial features of the second speaker in the extended speech image data.

5. The data construction device described in claim 4, characterized in that the learning unit trains the learning model using the learning data, the impression label, the speech image data and the facial image data constructed by the data construction unit, and inputs the impression label, the speech image data and the facial image data, and outputs features of the changes in facial expressions of the second speaker in the extended speech image data.

6. The data construction device described in claim 5, characterized in that the learning unit trains the learning model which outputs the difference between the facial features of the second speaker and the average facial features of the second speaker as the features of the changes in facial expression of the second speaker.

7. A data construction method executed by a data construction device, comprising: a data acquisition step of acquiring speech image data relating to an utterance by a first speaker and facial image data relating to the face of a second speaker; a data expansion step of generating extended speech image data relating to the utterance by the second speaker based on the speech image data and the facial image data acquired by the data acquisition step; and a data construction step of constructing a pair of the extended speech image data generated by the data expansion step and an impression label relating to an impression of the utterance as learning data for training a learning model.

8. A data construction program that causes a computer to execute the following steps: a data acquisition step of acquiring speech image data relating to an utterance by a first speaker and facial image data relating to the face of a second speaker; a data expansion step of generating extended speech image data relating to the utterance by the second speaker based on the speech image data and the facial image data acquired by the data acquisition step; and a data construction step of constructing a pair of the extended speech image data generated by the data expansion step and an impression label relating to an impression of the utterance as learning data for training a learning model.