Estimation device, estimation method, estimation program, learning device, and learning method

Abstract

An estimation device (10) includes: an acquisition unit (13) that acquires an utterance image feature pertaining to a feature of an utterance image to be converted, to-be-processed content information pertaining to utterance content corresponding to the utterance image, and designated impression information indicating a desired impression to be given by the converted utterance image; and an estimation unit (18) that estimates an impression-converted utterance image feature converted according to the designated impression information and the to-be-processed content information on the basis of a pretrained machine learning model, with the utterance image feature, the to-be-processed content information, and the designated impression information used as inputs.

Description

Estimation device, estimation method, estimation program, learning device, and learning method

[0001] The present invention relates to an estimation device, an estimation method, an estimation program, a learning device, and a learning method.

[0002] Conventionally, there are known techniques for generating speech images based on image information, audio information, and emotional information. For example, there is known a technique for inputting facial expression features relating to facial expression features extracted from image information of a speaker's face with a neutral expression, such as a straight face, audio features relating to audio features extracted from audio information, and emotional information, and synthesizing a speech video including audio information and emotional information (see, for example, Non-Patent Document 1).

[0003] Sanjana Sinha, Sandika Biswas, Ravindra Yadav, Brojeshwar Bhowmick, “Emotion-Controllable Generalized Talking Face Generation”, arXiv:2205.01155v1 [cs.CV] 2 May 2022

[0004] However, conventional techniques have had the problem that the generated speech image may not match the content of what is being said. For example, conventional techniques may generate a speech image that gives a different impression than the desired impression because they do not take into account the content of the speech.

[0005] In order to solve the above-mentioned problems, the present invention includes an acquisition unit that acquires speech image features related to the features of a speech image to be converted, processing target content information related to the speech content corresponding to the speech image, and designated impression information that indicates the impression that is desired to be given by the converted speech image, and an estimation unit that uses the speech image features, the processing target content information, and the designated impression information as inputs and estimates, based on a pre-trained machine learning model, post-impression-converted speech image features that have been converted in accordance with the designated impression information and the processing target content information.

[0006] According to the present invention, it is possible to generate a speech image that gives an impression suited to the speech content.

[0007] FIG. 1 is a diagram illustrating an example of an outline of a learning process performed by an estimation system according to a first embodiment. FIG. 2 is a diagram illustrating an example of an outline of an estimation process performed by an estimation system according to the first embodiment. FIG. 3 is a diagram illustrating an example of a configuration of an estimation device according to the first embodiment. FIG. 4 is a flowchart illustrating an example of a flow of a learning process performed by an estimation device according to the first embodiment. FIG. 5 is a flowchart illustrating an example of a flow of an estimation process performed by an estimation system according to the first embodiment. FIG. 6 is a diagram illustrating an example of an outline of a learning process performed by an estimation system according to a second embodiment. FIG. 7 is a diagram illustrating an example of an outline of an estimation process performed by an estimation system according to the second embodiment. FIG. 8 is a diagram illustrating an example of a configuration of an estimation device according to the second embodiment. FIG. 9 is a diagram illustrating an example of extraction. FIG. 10 is a diagram illustrating an example of a configuration of a computer that executes an estimation program.

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments. In addition, in the description of the drawings, the same parts are given the same reference numerals, and duplicated explanations are omitted, and explanations of members having similar functions and similar processes are also omitted.

[0009] (First embodiment) (Example of overview) An estimation device 10 of an estimation system 1 according to a first embodiment acquires speech image features related to the features of a speech image to be converted, processing target content information related to the speech content corresponding to the speech image, and designated impression information indicating the impression a user wants to give by the converted speech image, and estimates post-impression-conversion speech image features based on a pre-trained machine learning model using the speech image features, processing target content information, and designated impression information as inputs. Then, the estimation device 10 uses the estimated post-impression-conversion speech image features to generate a converted speech image that takes into account the speech content and is an image that gives the impression a user wants to give.

[0010] In this way, the estimation device 10 executes an estimation process of impression-converted utterance image features based on a machine learning model, but may also execute a learning process to train the machine learning model used in the estimation process. Note that, here, "utterance image" refers to image data including a speaker. Furthermore, "utterance video" refers to video data that includes one or more utterance images and the corresponding utterance content (and, during learning, their impression information).

[0011] (Explanation of Terms) Here, each term used in the first embodiment will be explained. Speech image feature refers to information related to the feature of a speech image, for example, information related to the feature of a frame in a speech video. Designated impression information refers to information indicating the impression desired to be given by a speech image after conversion, for example, an impression label indicating the impression desired to be given by a speech image after conversion.

[0012] The processing content information refers to information related to the content of the utterance corresponding to the speech image, and may include, for example, polarity information and emotional information. Polarity information refers to information in which the content of the utterance is classified according to linguistic polarity, and may include, for example, one-dimensional information in which the content of the utterance is classified into one of two values, positive and negative, or one-dimensional information in which the content of the utterance is classified into one of three values, positive, neutral, and negative. Emotional information refers to information in which the content of the utterance is classified according to emotion, and may include, for example, one-dimensional information in which the content of the utterance is classified into one of four values, joy, anger, sadness, and happiness.

[0013] The machine learning model refers to a model that receives, for example, speech image features, processing target content information, and specified impression information as inputs, and outputs impression-converted speech image features. Examples of the machine learning model include a sequence transformation model such as a neural network model including LSTM (Long Short-Term Memory), a model based on Transformer, and any other model capable of sequence transformation.

[0014] The post-impression-conversion speech image feature refers to information on the feature converted in accordance with the designated impression information and the content information to be processed, for example, information in which the speech image feature has been converted to match the content information to be processed and the designated impression information.

[0015] (Example of Overview of Learning Process) Fig. 1 is a diagram for explaining an example of an overview of a learning process performed by an estimation system according to the first embodiment. In the example shown in Fig. 1, the estimation device 10 executes a learning process to train a machine learning model using a first speech video and a second speech video.

[0016] The first utterance video is, for example, a video that does not match the content of the utterance or a video that gives a different impression than the desired impression to the conversation partner. More specifically, the first utterance video is, for example, a video in which negative content is spoken with a strong smile or a video in which the speaker's facial expression does not match the content of the utterance.

[0017] The second utterance video is, for example, a video whose speech content is the same as that of the first utterance video, that is suitable for the speech content, and that is suitable for the impression that the user wants to give to the other person. More specifically, the second utterance video is, for example, a video in which the user speaks positive content that gives a likable impression with a smile, and to which an impression label of a likable impression has been assigned.

[0018] First, the time synchronization processing unit 12 executes time synchronization processing on, for example, the first utterance video and the second utterance video. As an example, the time synchronization processing unit 12 synchronizes the time of the second utterance video with the first utterance video as a reference by a general time synchronization processing used when matching between two series, such as DP (Dynamic Programming) matching.

[0019] Next, the estimation device 10 acquires training data for training a machine learning model from, for example, the first utterance video and the second utterance video. As an example, the estimation device 10 extracts first utterance image features related to the features of the first utterance image from the first utterance video by any existing method.

[0020] Furthermore, for example, the estimation device 10 extracts content information to be processed that includes polarity information by obtaining polarity information that has been manually assigned to the first speech video, or by reading (analyzing) whether linguistic information such as speech text or audio information contained in the first speech video is positive or negative.

[0021] The estimation device 10 also extracts second utterance image features corresponding to impression information and processing target content information from second image information of the second utterance video that has been time-synchronized. The estimation device 10 also acquires impression information by separating impression information such as impression labels from frames of the second utterance video using any existing method.

[0022] Next, the learning unit 17 of the estimation device 10 trains a machine learning model using the first utterance image feature amount, the processing target content information, and the impression information as input information and the second utterance image feature amount as correct answer information.

[0023] As an example, the learning unit 17 first generates an input vector by converting the first utterance image feature, the processing target content information including polarity information, and the impression information into numerical information, and then generates a correct vector by converting the second utterance image feature into numerical information.The learning unit 17 then updates the parameters of the machine learning model so that the correct vector is output when the input vector is input.

[0024] In other words, the estimation device 10 uses a set of the first utterance image feature, processing target content information, and impression information as input information, and learns the second utterance image feature as correct answer information, thereby causing the machine learning model to learn what second utterance image feature should be output when what input information (a set of the first utterance image feature, processing target content information, and impression information) is input. The machine learning model trained in this manner is used in the estimation process described below. Note that the machine learning model used in the estimation process is not limited to being trained by the estimation device 10, and for example, a model trained in advance by another device may be used.

[0025] The first utterance video and the second utterance video that are the basis of the learning data used in the learning process are not limited to the above examples, and may, for example, be any that satisfy the following conditions: - The speech content is the same. - The impressions are different. - The impression of the second utterance video corresponds to the desired impression (impression label).

[0026] In addition to the above-mentioned methods, the learning process may also be carried out in a manner such that, for example, when there is a first dataset consisting of a first speech video and a first impression of that video, and a second dataset consisting of a second speech video and a second impression of that video, a set of data with identical speech content is extracted from each dataset, and if the first impression and second impression are different, a machine learning model is trained to generate a second speech video using the first speech video and the second impression as input.

[0027] (Example of Overview of Estimation Process) Fig. 2 is a diagram for explaining an example of an overview of the estimation process performed by the estimation system according to the first embodiment. In the example shown in Fig. 2, the estimation device 10 of the estimation system 1 executes the estimation process of estimating impression-converted speech image features using a machine learning model.

[0028] The estimation device 10 acquires speech image features related to the features of the speech image to be converted, processing target content information related to the speech content corresponding to the speech image, and designated impression information indicating the impression desired to be given by the converted speech image.

[0029] Next, the estimation unit 18 of the estimation device 10 estimates, for example, speech image features, processing target content information, and designated impression information as inputs, and estimates impression-converted speech image features based on a pre-trained machine learning model. As an example, the estimation unit 18 first generates an input vector by converting the speech image features, processing target content information including polarity information, and designated impression information into numerical information. Next, the estimation unit 18 inputs, for example, the input vector to the machine learning model, and estimates output information such as an output vector output from the machine learning model as impression-converted speech image features.

[0030] Next, the generation unit 19 of the estimation device 10 generates an impression-converted speech video converted according to the designated impression information and the processing target content information, for example, based on the impression-converted speech image feature quantities. As an example, the generation unit 19 generates an impression-converted speech video in which the speaker's facial expression is converted according to the designated impression information and the processing target content information, using a video generation technology such as a rendering technology (see, for example, references). Reference: Aliaksandr Siarohin, Stephane Lathuiliere, Sergey Tulyakov, Elisa Ricci, Nicu Sebe, "First Order Motion Model for Image Animation", NeurIPS 2019

[0031] (Example of Effect) As described above, the estimation device 10 uses a set of speech image feature amounts, processing target content information, and impression information as input information, and estimates impression-converted speech image feature amounts based on a pre-trained machine learning model, thereby taking the speech content into consideration and generating a speech image that gives the impression the user wants to give. As a result, it is possible to generate a speech image that gives an impression that suits the image speech content. Furthermore, because the estimation device 10 acquires processing target content information that includes polarity information such as linguistic information, it is possible to generate a speech image that gives a desired impression without the user having to determine whether the speech text is positive or negative and input it into the machine learning model.

[0032] (Example of configuration of estimation device) Fig. 3 is a diagram showing an example of the configuration of the estimation device according to the first embodiment. In the example shown in Fig. 3, the estimation device 10 includes a control unit 11, an output unit 20, and a storage unit 21.

[0033] (Control Unit) The control unit 11 controls the entire estimation device 10. As an example, the control unit 11 is configured with one or more processors having an internal memory storing programs defining the flow of various processes and control information, and the processor executes each process using the programs and the internal memory. The control unit 11 is realized by, for example, electronic circuits such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), or integrated circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). In the example shown in FIG. 3 , the control unit 11 includes a time synchronization processing unit 12, an acquisition unit 13, a learning unit 17, an estimation unit 18, and a generation unit 19.

[0034] (Time synchronization processing unit) The time synchronization processing unit 12 executes time synchronization processing on the first utterance video and the second utterance video. For example, the time synchronization processing unit 12 synchronizes the time of the second utterance video with the first utterance video as a reference by a general time synchronization processing used when matching between two series such as DP matching.

[0035] As an example, the time synchronization processing unit 12 DP matches the second audio associated with the second speech video with the first audio associated with the first speech video, removes the excess part of the second audio compared to the first audio, and inserts information into the second audio that is similar to that of the first audio for the missing part, thereby aligning the lengths of the second audio and the first audio, and adjusting the lengths of the second speech video and the first speech video to the lengths of the DP-matched second audio and first audio.

[0036] As another example, the time synchronization processing unit 12 first performs processing in units of a predetermined frame, DP-matches the second utterance video with the first utterance video, removes the excess portion of the second utterance video relative to the first utterance video, and inserts the same information as the first utterance video into the second utterance video for the missing portion, thereby aligning the lengths of the second utterance video and the first utterance video. In this way, the time synchronization processing unit 12 adjusts the number of frames of the second utterance video to be the same as the number of frames of the first utterance video.

[0037] (Acquisition Unit) The acquisition unit 13 acquires speech image feature amounts relating to the feature amounts of the speech image to be converted, processing target content information relating to the speech content corresponding to the speech image, and designated impression information indicating the impression to be given by the speech image after conversion. In the example shown in Figure 3, the acquisition unit 13 has an image feature extraction unit 14, a language feature extraction unit 15, and an impression information acquisition unit 16.

[0038] For example, the image feature extraction unit 14 extracts speech image features from image information included in the speech video. The language feature extraction unit 15 extracts processing target content information including polarity information from language information included in the speech video. The impression information acquisition unit 16 acquires designated impression information such as a designated impression label indicating the impression desired to be given by the converted speech image.

[0039] As another example, the acquiring unit 13 acquires a set of processing target content information including pre-extracted speech image feature amounts and polarity information, and designated impression information such as a designated impression label.

[0040] For example, the acquiring unit 13 may acquire training data for training a machine learning model. That is, the acquiring unit 13 may acquire first utterance image feature amounts related to feature amounts of a first utterance image, processing target content information related to the utterance content corresponding to the first utterance image, impression information indicating the impression desired to be given by the second utterance image, and second utterance image feature amounts corresponding to the impression information and the processing target content information.

[0041] As an example, the image feature extraction unit 14 extracts first utterance image features from the first utterance video using any existing method. As an example, the language feature extraction unit 15 acquires polarity information manually assigned to the first utterance video. Alternatively, the language feature extraction unit 15 extracts processing target content information containing polarity information by analyzing whether linguistic information such as speech text and audio information contained in the first utterance video is positive or negative. Furthermore, the image feature extraction unit 14 extracts second utterance image features from, for example, second image information of the second utterance video that has been subjected to time synchronization processing. As an example, the impression information acquisition unit 16 acquires impression information by separating impression information such as impression labels from frames of the second utterance video using any existing method.

[0042] As another example, the acquisition unit 13 acquires a pair of pre-extracted first speech image features and processing target content information including polarity information, and acquires impression information such as pre-separated impression labels from the second speech video.

[0043] (Learning Unit) The learning unit 17 trains a machine learning model using the first utterance image feature amount, processing target content information, and impression information as input information and the second utterance image feature amount as correct answer information.

[0044] As an example, the learning unit 17 may perform supervised learning of a machine learning model using as input information a first utterance image feature amount included in a first utterance image, processing target content information related to the utterance content corresponding to the first utterance image, and impression information included in a second utterance image whose utterance content is the same as that of the first utterance image, and using as correct answer information a second utterance image feature amount included in the second utterance image. That is, the machine learning model may be a model that receives as input information a first utterance image feature amount related to the feature amount of the first utterance image, processing target content information related to the utterance content corresponding to the first utterance image, and impression information indicating the impression desired to be given by a second utterance image whose utterance content is the same as that of the first utterance image, and trains as correct answer information a second utterance image feature amount corresponding to the impression information and the processing target content information.

[0045] Because the speech content of the first speech image and the second speech image is the same, even if a machine learning model is trained using two different speech images, a speech image with an impression that suits the speech content can be generated based on the machine learning model.

[0046] (Estimation unit) The estimation unit 18 receives the speech image feature, the processing target content information, and the designated impression information as input, and estimates the impression-converted speech image feature that has been converted according to the designated impression information and the processing target content information based on a pre-trained machine learning model.

[0047] As an example, the estimation unit 18 generates an input vector from the speech image feature, processing target content information including polarity information, and specified impression information, inputs the input vector to a pre-trained machine learning model, and estimates output information such as an output vector output from the machine learning model as the impression-converted speech image feature. In this way, when polarity information is included in the processing target information, etc., the estimation unit 18 can generate a speech image with an impression appropriate for the speech content with less processing load and cost than when language feature is included in the processing target content information.

[0048] The generation unit 19 generates an impression-converted speech image based on the impression-converted speech image feature amount. For example, the generation unit 19 uses a moving image generation technique such as a rendering technique to generate an impression-converted speech image in which the facial expression of the speaker is converted according to the designated impression information and the processing target content information.

[0049] (Output Unit) The output unit 20 outputs various information. As an example, the output unit 20 outputs an impression-converted speech video. The output unit 20 is realized by, for example, a display unit such as a display.

[0050] (Storage Unit) The storage unit 21 stores various types of information. As an example, the storage unit 21 stores various types of information such as speech images, impression-converted speech images, speech image feature amounts, impression-converted speech image feature amounts, designated impression information, processing target content information, machine learning models, an OS (Operating System) and various programs executed by the estimation device 10, etc. The storage unit 21 is realized by, for example, a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an optical disk, or an information-rewritable semiconductor memory such as RAM (Random Access Memory), a flash memory, or NVSRAM (Non-Volatile Static Random Access Memory), etc.

[0051] 4 is a flowchart showing an example of the flow of a learning process by the estimation system according to the first embodiment. In step S1, the acquisition unit 13 acquires learning data for training a machine learning model. That is, the acquisition unit 13 acquires first utterance image feature amounts related to the feature amounts of the first utterance image, processing target content information related to the speech content corresponding to the first utterance image, impression information indicating the impression desired to be given by the second utterance image, and second utterance image feature amounts corresponding to the impression information and the processing target content information.

[0052] As an example, when it is determined that a learning process is to be executed, the acquisition unit 13 acquires the first speech image feature and processing target content information from the first speech video, and acquires impression information from the second speech video.

[0053] In step S2, the learning unit 17 uses the first utterance image feature amount, the processing target content information, and the impression information as input information, and trains a machine learning model using the second utterance image feature amount as correct answer information. For example, the learning unit 17 uses the first utterance image feature amount included in the first utterance image, the processing target content information related to the utterance content corresponding to the first utterance image, and the impression information included in a second utterance image whose utterance content is the same as that of the first utterance image as input information, and trains a machine learning model using the second utterance image feature amount included in the second utterance image as correct answer information.

[0054] 5 is a flowchart showing an example of the flow of the estimation process by the estimation system according to the first embodiment. In step S11, the acquisition unit 13 acquires speech image feature amounts related to the feature amounts of the speech image to be converted, processing target content information related to the speech content corresponding to the speech image, and designated impression information indicating the impression to be given by the converted speech image.

[0055] For example, when it is determined that the estimation process is to be performed, the image feature extraction unit 14 extracts speech image features from image information included in the speech video. The language feature extraction unit 15 extracts processing target content information including polarity information from language information included in the speech video. Furthermore, the impression information acquisition unit 16 acquires designated impression information such as a designated impression label.

[0056] In step S12, the estimation unit receives the speech image feature values, the processing target content information, and the designated impression information as input, and estimates impression-converted speech image feature values ​​that have been converted in accordance with the designated impression information and the processing target content information, based on a pre-trained machine learning model.

[0057] As an example, the estimation unit 18 generates an input vector from the speech image feature, processing target content information including polarity information, and specified impression information, inputs the input vector to a pre-trained machine learning model, and estimates output information such as an output vector output from the machine learning model as the speech image feature after impression conversion.

[0058] (Second embodiment) (Example of overview) In the above example, a case is described in which the content information to be processed may include polarity information, but the content information to be processed used in the estimation device 10X of the estimation system 1X according to this embodiment includes linguistic features in which linguistic features included in the utterance content are expressed by a vector of a predetermined dimension.

[0059] 6 is a diagram illustrating an example of an overview of a learning process performed by the estimation system according to the second embodiment. In the example shown in Fig. 6, the language feature extraction unit 15X extracts processing target content information including language features from language information included in the first speech video using a natural language processing technique such as a language model, for example, BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-trained Transformer)3.

[0060] 7 is a diagram for explaining an example of an outline of the estimation process by the estimation system according to the second embodiment. In the example shown in Fig. 7, the language feature extraction unit 15X extracts processing target content information including language features from language information included in the speech video, for example, by using a natural language processing technique similar to that used in the learning process.

[0061] (Example of Effect) By including a linguistic feature having a larger amount of information than polarity information in the content information to be processed, the estimation device 10X can generate an utterance image that gives an impression that is more appropriate for the utterance content than when polarity information is included in the content information to be processed. For example, the estimation device 10X can generate an utterance image with a facial expression that matches the linguistic information included in the utterance image.

[0062] (Example of configuration of estimation device) Fig. 8 is a diagram showing an example of the configuration of an estimation device according to the second embodiment. In the example shown in Fig. 8, an estimation device 10X has a control unit 11X instead of the control unit 11. The control unit 11X has an acquisition unit 13X instead of the acquisition unit 13, and further has a parameter learning unit 22. The acquisition unit 13X has a language feature extraction unit 15X instead of the language feature extraction unit 15.

[0063] (Language Feature Extraction Unit) The language feature extraction unit 15X extracts processing target content information that includes language features. For example, the language features may be features obtained by compressing high-dimensional language features extracted from utterance content by a language model using an intermediate layer of a neural network model that has been trained in advance to estimate a predetermined task. The predetermined task refers to, for example, a task for estimating information that should be focused on as linguistic information, and examples include polarity information such as positive / negative, and emotional information that correspond to the high-dimensional language features.

[0064] An example of extraction by the language feature extraction unit 15X when the language features are compressed as described above will be described below with reference to Fig. 9. Fig. 9 is a diagram for explaining one example of extraction. In the example shown in Fig. 9, before extraction by the language feature extraction unit 15X, the parameter learning unit 22 first trains a neural network model that estimates (outputs) a predetermined task using, as input, high-dimensional language features extracted by a language model from language information (utterance content) such as speech text included in a speech video.

[0065] As an example, the parameter learning unit 22 performs supervised learning of the model parameters of the neural network model using learning data that is a pair of linguistic information and polarity information that is correct answer information. In this case, the parameter learning unit 22 learns the model parameters of the neural network model so that, for example, a predetermined task that is output information from the neural network model approaches the polarity information that is correct answer information. In this way, the model parameters of the neural network model are trained from model parameters currently being trained to trained model parameters.

[0066] After training the model parameters of the neural network model, the language feature extraction unit 15X extracts low-dimensional language features of a predetermined number of dimensions or less that are output from a bottleneck, which is, for example, one intermediate layer of the pre-trained neural network model that has fewer nodes than at least the input layer.

[0067] In this way, the language feature extraction unit 15X extracts language features whose dimensionality has been reduced by a neural network model that estimates a predetermined task, such as polarity information, thereby enabling extraction of low-dimensional language features suitable for estimating the task from high-dimensional language features. For example, the language feature extraction unit 15X can extract useful language features that should be focused on as language information in a lower-dimensional state compared to when extracting language features extracted from language information by a language model. Similarly, the language feature extraction unit 15X can extract useful language features that should be focused on as language information in a lower-dimensional state compared to when processing layers after the bottleneck of the neural network model and extracting language features output from the output layer.

[0068] Furthermore, by extracting language features whose dimensionality has been compressed by the above-mentioned neural network model, the language feature extraction unit 15X can reduce the amount of training data, the amount of processing required for the training process and the estimation process, etc., compared to when high-dimensional language features are extracted from the output layer of a language model or a neural network model, thereby reducing costs.

[0069] (Modifications) The components of each illustrated unit are conceptual functional units and do not necessarily have to be physically configured as illustrated. In other words, the specific form of distribution and integration of the device is not limited to that illustrated, and all or part of it can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by the device can be realized by a CPU and a program executed by the CPU, or can be realized as hardware using wired logic.

[0070] In the example shown in Figure 3, the estimation device 10 is configured as a device having a learning unit 17 and an estimation unit 18, but it may also be configured as, for example, a learning device having the learning unit 17 and an estimation device having the estimation unit 18.

[0071] Furthermore, among the above processes, all or part of the processes described as being performed automatically can be performed manually, and all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the processing procedures, specific names, various information, parameters, etc. shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0072] (Program) The estimation device 10 and the estimation device 10X can be implemented by installing an estimation program on a computer as packaged software or online software. For example, a computer can function as the estimation device 10 and the estimation device 10X by executing the estimation program. An example of the configuration of a computer that executes the estimation program will be described below with reference to FIG. 10 .

[0073] Fig. 10 is a diagram showing an example of the configuration of a computer that executes an estimation program. In the example shown in Fig. 10, a computer 1000 has a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0074] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0075] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes executed by the estimation device 10 and the estimation device 10X are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to the functional configurations of the estimation device 10 and the estimation device 10X are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD.

[0076] Data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. The CPU 1020 then reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary and executes them.

[0077] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0078] (Supplementary Note) The following supplementary note is further disclosed regarding the above embodiment.

[0079] (Supplementary Item 1) An estimation device comprising: a memory; and at least one processor connected to the memory, wherein the processor acquires speech image features relating to features of a speech image to be converted, processing target content information relating to speech content corresponding to the speech image, and designated impression information indicating an impression to be given by the converted speech image, and estimates, using the speech image features, the processing target content information, and the designated impression information as inputs, post-impression converted speech image features converted in accordance with the designated impression information and the processing target content information based on a machine learning model that has been trained in advance.

[0080] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute the estimation process by the estimation device according to Supplementary Item 1.

[0081] (Supplementary Item 3) A learning device comprising: a memory; and at least one processor connected to the memory, wherein the processor acquires first speech image features relating to features of a first speech image, processing target content information relating to speech content corresponding to the first speech image, impression information indicating an impression desired to be given by a second speech image, and second speech image features corresponding to the impression information and the processing target content information; and trains a machine learning model using the first speech image features, the processing target content information, and the impression information as input information and the second speech image features as correct answer information.

[0082] (Supplementary Item 4) A non-transitory storage medium storing a program executable by a computer to execute the learning process by the learning device according to Supplementary Item 3.

[0083] REFERENCE SIGNS LIST 1 Estimation system 10 Estimation device 11 Control unit 12 Time synchronization processing unit 13 Acquisition unit 14 Image feature extraction unit 15 Language feature extraction unit 16 Impression information acquisition unit 17 Learning unit 18 Estimation unit 19 Generation unit 20 Output unit 21 Storage unit

Claims

1. An estimation device comprising: an acquisition unit that acquires speech image features related to features of a speech image to be converted, processing target content information related to speech content corresponding to the speech image, and designated impression information indicating an impression desired to be given by the converted speech image; and an estimation unit that uses the speech image features, the processing target content information, and the designated impression information as inputs, and estimates post-impression converted speech image features converted in accordance with the designated impression information and the processing target content information based on a machine learning model that has been trained in advance.

2. The estimation device according to claim 1, wherein the processing content information includes polarity information that classifies the speech content according to linguistic polarity.

3. The estimation device according to claim 1, characterized in that the processing target content information includes linguistic features in which linguistic features contained in the utterance content are expressed by vectors of a predetermined dimension.

4. The estimation device described in claim 3, characterized in that the language features are high-dimensional language features extracted from the speech content by a language model and compressed by an intermediate layer of a neural network model that has been trained in advance to estimate a specified task.

5. An estimation method executed by an estimation device, comprising: an acquisition step of acquiring speech image features related to features of a speech image to be converted, processing target content information related to speech content corresponding to the speech image, and designated impression information indicating an impression to be given by the converted speech image; and an estimation step of estimating, using the speech image features, the processing target content information, and the designated impression information as inputs, post-impression converted speech image features converted in accordance with the designated impression information and the processing target content information based on a machine learning model that has been trained in advance.

6. An estimation program for causing a computer to execute the following steps: an acquisition step for acquiring speech image features relating to features of a speech image to be converted, processing target content information relating to the speech content corresponding to the speech image, and designated impression information indicating the impression to be given by the converted speech image; and an estimation step for estimating, using the speech image features, the processing target content information, and the designated impression information as inputs, post-impression converted speech image features converted in accordance with the designated impression information and the processing target content information based on a previously trained machine learning model.

7. A learning device comprising: an acquisition unit that acquires a first speech image feature relating to a feature of a first speech image, processing target content information relating to speech content corresponding to the first speech image, impression information indicating an impression desired to be given by a second speech image, and a second speech image feature corresponding to the impression information and the processing target content information; and a learning unit that uses the first speech image feature, the processing target content information, and the impression information as input information and trains a machine learning model using the second speech image feature as correct answer information.

8. A learning method executed by a learning device, comprising: an acquisition step of acquiring first speech image features relating to features of a first speech image, processing target content information relating to speech content corresponding to the first speech image, impression information indicating an impression desired to be given by a second speech image, and second speech image features corresponding to the impression information and the processing target content information; and a learning step of training a machine learning model using the first speech image features, the processing target content information, and the impression information as input information and the second speech image features as correct answer information.

Images