Information processing device, information processing method, and program

Abstract

[Problem] To realize a technique capable of improving quality of life for a subject such as a patient who is impaired in vocalization. [Solution] An information processing device 1 includes a moving image data acquisition unit 11a, a preprocessing unit 11b, a feature amount acquisition unit 11c, a learning model generation unit 11d, an utterance content estimation unit 11e, and a substitute voice output unit 11f. The moving image data acquisition unit 11a acquires a moving image including lips of a subject. The preprocessing unit 11b extracts a lip portion in the moving image. The feature amount acquisition unit 11c acquires a feature amount of the extracted lip portion. The utterance content estimation unit 11e estimates an utterance content of the subject corresponding to the lip portion extracted by the preprocessing unit 11b, on the basis of a learning model that has been machine-learned in advance by associating a feature amount of a lip portion in a moving image for machine learning including the movement of lips during utterance with the utterance content uttered by the lip portion.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] Patients suffering from diseases such as laryngeal cancer may lose their voice due to total laryngectomy or pharyngopharyngoesophagectomy. For postoperative speech communication, electrolarynx, esophageal speech generation, and tracheoesophageal shunts are used as substitute speech devices, but they have problems such as intelligibility and simplicity. In response to this, lip-reading technology that estimates the content of speech from the movement of a speaker's lips has been attracting attention. For example, Patent Document 1 describes a speech recognition system that extracts the lip region and recognizes the shape of the mouth using a model that has previously trained a large amount of data using a neural network or the like.

[0003] Japanese Patent Application Laid-Open No. 2023-46127

[0004] However, conventional lip-reading technologies require a relatively long time to estimate spoken speech based on lip images. Furthermore, the accuracy of the speech estimation results based on lip images is not sufficient. Therefore, conventional lip-reading technologies cannot estimate spoken speech from lip images in real time and output substitute speech to enable smooth conversation, making it difficult to improve the quality of life of patients.

[0005] An object of the present invention is to realize a technology that can improve the quality of life of subjects such as patients with speech disorders.

[0006] In order to achieve the above object, an information processing device of one embodiment of the present invention is characterized by comprising: a video acquisition means for acquiring a video including the lips of a subject; an extraction processing means for extracting the lip portion from the video; a feature acquisition means for acquiring features of the extracted lip portion; and a speech content estimation means for estimating the speech content of the subject corresponding to the lip portion extracted by the extraction processing means, based on a learning model that has been machine-learned in advance by associating the features of the lip portion in a video for machine learning that includes lip movement during speech with the speech content uttered by the lip portion.

[0007] According to the present invention, it is possible to realize a technology that can improve the quality of life of subjects such as patients with speech disorders.

[0008] 1 is a schematic diagram showing an example of a usage state of an information processing device 1 according to an embodiment of the present invention. FIG. 2 is a schematic diagram (front view) showing an example of the external configuration of the information processing device 1. FIG. 3 is a block diagram showing the hardware configuration of the information processing device 1. FIG. 4 is a schematic diagram showing landmarks set in various parts of the face. FIG. 5 is a schematic diagram showing an example of a correspondence between landmarks set in the lip area and an image of the lips. FIG. 6 is a flowchart explaining the flow of a learning model generation process executed by the information processing device 1. FIG. 7 is a flowchart explaining the flow of a substitute voice output process executed by the information processing device 1. FIG. 8 is a diagram showing experimental results indicating the estimation accuracy of spoken voice using a conventional method. FIG. 9 is a diagram showing experimental results indicating the estimation accuracy of spoken voice according to the present invention. FIG. 10 is a block diagram showing the hardware configuration of the information processing device 1 in Modification 4. FIG. 11 is a flowchart explaining the flow of a learning model generation process executed by the information processing device 1 in Modification 4. FIG. 12 is a flowchart explaining the flow of a substitute voice output process executed by the information processing device 1 in Modification 4.

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Basic Principle of the Present Invention] First, the basic principle of the present invention will be described. The present invention acquires a sample video including lip movements during speech (e.g., a video including the lips of a subject whose speech is to be analyzed while speaking), and estimates the subject's speech content from the lip movements using lip reading technology that applies image analysis technology. Specifically, a sample video including lip movements during speech (e.g., a video including the lips of the subject while speaking) is acquired, and feature values ​​for each viseme (the shape of the lips when pronouncing a phoneme) are calculated using image analysis technology. Furthermore, machine learning is performed using AI (artificial intelligence) to associate the extracted feature values ​​with the speech content (here, text data representing a language). Then, feature values ​​for each viseme are sequentially calculated from a video including the subject's lips moving when intending to speak, and the speech content the subject intends to utter is estimated using a learning model generated by machine learning. At this time, the dimensionality of the features of the visemes is sequentially reduced to speed up the process of estimating the speech content. Furthermore, the speech of the estimated speech content is synthesized, and substitute speech is sequentially output by a speech output device. In this invention, in the machine learning and speech content estimation process, a predetermined template representing lip shape is prepared in advance, and by using this template, the lip area in the video is extracted with high speed and accuracy, and lip movement including transitions between single sounds is recognized. This makes it possible to output substitute speech that matches the intention of the speech more quickly and accurately based on the lip shape during speech. Therefore, it is possible to provide a technology that can improve the quality of life of subjects such as patients with speech disorders.

[0010] [Configuration] Fig. 1A is a schematic diagram showing an example of a usage state of an information processing device 1 according to an embodiment of the present invention, and Fig. 1B is a schematic diagram (front view) showing an example of the external configuration of the information processing device 1. Fig. 2 is a block diagram showing the hardware configuration of the information processing device 1. The information processing device 1 is configured by an information processing device such as a smartphone or tablet terminal, and is placed in a position where it can capture an image of the subject's face using a wearing device. Note that the information processing device 1 can also be configured by a small PC (Personal Computer) or digital camera equipped with an imaging function.

[0011] As shown in FIGS. 1A, 1B, and 2, the information processing device 1 includes a processor 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input unit 15, an output unit 16, a storage unit 17, a communication unit 18, a drive 19, and an imaging unit 20.

[0012] The processor 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 17 to the RAM 13. The RAM 13 also stores data and the like necessary for the processor 11 to execute various processes.

[0013] The processor 11, the ROM 12, and the RAM 13 are connected to one another via a bus 14. To the bus 14, an input unit 15, an output unit 16, a storage unit 17, a communication unit 18, a drive 19, and an imaging unit 20 are connected.

[0014] The input unit 15 is composed of various buttons and the like, and inputs various information in response to instruction operations. The output unit 16 is composed of a display, a speaker, and the like, and outputs images and sounds. The storage unit 17 is composed of a non-volatile storage device such as a hard disk or a flash memory, and stores various programs and the like for controlling the information processing device 1. The storage unit 17 also stores various data used by the information processing device 1 (e.g., learning model data, voice data for synthesis, etc.) and various data generated by processing of the information processing device 1 (e.g., data on estimated utterance content, etc.). The communication unit 18 controls communication with other devices via a network.

[0015] Removable media 31, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately attached to the drive 19. Programs read from the removable media 31 by the drive 19 are installed in the storage unit 17 as needed. The imaging unit 20 is configured as an imaging device equipped with a lens, an imaging element, etc., and captures digital images of a subject. The input unit 15 may be configured as a touch sensor, and may be placed over the display of the output unit 16 to form a touch panel.

[0016] 2, various programs are executed, causing a video data acquisition unit 11a, a preprocessing unit 11b, a feature acquisition unit 11c, a learning model generation unit 11d, an utterance content estimation unit 11e, and a substitute voice output unit 11f to function in the processor 11 of the information processing device 1. In addition, the memory unit 17 includes a learning model memory unit 17a, a synthesis voice data memory unit 17b, and an estimation result memory unit 17c.

[0017] The learning model storage unit 17a stores a learning model that has been machine-learned by associating features for each viseme in lip movements during speech with the content of the speech (here, text data representing a language). In this embodiment, the learning model storage unit 17a stores a learning model that has been machine-learned by associating unique features for each viseme (described later) calculated from a sample video including the lips of a subject whose speech is to be analyzed with the content of the actual speech (here, text data representing a language). However, if it is not possible to prepare a sample video including the lips of a subject whose speech is to be analyzed while speaking (for example, in the case of a patient whose vocal cords have been removed and who is unable to speak), a versatile learning model that has been generated using sample videos including lip movements during speech of multiple subjects who are able to speak may be stored in the learning model storage unit 17a.

[0018] The synthesis voice data storage unit 17b stores synthesis voice data used to synthesize language representing the content of an utterance as voice. In this embodiment, data on the voice uttered by the subject whose speech is to be analyzed is stored as synthesis voice data in the synthesis voice data storage unit 17b. However, if data on the voice uttered by the subject whose speech is to be analyzed cannot be prepared (for example, in the case of a patient whose vocal cords have been removed and who is unable to speak), data on arbitrarily generated voice may be stored as synthesis voice data in the synthesis voice data storage unit 17b.

[0019] The estimation result memory unit 17c stores data on the speech content (here, text data representing language) estimated from a video including the lips of the subject whose speech is being analyzed as they move to intentionally speak.

[0020] The video data acquisition unit 11a acquires sample video data including lip movements during speech, which is used in the machine learning processing, via the imaging unit 20, the communication unit 18, etc. The sample video acquired at this time is sample video data including the lips of the subject whose speech is to be analyzed while speaking, as described above. However, if it is not possible to prepare a sample video including the lips of the subject whose speech is to be analyzed while speaking (for example, in the case of a patient whose vocal cords have been removed and who is unable to speak), it is also possible to acquire sample videos including lip movements during speech of multiple subjects who are able to speak. Furthermore, the video data acquisition unit 11a acquires video data including the lips of the subject when they move with the intention of speaking, via the imaging unit 20.

[0021] The preprocessing unit 11b performs preprocessing on the video data acquired by the video data acquisition unit 11a to calculate feature amounts. In this embodiment, the preprocessing unit 11b recognizes facial areas included in the video data and sets landmarks representing each part of the face.

[0022] FIG. 3 is a schematic diagram showing landmarks set in each part of the face. FIG. 4 is a schematic diagram showing an example of the correspondence between landmarks set in the lip area and an image of the lips. As shown in FIGS. 3 and 4 , a unique landmark is set in each part of the face, and coordinates are assigned to each landmark. The facial area in a video can be tracked while identifying each part using the landmarks. In this embodiment, the landmarks set in the lip area are used as a reference to specify the search range for the lips in the face image.

[0023] The preprocessing unit 11b also extracts the lip area from the face region using a prepared template representing the shape of the lips. At this time, the preprocessing unit 11b performs template-based extraction processing on a lip search range identified by landmarks set in the lip area. In this embodiment, the preprocessing unit 11b extracts the lip area from the video using a lip template image based on brightness values. The preprocessing unit 11b can also extract the lip area from the face region by supplementally using the R, G, and B components of the image. For example, the lip color range can be defined by the R, G, and B components, and a region obtained by template matching of brightness values ​​can be further extracted as the lip area by matching the region that matches the lip color range. Furthermore, to extract the lip area from the face region, instead of using a template representing the shape of the lips, a cascade classifier or the like can be used to extract objects.

[0024] The feature acquisition unit 11c calculates lip shape features by referencing the image of the lip portion extracted by the preprocessing unit 11b. The lip shape features can be calculated, for example, as the difference in the position of each part of the lip when compared with an average lip image (a reference lip shape image). Furthermore, the feature acquisition unit 11c calculates unique features by reducing the dimension of the calculated features using principal component analysis (PCA). The unique features correspond to the difference (relative value) between the features of a viseme of a target sound and the features of visemes other than the target sound. Once the unique features are determined, it is possible to identify which viseme produces which sound. Note that the dimensionality of the features may be reduced by methods other than principal component analysis (e.g., Hough transform, inner product, etc.) as long as they can acquire unique features. By replacing the features calculated from lip shape with unique features, it is possible to reduce the amount of calculation required when performing machine learning from sample videos and when estimating speech content from videos in which a subject is attempting to speak.

[0025] The learning model generation unit 11d performs machine learning to associate features (here, inherent features with reduced dimensions) for each viseme in the sample video acquired by the feature acquisition unit 11c with the speech content (text data representing phonemes) corresponding to the viseme. For example, the learning model generation unit 11d can perform machine learning using a CNN (Convolutional Neural Network) to associate features (here, inherent features) for each viseme with the speech content (text data representing phonemes) corresponding to the viseme. However, machine learning may be performed using a method other than CNN. The learning model generation unit 11d also stores the learning model generated by machine learning in the learning model storage unit 17a.

[0026] When the video data acquisition unit 11a acquires video data including the subject's lips moving with the intention of speaking, the utterance content estimation unit 11e uses as input the features (here, unique features) calculated by the feature acquisition unit 11c for the video and applies the learning model stored in the learning model storage unit 17a to estimate the utterance content (text data representing the language). The utterance content estimation unit 11e stores data representing the estimated utterance content in the estimation result storage unit 17c. The utterance content (text data representing the language) estimated by the utterance content estimation unit 11e may be displayed on the display of the output unit 16 so that the subject user can confirm the estimation result.

[0027] The substitute voice output unit 11f uses the synthesis voice data stored in the synthesis voice data storage unit 17b to output substitute voice representing the speech content (text data representing the language) estimated by the speech content estimation unit 11e.

[0028] [Operation] Next, a description will be given of the operation of the information processing device 1. [Learning model generation process] Fig. 5 is a flowchart illustrating the flow of the learning model generation process executed by the information processing device 1. The learning model generation process is started in response to an instruction to execute the learning model generation process being input via the input unit 15.

[0029] When the learning model generation process is started, in step S1, the video data acquisition unit 11a acquires sample video data including lip movement during speech to be used in the machine learning process via the imaging unit 20 or the communication unit 18. In step S2, the preprocessing unit 11b performs preprocessing on the video data acquired by the video data acquisition unit 11a to calculate feature amounts. In the preprocessing, the facial area included in the video data is recognized, landmarks representing each part of the face are set, and the lip area in the video is extracted using a template, with the lip search range specified by the landmarks set in the lip area as the target.

[0030] In step S3, the feature acquisition unit 11c calculates lip shape features by referencing the image of the lip portion extracted by the preprocessing unit 11b. Furthermore, the feature acquisition unit 11c calculates unique features by reducing the dimensions of the calculated features using principal component analysis (PCA). In step S4, the learning model generation unit 11d performs machine learning to associate the features of each viseme in the sample video acquired by the feature acquisition unit 11c (here, the unique features with reduced dimensions) with the speech content corresponding to that viseme (text data representing phonemes).

[0031] In step S5, the learning model generation unit 11d stores the learning model generated (or updated) by machine learning in the learning model storage unit 17a. After step S5, the learning model generation process is repeated until learning of the video data to be learned is completed.

[0032] 6 is a flowchart illustrating the flow of the substitute voice output process executed by the information processing device 1. The substitute voice output process is started in response to an instruction to execute the substitute voice output process being input via the input unit 15.

[0033] When the substitute voice output process starts, in step S11, video data including the lips of a subject moving with the intention of speaking is acquired via the imaging unit 20. In step S12, the preprocessing unit 11b performs preprocessing for calculating features on the video data acquired by the video data acquisition unit 11a. In the preprocessing, the facial area included in the video data is recognized, landmarks representing each part of the face are set, and the lip area in the video is extracted using a template within a lip search range identified by the landmarks set in the lip area.

[0034] In step S13, the feature acquisition unit 11c calculates lip shape features by referencing the image of the lip portion extracted by the preprocessing unit 11b. Furthermore, the feature acquisition unit 11c calculates characteristic features by reducing the dimensions of the calculated characteristics using principal component analysis (PCA). In step S14, the utterance content estimation unit 11e uses the characteristic features as input and applies a learning model stored in the learning model storage unit 17a to estimate the utterance content (text data representing the language). In step S15, the substitute voice output unit 11f outputs substitute voice representing the utterance content (text data representing the language) estimated by the utterance content estimation unit 11e using the synthesis voice data stored in the synthesis voice data storage unit 17b. After step S15, the substitute voice output process is repeated until an instruction to end the substitute voice output process is received.

[0035] As described above, the information processing device 1 to which the present invention is applied extracts lip areas from sample videos containing lip movements during speech using a template based on brightness values ​​and calculates lip shape features. Furthermore, the information processing device 1 calculates unique features by reducing the dimension of the lip shape features using principal component analysis, and performs machine learning to associate the unique features of each viseme of a single sound with the speech content (text data representing phonemes) corresponding to that viseme. The information processing device 1 then extracts lip areas from videos containing lips when a subject moves their lips to speak using a template based on brightness values ​​and calculates lip shape features. Furthermore, the information processing device 1 calculates unique features by reducing the dimension of the lip shape features using principal component analysis, and uses the calculated unique features as input to apply a learning model generated by machine learning to estimate the speech content (text data representing language) of the subject corresponding to the lip shape. When extracting the lip areas, an extraction process using a template is performed within a search range identified by landmarks set in the lip area. This process enables the information processing device 1 to estimate the speech content of the subject more quickly and accurately based on the shape of the lips during speech, and to output substitute speech that matches the intention of the speech. Therefore, it is possible to provide technology that can improve the quality of life of subjects such as patients with speech disorders.

[0036] [Verification of Effect] FIG. 7 shows experimental results showing the speech estimation accuracy of the conventional method. FIG. 8 shows experimental results showing the speech estimation accuracy of the present invention. The experimental results shown in FIGS. 7 and 8 represent the estimation accuracy when similar experiments were conducted on two common subjects using an existing method (conventional method) for estimating speech content (speech) from images of the lip area and the method of the present invention. The conventional method used here is the method described in the literature "Shinsuke Okita, Yasue Mitsukura, Nozomu Hamada, 'Augmented classification of Japanese visemes and hierarchical weighted discrimination for visual speech recognition,' IEEE Conference on Systems, Process & Control (ICSPC), 2013." As shown in FIG. 7 , the conventional method achieved an estimation accuracy of 63.89% for subject A and 51.16% for subject B.

[0037] In contrast, as shown in Figure 8, with the method of the present invention, after 10 trials, the estimation accuracy for subject A was 76.00% and the estimation accuracy for subject B was 61.50%. Furthermore, with the method of the present invention, after 20 trials, the estimation accuracy for subject A was 90.00% and the estimation accuracy for subject B was 69.47%. As can be seen from the estimation accuracy shown in Figures 7 and 8, the method of the present invention has a higher estimation accuracy for estimating speech content (text data representing language) from images of the lip region than the conventional method, and by increasing the number of trials, the estimation accuracy improves as the characteristics of the subject are learned. In addition to the above verification, when a learning model for a specific healthy subject was used to estimate the speech content of other healthy subjects, the estimation accuracy was an average of 80% or more. This indicates that the speech content estimation method of the present invention can achieve high accuracy even when subjects are generalized to a specific learning model. Furthermore, even if the estimation accuracy falls below 80%, it is possible to improve the estimation accuracy by updating the learning model through trials using videos of the subject himself or herself, or by adjusting parameters.

[0038] [Variation 1] In the above embodiment, an example was described in which a learning model for estimating speech content for each viseme of a single phoneme was generated. However, it is also possible to calculate lip shape features when a combination of two or more single phones is spoken consecutively and generate a learning model for estimating the speech content. This allows the learning model to be generated by including the combination of the lip shape of a consonant and the lip shape of a vowel (i.e., the lip shape when pronouncing a consonant) and the transition from the lip shape of a vowel to the lip shape of a consonant or vowel (i.e., the lip shape when pronouncing the next single phone from a single phone that converges as a vowel). This makes it possible to more accurately estimate speech content (text data representing a language) from the lip shape.

[0039] [Variation 2] In the above-described embodiment, in addition to machine learning the speech content for each viseme of a single sound, machine learning may be performed by associating the feature of the lip shape when a word is spoken with text data (speech content) representing the spoken word, and a learning model may be generated that includes these machine learning results. This makes it easier to estimate the speech content from the lip shape when a specific word is spoken, and enables increased accuracy in estimating the speech content.

[0040] [Variation 3] In the above-described embodiment, when estimating speech content (text data representing language) from video data including the lips of a subject moving to speak, the estimation method may be adjusted according to the subject's attributes. For example, speech content can be estimated by taking into account differences between a subject who previously had normal speech but now has difficulty speaking due to illness or other factors, and a subject who is congenitally unable to speak normally. Note that various classifications can be used as subject attributes, such as the degree of disability, the history of the disability, age, language used, and race. In this case, depending on the subject's attributes, weighting can be applied to the feature elements representing lip shape, a template representing lip shape can be switched, or the area of ​​the image extracted as the lip area can be expanded or reduced. This makes it possible to more accurately estimate speech content (text data representing language) from lip shape tailored to the subject, thereby improving convenience for the user.

[0041] [Variation 4] In the above-described embodiment, the process of identifying words from lip movements can be modified in various ways to improve functionality. FIG. 9 is a block diagram showing the hardware configuration of an information processing device 1 in this variation. The hardware configuration shown in FIG. 9 is substantially the same as that of the information processing system 1 in the first embodiment shown in FIG. 2. However, by executing various programs, the processor 11 of the information processing device 1 shown in FIG. 9 functions as follows: a video data acquisition unit 11a, a preprocessing unit 11b, a data expansion unit 11g, a feature acquisition unit 11c, a learning model generation unit 11d, an utterance content estimation unit 11e, and a substitute voice output unit 11f. The memory unit 17 also includes a learning model storage unit 17a, a speech data for synthesis storage unit 17b, and an estimation result storage unit 17c. The configurations of these units, other than the data expansion unit 11g, the feature acquisition unit 11c, the learning model generation unit 11d, and the utterance content estimation unit 11e, are the same as those of the information processing system 1 in the first embodiment. Therefore, the following mainly describes the parts that are different from the first embodiment, namely, the data expansion unit 11g, the feature acquisition unit 11c, the learning model generation unit 11d, and the utterance content estimation unit 11e.

[0042] The data expansion unit 11g performs data processing on the image of the lip portion extracted by the preprocessing unit 11b to reduce the effects of the shooting environment. For example, the data expansion unit 11g corrects the image of the lip portion extracted by the preprocessing unit 11b to a standard shape (e.g., the shape of the lips when viewed from the front) using affine transformation. Furthermore, for example, the data expansion unit 11g performs image processing on the image of the lip portion after the affine transformation. For example, the data expansion unit 11g performs image processing on the image of the lip portion after the affine transformation to reduce changes in the image due to differences in the shooting environment, such as adjusting color characteristics, brightness, contrast, saturation, and hue, posterizing the image, flattening the brightness histogram, and applying a Gaussian filter.

[0043] The feature acquisition unit 11c acquires lip shape features by referencing the image of the lip portion processed by the data expansion unit 11g. In this modification, the feature acquisition unit 11c acquires lip shape features from the image of the lip portion using a machine learning model (such as a CNN) that has previously performed machine learning on the image of the lip portion and the feature of the lip shape. For example, the feature acquisition unit 11c uses ResNet (Residual Networks) 18 to acquire lip shape features from the image of the lip portion processed by the data expansion unit 11g. As in the first embodiment, the lip shape feature can be, for example, the difference in the position of each part of the lip compared to an average lip image (a reference lip shape image). In this modification, the machine learning model used in the feature acquisition unit 11c (here, ResNet 18) generates learning results (machine learning model) for a larger number of words by transfer learning from learning results for a smaller number of words. This makes it possible to generate machine learning models more efficiently, and also to generate highly versatile machine learning models that can handle a greater number of words.

[0044] The learning model generation unit 11d receives as input the features for each viseme in the sample video acquired by the feature acquisition unit 11c, and generates a Transformer model that outputs a highly probable utterance corresponding to the viseme (text data representing phonemes). As an example, the output estimated by the Transformer model can be word-by-word. The Transformer model can obtain an output (estimated utterance) that reflects not only the features for each viseme alone, but also the context formed by other data (features for each viseme) included in the sequence.

[0045] When the video data acquisition unit 11a acquires video data including the subject's lips moving with the intention of speaking, the utterance content estimation unit 11e uses the features calculated by the feature acquisition unit 11c for this video as input and applies a learning model (Transformer model) stored in the learning model storage unit 17a to estimate the utterance content (text data representing language). The utterance content estimation unit 11e stores data representing the estimated utterance content in the estimation result storage unit 17c. The utterance content (text data representing language) estimated by the utterance content estimation unit 11e may be displayed on the display of the output unit 16 so that the subject user can confirm the estimation result.

[0046] FIG. 10 is a flowchart illustrating the flow of the learning model generation process executed by the information processing device 1 of this modified example. The learning model generation process shown in FIG. 10 differs from the learning model generation process of the first embodiment shown in FIG. 5 in the processes of steps S21, S22, and S23. Note that steps S1, S2, and S5 are the same as the learning model generation process of the first embodiment shown in FIG. 5. Below, steps S21, S22, and S23, which are different from the first embodiment, will be mainly described. In step S21, the data expansion unit 11g performs data processing on the image of the lip portion extracted by the preprocessing unit 11b to reduce the influence of the shooting environment. For example, the data expansion unit 11g corrects the image of the lip portion to a standard shape (e.g., the shape of the lips when viewed from the front) using affine transformation, or performs image processing on the image of the lip portion after affine transformation to reduce image changes due to differences in the shooting environment. In step S22, the feature acquisition unit 11c acquires lip shape features by referencing the image of the lip portion processed by the data expansion unit 11g. For example, the feature acquisition unit 11c acquires lip shape features by inputting the image of the lip portion processed by the data expansion unit 11g using ResNet (Residual Networks) 18. In step S23, the learning model generation unit 11d receives input of features for each viseme in the sample video acquired by the feature acquisition unit 11c and generates a Transformer model that outputs a highly likely utterance content (text data representing phonemes) corresponding to the viseme.

[0047] FIG. 11 is a flowchart illustrating the flow of the substitute voice output process executed by the information processing device 1 of this modified example. The substitute voice output process shown in FIG. 11 differs from the substitute voice output process of the first embodiment shown in FIG. 6 in the processes of steps S33, S34, and S35. Note that the processes of steps S11, S12, and S15 are the same as those of the substitute voice output process of the first embodiment shown in FIG. 6. Below, steps S33, S34, and S35, which are different from the first embodiment, will be mainly described. In step S33, the data expansion unit 11g performs data processing on the image of the lip portion extracted by the preprocessing unit 11b to reduce the influence of the shooting environment. For example, the data expansion unit 11g corrects the image of the lip portion to a standard shape (e.g., the shape of the lips when viewed from the front) using affine transformation, or performs image processing on the image of the lip portion after affine transformation to reduce image changes due to differences in the shooting environment. In step S34, the feature acquisition unit 11c acquires lip shape feature quantities by referring to the image of the lip portion processed by the data expansion unit 11g. For example, the feature acquisition unit 11c acquires lip shape feature quantities using ResNet (Residual Networks) 18, with the image of the lip portion processed by the data expansion unit 11g as input. In step S35, when the video data acquisition unit 11a acquires video data including the subject's lips moving when intending to speak, the utterance content estimation unit 11e uses the feature quantities calculated by the feature acquisition unit 11c for the video as input and applies the Transformer model stored in the learning model storage unit 17a to estimate the utterance content (text data representing language).

[0048] FIG. 12 is a diagram showing the estimation accuracy of speech when the information processing system 1 of this modified example is used. FIG. 12 shows the relationship between the number of words recognizable by the machine learning model and the estimation loss value (Test Loss) and accuracy rate (Test Accuracy). The machine learning model (ResNet18) used to acquire the features here is one in which the number of recognizable words is expanded to 140 [words] through transfer learning. Furthermore, a Transformer model is used to estimate the speech content (text data representing phonemes). Verification results show that when the information processing system 1 of this modified example is used, the estimation accuracy of speech improves to 86.82 [%].

[0049] As described above, the information processing device 1 according to this embodiment includes a video data acquisition unit 11a, a preprocessing unit 11b, a feature acquisition unit 11c, a learning model generation unit 11d, an utterance content estimation unit 11e, and a substitute voice output unit 11f. The video data acquisition unit 11a acquires a video including the subject's lips. The preprocessing unit 11b extracts the lip portion from the video. The feature acquisition unit 11c acquires the feature of the extracted lip portion. The utterance content estimation unit 11e estimates the subject's utterance corresponding to the lip portion extracted by the preprocessing unit 11b based on a learning model previously trained by machine learning, which associates the feature of the lip portion in a machine learning video including lip movement during speech with the utterance content uttered by the lip portion. This enables the subject's utterance content to be estimated more quickly and accurately based on the shape of the lips during speech. Therefore, a technology that can improve the quality of life of subjects, such as patients with speech disorders, can be provided.

[0050] The substitute speech output unit 11f outputs the speech content of the target person estimated by the speech content estimation unit 11e as a voice, thereby enabling the output of a substitute speech that matches the intention of the target person's speech more quickly and accurately based on the shape of the lips during the speech.

[0051] The video data acquisition unit 11a acquires video for machine learning, including lip movements during speech. The preprocessing unit 11b extracts the lip area from the video for machine learning. The learning model generation unit 11d acquires a learning model through machine learning by associating the feature values ​​of the lip area in the video for machine learning with the speech content uttered by the lip area in the video for machine learning. This makes it possible to estimate the speech content using AI from the video of the lip area, thereby realizing a technology for easily estimating the speech content.

[0052] The preprocessing unit 11b extracts the lip area from the video acquired by the video data acquisition unit 11a using a template representing the shape of the lips, thereby enabling faster extraction of the lip area from a video that includes the lip area.

[0053] The preprocessing unit 11b sets landmarks in the face area included in the video acquired by the video data acquisition unit 11a, and specifies the range for searching for the lip area using a template based on the landmarks set in the lip area. This makes it possible to appropriately limit the range for searching for the lip area, thereby enabling the lip area to be extracted more quickly.

[0054] The feature acquisition unit 11c acquires unique features represented by the difference between the features of the lip part uttering a predetermined single sound and the features of the lip part uttering sounds other than the single sound. The learning model generation unit 11d acquires a learning model in which machine learning has been performed to associate the unique features of the lip part for each single sound with the utterance content uttered by the lip part. This makes it possible to use a learning model that can estimate the utterance content using unique features represented by a smaller amount of information, thereby enabling faster estimation of the utterance content.

[0055] The data expansion unit 11g performs data processing on the image of the lip area extracted by the preprocessing unit 11b to reduce the influence of the shooting environment. The feature acquisition unit 11c acquires feature amounts of the lip area based on the image of the lip area processed by the data expansion unit 11g and a machine learning model that has previously machine-learned the image of the lip area and the feature amounts of the lip area. This makes it possible to use an image of the lip area in which variation due to the shooting environment is reduced, and to estimate the speech content of the subject with higher accuracy.

[0056] The present invention can be modified, improved, and the like as appropriate within the scope of the effects of the present invention, and is not limited to the above-described embodiment. For example, in the above-described embodiment, part of the processing performed by the information processing device 1 may be performed by another device (e.g., a server device) that can communicate via a network, and the processing results may be transmitted to the information processing device 1. As an example, the server device may perform the processing by the information processing device 1 to generate a learning model through machine learning, and the information processing device 1 may receive and use the generated learning model.

[0057] Furthermore, the various methods used in the above-described embodiments are merely examples, and other methods can be used as long as they can achieve the same purpose. Furthermore, the present invention can be implemented by appropriately combining the examples described in the above-described embodiments and modifications.

[0058] The above-described series of processes can be executed by hardware or software. In other words, the functional configuration in the above-described embodiment is merely an example and is not particularly limited. That is, it is sufficient for the information processing device 1 to be provided with a function that can execute the above-described series of processes as a whole, and the type of functional block used to realize this function is not limited to the example shown. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof.

[0059] Furthermore, the recording medium containing the program for executing the above-mentioned series of processes may be configured not only as a removable medium distributed separately from the device main body in order to provide the program to the user, but also as a recording medium provided to the user in a state where it is pre-installed in the device main body.

[0060] The above embodiment shows an example of application of the present invention and does not limit the technical scope of the present invention. In other words, the present invention can be modified in various ways, such as by omission or substitution, without departing from the gist of the present invention, and various embodiments other than the above embodiment can be adopted. The various embodiments and modifications that the present invention can adopt are included in the scope of the invention described in the claims and their equivalents.

[0061] 1 Information processing device, 11 Processor, 12 ROM, 13 RAM, 14 Bus, 15 Input unit, 16 Output unit, 17 Storage unit, 18 Communication unit, 19 Drive, 20 Imaging unit, 31 Removable media, 11a Video data acquisition unit, 11b Preprocessing unit, 11c Feature acquisition unit, 11d Learning model generation unit, 11e Speech content estimation unit, 11f Substitute voice output unit, 11g Data expansion unit, 17a Learning model storage unit, 17b Synthesis voice data storage unit, 17c Estimation result storage unit

Claims

1. An information processing device comprising: a video acquisition means for acquiring a video including the lips of a subject; an extraction processing means for extracting the lip portion from the video; a feature acquisition means for acquiring features of the extracted lip portion; and a speech content estimation means for estimating the speech content of the subject corresponding to the lip portion extracted by the extraction processing means, based on a learning model that has been machine-learned in advance by associating the features of the lip portion in a video for machine learning that includes lip movement during speech with the speech content uttered by the lip portion.

2. The information processing apparatus according to claim 1, further comprising a voice output means for outputting the speech content of the subject estimated by the speech content estimation means by voice.

3. The information processing device described in claim 1 or 2, characterized in that the video acquisition means acquires the video for machine learning that includes lip movement during speech, the extraction processing means extracts the lip area in the video for machine learning, and the information processing device further comprises a learning model acquisition means that acquires the learning model that has been machine-learned by correlating the features of the lip area in the video for machine learning with the speech content uttered by the lip area in the video for machine learning.

4. An information processing device according to claim 1 or 2, characterized in that the extraction processing means extracts the lip portion from the video acquired by the video acquisition means using a template representing the shape of the lips.

5. The information processing device described in claim 4, characterized in that the extraction processing means sets landmarks in the facial area contained in the video acquired by the video acquisition means, and uses the landmarks set in the lip area as a reference to determine the range in which to search for the lip area using the template.

6. The information processing device according to claim 3, characterized in that the feature acquisition means acquires a unique feature represented by the difference between the feature of the lip part that produces a predetermined single sound and the feature of the lip part that produces a sound other than the predetermined single sound, and the learning model acquisition means acquires the learning model in which machine learning has been performed to associate the unique feature of the lip part for each single sound with the speech content produced by the lip part.

7. An information processing system as described in claim 1 or 2, characterized in that it comprises a lip image data processing means that performs data processing on the image of the lip area extracted by the extraction processing means to reduce the influence of the shooting environment, and the feature acquisition means acquires the feature of the lip area based on the image of the lip area data processed by the lip image data processing means and a machine learning model that has previously machine-learned the image of the lip area and the feature of the lip area.

8. An information processing method executed by an information processing device, comprising: a video acquisition step of acquiring a video including the lips of a subject; an extraction processing step of extracting the lip portion from the video; a feature acquisition step of acquiring features of the extracted lip portion; and a speech content estimation step of estimating the speech content of the subject corresponding to the lip portion extracted in the extraction processing step, based on a learning model that has been machine-learned in advance by correlating the features of the lip portion in a machine-learning video including lip movement during speech with the speech content uttered by the lip portion.

9. A program that causes a computer to implement the following functions: a video acquisition function that acquires a video including the lips of a subject; an extraction processing function that extracts the lip area from the video; a feature acquisition function that acquires features of the extracted lip area; and a speech content estimation function that estimates the speech content of the subject corresponding to the lip area extracted by the extraction processing function based on a learning model that has been machine-learned in advance by correlating the features of the lip area in a video for machine learning that includes lip movement during speech with the speech content emitted by the lip area.

Images