Facial synthesis lip reading device and facial synthesis lip reading method

Abstract

To provide a highly practical face synthesis lip reading device and face synthesis lip reading method capable of improving recognition accuracy of lip reading by image processing a face image of a speaker.SOLUTION: An image processing unit 14 that performs image processing on a learning target image and a recognition target image includes: face detection means 22 that detects a face image in learning of a learning object speaker and a face image in recognition of a recognition object speaker from a learning target image and a recognition target image; face synthesis means 23 that converts the face image in learning and the face image in recognition into a synthetic face image in learning and a synthetic face image in recognition using a face image of a specific speaker; lip region extraction means 24 that extracts a lip region in learning and a lip region in recognition from the synthetic face image in learning and the synthetic face image in recognition; and feature extracting means 25 that extracts a lip feature in learning from the lip region in learning as target data in learning, and extracts a lip feature in recognition from the lip region in recognition as recognition target data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a face synthesis lip reading device and a face synthesis lip reading method that can predict the speech content of an unspecified speaker to be recognized with high accuracy using machine learning. [Background technology]

[0002] Conventionally, speech recognition technology that converts speech information into text has a sufficient recognition rate in low-noise environments such as laboratories, and is gradually becoming more widespread, but it is difficult to use in noisy environments such as offices and outdoors where it is easily affected by surrounding noise, or in public places such as trains and hospitals where it is difficult to speak, and there is a problem that it lacks practicality. In addition, speech recognition technology cannot be used by people with speech disorders who have difficulty speaking, and there is also a problem that it lacks versatility. In contrast, lip-reading technology can infer the content of speech from the movement of the speaker's lips, and does not require the speaker to emit sound (no audio information is required). It can infer the content of speech from video alone, which is expected to make it usable not only in noisy environments and public places, but also by people with speech disabilities. In particular, lip-reading technology using a computer does not require special training and can be easily used by anyone, so it is expected to become more widespread. For example, Patent Document 1 discloses a word spotting lip reading device that includes an imaging means for acquiring a facial image including the lip area, an area extraction means for extracting the lip area from the acquired image, a feature measurement means for measuring shape features from the extracted lip area, a keyword DB for registering features of keyword speech scenes measured in a registration mode, a determination means for performing a recognition process in a recognition mode to recognize (infer) the speech content of the lips by comparing the features of the registered keywords with features measured for speech scenes in a sentence, thereby performing word spotting lip reading to recognize keywords from within a sentence, and a display means for displaying the recognition results performed by the determination means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-59017 A Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional lip reading technology using a computer, such as that described in Patent Document 1, in a registration mode (learning mode, during learning), a model is learned by machine learning using speech scenes of a speaker (learning target speaker) to be registered (learned), and in a recognition mode, the speech content of a speaker (recognition target speaker) to be recognized is inferred. This recognition task generally includes two types: specific speaker recognition, in which the recognition target speaker is included in the learning target speakers, and unspecified speaker recognition, in which the recognition target speaker is not included in the learning target speakers. In specific speaker recognition, the learning target speaker and the recognition target speaker are specified (limited), so the recognition accuracy is usually higher than that of unspecified speaker recognition, but there is a problem that it is necessary to pre-register (learn) speech scenes in which the user (recognition target speaker) is the learning target speaker, and it takes time to build a learning model. In contrast, speaker-independent recognition does not require prior registration as in speaker-specific recognition (any speaker can be the target speaker), and learning can be performed on speech scenes of unspecified speakers (using existing databases, etc.). This makes it easier to build a learning model, but it has the drawback that recognition is more difficult than speaker-specific recognition. The present invention has been made in consideration of the above circumstances, and aims to provide a highly practical face synthesis lip reading device and face synthesis lip reading method that can improve the accuracy of lip reading recognition by processing the speaker's facial image. [Means for solving the problem]

[0005] A face synthesis lip reading device according to a first invention that meets the above-mentioned objective comprises an image acquisition unit that reads, during learning, a learning target image in which a speech scene of a learning target speaker is recorded, and, during recognition, reads a recognition target image in which a speech scene of the recognition target speaker is recorded; an image processing unit that performs image processing on the learning target image and the recognition target image read by the image acquisition unit to extract learning target data and recognition target data, respectively; a learning processing unit that performs machine learning of lip reading based on the learning target data during learning and constructs a learning model; a lip reading database that stores the learning model; and a recognition processing unit that, during recognition, infers the speech content of the recognition target speaker by machine learning from the recognition target data and the learning model stored in the lip reading database, The image processing unit includes a face detection means for detecting a training-time facial image of the training target speaker from the training target image and a recognition-time facial image of the recognition target speaker from the recognition target image, a face synthesis means for converting the training-time facial image and the recognition-time facial image detected by the face detection means into a training-time synthetic facial image and a recognition-time synthetic facial image, respectively, using a facial image of a specific speaker, a lip area extraction means for extracting a training-time lip area and a recognition-time lip area from the training-time synthetic facial image and the recognition-time synthetic facial image created by the face synthesis means, respectively, and a feature extraction means for extracting training-time lip features from the training-time lip area as the training target data and extracting recognition-time lip features from the recognition-time lip area as the recognition target data.

[0006] In the face synthesis lip reading device of the first invention, the image processing unit may have a facial feature point detection means for detecting training-time facial feature points and recognition-time facial feature points from the training-time synthetic face image and the recognition-time synthetic face image, respectively, and the lip area extraction means may extract the training-time lip area and the recognition-time lip area from the training-time facial feature points and the recognition-time facial feature points, respectively.

[0007] In the face synthesis lip reading device of the first invention, the feature extraction means extracts, as the learning target data, in addition to the learning lip features, learning-time facial expression features from the learning-time facial feature points, and can also extract, as the recognition target data, in addition to the recognition-time lip features, recognition-time facial expression features from the recognition-time facial feature points.

[0008] The face synthesis lip reading device of the first invention can include an imaging means for imaging the speech scenes of the learning target speaker and the recognition target speaker, and a recognition result output unit for outputting the speech content of the recognition target speaker estimated by the recognition processing unit.

[0009] In the face synthesis lip reading device of the first invention, it is preferable that the recognition result output unit includes a display that displays the spoken content of the recognition target speaker estimated by the recognition processing unit in text and / or a speaker that outputs the spoken content in audio.

[0010] A face synthetic lip reading method according to a second invention in accordance with the above object includes a first learning step of reading a learning target image in which a speech scene of a learning target speaker is recorded, a second learning step of detecting a learning face image of the learning target speaker from the learning target image, a third learning step of converting the learning face image into a learning synthetic face image using a face image of a specific speaker, a fourth learning step of extracting a learning lip area from the learning synthetic face image, a fifth learning step of extracting learning lip features from the learning lip area as learning target data, a sixth learning step of repeating the first learning step to the fifth learning step, performing machine learning of lip reading based on the learning target data, and constructing a learning model, and the seventh step at learning time of saving the learning model; the first step at recognition time of reading the saved learning model; the second step at recognition time of reading a recognition target image in which a speech scene of a recognition target speaker is recorded; the third step at recognition time of detecting a recognition time facial image of the recognition target speaker from the recognition target image; the fourth step at recognition time of converting the recognition time facial image into a recognition time synthetic facial image using a facial image of a specific speaker; the fifth step at recognition time of extracting a recognition time lip area from the recognition time synthetic facial image; the sixth step at recognition time of extracting recognition time lip features from the recognition time lip area as recognition target data; and the seventh step at recognition time of inferring the speech content of the recognition target speaker by machine learning from the recognition target data and the learning model.

[0011] In the face synthesis lip reading method of the second invention, in the fourth learning step, learning facial feature points are detected from the learning synthetic face image and the learning lip area is extracted from the learning facial feature points, and in the fifth recognition step, recognition facial feature points are detected from the recognition synthetic face image and the recognition lip area is extracted from the recognition facial feature points.

[0012] In the face synthesis lip reading method of the second invention, in the fifth learning step, in addition to the learning lip features, learning facial expression features are extracted from the learning facial feature points as the learning target data, and in the sixth recognition step, in addition to the recognition lip features, recognition facial expression features can also be extracted from the recognition facial feature points as the recognition target data. Effect of the Invention

[0013] The facial synthesis lip reading device of the first invention and the facial synthesis lip reading method of the second invention convert the facial image of an unspecified speaker (the learning target speaker and the recognition target speaker are collectively referred to as speakers) into the facial image of a specific speaker and then perform learning and recognition processes, thereby making it possible to predict the spoken content with high accuracy during recognition, thereby improving recognition accuracy. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of a face synthesis lip reading device according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing functions of an image processing unit of the face synthesis lip reading device. [Diagram 3] 5 is a flowchart showing an operation during learning of the face synthesis and lip reading method according to the embodiment of the present invention. [Figure 4] 11 is a flowchart showing an operation during recognition of the same face synthesis lip reading method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Next, specific embodiments of the present invention will be described for better understanding of the present invention. The face synthesis lip reading device 10 and face synthesis lip reading method according to one embodiment of the present invention shown in FIG. 1 are lip reading technologies using a computer (machine learning), which convert the facial images of an unspecified learning target speaker and an unspecified recognition target speaker into facial images of a specific speaker and then perform learning and recognition processes to predict the spoken content of the unspecified recognition target speaker with high accuracy, thereby improving recognition accuracy (lip reading accuracy). As shown in Fig. 1, the face synthesis lip reading device 10 includes an image capture unit 11 for capturing (recording) speech scenes of a learning target speaker and a recognition target speaker. The face synthesis lip reading device 10 also includes an image capture unit 13 for reading a learning target image in which a speech scene of the learning target speaker is recorded from the image capture unit 11 during learning, and for reading a recognition target image in which a speech scene of the recognition target speaker is recorded from the image capture unit 11 during recognition. The face synthesis lip reading device 10 also includes an image processing unit 14 for processing the learning target image and the recognition target image loaded into the image capture unit 13, respectively, to extract learning target data and recognition target data required for machine learning. The face synthesis lip reading device 10 also includes a learning processing unit 15 for performing machine learning of lip reading based on the learning target data during learning, and constructing a learning model, and a lip reading database 16 for storing the learning model. The face synthesis lip reading device 10 is equipped with a recognition processing unit 17 that, during recognition, infers the speech content of the speaker to be recognized through machine learning from the recognition target data and a learning model stored in the lip reading database 16.

[0016] Here, as shown in Fig. 1, the face synthesis lip reading device 10 is configured to include an image acquisition unit 13, an image processing unit 14, a learning processing unit 15, a lip reading database 16, and a recognition processing unit 17, and a program for executing the face synthesis lip reading method used in the face synthesis lip reading device 10 is installed in a computer 18, and the CPU of the computer 18 executes the program, thereby making the computer 18 function as the image acquisition unit 13, the image processing unit 14, the learning processing unit 15, the lip reading database 16, and the recognition processing unit 17. As for the form of the computer, a desktop type or a notebook type is preferably used, but is not limited thereto, and can be appropriately selected. Note that some or all of the image acquisition unit 13, the image processing unit 14, the learning processing unit 15, the lip reading database 16, and the recognition processing unit 17 can also be used through a network by cloud computing. In addition, a video camera is preferably used as the photographing means, but the face synthesis lip reading device does not need to be equipped with a dedicated photographing means, and various photographing means that photograph the speech scene or various storage means (storage media) in which the speech scene is recorded can be connected to a computer (image acquisition unit) to read the learning target image or the recognition target image. Therefore, a smartphone or the like equipped with a video shooting function may be used as the photographing means, and instead of connecting the photographing means to a computer (image acquisition unit) to read the image, a storage medium such as a memory card built into the photographing means can be inserted from the photographing means to the computer (image acquisition unit) to read the image.

[0017] The face synthesis lip reading device 10 also includes a recognition result output unit 19 that outputs the speech content of the recognition target speaker estimated by the recognition processing unit 17. In this embodiment, the recognition result output unit 19 includes a display 20 that displays the speech content of the recognition target speaker estimated by the recognition processing unit 17 in characters and a speaker 21 that outputs it in audio, but either or both of the display 20 and the speaker 21 can be appropriately selected and used depending on the place and environment in which the face synthesis lip reading device 10 is used. The display and the speaker may be accessories or built-in components of the computer, or may be separately attached (externally) to the computer. The recognition result output unit may also be configured to include only one of the display and the speaker.

[0018] Next, the image processing unit 14 will be described in detail with reference to FIG. The image processing unit 14 includes face detection means 22 for detecting a learning face image of the learning target speaker from the learning target image during learning, and detecting a recognition face image of the recognition target speaker from the recognition target image during recognition. The image processing unit 14 also includes face synthesis means 23 for converting the learning face image and the recognition face image detected by the face detection means 22 into a learning composite face image and a recognition composite face image using the face image of a specific speaker, respectively, and lip area extraction means 24 for extracting a learning lip area and a recognition lip area from the learning composite face image and the recognition composite face image created by the face synthesis means 23, respectively. The image processing unit 14 also includes feature extraction means 25 for extracting learning lip features from the learning lip area as learning target data, and for extracting recognition lip features from the recognition lip area as recognition target data.

[0019] In the face synthesis means 23, the face image (still image) of a certain specific person can be moved (animated) in accordance with the movement of the learning face image and the recognition face image of an unspecified person when speaking. In other words, the learning face image and the recognition face image of an unspecified speaker can be converted into a learning synthetic face image and a recognition synthetic face image using the face image of a specific speaker, so that the recognition accuracy of lip reading can be improved. As the face image of the specific speaker, for example, a face image of a person registered in an existing database or the like can be used, or a face image of another person can be used. In addition to a real person, the face image can be a face image of a person drawn by computer graphics or animation, or for example, a face image of a person other than a person, such as a robot, a car, an animal, or an insect, which is personified. In addition, the specific speaker is not limited to one person, and may be a small number of specified people. The recognition accuracy is improved by consolidating the target of the learning target data and the recognition target data from a large number of unspecified speakers to one or a small number of specific speakers. When a specific speaker is selected from the learning target speakers or the recognition target speakers, the face image of the learning target speaker or the recognition target speaker selected as the specific speaker may or may not be converted to the face image of the specific speaker, like other learning target speakers or the recognition target speakers. Note that the face synthesis means 23 preferably uses a First Order Motion Model (FOMM), but is not limited to this.

[0020] The image processing unit 14 extracts the learning-time lip region and the recognition-time lip region by image processing the learning-time synthetic face image and the recognition-time synthetic face image with the lip region extraction means 24, and the means and method for extracting the learning-time lip region and the recognition-time lip region are appropriately selected. For example, the image processing unit 14 can detect the learning-time facial feature points and the recognition-time facial feature points from the learning-time synthetic face image and the recognition-time synthetic face image, respectively, with the facial feature point detection means, and then extract the learning-time lip region and the recognition-time lip region based on the learning-time facial feature points and the recognition-time facial feature points with the lip region extraction means. The learning-time facial feature points and the recognition-time facial feature points represent, for example, the facial outline of a specific speaker and the positions and shapes of the eyebrows, eyes, nose, and mouth, and the number of feature points is appropriately selected. In addition, as in this embodiment, when the face synthesis means 23 first converts the learning-time facial image of the learning target speaker and the recognition-time facial image of the recognition target speaker into a learning-time synthetic face image and a recognition-time synthetic face image using the facial image of a specific speaker, the number of learning-time facial feature points and recognition-time facial feature points can be reduced to speed up or simplify processing in the facial feature point detection means, or the facial feature point detection means can be omitted to simplify processing in the image processing unit.

[0021] In addition, the feature extraction means extracts learning-time lip features from the learning-time lip area as learning target data, and extracts recognition-time lip features from the recognition-time lip area as recognition target data, so that the learning processing unit and the recognition processing unit can perform machine learning based on the respective lip features (movement characteristics of the area around the lips), construct a learning model, and infer the content of the speech. As mentioned above, when the image processing unit has a facial feature point detection means and learning-time facial feature points and recognition-time facial feature points are detected from the learning-time synthetic face image and the recognition-time synthetic face image, respectively, the feature extraction means can extract learning-time facial expression features from the learning-time facial feature points in addition to the learning-time lip features as the learning target data, and extract recognition-time facial expression features from the recognition-time facial feature points in addition to the recognition-time lip features as the recognition target data.It is also possible to further improve the recognition accuracy (recognition rate) of the speech content by performing machine learning taking into account the facial expression features of the entire face (e.g., changes in the position, shape, angle, etc. of the eyebrows, eyes, mouth, etc.) in addition to the lip features; however, sufficient recognition accuracy can be obtained with just the learning-time lip features and the recognition-time lip features.

[0022] Next, the operation during learning of the face synthesis and lip reading method according to the embodiment of the present invention will be described with reference to FIG. First, in a first learning step, a learning target image in which a speech scene of a learning target speaker is recorded is read into the image acquisition unit 13 (S1). Next, in a second learning step, the face detection means 22 of the image processing unit 14 detects a learning face image of the learning target speaker from the learning target image (S2). Then, in a third learning step, the face synthesis means 23 of the image processing unit 14 converts the learning face image into a learning composite face image using the face image of the specific speaker (S3), and in a fourth learning step, the lip region extraction means 24 of the image processing unit 14 extracts a learning lip region from the learning composite face image (S4). Furthermore, in a fifth learning step, the feature extraction means 25 of the image processing unit 14 extracts learning lip features from the learning lip region as learning target data (S5). The above first to fifth learning steps are repeated as many times as the number of speech scenes to be learned. Then, in a sixth learning step, the learning processing unit 15 performs machine learning of lip reading based on the learning target data extracted from each speech scene, and constructs a learning model (S6). The learning model thus constructed is stored in the lip reading database 16 in a seventh learning step (S7). As described above, the means and method for extracting the learning lip area are appropriately selected. For example, in the fourth learning step, the face feature point detection means of the image processing unit detects the learning face feature points from the learning composite face image, and the lip area extraction means extracts the learning lip area based on the learning face feature points. If the learning face feature points are detected, in the fifth learning step, the feature extraction means of the image processing unit extracts learning facial expression features from the learning face feature points in addition to the learning lip area as learning target data, and machine learning can be performed taking into account the facial expression features of the entire face in addition to the lip features.

[0023] Next, the operation during recognition in the face synthesis and lip reading method will be described with reference to FIG. First, in a first step during recognition, a learning model (trained model) stored in the lip reading database 16 is read (S1). Then, in a second step during recognition, a recognition target image in which a speech scene of a recognition target speaker is recorded is read into the image acquisition unit 13 (S2). Next, in a third step during recognition, the face detection means 22 of the image processing unit 14 detects a recognition time face image of the recognition target speaker from the recognition target image (S3). Then, in a fourth step during recognition, the face synthesis means 23 of the image processing unit 14 converts the recognition time face image into a recognition time composite face image using the face image of the specific speaker (S4), and in a fifth step during recognition, the lip area extraction means 24 of the image processing unit 14 extracts a recognition time lip area from the recognition time composite face image (S5). Furthermore, in a sixth step during recognition, the feature extraction means 25 of the image processing unit 14 extracts recognition time lip features from the recognition time lip area as recognition target data (S6). Then, in the seventh step of recognition, the speech content of the speaker to be recognized is inferred from the recognition target data and the learning model by machine learning (lip reading processing) (S7). The inferred speech content (evaluation result) is converted into characters and / or voice, and is output from the display 20 and / or speaker 21 of the evaluation result output unit 19 (S8). As described above, if the image processing unit has a facial feature point detection means, in the fifth recognition step, the recognition facial feature points are detected from the recognition composite face image, and the recognition lip area can be extracted based on the recognition facial feature points. In the sixth recognition step, the feature extraction means can extract recognition facial expression features from the recognition facial feature points in addition to the recognition lip features as recognition target data, and can perform machine learning taking into account the facial expression features of the entire face in addition to the lip features. EXAMPLES

[0024] Next, the results of evaluations carried out to confirm the effects of the present invention will be described. Comparative Example 1 A conventional lip-reading method was evaluated using the database CUAVE, which is open to the public for lip-reading. CUAVE contains scenes in which 36 registered speakers (19 men and 17 women) each speak 10 numbers from 0 to 9 in English. As the conventional lip-reading method, the third learning step and the fourth recognition step of the face synthesis lip-reading method of the present invention were omitted, and in the subsequent steps, the learning face image and the recognition face image were used as they were, instead of the learning synthetic face image and the recognition synthetic face image. As the evaluation method, the existing method of the leave-one-person method was used. In other words, one recognition target speaker was selected from the 36 registered speakers, and the remaining 35 registered speakers were used as the learning target speakers for learning and evaluation. The recognition target speaker was changed, and 36 recognition experiments (one pattern) were performed, and the average recognition accuracy was 73%.

[0025] Example 1 The same database CUAVE as in Comparative Example 1 was used, and the face synthesis lip reading method of the present invention was used as the lip reading method. First, in order to select the face image of the specific speaker to be used in the third step of learning and the fourth step of recognition, one frame image was taken out from each registered speaker's speech scene to prepare face images (still images) of 36 registered speakers, and one face image of the specific speaker was selected from the face images of each registered speaker. 36 recognition experiments (one pattern) were performed for each specific speaker by the leave-one-out method similar to Comparative Example 1. Of the total 36 recognition experiments for the 36 specific speakers, 35 (97%) showed improved recognition accuracy compared to the recognition experiments in Comparative Example 1, with an average recognition accuracy of 83%. In addition, when the results of 36 (one) recognition experiments for one specific speaker with the highest recognition accuracy out of 36 recognition experiments (36 specific speakers) were compared with the results of the recognition experiments in Comparative Example 1 for each target speaker, it was found that the recognition accuracy was improved compared to the recognition experiments in Comparative Example 1 in 27 of the 36 recognition experiments, that is, in the recognition experiments for 27 out of 36 target speakers (75%).

[0026] Comparative Example 2 OuluVS was used as the database instead of CUAVE. OuluVS contains scenes in which 20 registered speakers (17 males and 3 females) each utter 10 different greetings in English. Evaluation was performed in the same manner as in Comparative Example 1, except that the number of registered speakers was different. In other words, one recognition target speaker was selected from the 20 registered speakers, and the remaining 19 registered speakers were used as learning targets for learning and evaluation. The recognition target speaker was changed and 20 (one pattern) recognition experiments were performed, and the average recognition accuracy was 81%.

[0027] Example 2 Evaluation was performed in the same manner as in Example 1, except that the same database OuluVS as in Comparative Example 2 was used. That is, a face image of a specific speaker was selected one by one from the prepared face images (still images) of 20 registered speakers, and 20 (one-way) recognition experiments were performed for each specific speaker by the leave-one-out method. Of the total 20 recognition experiments for the 20 specific speakers, the recognition accuracy was improved in 17 (85%) recognition experiments compared to the recognition experiment in Comparative Example 2, and the average recognition accuracy was 87%. In addition, when the results of the 20 (one-way) recognition experiments for the one specific speaker with the highest recognition accuracy among the 20 recognition experiments (20 specific speakers) were compared with the results of the recognition experiment in Comparative Example 2 for each recognition target speaker, the recognition accuracy was improved in 12 of the 20 recognition experiments, that is, in the recognition experiments for 12 of the 20 recognition experiments (60%) of the recognition target speakers, compared to the recognition experiment in Comparative Example 2.

[0028] From the above, it was found that the use of the face synthesis lip reading method of the present invention improves the recognition accuracy of lip reading by about 6 to 7% on average compared to the conventional lip reading method, and the effectiveness of the face synthesis lip reading method of the present invention was confirmed. Note that in Examples 1 and 2, the recognition experiment was performed by converting all the face images of the learning target speaker and the recognition target speaker into the face image of one specific speaker, but even if there are multiple specific speakers, it is considered that the recognition accuracy can be improved as long as the number of specific speakers is at least less than the number of learning target speakers.

[0029] The present invention has been described above with reference to an embodiment, but the present invention is not limited to the configurations described in the above embodiment, and also includes other embodiments and modifications that are possible within the scope of the matters described in the claims. For example, in this embodiment, a gated recurrent unit (GRU), which is a type of deep learning, is used for machine learning, but the machine learning algorithm may be selected as appropriate. [Explanation of symbols]

[0030] 10: face synthesis lip reading device, 11: photographing means, 13: image acquisition unit, 14: image processing unit, 15: learning processing unit, 16: lip reading database, 17: recognition processing unit, 18: computer, 19: recognition result output unit, 20: display, 21: speaker, 22: face detection means, 23: face synthesis means, 24: lip area extraction means, 25: feature extraction means

Claims

1. an image acquisition unit that loads a learning target image in which a speech scene of a learning target speaker is recorded during learning, and loads a recognition target image in which a speech scene of the recognition target speaker is recorded during recognition, an image processing unit that performs image processing on the learning target image and the recognition target image loaded into the image acquisition unit to extract learning target data and recognition target data, respectively, a learning processing unit that performs machine learning of lip reading based on the learning target data during learning and constructs a learning model, a lip reading database that stores the learning model, and a recognition processing unit that infers the speech content of the recognition target speaker by machine learning from the recognition target data and the learning model stored in the lip reading database during recognition, said image processing unit comprises face detection means for detecting a training-time facial image of the training target speaker from said training target image and for detecting a recognition-time facial image of the recognition target speaker from said recognition target image; face synthesis means for converting the training-time facial image and the recognition-time facial image detected by said face detection means into a training-time synthetic facial image and a recognition-time synthetic facial image, respectively, by using a facial image of a specific speaker; lip area extraction means for extracting a training-time lip area and a recognition-time lip area from the training-time synthetic facial image and the recognition-time synthetic facial image created by said face synthesis means, respectively; and feature extraction means for extracting learning-time lip features from the training-time lip area as said training target data and extracting recognition-time lip features from the recognition-time lip area as said recognition target data.

2. 2. A facial synthesis lip reading device according to claim 1, wherein said image processing unit has a facial feature point detection means for detecting training-time facial feature points and recognition-time facial feature points from said training-time synthetic face image and said recognition-time synthetic face image, respectively, and said lip area extraction means extracts said training-time lip area and said recognition-time lip area from said training-time facial feature points and said recognition-time facial feature points, respectively.

3. 3. A facial synthesis lip reading device according to claim 2, wherein said feature extraction means extracts, as said learning target data, in addition to said learning time lip features, learning time facial expression features from said learning time facial feature points, and, as said recognition target data, in addition to said recognition time lip features, recognition time facial expression features from said recognition time facial feature points.

4. 4. The facial synthesis lip reading device according to claim 1, further comprising: an imaging means for imaging the speech scenes of the learning target speaker and the recognition target speaker; and a recognition result output unit for outputting the speech content of the recognition target speaker estimated by the recognition processing unit.

5. 5. The facial synthesis lip reading device according to claim 4, wherein the recognition result output unit is provided with a display for displaying the speech content of the speaker to be recognized, which is estimated by the recognition processing unit, in characters and / or a speaker for outputting the speech content in audio.

6. A face synthesis and lip reading method using computerized machine learning, comprising: a learning step of reading a learning target image in which a speech scene of a learning target speaker is recorded; a learning step of detecting a learning face image of the learning target speaker from the learning target image; a learning step of converting the learning face image into a learning synthetic face image using a face image of a specific speaker; a learning step of extracting a learning lip area from the learning synthetic face image; a learning step of extracting learning lip features from the learning lip area as learning target data; a learning step of repeating the learning steps from the learning step to the learning step, and constructing a learning model based on the learning target data; a learning step of storing the learning model; a third step at recognition time of detecting a time-of-recognition face image of the recognition target speaker from the recognition target image; a fourth step at recognition time of converting the time-of-recognition face image into a time-of-recognition synthetic face image using a face image of a specific speaker; a fifth step at recognition time of extracting a time-of-recognition lip area from the time-of-recognition synthetic face image; a sixth step at recognition time of extracting a time-of-recognition lip feature from the time-of-recognition lip area as recognition target data; and a seventh step at recognition time of inferring the speech content of the recognition target speaker from the recognition target data and the learning model by machine learning.

7. 7. The face synthesis lip reading method according to claim 6, wherein in the fourth learning step, learning-time facial feature points are detected from the learning-time synthetic face image and the learning-time lip area is extracted from the learning-time facial feature points, and in the fifth recognition step, recognition-time facial feature points are detected from the recognition-time synthetic face image and the recognition-time lip area is extracted from the recognition-time facial feature points.

8. 8. The face synthesis lip reading method according to claim 7, wherein in the fifth learning step, in addition to the learning lip features, learning facial expression features are extracted from the learning facial feature points as the learning target data, and in the sixth recognition step, in addition to the recognition lip features, recognition facial expression features are extracted from the recognition facial feature points as the recognition target data.

Images