Speech content recognition device, method, and program

Abstract

To recognize the details of a speech with high accuracy for lip image data of various types (various imaging directions) in a number exceeding the number of corresponding directions.SOLUTION: A speech detail recognition device 100 recognizes the details of a speech made by a speaker, and has: an input unit 111 that receives input of the speaker's lips image data; a plurality of lip-reading units 131, 132 that have high lip-reading accuracy for the lips image data picked up from corresponding directions (corresponding angles); and an integration and generation unit 141 that integrates results of lip-reading processing performed by the plurality of lip-reading units for the lips image data input to the input unit, and generates a result of recognition of the details of the speech made by the speaker based on a result of the integration. At least one lip-reading unit of the plurality of lip-reading units is configured to include, in the corresponding directions, a direction not included in corresponding directions in the other lip-reading unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a speech recognition device. Place, The present invention relates to a method and a program. [Background technology]

[0002] Conventionally, there is known a speech content recognition device that recognizes the contents of a speaker's speech. For example, a device is known that inputs lip image data of a speaker and recognizes the contents of a speaker's speech using a lip reading unit that has high lip reading accuracy for the lip image data captured from a corresponding direction.

[0003] Non-Patent Document 1 also discloses a method called "View2View" using an encoder-decoder model that uses a convolutional neural network. In this method, lip-reading results are output using a machine lip-reading model that has been trained in advance on image data of a frontal face (image data of lips captured from the front of the face). When non-frontal image data is input, it is converted to image data of a frontal face and then input to the machine lip-reading model, which then outputs the lip-reading results.

[0004] In addition, Non-Patent Document 2 discloses an end-to-end lip-reading method that uses a deep learning technology called bidirectional long-short-term memory. This Non-Patent Document 2 describes that the lip-reading performance of a lip-reading model is improved by learning by combining face image data captured from multiple imaging angles, such as a frontal face and a side face. Summary of the Invention [Problem to be solved by the invention]

[0005] In a conventional speech content recognition device equipped with a lip reading unit, correct lip reading results can be obtained with high accuracy when using lip image data captured from a specific direction (corresponding direction), but there is a problem in that accuracy decreases when using lip image data captured from a direction other than the corresponding direction. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention provides a speech content recognition device that recognizes the speech content of a speaker, comprising an input unit that inputs lip image data of the speaker, a plurality of lip reading units that have high lip reading accuracy for lip image data captured from corresponding directions, and an integration generation unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result, and at least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units. The plurality of lip reading units include a multi-directional lip reading unit having two or more corresponding directions. It is characterized by: Effect of the Invention

[0007] According to the present invention, correct lip reading results can be obtained with high accuracy even for lip image data captured from a direction that does not match the corresponding direction of the lip reading unit, so that the speech content can be recognized with high accuracy for various types (various imaging directions) of lip image data that exceed the number of corresponding directions. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a lip reading device according to a first embodiment. [Diagram 2] FIG. 1 is an explanatory diagram showing an example of a model of one neuron that constitutes a neural network. [Diagram 3] FIG. 1 is an explanatory diagram showing an example of a neural network having a multi-layer structure. [Figure 4] FIG. 2 is an explanatory diagram showing an overview of a method for creating a machine lip-reading model (trained model) (training mode) in the first embodiment. [Diagram 5] FIG. 4 is a block diagram showing another example of the lip reading device according to the first embodiment. [Figure 6] FIG. 4 is a block diagram showing still another example of the lip reading device according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing a lip reading device according to a first modified example. [Figure 8] FIG. 11 is a block diagram showing a lip reading device according to a second modification. [Figure 9]FIG. 11 is a block diagram showing a multimodal speech recognition device according to a second embodiment. [Figure 10] FIG. 11 is an explanatory diagram showing the configuration of a learning data collection system according to a third embodiment. [Figure 11] FIG. 2 is an explanatory diagram showing the camera array of the learning data collection system viewed vertically from above. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment 1] Hereinafter, an embodiment in which the present invention is applied to a lip reading device as a speech content recognition device (hereinafter, this embodiment will be referred to as "embodiment 1") will be described. The lip reading device of this embodiment 1 inputs facial image data obtained by capturing an image of a speaker's face as lip image data, analyzes the lip portion of the input facial image data, and outputs a recognition result (lip reading result) of the speech content uttered by the speaker.

[0010] FIG. 1 is a block diagram showing a lip reading device according to the first embodiment. The lip reading device 100 of this embodiment 1 is mainly composed of an image input unit 111 as an input unit, two single-angle compatible lip reading units 131 and 132 as multiple lip reading units, and a lip reading result integration unit 141 as an integration generation unit.

[0011] The image input unit 111 accepts input of face image data (lip image data) of a speaker whose speech content is to be recognized. The image input unit 111 of the present embodiment 1 is communicatively connected, via wire or wirelessly, to a camera 1, which is an imaging device that captures an image of the speaker's face, and a storage medium 2 that stores the face image data. Real-time face image data of the speaker currently speaking is input from the camera 1 to the image input unit 111. The storage medium 2 stores face image data of past utterances by the speaker, and the past face image data is input from the storage medium 2 to the image input unit 111.

[0012] The image input unit 111 processes the input face image data as necessary before inputting the data to the two single-angle compatible lip reading units 131, 132, and transfers the data to each of the single-angle compatible lip reading units 131, 132. For example, the image input unit 111 extracts lip image portions from the input face image data by arranging them in chronological order, and transfers the data of the lip image portions to each of the single-angle compatible lip reading units 131, 132.

[0013] There are no particular limitations on the imaging direction of the lip image data input to image input unit 111, so long as the image data is imaged so as to include the lips of the speaker. Furthermore, the lip image data input to the image input unit 111 may be in an image data format, or may be in a non-image data format obtained by processing or calculating the lip image data. In addition, lip image data is typically captured image data obtained by capturing an image of a real speaker using an imaging device, etc., but it may also be image data of a virtual speaker (such as one created using computer graphics, etc.) viewed from a specified viewpoint.

[0014] The two single-angle lip-reading units 131 and 132 each perform lip-reading processing with high lip-reading accuracy on lip image data captured from a specific direction (corresponding direction) and generate a lip-reading processing result. The two single-angle lip-reading units 131 and 132 are configured so that their corresponding directions include directions that are not included in the corresponding directions of the other single-angle lip-reading unit.

[0015] In the first embodiment, the corresponding direction is expressed as an angle (hereinafter referred to as the "corresponding angle") around a vertical axis with the imaging direction when the speaker's face is imaged from the front direction as the reference (0°). In this case, the first single-angle corresponding lip reading unit 131 has high lip reading accuracy for face image data when the speaker's face is imaged from the front direction (exceeding the accuracy threshold that meets the user's required level), and its corresponding angle (angle with high lip reading accuracy) is 0°. On the other hand, the second single-angle corresponding lip reading unit 132 has high lip reading accuracy for face image data when the speaker's face is imaged from a direction shifted 30° to the side from the front direction, and its corresponding angle (angle with high lip reading accuracy) is 30°.

[0016] The single-angle compatible lip reading units 131, 132 of the present embodiment 1 execute a predetermined lip reading processing program (estimation program) on a computer to perform lip reading processing on the face image data input to the image input unit 111 and generate a lip reading processing result. The lip reading processing program of the present embodiment 1 uses a machine lip reading model (trained model) trained using training data including the face image data of the speaker, but a lip reading processing program programmed by a programmer may also be used.

[0017] The machine lip-reading model (trained model) in the present embodiment 1 estimates the contents of a speaker's speech from input data (face image data), and there is no particular restriction on the format of the estimation result (lip-reading processing result) output from the machine lip-reading model. As an example, in the present embodiment 1, a case will be described in which data including one or more speech content candidates (such as one character, one word, or word sequence) and reliability information for each of the speech content candidates (hereinafter referred to as "reliability score") is output as the lip-reading processing result for the input data (face image data).

[0018] A machine lip reading model specialized for a predetermined corresponding angle (a trained model with high lip reading accuracy for face image data at the corresponding angle) can be created, for example, by performing machine learning or deep learning using a large amount of face image data captured from the corresponding angle as training data. For example, by using such training data and performing supervised learning on a predetermined model, it is possible to obtain a machine lip reading model (trained model) that outputs data including one or more speech content candidates and a reliability score of each speech content candidate as an estimation result according to the features learned from the training data when unknown face image data is input. In this embodiment 1, a neural network model is adopted as the predetermined model, but other machine learning models can also be used.

[0019] In "supervised learning," a machine learning device is generally given a large amount of data pairs of inputs and results (labels), which allows it to learn the features of those data sets and inductively acquire a model that estimates results from inputs, i.e., the relationships between them. This can be achieved using algorithms such as neural networks and SVMs (Support Vector Machines), which are described below.

[0020] A neural network is composed of, for example, an arithmetic device and a memory that realize a neural network that imitates a neuron model as shown in FIG. 2. As shown in FIG. 2, a neuron outputs an output y for a plurality of inputs x (inputs x1 to x3 are shown here as an example, but the number of inputs may be less or more). Each of the inputs x1 to x3 is multiplied by a weight W (W1 to W3) corresponding to each input x. As a result, the neuron outputs an output y expressed by the following formulas (1) and (2). In formulas (1) and (2), θ is a bias, and fk is an activation function.

[0021] y = fk(v) (1) v = Σ(W × x) - θ (2)

[0022] The neural network operates in a learning mode and an evaluation mode. In the learning mode, weights W are learned using learning data, and in the evaluation mode, the weights W are used to obtain an output for an input of evaluation data (in this embodiment 1, utterance content candidates and their respective reliability scores). The weights W1 to W3 can be learned by backpropagation or the like. The backpropagation is a technique for adjusting (learning) the weights for each neuron so as to reduce the difference between the output y when an input x is input and the correct output y (correct label).

[0023] Neural networks can have a multi-layer structure known as deep learning, as shown in Figure 3. The example in Figure 3 shows a three-layer structure of intermediate layers (hidden layers). Each layer is made up of multiple nodes (neurons), and the nodes between layers are connected with different weights W. Inputs x1 to x6 input to the input layer are combined while being weighted by the weight W as they pass through the nodes in the intermediate layers with different weights W, and then pass through the output layer to derive the output y.

[0024] In this embodiment 1, a neural network model consisting of a multi-layered neural network as shown in FIG. 3 is adopted, and a machine lip reading model (trained model) is created by using training data including facial image data of a speaker who speaks known speech content and using the correct speech content as a correct answer label to perform supervised learning.

[0025] FIG. 4 is an explanatory diagram showing an overview of a method (learning mode) for creating a machine lip-reading model (trained model) in the first embodiment. In the learning mode of the present embodiment 1, as shown in Fig. 4, a speaker speaks a specified utterance content, and this is captured by each of the recording cameras 31-1 and 31-2 from the corresponding angles (0° and 30° in the present embodiment 1). The facial image data captured in this manner is stored in the learning data storage medium 32 for each corresponding angle. The facial image data stored in the learning data storage medium 32 is stored in a format that allows a time series to be specified. Therefore, the facial image data stored in the learning data storage medium 32 can be associated with the speaker's utterance content by checking it against the time when the speaker spoke (the time when each utterance content was spoken), and can be used as learning data.

[0026] The learning data thus stored in the learning data storage medium 32 is used by the first learning unit 33-1 for face image data with a corresponding angle of 0°, and by the second learning unit 33-2 for face image data with a corresponding angle of 30°. In addition, the learning data may include other information useful as a feature for identifying (estimating) the contents of speech in order to further improve accuracy. In the first learning unit 33-1, a machine lip-reading model (trained model) with a corresponding angle of 0° is generated from the input learning data, and the generated machine lip-reading model is installed in the first single-angle compatible lip-reading unit 131 of the present embodiment 1. Similarly, in the second learning unit 33-2, a machine lip-reading model (trained model) with a corresponding angle of 30° is generated from the input learning data, and the generated machine lip-reading model is installed in the second single-angle compatible lip-reading unit 132 of the present embodiment 1.

[0027] For the generated machine lip reading model (trained model), parameter tuning may be performed by repeatedly trying to create the trained model (training mode). The parameters to be adjusted (tuned) in parameter tuning refer to the setting values ​​and limit values ​​(hyperparameters) in the trained model. Parameter tuning is, for example, the task of scanning and setting parameters that allow the model to produce an optimal solution. Types of parameter tuning that can be used include grid search and random search methods.

[0028] In addition, model evaluation may be performed on the machine lip reading model (trained model). For this model evaluation, for example, cross-validation or hold-out method can be used. Model evaluation can also be performed by using the hold-out method and cross-validation in combination.

[0029] Specifically, in the hold-out method, the original data is divided in advance into training data to be used in the training mode and test data to be used in the evaluation mode, and a trained model is created using only the training data. After that, the test data is input to the trained model, and the output result is compared with the correct label of the test data (error = estimated accuracy) to evaluate the model.

[0030] In cross-validation, the original data is divided into, for example, five groups, and the first time, one of the groups is used as test data and the other groups are used as training data to create a trained model and evaluate the model. The second time, a group different from the first time is used as test data, and the third time, a group different from the first and second times is used as test data to create a trained model and evaluate the model in the same way. This is done for all five groups, and the model evaluations (estimation accuracy) evaluated each time are averaged.

[0031] In addition, the estimation program (trained model) of the present embodiment 1 can be distilled to create a new estimation program (distilled model) with similar functions. Specifically, face image data with known speech content is input as distillation input data to the estimation program (trained model) of the present embodiment 1, and the reliability score is output. Then, learning data for distillation is created with the output reliability score as the correct answer label of the distillation input data, and a new estimation program (distilled model) with similar functions to the estimation program (trained model) of the present embodiment 1 is created by having a model learn using the learning data for distillation. The new estimation program (distilled model) created in this way is generally lighter than the estimation program (trained model) of the present embodiment 1. In addition, it is also possible to improve the estimation accuracy more than the estimation program (trained model) of the present embodiment 1 by devising the distillation input data.

[0032] Note that the machine lip-reading model of the first embodiment is an example in which data including utterance content candidates and the reliability scores of each utterance content candidate is used as the lip-reading processing result, but intermediate representation data conforming to the data format used in the subsequent lip-reading result integration unit 141 may also be used as the lip-reading processing result. Specifically, vector data recording the internal state of the machine lip-reading model when lip-reading processing is performed using the machine lip-reading model may also be used as the lip-reading processing result.

[0033] In this embodiment 1, the two single-angle lip-reading units 131 and 132, which have high lip-reading accuracy for lip image data captured from a specific direction (corresponding direction), are used to perform lip-reading processing on lip image data. The two single-angle lip-reading units 131 and 132 used in this embodiment 1 include angles (angles at which high lip-reading accuracy can be obtained) that are not included in the corresponding angles of the other single-angle lip-reading unit. Therefore, if the lip image data is captured from an angle that matches one of the corresponding angles (0° and 30°) of these single-angle lip-reading units, these single-angle lip-reading units can obtain high lip-reading accuracy, and the speech content can be recognized with high accuracy from the lip-reading processing result. Therefore, according to this embodiment 1, the speech content can be recognized with high accuracy for lip image data captured from angles of 0° and 30°.

[0034] Here, for lip image data captured from an angle (for example, 15° or 45°) that does not match any of the angles supported by the two single-angle lip readers 131 and 132, sufficient lip reading accuracy cannot be obtained by the lip reading process of each of the single-angle lip readers 131 and 132. Therefore, if only the lip reading process result of one of the single-angle lip readers 131 and 132 is used, the speech content of such lip image data cannot be recognized with high accuracy.

[0035] Therefore, in the present embodiment 1, a lip reading result integration unit 141 is provided to integrate the lip reading process results obtained by the two single-angle compatible lip reading units 131, 132, and generate a recognition result of the speaker's utterance content as a final lip reading result based on the integration result. As a result, even if the lip reading process results of the individual single-angle compatible lip reading units 131, 132 are such that the reliability (reliability score) of the correct utterance content is not significantly higher than the reliability of the other incorrect utterance contents or, conversely, is lower, by integrating these lip reading process results, it becomes possible to highlight the reliability (reliability score) of the correct utterance content and process it so that it is significantly higher than the reliability of the other incorrect utterance contents.

[0036] There are no particular limitations on the integration process performed by the lip reading result integration unit 141, so long as it is a process of integrating the lip reading process results obtained by the two single-angle compatible lip reading units 131, 132 so as to obtain highly accurate recognition results (in the case of this embodiment 1, so that the reliability score of the correct speech content becomes relatively high).

[0037] As an example of the integration process performed by the lip reading result integration unit 141, for example, the integration result may be the utterance content candidate with the highest reliability score among the common utterance content candidates (such as words and word sequences) included in both of the lip reading process results obtained by the two single-angle compatible lip reading units 131 and 132. Alternatively, the integration result may be two or more utterance content candidates in descending order of reliability score. Also, for example, the total value of the reliability score for each utterance content candidate included in the lip reading processing results obtained by the two single-angle compatible lip reading units 131 and 132 may be calculated, and the utterance content candidate with the highest total value may be used as the integrated result. Alternatively, two or more utterance content candidates for which the total value of the reliability score has been calculated may be used as the integrated result.

[0038] As another example of the integration process performed by the lip-reading result integration unit 141, a trained model (integration model) for obtaining a recognition result from intermediate representations in the two single-angle compatible lip-reading units 131 and 132 is trained in advance, for example, by a neural network model. Then, the intermediate representations (vectors, etc.) included in the lip-reading process results obtained by the two single-angle compatible lip-reading units 131 and 132 are linked to generate one intermediate representation, and the trained integration model is executed by the computer of the lip-reading result integration unit 141 to obtain a recognition result of one or more utterance contents from the one intermediate representation, which is used as the integration result. Alternatively, the recognition result of one or more utterance contents from the one intermediate representation may include each reliability score.

[0039] According to the lip reading device 100 of the first embodiment, by integrating the lip reading results of the two single-angle lip reading units 131, 132 in the lip reading result integration unit 141, it becomes possible to recognize the speech content with high accuracy not only for the corresponding angles of 0° and 30° at which highly accurate lip reading results can be obtained by the lip reading process of each single-angle lip reading unit 131, 132, but also for lip image data at angles outside these corresponding angles (for example, 15° or 45°). As a result, it becomes possible to recognize the speech content with high accuracy for lip image data from various angles that exceed the number of corresponding angles (0° and 30°) of the two single-angle lip reading units 131, 132.

[0040] The recognition result generated by the lip reading result integration unit 141 is output to a downstream information processing device that uses the recognition result of the speaker's speech content, or to an information storage device that stores the recognition result of the speaker's speech content. There is no particular limitation on the output mode, and for example, the recognition result may be displayed on a display unit provided in the lip reading device 100 of the first embodiment, or may be output as audio from an audio output unit provided in the lip reading device 100.

[0041] In the above-mentioned first embodiment, an example is given in which there are two lip reading units, but an example in which there are three or more lip reading units may be used. For example, as shown in Fig. 5, a configuration may be used in which a third single-angle compatible lip reading unit 133 is added, which has high lip reading accuracy for face image data captured from a direction shifted 60° to the side from the front direction of the speaker's face.

[0042] It is also possible to create a multiple-angle lip-reading unit capable of obtaining highly accurate lip-reading results at two or more corresponding angles using a single lip-reading unit. Specifically, for example, as shown in Fig. 4, a learning data storage medium 32 storing face image data captured from two corresponding angles (0° and 30°) is used, and these are input as learning data to a single learning unit for learning, thereby making it possible to generate a machine lip-reading model (trained model) capable of highly accurate lip-reading processing at two corresponding angles, 0° and 30°.

[0043] Therefore, for example, as shown in Fig. 6, a first multiple-angle lip reading unit 134 capable of highly accurate lip reading at two corresponding angles, 0° and 30°, may be provided in place of the above-mentioned second single-angle lip reading unit 132. Furthermore, as shown in Fig. 6, a second multiple-angle lip reading unit 135 capable of highly accurate lip reading at three corresponding angles, 0°, 30°, and 60°, may be provided in place of the above-mentioned third single-angle lip reading unit 133.

[0044] However, a multi-angle lip-reading unit usually has the advantage of being able to improve lip-reading accuracy for a wider range of angles compared to creating multiple single-angle lip-reading units with the multiple corresponding angles of the multi-angle lip-reading unit as the corresponding angle, but has the disadvantage of increased learning costs. This disadvantage can be compared, for example, with the amount of training data required to build a machine lip-reading model that executes these lip-reading processes. In order to obtain a predetermined high accuracy (accuracy exceeding a predetermined accuracy threshold) for each of the multiple corresponding angles, the machine lip-reading model of the multi-angle lip-reading unit usually requires much more training data than the total amount of training data required to build each machine lip-reading model of the multiple single-angle lip-reading units. In addition, as the training data increases, the cost of parameter tuning and the like also increases.

[0045] On the other hand, considering that the multiple-angle lip reading unit has the above-mentioned advantages, it is preferable to use a configuration in which a single-angle lip reading unit and a multiple-angle lip reading unit are mixed, as in the example of Fig. 6. In this way, for example, the single-angle lip reading unit can handle the most frequent imaging angle and its neighboring angles (corresponding angles that provide high lip reading accuracy) in the input face image data (lip image data), and the multiple-angle lip reading unit can handle the other angles, and by integrating these lip reading process results in the lip reading result integration unit 141, it becomes possible to recognize the spoken content with high accuracy for lip image data from a wider variety of angles.

[0046] In this respect, it can be said that, theoretically, it is possible to construct a single lip-reading unit capable of recognizing speech content with high accuracy for lip image data captured from various angles similar to that of the first embodiment. However, it is extremely difficult to realize such a single lip-reading unit by a lip-reading program programmed by a programmer. Moreover, in order to realize such a single lip-reading unit by a machine lip-reading model (trained model), a huge amount of training data is required, and it is also difficult to construct such a machine lip-reading model.

[0047] In contrast, the lip reading unit used in the lip reading device 100 of the present embodiment 1 is a lip reading unit specialized for one corresponding angle (single-angle compatible lip reading unit) or a lip reading unit specialized for multiple (several) corresponding angles (multiple-angle compatible lip reading unit), and it is relatively easy to construct such a lip reading unit. Therefore, according to the present embodiment 1, there is also an advantage that a lip reading device that can recognize the speech content with high accuracy from lip image data from various angles can be more easily created.

[0048] In addition, the corresponding angles of the lip readers whose lip reading process results are integrated by the lip reading result integration unit 141 may overlap partially. In other words, the corresponding angles of the multiple-angle lip readers may include some or all of the corresponding angles of other single-angle lip readers or other multiple-angle lip readers, as long as they are not exactly the same combination. For example, as shown in Fig. 6, 0° may be the corresponding angle of all the lip readers 131, 134, and 135, and 30° may be the corresponding angle of the two multiple-angle lip readers 134 and 135.

[0049] [Variation 1] Next, a modification of the lip reading apparatus 100 in the above-mentioned first embodiment (hereinafter, this modification will be referred to as "Modification 1") will be described. FIG. 7 is a block diagram showing a lip reading device 100 in the first modification. As shown in FIG. 7, the lip reading device 100 in this variant example 1 has three lip reading units: a first single-angle compatible lip reading unit 131 capable of highly accurate lip reading processing at a corresponding angle of 0°; a second single-angle compatible lip reading unit 136 capable of highly accurate lip reading processing at a corresponding angle of 45°; and a multiple-angle compatible lip reading unit 137 capable of highly accurate lip reading processing at two corresponding angles of 0° and 45°.

[0050] The lip reading device 100 in the present modified example 1 includes angle conversion units 121, 122, and 123 as data conversion units that generate data converted based on the lip image data input to the image input unit 111 so that the imaging direction becomes the corresponding angle of at least one of the lip reading units 131, 136, and 137. In the example of Fig. 7, the angle conversion units 121, 122, and 123 are provided for all of the three lip reading units 131, 136, and 137 as processing units in the preceding stage, and each angle conversion unit performs conversion processing of the lip image data input to the image input unit 111 so that the imaging direction matches one of the corresponding angles of the lip reading units 131, 136, and 137. That is, each angle conversion unit 121, 122, 123 performs a conversion process so that the lip image of the lip image data input to the image input unit 111 becomes pseudo-equivalent to the lip image captured from the corresponding angle of each lip reading unit 131, 136, 137.

[0051] For example, when lip image data captured at an angle of 30° is input to image input unit 111, first angle conversion unit 121 performs conversion processing of the lip image data input to image input unit 111 so that the imaging direction matches 0°, which is the corresponding angle of first single angle compatible lip reading unit 131. In this case, similarly, second angle conversion unit 122 performs conversion processing of the lip image data input to image input unit 111 so that the imaging direction matches 45°, which is the corresponding angle of second single angle compatible lip reading unit 136. Moreover, third angle conversion unit 123 performs conversion processing of the lip image data input to image input unit 111 so that the imaging direction matches either 0° or 45° (here, 0°), which are the corresponding angles of multiple angle compatible lip reading unit 137.

[0052] Each of the angle conversion units 121, 122, and 123 may perform conversion using a linear mapping such as an affine transformation, or may perform conversion using a conversion model based on machine learning or deep learning. Furthermore, the data converted by each of the angle conversion units 121, 122, and 123 only needs to correspond to the input data format of each of the lip reading units 131, 136, and 137, and may be in the format of image data or an intermediate representation format of a conversion model, for example.

[0053] According to this first modification, the data (image data or intermediate representation) received by each of the lip reading units 131, 136, 137 from the angle conversion units 121, 122, 123 matches the corresponding angle of each of the lip reading units 131, 136, 137. Therefore, each of the lip reading units 131, 136, 137 can perform lip reading processing with high accuracy, and a highly accurate lip reading processing result can be obtained from each of the lip reading units 131, 136, 137. As a result, the final lip reading result (recognition result of the speech content) obtained by integrating these lip reading processing results by the lip reading result integration unit 141 is also highly accurate.

[0054] [Modification 2] Next, another modification of the lip reading device 100 in the above-mentioned first embodiment (hereinafter, this modification will be referred to as "Modification 2") will be described. FIG. 8 is a block diagram showing a lip reading device 100 in the second modification. As shown in FIG. 8, the lip reading device 100 in this variant example 2 has three lip reading units: a first single-angle compatible lip reading unit 131 capable of high-precision lip reading processing at a corresponding angle of 0°; a second single-angle compatible lip reading unit 136 capable of high-precision lip reading processing at a corresponding angle of 45°; and a multiple-angle compatible lip reading unit 134 capable of high-precision lip reading processing at two corresponding angles of 0° and 30°.

[0055] The lip reading device 100 in the present modified example 2 includes an angle estimation unit 112 as an imaging direction estimation unit that estimates the imaging direction of the lip image data input to the image input unit 111. For example, a model that estimates the imaging angle of the input face image data is learned by machine learning or deep learning using a large amount of face image data captured from various angles and stored in the above-mentioned learning data storage medium 32. The angle estimation model thus generated is then executed by the computer of the angle estimation unit 112 to estimate the imaging direction of the lip image data input to the image input unit 111.

[0056] When the angle estimation unit 112 of the present modified example 2 receives face image data from the image input unit 111, it estimates the imaging angle of the face image data and outputs a certainty factor indicating the probability for each preset angle as an angle estimation result. For example, when the preset angles are 0°, 30°, 45°, and 60°, the angle estimation unit 112 outputs information such as a certainty factor of 0° of 0.3, a certainty factor of 30° of 0.4, a certainty factor of 45° of 0.2, and a certainty factor of 60° of 0.1 as an estimation result of the imaging angle of the input face image data,

[0057] In this second modification, the angle estimation result of the angle estimation unit 112 is sent to the lip-reading result integration unit 141. The lip-reading result integration unit 141 of this second modification integrates the lip-reading process results obtained by the three lip-reading units 131, 136, and 134 using the angle estimation result received from the angle estimation unit 112, and generates a recognition result of the speaker's utterance content as a final lip-reading result.

[0058] As an example of the integration process in the present modified example 2, for example, the reliability score of each lip reading process result obtained by the three lip reading units 131, 136, 134 is multiplied by the certainty of the estimated angle that matches the corresponding angle of each lip reading unit included in the angle estimation result of the angle estimation unit 112. For example, in the above example, for the first single angle compatible lip reading unit 131 whose corresponding angle is 0°, the reliability score is multiplied by 0.3, for the second single angle compatible lip reading unit 136 whose corresponding angle is 45°, the reliability score is multiplied by 0.2, and for the multiple angle compatible lip reading unit 134 whose corresponding angles are 0° and 30°, the reliability score is multiplied by 0.4 using the higher of the certainty of 0° and 30°.

[0059] After using the angle estimation result in this way, the lip reading result integration unit 141 may take as the integrated result the speech content candidate with the highest reliability score among the common speech content candidates included in all of the lip reading process results obtained by the three lip reading units 131, 136, and 134, as in the above-mentioned first embodiment. Alternatively, two or more speech content candidates in descending order of reliability score may be taken as the integrated result. Also, for example, after using the angle estimation result, the lip reading result integration unit 141 may calculate the total value of the reliability score for each of the speech content candidates included in the lip reading process results obtained by the three lip reading units 131, 136, and 134, and take as the integrated result the speech content candidate with the highest total value. Alternatively, two or more speech content candidates for which the total value of the reliability score has been calculated may be taken as the integrated result.

[0060] According to this second modification, the lip reading results of each of the lip reading units 131, 136, and 134 can be weighted using the angle estimation result obtained by estimating the imaging angle of the face image data from the image input unit 111. In other words, the angle estimation result from the angle estimation unit 112 is used, and the lip reading results of a lip reading unit with a higher degree of certainty of an estimated angle that matches the corresponding angle are weighted more heavily, thereby increasing the influence of the lip reading results of the lip reading unit on the recognition result of the utterance content. This makes it possible to achieve a higher accuracy in the final lip reading result (recognition result of the utterance content) obtained by the lip reading result integration unit 141.

[0061] [Embodiment 2] Next, an embodiment in which the present invention is applied to a multimodal speech recognition device as a speech content recognition device (hereinafter, this embodiment will be referred to as "embodiment 2") will be described. The multimodal speech recognition device of the second embodiment uses two types of speech content recognition processing, namely lip reading processing and speech recognition processing, to output a recognition result of the content of an utterance uttered by a speaker.

[0062] FIG. 9 is a block diagram showing a multimodal speech recognition device according to the second embodiment. The multimodal speech recognition device 300 of the second embodiment is composed of a lip-reading recognition processing unit 101, a speech recognition processing unit 201, and a recognition result integration unit 301.

[0063] The configuration of the lip reading recognition processing unit 101 can be the same as that of the lip reading device 100 of the above-mentioned embodiment 1. The example of Fig. 9 employs the configuration of the lip reading device 100 shown in Fig. 1.

[0064] The voice recognition processing unit 201 is mainly composed of a voice input unit 211 and a voice recognition unit 231 .

[0065] The voice input unit 211 accepts input of voice data of a speaker who is speaking. The voice input unit 211 of the second embodiment is connected to a microphone 3 that collects the voice of the speaker and a storage medium 2 that stores the voice data of the speaker in a wired or wireless manner so as to be able to communicate with each other. Real-time voice data of the speaker currently speaking is input to the voice input unit 211 from the microphone 3. The storage medium 2 stores voice data of past utterances by the speaker, and the past voice data is input from the storage medium 2 to the voice input unit 211.

[0066] The voice input unit 211 processes the input voice data as necessary so that it corresponds to the input of the voice recognition unit 231, and passes the processed data to the voice recognition unit 231. For example, a voice signal from which noise has been removed is extracted from the input voice data, and the data of the voice signal is passed to the voice recognition unit 231.

[0067] The voice recognition unit 231 of the present embodiment 2 executes a predetermined voice recognition program on a computer, thereby executing voice recognition processing on voice data input to the voice input unit 211 and generating a voice recognition result. The voice recognition program of the present embodiment 2 uses a voice recognition model (trained model) trained using training data including the speaker's voice data, but a voice recognition program programmed by a programmer may also be used.

[0068] The speech recognition model (trained model) in the second embodiment estimates the contents of a speaker's speech from input data (speech data), and the format of the estimation result (speech recognition result) output from the speech recognition model is not particularly limited, as in the case of the machine lip reading model described above. As an example, in the second embodiment, data including one or more speech content candidates (such as one word or word sequence) and a reliability score for each of the speech content candidates is output as a speech recognition result for the input data (speech data) in accordance with the format of the lip reading recognition processing unit 101.

[0069] The recognition result integration unit 301 integrates the recognition result (lip reading result) output from the lip reading result integration unit 141 of the lip reading recognition processing unit 101 and the recognition result (voice recognition result) output from the voice recognition unit 231 of the voice recognition processing unit 201, and outputs the final recognition result of the speech content.

[0070] There are no particular limitations on the integration process performed by the recognition result integration unit 301, so long as it is a process of integrating the recognition result (lip reading result) of the lip reading recognition processing unit 101 and the recognition result (voice recognition result) of the voice recognition processing unit 201 so as to obtain a highly accurate recognition result (for example, so that the reliability score of the correct speech content becomes relatively high).

[0071] As an example of the integration process performed by the recognition result integration unit 301, for example, the integration result may be an utterance content candidate with the highest reliability score among common utterance content candidates included in both of the recognition results of the two recognition processing units 101 and 201. Alternatively, the integration result may be two or more utterance content candidates in descending order of reliability score. Also, for example, the total value of the reliability score for each utterance content candidate included in the recognition results of the two recognition processing units 101 and 201 may be calculated, and the utterance content candidate with the highest total value may be used as the integrated result. Alternatively, two or more utterance content candidates for which the total value of the reliability score has been calculated may be used as the integrated result.

[0072] As another example of the integration process performed by the recognition result integration unit 301, a trained model (integration model) for obtaining a recognition result from intermediate representations in the two recognition processing units 101 and 201 is trained in advance, for example, by a neural network model. Then, each intermediate representation (vector, etc.) included in the recognition results obtained by the two recognition processing units 101 and 201 is linked to generate one intermediate representation, and the trained integration model is executed by the computer of the recognition result integration unit 301 to obtain a recognition result of one or more utterance contents from the one intermediate representation, which is set as an integration result. Alternatively, the recognition result of one or more utterance contents from the one intermediate representation may include each confidence score.

[0073] In the multimodal speech recognition device 300 of the second embodiment, the lip-reading recognition processing unit 101 employs the configuration of the lip-reading device 100 of the first embodiment described above, and therefore, lip-reading results can be obtained with high accuracy.

[0074] In addition, the multimodal speech recognition device 300 of the second embodiment obtains a recognition result using two different types of speech content recognition methods, namely, a lip reading device and a voice recognition device, and integrates these recognition results to output a final recognition result of the speech content. Therefore, even in a situation where the speech recognition processing unit 201 has difficulty in recognizing the speaker's speech content with high accuracy (for example, a noisy environment, a conference environment where multiple speakers often speak at the same time, etc.), the lip reading recognition processing unit 101 can recognize the speech content with high accuracy. Also, even in a situation where the lip reading recognition processing unit 101 has difficulty in recognizing the speaker's speech content with high accuracy (for example, a dark environment with low lighting, an environment where it is difficult to capture the speaker's lips due to the speaker's movement, etc.), the speech recognition processing unit 201 can recognize the speech content with high accuracy.

[0075] In this way, the multimodal speech recognition device 300 of the second embodiment can realize a robust speech content recognition device in which the recognition accuracy of the speaker's speech content is not easily affected by the speaker's environment. Specifically, such a multimodal speech recognition device 300 can be suitably used as an automatic meeting minutes generation system in a conference room or online, or as a speech input interface in a smartphone.

[0076] [Embodiment 3] Next, we will explain one embodiment of a learning data collection system (hereinafter, this embodiment will be referred to as "embodiment 3") that collects learning data for constructing a machine lip reading model used in the lip reading recognition processing unit 101 and a speech recognition model used in the speech recognition processing unit 201 in the multimodal speech recognition device 300 of the above-mentioned embodiment 2.

[0077] FIG. 10 is an explanatory diagram showing the configuration of a learning data collection system in the third embodiment. The learning data collection system of the present embodiment 3 includes a camera array 31 having a plurality of imaging devices, a recording microphone 21 as a voice acquisition device, a display 42 as an instruction device, and a control device 43. In addition, the learning data collection system of the present embodiment 3 includes a reporting unit 41 and a learning data storage medium 32 as a storage device.

[0078] The camera array 31 is composed of a plurality of cameras (imaging devices) 31-1 to 31-10 that capture images of a speaker at a predetermined position from a plurality of different imaging directions. In the third embodiment, as shown in FIG. 11, ten recording cameras 31-1 to 31-10 are arranged at equal intervals. Specifically, assuming that the imaging angle is an angle around a vertical axis with the imaging direction when the speaker's face is imaged from the front direction as a reference (0°), the recording cameras 31-1 to 31-10 are arranged at 10° intervals between 0° and 90°. This camera array 31 allows the recording cameras 31-1 to 31-10 to simultaneously capture images of the lips of a speaker who is speaking from their respective imaging angles.

[0079] The camera array 31 is connected to a learning data storage medium 32, and the face image data (lip image data) of the speaker captured by each of the recording cameras 31-1 to 31-10 is stored and accumulated in the learning data storage medium 32. The camera array 31 is also connected to a control device 43, and the image capturing operation is controlled by the control device 43.

[0080] There is no particular limit to the number of cameras constituting the camera array 31. In the third embodiment, the cameras constituting the camera array 31 are arranged along a horizontal plane so that the imaging angles around the vertical axis are different from each other, but the arrangement is not limited to this. For example, the recording cameras 31-1 to 31-10 may be arranged so that the imaging angles around the horizontal axis or the inclined axis inclined with respect to the vertical axis are different from each other.

[0081] The recording microphone 21 acquires the voice of a speaker who is the subject of imaging by the camera array 31. The recording microphone 21 is connected to a learning data storage medium 32, and the voice data collected by the recording microphone 21 is stored and accumulated in the learning data storage medium 32. The recording microphone 21 is also connected to a control device 43, and the operation of the recording microphone 21 is controlled by the control device 43. The recording microphone 21 may be, for example, a lapel microphone installed on the speaker's collar, or a stand microphone may be installed near the speaker. In this case, it is desirable to install the recording microphone 21 so as not to get in the way when the recording cameras 31-1 to 31-10 of the camera array 31 capture the speaker's lip image.

[0082] In addition, when the cameras constituting the camera array 31 have a function as a voice acquisition device, the voice acquisition device of the camera may be used as the recording microphone 21.

[0083] The display 42 displays the spoken instructions to the speaker. The display contents of the display 42 are controlled by a control device 43 connected by wire or wirelessly, and under the control of the control device 43, the display 42 displays the spoken instructions to the speaker, instructs the speaker to repeat the utterance, etc.

[0084] The operation of the notification unit 41 is controlled by a control device 43 connected by wire or wirelessly, and under the control of the control device 43, notifies the speaker of the start and end timing of speech by light, sound, etc.

[0085] As described above, the learning data storage medium 32 stores the face image data of the speaker captured by each of the recording cameras 31-1 to 31-10 of the camera array 31 and the voice data of the speaker collected by the recording microphone 21 in a format that allows a time series to be specified. Specifically, the speech start synchronization signal and the speech start synchronization signal emitted by the notification unit 41 are embedded in the face image data captured by each of the recording cameras 31-1 to 31-10 and the voice data collected by the recording microphone 21. This allows the face image data and voice data stored in the learning data storage medium 32 to be compared with the time when the speaker spoke the speech content instructed by the display 42. Therefore, the speech content instructed to the speaker via the display 42 is associated with the face image data and voice data of the speaker at the time when the speech content was spoken.

[0086] The learning data storage medium 32 may be a distributed storage device consisting of multiple storage media connected to each of the recording cameras 31-1 to 31-10 and the recording microphone 21 of the camera array 31, or may be composed of a single storage device (such as a file server).

[0087] The control device 43 is composed of an information processing device such as a personal computer, and controls the entire system. Specifically, the control device 43 starts recording with the camera array 31 and the recording microphone 21 in response to an instruction operation by the operator, issues a speech start synchronization signal through the notification unit 41 and instructs the speaker when to start speaking, displays the speech content on the display 42, and causes the speaker to speak the speech content. The control device 43 also issues a speech end synchronization signal through the notification unit 41 and instructs the speaker when to end speaking, and ends recording with the camera array 31 and the recording microphone 21. The control device 43 also instructs the speaker to respeak through the display 42 in response to an instruction operation by the operator.

[0088] According to the learning data collection system of the third embodiment, the lip images of a speaker who is speaking the utterance content instructed by the display 42 are simultaneously captured from different imaging angles by the recording cameras 31-1 to 31-10, and the voice of the speaker at that time is acquired by the recording microphone 21, and these can be stored in the learning data storage medium 32. This makes it possible to rapidly collect a large amount of face image data (lip image data) captured from different imaging angles and the corresponding voice data. Therefore, it is possible to easily and rapidly collect a large amount of learning data required to construct a machine lip-reading model used in the lip-reading recognition processing unit 101 and a voice recognition model used in the voice recognition processing unit 201 in the multimodal voice recognition device 300 of the above-mentioned second embodiment.

[0089] The learning data collection system of the present embodiment 3 is also useful for easily and quickly collecting a large amount of learning data required to construct a machine lip-reading model used in the lip-reading device 100 of the above-mentioned embodiment 1. In this case, recording by the recording microphone 21 is not necessarily required.

[0090] The above description is merely an example, and the present invention provides unique effects for each of the following aspects. [First aspect] The first aspect is a speech content recognition device (e.g., lip reading device 100, multimodal speech recognition device 300) that recognizes the speech content of a speaker, and includes an input unit (e.g., image input unit 111) that inputs lip image data (e.g., face image data) of the speaker, a plurality of lip reading units (e.g., single-angle compatible lip reading units 131, 132, 133, 136, multiple-angle compatible lip reading units 134, 135, 137) that have high lip reading accuracy for lip image data captured from corresponding directions (e.g., corresponding angles), and an integration generation unit (e.g., lip reading result integration unit 141) that integrates the lip reading processing results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speaker's speech content based on the result of the integration, and is characterized in that at least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units. In general, a lip reading unit that performs lip reading processing of lip image data input to an input unit can perform lip reading processing with high accuracy (accuracy that meets the level required by the user) when the imaging direction of the input lip image data is a specific direction (corresponding direction), and the recognition accuracy of the spoken content is high. Specifically, for example, a lip reading unit whose corresponding direction is the front direction of the speaker's face has high lip reading accuracy when lip image data obtained when the speaker's face is imaged from the front is input. On the other hand, when lip image data obtained when the speaker's face is imaged from the side or diagonally is input to this lip reading unit, the lip reading accuracy drops, and in many cases high lip reading accuracy cannot be obtained. In this embodiment, lip reading processing is performed on lip image data input to an input unit using a plurality of lip reading units having different corresponding directions. At least one of the plurality of lip reading units used in this embodiment includes a corresponding direction (a direction in which high lip reading accuracy can be obtained) that is not included in the corresponding directions of any of the other lip reading units. Therefore, if lip image data is captured from a direction that matches one of the corresponding directions of these lip reading units, the lip reading unit can obtain high lip reading accuracy, and the speech content can be recognized with high accuracy from the lip reading processing result. Therefore, according to this embodiment, the imaging directions of lip image data in which the speech content can be recognized with high accuracy can be increased by the number of corresponding directions of these plurality of lip reading units. Here, when lip image data captured from a direction that does not match any of the corresponding directions of the multiple lip readers is input to the input unit, the lip reading process of each lip reader cannot obtain sufficient lip reading accuracy. Therefore, if only the lip reading process result of one of the lip readers is used, the speech content of such lip image data cannot be recognized with high accuracy. Therefore, in this embodiment, the integration generating unit integrates the results of the lip reading processes of the multiple lip reading units for the lip image data input to the input unit, and generates a recognition result of the speech content based on the integration result. As a result, even if the results of the lip reading processes of the individual lip reading units show that the reliability (confidence) of the correct speech content is not significantly higher than the reliability of the other incorrect speech content, or conversely, is lower, the integration of these lip reading results can highlight the reliability of the correct speech content and create a significant difference from the reliability of the other incorrect speech content. For example, by accumulating the reliability included in the results of the lip reading processes of the individual lip reading units for each speech content candidate, it is possible to derive a reliability of the correct speech content that is significantly different from the reliability of the other incorrect speech content. Therefore, by integrating the results of the lip reading processes of the multiple lip reading units described above and generating a recognition result of the speech content based on the integration result, it is possible to recognize the speech content with high accuracy for lip image data captured from a direction in which sufficient lip reading accuracy cannot be obtained by the lip reading processes of the individual lip reading units. Therefore, according to this aspect, it is possible to recognize the speech content with high accuracy for various types (imaging directions) of lip image data, which exceeds the number of corresponding directions of the above-mentioned plurality of lip reading units.

[0091] [Second aspect] A second aspect is characterized in that in the first aspect, the plurality of lip readers include a unidirectional lip reader having one corresponding direction (for example, single-angle corresponding lip readers 131, 132, 133, 136). A unidirectional lip reader that supports one direction can be constructed more easily than a multi-directional lip reader that supports two or more directions, and therefore an utterance content recognition device can be easily realized.

[0092] [Third aspect] The third aspect is characterized in that in the first or second aspect, the multiple lip reading units include a multi-directional lip reading unit (e.g., multiple angle compatible lip reading units 134, 135, 137) having two or more corresponding directions. This allows the number of lip readers to be reduced, making it easier to realize a simple speech content recognition device.

[0093] [Fourth aspect] A fourth aspect is characterized in that, in any of the first to third aspects, the device has a data conversion unit (e.g., angle conversion units 121-123) which generates data converted based on lip image data input to the input unit so that the imaging direction corresponds to the corresponding direction of at least one of the multiple lip reading units, and the at least one lip reading unit performs lip reading processing using the data converted by the data conversion unit. According to this, since the lip reading units are each input with lip image data of an imaging direction that matches the corresponding direction, each lip reading unit can obtain a lip reading process result with high accuracy. As a result, the recognition result of the speech content obtained by integrating these lip reading process results by the integration generation unit can also be highly accurate.

[0094] [Fifth aspect] A fifth aspect is characterized in that, in any of the first to fourth aspects, it has an imaging direction estimation unit (e.g., an angle estimation unit 112) that estimates the imaging direction of the lip image data input to the input unit, and the integration generation unit uses the estimation result of the imaging direction estimation unit to generate a recognition result of the speaker's speech content. According to this, the lip reading process results of each lip reading unit can be weighted using the estimation result of the imaging direction estimation unit estimating the imaging angle of the face image data from the input unit. In other words, using the estimation result of the imaging direction estimation unit, the lip reading process results of the lip reading unit with a higher degree of certainty of the estimated angle matching the corresponding direction can be weighted more heavily, thereby increasing the influence of the lip reading process results of the lip reading unit on the recognition result of the utterance content. This makes it possible to increase the accuracy of the recognition result of the utterance content obtained by the integration generation unit.

[0095] [Sixth aspect] A sixth aspect is characterized in that, in any of the first to fifth aspects, the multiple lip reading units generate lip reading processing results including one or more speech content candidates estimated by the lip reading process and reliability information (e.g., a reliability score) for each speech content candidate, and the integration generation unit integrates the reliability information included in each lip reading processing result of the multiple lip reading units for each speech content candidate. This makes it possible to obtain a highly accurate recognition result of the speech content in the integration generation unit using a simple method.

[0096] [Seventh aspect] A seventh aspect is characterized in that, in any of the first to fifth aspects, the multiple lip reading units generate intermediate information (e.g., intermediate expressions) for estimating candidate speech content as a lip reading processing result by the lip reading process, and the integration generation unit integrates the intermediate information contained in each lip reading processing result of the multiple lip reading units. According to this, the lip reading processing results of multiple lip reading units can be integrated using a trained model (integrated model) trained using intermediate information learning data, and the integrated generation unit can obtain highly accurate recognition results of the speech content.

[0097] [Eighth aspect] The eighth aspect is characterized in that, in any of the first to seventh aspects, it has a voice recognition unit (e.g., the voice recognition processing unit 201) that recognizes the contents of the speaker's speech from the voice data of the speaker, and the integration generation unit (e.g., the lip reading result integration unit 141 and the recognition result integration unit 301) generates a recognition result of the contents of the speaker's speech using the recognition result of the voice recognition unit. This makes it possible to realize a multimodal speech content recognition device (e.g., the multimodal speech recognition device 300) that outputs a recognition result of the speech content uttered by a speaker using two types of speech content recognition processing, namely lip reading processing and speech recognition processing. This makes it possible to realize a speech content recognition device with excellent robustness in which the recognition accuracy of the speaker's speech content is less affected by the speaker's environment.

[0098] [Ninth aspect] A ninth aspect is characterized in that, in any one of the first to eighth aspects, the plurality of lip reading units include a lip reading unit that performs lip reading processing of lip image data input to the input unit by having a computer execute a machine lip reading model trained using training data including lip image data of a speaker. According to this, lip reading processing is performed using a machine lip reading model, so that lip reading processing results with higher accuracy can be obtained.

[0099] [Tenth aspect] The tenth aspect is a learning data collection system that collects learning data for constructing the machine lip reading model used in the speech content recognition device of the ninth aspect, characterized in that it has a plurality of imaging devices (e.g., recording cameras 31-1 to 31-10) that image a speaker at a predetermined position from a plurality of different imaging directions, a voice acquisition device (e.g., recording microphone 21) that acquires the voice of the speaker, an instruction device (e.g., display 42) that instructs the speaker on the content to be spoken, and a control device 43 that causes the instruction device to instruct the content of the speech, simultaneously images the lips of the speaker speaking the instructed content by the plurality of imaging devices, and acquires the voice of the speaker by the voice acquisition device, and executes control to store the obtained lip image data and voice data in a storage device (e.g., learning data storage medium 32). According to this, the lip images of the speaker who speaks the utterance contents instructed by the instruction device can be simultaneously captured from different imaging angles by the multiple imaging devices, and the voice of the speaker at that time can be acquired by the voice acquisition device and stored in the storage device. This makes it possible to quickly and massively collect lip image data captured from different imaging angles and corresponding voice data. Therefore, it is possible to easily and quickly collect a large amount of learning data required to build a machine lip reading model used in the lip reading process and a voice recognition model used in the voice recognition process in the multimodal utterance content recognition device of the above-mentioned ninth aspect.

[0100] [Eleventh aspect] An eleventh aspect is a method for recognizing the content of a speaker's utterance by a speech content recognition device, comprising: an input step of inputting lip image data of the speaker into the speech content recognition device; a lip reading step in which the speech content recognition device performs lip reading processing of the lip image data input in the input step using a plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions; and an integration generation step in which the speech content recognition device integrates the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generates a recognition result of the speaker's speech content based on the result of the integration, wherein at least one lip reading unit among the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units. According to this aspect, it is possible to recognize the speech content with high accuracy for various types (imaging directions) of lip image data, which exceeds the number of corresponding directions in the above-mentioned plurality of lip reading units.

[0101] [12th aspect] A twelfth aspect is a program executed by a computer of a speech content recognition device that recognizes the content of a speaker's speech, which causes the computer to function as an integration generation means that integrates the results of lip reading processing performed by multiple lip reading means with high lip reading accuracy on lip image data captured from corresponding directions input to the speech content recognition device, and generates a recognition result for the content of the speaker's speech based on the result of the integration, and is characterized in that at least one lip reading unit among the multiple lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units. According to this aspect, it is possible to recognize the speech content with high accuracy for various types (imaging directions) of lip image data, which exceeds the number of corresponding directions in the above-mentioned plurality of lip reading units.

[0102] [13th aspect] A thirteenth aspect is a method for collecting learning data for constructing the machine lip reading model used in the speech content recognition device by the learning data collection system of the tenth aspect, characterized in that the instruction device is caused to indicate the speech content, lip images of a speaker speaking the indicated speech content are simultaneously captured by the multiple imaging devices, and the speaker's voice is acquired by the voice acquisition device, and the obtained lip image data and voice data are stored in a storage device. According to this aspect, it is possible to easily and quickly collect a large amount of training data required to construct a machine lip reading model used in the lip reading process and a voice recognition model used in the voice recognition process in the multimodal speech content recognition device of the ninth aspect described above.

[0103] [14th aspect] A 14th aspect is a program executed on a computer of the control device in the learning data collection system of the 10th aspect, which causes the instructing device to instruct the speech content, simultaneously captures lip images of a speaker speaking the instructed speech content using the multiple imaging devices, acquires the speaker's voice using the voice acquisition device, and causes the obtained lip image data and voice data to function as a control means that executes control to store in a storage device. According to this aspect, it is possible to easily and quickly collect a large amount of training data required to construct a machine lip reading model used in the lip reading process and a voice recognition model used in the voice recognition process in the multimodal speech content recognition device of the ninth aspect described above. [Explanation of symbols]

[0104] 1: Camera 2:Storage medium 3: Microphone 21: Recording microphone 31: Camera array 31-1~31-10: Recording camera 32: Learning data storage medium 33-1: First Learning Section 33-2: Second Learning Section 41: Reporting Department 42: Display 43: Control device 100: Lip reading device 101: Lip recognition processing unit 111: Image input unit 112: Angle estimation section 121~123: Angle conversion section 131, 132, 133, 136: Single angle lip reader 134, 135, 137: Multi-angle lip reader 141: Lip reading result integration unit 201: Speech recognition processing unit 211: Audio input section 231: Voice recognition unit 300: Multimodal speech recognition device 301: Recognition result integration section [Prior art documents] [Non-patent literature]

[0105] [Non-Patent Document 1] A. Koumparoulis et al., "Deep view2view mapping for view-invariant lipreading", IEEE SLT, 2018, p.588-594 [Non-Patent Document 2] S. Petridis et al., "End-to-end Multiview Lip Reading", IEEE ICASSP, 2018, p.6548-6552

Claims

1. A speech content recognition device that recognizes the content of a speaker's speech, an input unit for inputting lip image data of a speaker; A plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions; an integration generating unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result; At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units; The speech content recognition device according to the present invention, wherein the plurality of lip reading units includes a multi-directional lip reading unit having two or more corresponding directions.

2. A speech content recognition device that recognizes the content of a speaker's speech, an input unit for inputting lip image data of a speaker; A plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions; an integration generating unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result; At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units; an imaging direction estimation unit that estimates a plurality of imaging directions of the lip image data input to the input unit and certainty factor information for each imaging direction; the integration generation unit integrates the lip reading process results using a plurality of estimation results from the imaging direction estimation unit, and generates a recognition result of the speaker's speech content based on the integration result.

3. A speech content recognition device that recognizes the content of a speaker's speech, an input unit for inputting lip image data of a speaker; A plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions; an integration generating unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result; At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units; the plurality of lip reading units generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates; The speech content recognition device, wherein the integration generation unit integrates reliability information included in each of the lip reading processing results of the plurality of lip reading units for each speech content candidate.

4. A speech content recognition device that recognizes the content of a speaker's speech, an input unit for inputting lip image data of a speaker; A plurality of lip reading units having high lip reading accuracy for lip image data captured from corresponding directions; an integration generating unit that integrates the lip reading process results of the plurality of lip reading units for the lip image data input to the input unit and generates a recognition result of the speech content of the speaker based on the integration result; At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units; the plurality of lip reading units generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates; a speech recognition unit that outputs, as a recognition result, speech content candidates of the speaker and reliability information for each of the speech content candidates from the speech data of the speaker; The speech content recognition device is characterized in that the integration generation unit integrates the lip reading processing results of the multiple lip reading units and the recognition results of the voice recognition unit, and generates a recognition result of the speaker's speech content based on the result of the integration.

5. 5. The speech recognition apparatus according to claim 1, The speech content recognition device, wherein the plurality of lip reading units include a unidirectional lip reading unit that corresponds to one direction.

6. 6. The speech recognition apparatus according to claim 1, a data conversion unit that generates data converted based on the lip image data input to the input unit so that the imaging direction corresponds to a corresponding direction of at least one of the plurality of lip reading units; The speech content recognition device according to claim 1, wherein the at least one lip reading unit performs lip reading processing using data converted by the data conversion unit.

7. 5. The speech recognition apparatus according to claim 1, 2 or 4, the plurality of lip reading units generate intermediate information for estimating speech content candidates as a lip reading process result by lip reading processing; The speech content recognition device, wherein the integration / generation unit integrates intermediate information included in each of the lip reading processing results of the plurality of lip reading units.

8. 8. The speech recognition apparatus according to claim 1, The speech content recognition device is characterized in that the multiple lip reading units include a lip reading unit that performs lip reading processing of the lip image data input to the input unit by having a computer execute a machine lip reading model trained using training data including the speaker's lip image data.

9. A method for recognizing a speaker's utterance content by a speech recognition device, comprising: an input step of inputting lip image data of a speaker into the speech recognition device; a lip reading process in which the speech content recognition device performs lip reading processing on the lip image data input in the input process using a plurality of lip reading units having high lip reading accuracy for the lip image data captured from corresponding directions; the speech content recognition device has an integration generating step of integrating the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generating a recognition result of the speech content of the speaker based on the integration result, At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units; The method according to claim 1, wherein the plurality of lip readers includes a multi-directional lip reader having two or more corresponding directions.

10. A method for recognizing a speaker's utterance content by a speech recognition device, comprising: an input step of inputting lip image data of a speaker into the speech recognition device; a lip reading process in which the speech content recognition device performs lip reading processing on the lip image data input in the input process using a plurality of lip reading units having high lip reading accuracy for the lip image data captured from corresponding directions; the speech content recognition device has an integration generating step of integrating the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generating a recognition result of the speech content of the speaker based on the integration result, At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units; an imaging direction estimating step of estimating a plurality of imaging directions of the lip image data inputted in the input step and pieces of certainty information for each imaging direction; The method is characterized in that in the integration / generation step, the lip reading process results are integrated using multiple estimation results from the imaging direction estimation step, and a recognition result of the speaker's speech content is generated based on the integration result.

11. A method for recognizing a speaker's utterance content by a speech recognition device, comprising: an input step of inputting lip image data of a speaker into the speech recognition device; a lip reading process in which the speech content recognition device performs lip reading processing on the lip image data input in the input process using a plurality of lip reading units having high lip reading accuracy for the lip image data captured from corresponding directions; the speech content recognition device has an integration generating step of integrating the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generating a recognition result of the speech content of the speaker based on the integration result, At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units; the plurality of lip reading units generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates; The method according to the present invention, characterized in that in the integrating and generating step, reliability information included in each of the lip reading processing results of the plurality of lip reading units is integrated for each speech content candidate.

12. A method for recognizing a speaker's utterance content by a speech recognition device, comprising: an input step of inputting lip image data of a speaker into the speech recognition device; a lip reading process in which the speech content recognition device performs lip reading processing on the lip image data input in the input process using a plurality of lip reading units having high lip reading accuracy for the lip image data captured from corresponding directions; the speech content recognition device has an integration generating step of integrating the lip reading processing results of the plurality of lip reading units obtained by the lip reading step and generating a recognition result of the speech content of the speaker based on the integration result, At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units; the plurality of lip reading units generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates; a speech recognition step of outputting, as a recognition result, speech content candidates of the speaker and reliability information for each of the speech content candidates from the speech data of the speaker; The method is characterized in that in the integration and generation process, the lip reading processing results of the multiple lip reading units and the recognition results of the voice recognition process are integrated, and a recognition result of the speaker's speech content is generated based on the result of the integration.

13. A program executed on a computer of an utterance content recognition device that recognizes the content of a speaker's utterance, The computer is caused to function as an integration generating means for integrating lip reading results of lip reading processes performed by a plurality of lip reading means having high lip reading accuracy on lip image data captured from corresponding directions, inputted into the speech content recognition device, and generating a recognition result of the speech content of the speaker based on the integration result, At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units, The program, wherein the plurality of lip reading means includes a multi-directional lip reading means having two or more corresponding directions.

14. A program executed on a computer of an utterance content recognition device that recognizes the content of a speaker's utterance, the computer is caused to function as an integration generating means for integrating lip reading results obtained by lip reading lip image data input to the speech content recognition device using a plurality of lip reading means each having a high lip reading accuracy for lip image data captured from corresponding directions, and generating a recognition result for the speech content of the speaker based on the integration result; and an imaging direction estimating means for estimating a plurality of imaging directions of the input lip image data and certainty information for each imaging direction, At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units, The program, characterized in that the integration generation means integrates the lip reading process results using multiple estimation results from the imaging direction estimation means, and generates a recognition result of the speaker's speech content based on the integration result.

15. A program executed on a computer of an utterance content recognition device that recognizes the content of a speaker's utterance, The computer is caused to function as an integration generating means for integrating lip reading results of lip reading processes performed by a plurality of lip reading means having high lip reading accuracy on lip image data captured from corresponding directions, inputted into the speech content recognition device, and generating a recognition result of the speech content of the speaker based on the integration result, At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units, the plurality of lip reading means generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates; The program, wherein the integrating and generating means integrates reliability information included in each lip reading process result of the plurality of lip reading means for each speech content candidate.

16. A program executed on a computer of an utterance content recognition device that recognizes the content of a speaker's utterance, The computer is caused to function as an integration generating means for integrating lip reading results obtained by lip reading lip image data input to the speech content recognition device using a plurality of lip reading means each having high lip reading accuracy for lip image data captured from corresponding directions, and generating a recognition result of the speaker's speech content based on the integration result, and a voice recognition means for outputting, as recognition results, speech content candidates of the speaker and reliability information for each of the speech content candidates from the voice data of the speaker, At least one of the plurality of lip reading units is configured to include a direction in its corresponding direction that is not included in the corresponding direction of any of the other lip reading units, the plurality of lip reading means generate a lip reading process result including one or more speech content candidates estimated by the lip reading process and reliability information for each of the speech content candidates; The program, wherein the integration and generation means integrates the lip reading processing results of the plurality of lip reading means and the recognition results of the voice recognition means, and generates a recognition result of the speaker's speech content based on the result of the integration.

Images