Speech recognition method and device based on multi-view lip movement image

Abstract

The invention discloses a speech recognition method and device based on a multi-view lip movement image. The method comprises the following steps: acquiring to-be-processed image information and angle information; the to-be-processed image information comprises M pieces of lip movement image information; the angle information comprises N lip angles; performing fusion processing on the to-be-processed image information and the angle information to obtain target image result information; the target image result information comprises L pieces of lip movement image result information; l is a positive integer greater than or equal to 1 and less than or equal to N; and based on the target image result information, determining voice recognition result information. Therefore, the voice information can be captured more comprehensively, so that the voice recognition performance in a real scene is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and human-computer interaction, and in particular to a speech recognition method and device based on multi-view lip movement images. Background Art

[0002] In today's digital age, the way people interact with computers is undergoing revolutionary changes, and one of the technologies that has attracted much attention is speech recognition. Speech recognition technology allows computer systems to understand and convert spoken language into actionable text or instructions, thus breaking the limitations of traditional input methods and providing users with a more natural and convenient interactive experience. With the development of deep learning, in addition to traditional audio information input, visual information has also been introduced into the field of speech recognition technology, promoting a type of technology that has attracted much attention - Visual Speech Recognition (Visual Speech Recognition) and its further evolution, namely audio-visual fusion speech recognition (Audio-Visual Speech Recognition).

[0003] VSR focuses on using the visual features of lip movement to achieve lip recognition through computer vision technology. By analyzing information such as mouth shape and lip movement, VSR can understand and recognize speech in the absence of sound. This is of great significance for speech recognition tasks in noisy environments, poor voice quality, or environments that require silence.

[0004] AVSR combines visual speech recognition with audio signals to further improve the performance of speech recognition. Since visual and audio signals are highly complementary, they are often used together as multimodal inputs of the model. When the signal-to-noise ratio of the audio signal is low (i.e., it contains more background noise), the visual context information can play a good complementary and enhancing role, while the audio signal can effectively alleviate the visual ambiguity problem of one-to-many visemes and phonemes. By using both audio and visual information, AVSR can more accurately restore the speaker's voice, especially in noisy environments.

[0005] When using lip reading recognition technology in real scenarios, whether it is VSR or AVSR, most studies only consider frontal or near-frontal lip movement images, that is, it is assumed that the speaker is always facing the camera during human-computer interaction. When the speaker's head posture changes or the lips are not facing the camera, the existing methods may face a series of challenges, such as information loss and serious reduction in recognition accuracy. Therefore, a speech recognition method and device based on multi-view lip movement images are provided to more comprehensively capture speech information and improve speech recognition performance in real scenarios. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a speech recognition method and device based on multi-view lip movement images to more comprehensively capture speech information and improve speech recognition performance in real scenarios.

[0007] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a speech recognition method based on multi-view lip movement images, the method comprising:

[0008] S1, obtaining image information and angle information to be processed; the image information to be processed includes M lip movement image information; the angle information includes N lip angles;

[0009] S2, fusing the image information to be processed and the angle information to obtain target image result information; the target image result information includes L lip movement image result information; L is an integer greater than or equal to 1 and less than or equal to N;

[0010] S3, determining speech recognition result information based on the target image result information.

[0011] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the fusing the image information to be processed and the angle information to obtain the target image result information includes:

[0012] S21, performing fusion processing on the image information to be processed and the angle information to obtain a target angle and target image information; the target image information includes a plurality of lip movement target image information;

[0013] S22, determining target model information based on the angle information;

[0014] S23, determining target image result information based on the target image information, the target angle and the target model information.

[0015] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the image information to be processed and the angle information are fused to obtain the target angle and the target image information, including:

[0016] S211, sorting the N lip angles in the angle information in descending order to obtain pre-processed angle information;

[0017] S212, performing feature extraction on any of the lip movement image information to obtain a first angle corresponding to the lip movement image information;

[0018] S213, determining whether the first angle corresponding to the lip movement image information is greater than the first lip angle in the pre-processed angle information, and obtaining a first determination result;

[0019] When the first judgment result is no, judging whether the first angle is smaller than the Nth lip angle in the preprocessing angle information, and obtaining a second judgment result;

[0020] When the second judgment result is no, determining that the lip movement image information is one of the lip movement target image information;

[0021] Based on the first angle and the pre-processed angle information, determining a second angle corresponding to the lip movement target image information;

[0022] When the second judgment result is no, ending the judgment process corresponding to the second judgment result;

[0023] When the first judgment result is no, ending the judgment process corresponding to the first judgment result;

[0024] S214, classifying the second angles corresponding to all the lip movement target image information in the target image information according to the angle size to obtain angle classification information; the angle classification information includes a plurality of angle classification values; the angle classification value represents the number of the same second angle;

[0025] S215, determining the second angle corresponding to the largest angle classification value as the target angle.

[0026] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the extracting features of any of the lip movement image information to obtain the first angle corresponding to the lip movement image information includes:

[0027] S2121, using a lip movement image calculation model, performing fusion processing on any of the lip movement image information to obtain pre-processed image information corresponding to the lip movement image information;

[0028] Wherein, the lip movement image calculation model is:

[0029]

[0030] Wherein, XS is the pre-processed image information, DXS is the first pixel value, XXS is the second pixel value, HD i is the i-th pixel value corresponding to the lip movement image information, JB i is the local mean of the i-th pixel value corresponding to the lip movement image information, CS is the first correction parameter, BZ i is the local variance of the i-th pixel value corresponding to the lip movement image information, i all is the total number of pixel values ​​of the lip movement graphic information;

[0031] S2122, performing fusion processing on the pre-processed image information corresponding to the lip movement image information to obtain feature information corresponding to the lip movement image information; the feature information includes first coordinate information, second coordinate information and third coordinate information;

[0032] S2123: Determine a first angle corresponding to the lip movement image information based on feature information corresponding to the lip movement image information.

[0033] As an optional implementation manner, in the first aspect of the embodiment of the present invention, determining the target model information based on the angle information includes:

[0034] S221, based on the angle information, obtaining pre-recognition model information and a first information set; the pre-recognition model information includes a lip pre-recognition model; the first information set includes P first information;

[0035] Wherein, the calculation expression of P is:

[0036]

[0037] Wherein, N is the number of lip angles in the angle information, It represents the number of combinations of i elements selected from N elements, 1≤i≤N;

[0038] S222, training the pre-recognition model information and the first information set to obtain target model information; the target model information includes the P lip recognition models.

[0039] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the determining the target image result information based on the target image information, the target angle and the target model information includes:

[0040] S231, acquiring a second information set based on the target angle;

[0041] S232, using any lip recognition model in the target model information, performing calculation processing on the second information set to obtain a recognition value corresponding to the lip recognition model;

[0042] S233, sorting the recognition values ​​of all the lip recognition models to obtain recognition value information;

[0043] S234, determining the lip recognition models corresponding to the front L recognition values ​​in the recognition value information as target recognition model information;

[0044] S235: Determine target image result information based on the target image information and the target recognition model information.

[0045] As an optional implementation manner, in the first aspect of the embodiment of the present invention, determining the target image result information based on the target image information and the target recognition model information includes:

[0046] S2351, using any of the lip recognition models in the target recognition model information, performing speech recognition processing on all the lip movement target image information in the target image information to obtain result information corresponding to the lip recognition model;

[0047] S2352, concatenating the result information corresponding to all lip recognition models in the target recognition model information to obtain target image result information.

[0048] A second aspect of an embodiment of the present invention discloses a speech recognition device based on multi-view lip movement images, characterized in that the device comprises:

[0049] An acquisition module, used to acquire image information to be processed and angle information; the image information to be processed includes M lip movement image information; the angle information includes N lip angles;

[0050] A first calculation module is used to perform a fusion process on the image information to be processed and the angle information to obtain target image result information; the target image result information includes L lip movement image result information; L is a positive integer greater than or equal to 1 and less than or equal to N;

[0051] The second calculation module is used to determine the speech recognition result information based on the target image result information.

[0052] A third aspect of an embodiment of the present invention discloses another speech recognition device based on multi-view lip movement images, characterized in that the device comprises:

[0053] processor;

[0054] a memory coupled to the processor and storing executable program code;

[0055] The processor calls the executable program code stored in the memory to execute part or all of the steps of the speech recognition method based on multi-view lip movement images disclosed in the first aspect of the embodiment of the present invention.

[0056] The fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and when the computer instructions are called, they are used to execute part or all of the steps of the speech recognition method based on multi-view lip movement images disclosed in the first aspect of the embodiment of the present invention.

[0057] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0058] In an embodiment of the present invention, image information to be processed and angle information are obtained; the image information to be processed includes M lip movement image information; the angle information includes N lip angles; the image information to be processed and the angle information are fused to obtain target image result information; the target image result information includes L lip movement image result information; L is a positive integer greater than or equal to 1 and less than or equal to N; based on the target image result information, speech recognition result information is determined. It can be seen that the present application is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 A flowchart of a speech recognition method based on multi-view lip movement images disclosed in an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of the structure of a speech recognition device based on multi-view lip movement images disclosed in an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of the structure of another speech recognition device based on multi-view lip movement images disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] It should be noted that the coordinate information in the present application is the coordinate in a two-dimensional plane coordinate system, that is, the vertex of the lower left corner of the position image information is taken as the origin, the direction from the lower left corner to the lower right corner is the positive direction of the X-axis, and the direction from the lower left corner to the upper left corner is the positive direction of the Y-axis. Each pixel point in the image is in the first quadrant of the coordinate system, and each pixel point corresponds to a coordinate point.

[0067] The present invention discloses a speech recognition method and device based on multi-view lip movement images, which is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenes. The following are detailed descriptions.

[0068] Embodiment 1

[0069] See also Figure 1 , Figure 1 is a flow chart of a method for speech recognition based on multi-view lip movement images disclosed in an embodiment of the present invention. Figure 1 The described speech recognition method based on multi-view lip movement images is applied to a speech recognition system based on multi-view lip movement images, such as a local server or cloud server for data processing and management of speech recognition based on multi-view lip movement images, and the embodiments of the present invention do not limit this. Figure 1 As shown, the speech recognition method based on multi-view lip movement images may include the following operations:

[0070] S1, obtaining image information and angle information to be processed; the image information to be processed includes M lip movement image information; the angle information includes N lip angles;

[0071] Preferably, N is 5, and the angle information includes 0°, 30°, 45°, 60°, and 90°.

[0072] S2, fusing the image information to be processed and the angle information to obtain target image result information; the target image result information includes L lip movement image result information; L is an integer greater than or equal to 1 and less than or equal to N;

[0073] Preferably, L is 3.

[0074] S3, determining speech recognition result information based on the target image result information.

[0075] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0076] In another optional embodiment, the fusing the image information to be processed and the angle information to obtain the target image result information includes:

[0077] S21, performing fusion processing on the image information to be processed and the angle information to obtain a target angle and target image information; the target image information includes a plurality of lip movement target image information;

[0078] S22, determining target model information based on the angle information;

[0079] S23, determining target image result information based on the target image information, the target angle and the target model information.

[0080] It should be noted that the target angle can also be obtained through a 2D convolutional neural network model. For example, 0°, 30°, 45°, 60°, and 90° are several angles set in the angle information. 2D CNN adopts a simple and general architecture, repeatedly applies convolution layers and pooling layers, and then uses several layers of fully connected layers after the Flatten layer to obtain the second angle of the lip movement image information. The convolution layer is the core component of the 2D CNN. It extracts local features by applying convolution kernels to the input data; the pooling layer takes the maximum value in the pooling window to reduce the spatial dimension of the feature map while retaining the most important information; the Flatten layer flattens the output of the convolution layer into a one-dimensional vector to input the fully connected layer; the fully connected layer converts the feature map output by the convolution layer into a one-dimensional vector, and then classifies it through a fully connected operation, and connects multiple fully connected layers after the convolution layer. In addition, a ReLu activation function is added after each convolution layer to introduce nonlinearity to help the network learn more complex patterns; the output layer uses a softmax activation function to generate a category probability distribution.

[0081] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0082] In yet another optional embodiment, the image information to be processed and the angle information are fused to obtain the target angle and target image information, including:

[0083] S211, sorting the N lip angles in the angle information in descending order to obtain pre-processed angle information;

[0084] S212, performing feature extraction on any of the lip movement image information to obtain a first angle corresponding to the lip movement image information;

[0085] S213, determining whether the first angle corresponding to the lip movement image information is greater than the first lip angle in the pre-processed angle information, and obtaining a first determination result;

[0086] When the first judgment result is no, judging whether the first angle is smaller than the Nth lip angle in the preprocessing angle information, and obtaining a second judgment result;

[0087] When the second judgment result is no, determining that the lip movement image information is one of the lip movement target image information;

[0088] Based on the first angle and the pre-processed angle information, determining a second angle corresponding to the lip movement target image information;

[0089] When the second judgment result is no, ending the judgment process corresponding to the second judgment result;

[0090] When the first judgment result is no, ending the judgment process corresponding to the first judgment result;

[0091] S214, classifying the second angles corresponding to all the lip movement target image information in the target image information according to the angle size to obtain angle classification information; the angle classification information includes a plurality of angle classification values; the angle classification value represents the number of the same second angle;

[0092] S215, determining the second angle corresponding to the largest angle classification value as the target angle.

[0093] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0094] In another optional embodiment, the extracting features of any of the lip movement image information to obtain the first angle corresponding to the lip movement image information includes:

[0095] S2121, using a lip movement image calculation model, performing fusion processing on any of the lip movement image information to obtain pre-processed image information corresponding to the lip movement image information;

[0096] Wherein, the lip movement image calculation model is:

[0097]

[0098] Wherein, XS is the pre-processed image information, DXS is the first pixel value, XXS is the second pixel value, HD i is the i-th pixel value corresponding to the lip movement image information, JB i is the local mean of the i-th pixel value corresponding to the lip movement image information, CS is the first correction parameter, BZ i is the local variance of the i-th pixel value corresponding to the lip movement image information, i alL is the total number of pixel values ​​of the lip movement graphic information;

[0099] It should be noted that the difference between the first pixel value and the second pixel value is 100, and the sum of the first pixel value and the second pixel value is 255, so that the pixel difference of the obtained pre-processed image information is extremely large, which is convenient for subsequent feature judgment;

[0100] It should be noted that the first correction parameter may be set by a user or acquired based on historical data.

[0101] S2122, performing fusion processing on the pre-processed image information corresponding to the lip movement image information to obtain feature information corresponding to the lip movement image information; the feature information includes first coordinate information, second coordinate information and third coordinate information;

[0102] It should be noted that the fusion processing of the pre-processed image information corresponding to any of the lip movement image information to obtain the feature information corresponding to the lip movement image information is calculated based on the existing lip recognition model, and the first coordinate information, the second coordinate information and the third coordinate information are the left endpoint coordinates, center point coordinates and right endpoint coordinates of the lips in the lip movement image information obtained after the fusion processing.

[0103] S2123: Determine a first angle corresponding to the lip movement image information based on feature information corresponding to the lip movement image information.

[0104] It should be noted that the first angle can also be calculated through a residual network or a Transformer network architecture.

[0105] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0106] In yet another optional embodiment, determining the first angle corresponding to the lip movement image information based on the feature information corresponding to the lip movement image information includes:

[0107] S21231, calculating and processing the feature information corresponding to the lip movement image information according to the first lip calculation model to obtain first lip information and second lip information;

[0108] Wherein, the first lip calculation model is:

[0109] DY=[X1-X2,Y1-Y2]

[0110] DE=[X2-X3,Y2-Y3]

[0111] Wherein, DY is the first lip information, DE is the second lip information, (X1, Y1), (X2, Y2), (X3, Y3) are the first coordinate information, the second coordinate information and the third coordinate information corresponding to the feature information;

[0112] S21232, calculating and processing the first lip information and the second lip information according to a second lip calculation model to obtain a first angle corresponding to the lip movement image information;

[0113] Wherein, the second lip calculation model is:

[0114] CB=acos(NJ(DY,DE) / (MC(DY)*MC(DE)))

[0115] In the formula, CB is the first angle corresponding to the lip movement image information, DY and DE are the first lip information and the second lip information, NJ(DY,DE) represents the inner product of DY and DE, MC(DY) and MC(DE) are the moduli of the first lip information and the second lip information.

[0116] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0117] In another optional embodiment, determining the second angle corresponding to the lip movement target image information based on the first angle and the pre-processed angle information includes:

[0118] S2131, calculating the absolute value of the first angle and any lip angle in the preprocessed angle information to obtain absolute value information corresponding to the lip angle;

[0119] S2132, comparing the absolute value information corresponding to each of the lip angles in the pre-processed angle information to obtain minimum absolute value information;

[0120] S2133, determining that the lip angle corresponding to the minimum absolute value information is the second angle corresponding to the lip movement target image information.

[0121] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0122] In another optional embodiment, determining the target model information based on the angle information includes:

[0123] S221, based on the angle information, obtaining pre-recognition model information and a first information set; the pre-recognition model information includes a lip pre-recognition model; the first information set includes P first information;

[0124] Wherein, the calculation expression of P is:

[0125]

[0126] Wherein, N is the number of lip angles in the angle information, It represents the number of combinations of i elements selected from N elements, 1≤i≤N;

[0127] It should be noted that, based on the angle information, the pre-recognition model information and the first information set are obtained by combining the lip angles in the angle information to obtain the P combinations, and then selecting the speech recognition training data set of the corresponding lip images through the corresponding lip angles in the P combinations as the first information set. This classification enables each model to focus on the lip images obtained from a specific angle or combination, so as to better capture the specific speech information of the viewing angle. Exemplarily, if one of the P combinations has a lip angle of 0° and 30°, then several lip image information of 0° and several lip image information of 30° are selected as the first information corresponding to the combination of 0° and 30° in the first information set.

[0128] It should be noted that the pre-recognition model information can be set by the user, or obtained based on historical data, or can be a three-dimensional convolutional neural network, a recurrent neural network, or a long short-term memory network.

[0129] S222, training the pre-recognition model information and the first information set to obtain target model information; the target model information includes the P lip recognition models.

[0130] It should be noted that the target model information is obtained by taking each first information in the first information set as input, and continuously training the lip pre-recognition model in the pre-recognition model information until convergence, thereby obtaining a trained model corresponding to the first information, i.e., a lip recognition model. Since there are P first information in the first information set, there are also P lip recognition models finally obtained, each of which corresponds to one of the different lip angle combinations in the angle information.

[0131] It should be noted that the target model information can also be obtained after training the three-dimensional convolutional neural network. For example, an independent three-dimensional convolutional neural network model is created for each lip angle classification, such as a model for 0°, 30°, 45°, 60° and 90°, and a model trained using specific combination data, such as 0°+30°, 0°+30°+45°, 0°+30°+45°+60°+90°, etc. There are 31 in total. This classification enables each model to focus on model training for a specific angle or combination, so as to better capture the specific voice information of the perspective. These 31 models are all separate three-dimensional convolutional neural networks (3D CNN). 3D CNN is a variant of CNN that is specifically designed to process three-dimensional data, such as video data. Unlike 2DCNN, 3D CNN takes into account the time dimension of the data, enabling it to capture features in spatiotemporal sequences such as videos. In 3DCNN, convolution operations occur in three dimensions: width, height, and time. This enables the network to learn spatiotemporal correlations in data such as videos and better capture the motion and evolution of objects. Similar to 2D CNNs, 3D CNNs typically include convolutional layers, pooling layers, and fully connected layers.

[0132] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0133] In yet another optional embodiment, determining the target image result information based on the target image information, the target angle and the target model information includes:

[0134] S231, acquiring a second information set based on the target angle;

[0135] S232, using any lip recognition model in the target model information, performing calculation processing on the second information set to obtain a recognition value corresponding to the lip recognition model;

[0136] S233, sorting the recognition values ​​of all the lip recognition models to obtain recognition value information;

[0137] S234, determining the lip recognition models corresponding to the front L recognition values ​​in the recognition value information as target recognition model information;

[0138] S235: Determine target image result information based on the target image information and the target recognition model information.

[0139] It should be noted that, based on the target angle, the second information set is obtained by first determining the target angle, and then obtaining the speech recognition verification set of the lip image information at the target angle as the second information set, so as to verify the accuracy of the speech recognition of the lip image information by the lip recognition model. Exemplarily, if the target angle is 0°, the speech recognition verification set of 0° is selected as the second information set.

[0140] It should be noted that, by using any lip recognition model in the target model information to calculate and process the second information set to obtain the recognition value corresponding to the lip recognition model, the second information set is used as the input of the lip recognition model, and the lip recognition model is used to calculate it to obtain the recognition value of the corresponding lip recognition model. Since there are P lip recognition models, the final recognition value also includes P.

[0141] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0142] In another optional embodiment, determining the target image result information based on the target image information and the target recognition model information includes:

[0143] S2351, using any of the lip recognition models in the target recognition model information, performing speech recognition processing on all the lip movement target image information in the target image information to obtain result information corresponding to the lip recognition model;

[0144] S2352, concatenating the result information corresponding to all lip recognition models in the target recognition model information to obtain target image result information.

[0145] It should be noted that, by using any of the lip recognition models in the target recognition model information to perform speech recognition processing on the target image information to obtain result information corresponding to the lip recognition model, the target image information is taken as input, and the input is calculated and processed by the lip recognition model in the target recognition model information to obtain target image result information corresponding to the target image information.

[0146] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0147] It should be noted that based on the target image result information, the speech recognition result information is determined by taking the target image result information as input, using the lip recognition model as a training model, and using two fully connected layers to output the speech recognition result information. When using a three-dimensional convolutional neural network, 50% is set as the loss between the fully connected layers.

[0148] It can be seen that implementing the speech recognition method based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0149] Embodiment 2

[0150] See also Figure 2 , Figure 2 is a structural schematic diagram of a speech recognition device based on multi-view lip movement images disclosed in an embodiment of the present invention. Figure 2 The described speech recognition device based on multi-view lip movement images is applied to a speech recognition system based on multi-view lip movement images, such as a local server or cloud server for data processing and management of speech recognition based on multi-view lip movement images, and the embodiments of the present invention do not limit this. Figure 2 As shown, the speech recognition device based on multi-view lip movement images includes:

[0151] The acquisition module 201 is used to acquire image information to be processed and angle information; the image information to be processed includes M lip movement image information; the angle information includes N lip angles;

[0152] The first calculation module 202 is used to perform a fusion process on the image information to be processed and the angle information to obtain target image result information; the target image result information includes L lip movement image result information; L is a positive integer greater than or equal to 1 and less than or equal to N;

[0153] The second calculation module 203 is used to determine the speech recognition result information based on the target image result information.

[0154] It can be seen that implementing the speech recognition device based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0155] Embodiment 3

[0156] See also Figure 3 , Figure 3 is a schematic diagram of the structure of another speech recognition device based on multi-view lip movement images disclosed in an embodiment of the present invention. Figure 3 The described speech recognition device based on multi-view lip movement images is applied to a speech recognition system based on multi-view lip movement images, such as a local server or cloud server for data processing and management of speech recognition based on multi-view lip movement images, and the embodiments of the present invention do not limit this. Figure 3 As shown, the speech recognition device based on multi-view lip movement images includes:

[0157] Processor 301;

[0158] A memory 302 coupled to the processor 301 and storing executable program codes;

[0159] The processor 301 calls the executable program code stored in the memory 302 to execute all or part of the steps in the speech recognition method based on multi-view lip movement images described in the first embodiment.

[0160] It can be seen that implementing the speech recognition device based on multi-view lip movement images described in the embodiment of the present invention is conducive to more comprehensive capture of speech information to improve speech recognition performance in real scenarios.

[0161] Embodiment 4

[0162] An embodiment of the present invention discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and when the computer instructions are called, they are used to execute all or part of the steps in the speech recognition method based on multi-view lip movement images described in Example 1.

[0163] Embodiment 5

[0164] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps of the speech recognition method based on multi-view lip movement images described in Example 1.

[0165] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0166] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0167] Finally, it should be noted that the speech recognition method and device based on multi-view lip movement images disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method based on multi-view lip movement images, characterized in that: The method comprises: S1, obtaining image information and angle information to be processed; the image information to be processed includes M lip movement image information; the angle information includes N lip angles; S2, fusing the image information to be processed and the angle information to obtain target image result information; the target image result information includes L lip movement image result information; L is an integer greater than or equal to 1 and less than or equal to N; S3, determining speech recognition result information based on the target image result information.

2. The method for speech recognition based on multi-view lip movement images according to claim 1, characterized in that: The fusing the image information to be processed and the angle information to obtain target image result information includes: S21, performing fusion processing on the image information to be processed and the angle information to obtain a target angle and target image information; the target image information includes a plurality of lip movement target image information; S22, determining target model information based on the angle information; S23, determining target image result information based on the target image information, the target angle and the target model information.

3. The method for speech recognition based on multi-view lip movement images according to claim 2, characterized in that: The image information to be processed and the angle information are fused to obtain target angle and target image information, including: S211, sorting the N lip angles in the angle information in descending order to obtain pre-processed angle information; S212, performing feature extraction on any of the lip movement image information to obtain a first angle corresponding to the lip movement image information; S213, determining whether the first angle corresponding to the lip movement image information is greater than the first lip angle in the pre-processed angle information, and obtaining a first determination result; When the first judgment result is no, judging whether the first angle is smaller than the Nth lip angle in the preprocessing angle information, and obtaining a second judgment result; When the second judgment result is no, determining that the lip movement image information is one of the lip movement target image information; Based on the first angle and the pre-processed angle information, determining a second angle corresponding to the lip movement target image information; When the second judgment result is no, ending the judgment process corresponding to the second judgment result; When the first judgment result is no, ending the judgment process corresponding to the first judgment result; S214, classifying the second angles corresponding to all the lip movement target image information in the target image information according to the angle size to obtain angle classification information; the angle classification information includes a plurality of angle classification values; the angle classification value represents the number of the same second angle; S215, determining the second angle corresponding to the largest angle classification value as the target angle.

4. The method for speech recognition based on multi-view lip movement images according to claim 3, characterized in that: The step of extracting features from any of the lip movement image information to obtain a first angle corresponding to the lip movement image information includes: S2121, using a lip movement image calculation model, performing fusion processing on any of the lip movement image information to obtain pre-processed image information corresponding to the lip movement image information; Wherein, the lip movement image calculation model is: Wherein, XS is the pre-processed image information, DXS is the first pixel value, XXS is the second pixel value, HD i is the i-th pixel value corresponding to the lip movement image information, JB i is the local mean of the i-th pixel value corresponding to the lip movement image information, CS is the first correction parameter, BZ i is the local variance of the i-th pixel value corresponding to the lip movement image information, i all is the total number of pixel values ​​of the lip movement graphic information; S2122, performing fusion processing on the pre-processed image information corresponding to the lip movement image information to obtain feature information corresponding to the lip movement image information; the feature information includes first coordinate information, second coordinate information and third coordinate information; S2123: Determine a first angle corresponding to the lip movement image information based on feature information corresponding to the lip movement image information.

5. The method for speech recognition based on multi-view lip movement images according to claim 2, characterized in that: The determining of target model information based on the angle information includes: S221, based on the angle information, obtaining pre-recognition model information and a first information set; the pre-recognition model information includes a lip pre-recognition model; the first information set includes P first information; Wherein, the calculation expression of P is: Wherein, N is the number of lip angles in the angle information, It represents the number of combinations of i elements selected from N elements, 1≤i≤N; S222, training the pre-recognition model information and the first information set to obtain target model information; the target model information includes the P lip recognition models.

6. The method for speech recognition based on multi-view lip movement images according to claim 2, characterized in that: The determining target image result information based on the target image information, the target angle and the target model information includes: S231, acquiring a second information set based on the target angle; S232, using any lip recognition model in the target model information, performing calculation processing on the second information set to obtain a recognition value corresponding to the lip recognition model; S233, sorting the recognition values ​​of all the lip recognition models to obtain recognition value information; S234, determining the lip recognition models corresponding to the front L recognition values ​​in the recognition value information as target recognition model information; S235: Determine target image result information based on the target image information and the target recognition model information.

7. The method for speech recognition based on multi-view lip movement images according to claim 6, characterized in that: The determining target image result information based on the target image information and the target recognition model information includes: S2351, using any of the lip recognition models in the target recognition model information, performing speech recognition processing on all the lip movement target image information in the target image information to obtain result information corresponding to the lip recognition model; S2352, concatenating the result information corresponding to all lip recognition models in the target recognition model information to obtain target image result information.

8. A speech recognition device based on multi-view lip movement images, characterized in that: The device comprises: An acquisition module, used to acquire image information to be processed and angle information; the image information to be processed includes M lip movement image information; the angle information includes N lip angles; A first calculation module is used to perform a fusion process on the image information to be processed and the angle information to obtain target image result information; the target image result information includes L lip movement image result information; L is a positive integer greater than or equal to 1 and less than or equal to N; The second calculation module is used to determine the speech recognition result information based on the target image result information.

9. A speech recognition device based on multi-view lip movement images, characterized in that: The device comprises: processor; a memory coupled to the processor and storing executable program code; The processor calls the executable program code stored in the memory to execute the speech recognition method based on multi-view lip movement images as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to execute the speech recognition method based on multi-view lip movement images as described in any one of claims 1-7.

Images