A cross-modal speech recognition method

Abstract

The application relates to the technical field of speech recognition, and discloses a cross-modal speech recognition method. The method comprises the following steps: obtaining video information to be analyzed and extracting call audio and visual information therefrom; obtaining a speech frame sequence; performing decoding operation on the speech frame sequence by using a speech recognition model to obtain corresponding text information; performing multiple feature extraction operation on the visual information to obtain a feature sequence of the visual information; obtaining a face information sequence by using a preset visual information extraction model; performing extraction analysis on the face information sequence by using a preset target detection model, so that further optimization is performed; performing decoding operation on the optimized lip sequence by using a trained lip movement conversion model to obtain a candidate word set; and comparing and fusing by using a trained fusion neural model to output final text information. The application can solve the problem that current speech recognition is prone to interference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a cross-modal speech recognition method. BACKGROUND

[0002] Speech is the most intuitive way to communicate in daily life, and through speech, humans can obtain a lot of information. However, in a noisy environment, there are multiple sound sources and background noise. Although the automatic speech recognition system has basically reached the same level as human perception, it is still susceptible to noise at the current stage, and when the signal source of speech information is destroyed, the system performance will be affected. SUMMARY

[0003] The present application provides a cross-modal speech recognition method to solve at least one of the above technical problems.

[0004] The present application provides a cross-modal speech recognition method, comprising the following steps:

[0005] Obtain video information to be analyzed and extract the call audio therefrom, perform a rectification and filtering operation on the call audio to obtain standard speech;

[0006] Frame processing is performed on the video information to obtain visual information;

[0007] Frame processing and endpoint detection are performed on the standard speech to obtain a speech frame sequence;

[0008] The speech frame sequence is decoded using a speech recognition model to obtain first text information corresponding to the speech frame sequence;

[0009] Multiple feature extraction operations are performed on the visual information to obtain a clear face feature sequence of the visual information;

[0010] The clear face feature sequence is extracted and analyzed using a preset target detection model to obtain a preliminary lip information sequence;

[0011] The preliminary lip information sequence is optimized using a generative adversarial network to obtain a corrected lip information sequence;

[0012] The corrected lip information sequence is decoded using a trained lip movement conversion model to obtain a second candidate word set corresponding to the corrected lip information sequence;

[0013] The first text information corresponding to the speech frame sequence and the second candidate word set of the corrected lip information sequence are compared and fused using a trained fusion neural model to output final text information.

[0014] Optionally, the step of obtaining the video information to be analyzed and extracting the conversation audio therefrom, rectifying and filtering the conversation audio to obtain the standard speech, comprises the following steps:

[0015] Converting the conversation audio into a sequence of speech frames, and configuring a sequence of key frames of the conversation audio according to the sequence of speech frames and a preset sampling frequency;

[0016] According to the sequence of key frames and the trend item of the sequence of speech frames, the sequence of key frames is removed from the sequence of signal data to obtain the rectified standard speech;

[0017] Optionally, the step of decoding the sequence of speech frames by using the speech recognition model to obtain the first text information corresponding to the sequence of speech frames comprises the following steps:

[0018] Decoding the sequence of speech frames into a sequence of candidate word sets by using a self-attention network in the speech recognition model;

[0019] Selecting a number of candidate words with higher weights in the sequence of candidate word sets to form a first candidate word set;

[0020] Selecting a candidate word with the largest weight in the sequence of the first candidate word set as a target candidate word, and arranging the target candidate word in a time sequence to form the first text information;

[0021] Optionally, the step of performing multi-feature extraction on the visual information to obtain a first feature sequence and a second feature sequence corresponding to the visual information comprises:

[0022] Extracting the first feature sequence from the visual information according to a preset face feature set;

[0023] Converting the first feature sequence into a clear and complete second feature sequence according to a preset lip feature set.

[0024] The step of performing local feature extraction on the second feature sequence by using a preset lip recognition model to obtain a preliminary lip information sequence comprises:

[0025] Encoding the second feature sequence into a state feature encoding sequence by using an encoder of a preset self-attention model;

[0026] Decoding a lip state sequence from the state feature encoding sequence by using a decoder of the attention model;

[0027] Performing weight and bias processing on the lip state sequence by using a multi-layer perceptron of the attention model to obtain a corresponding preliminary lip information sequence.

[0028] Optionally, the step of performing feature enhancement processing on the first lip sequence by using a preset image enhancement model to obtain a second lip sequence includes the following steps:

[0029] The edge information, contour information and contrast of the first lip sequence are highlighted or enhanced by using a preset image enhancement model, including but not limited to using image gray scale transformation, histogram correction and pseudo-color enhancement technology, thereby better displaying the second lip information sequence of the lip information.

[0030] Optionally, the step of decoding the second lip information sequence to obtain the corresponding second candidate word set of the lip information sequence includes the following steps:

[0031] The feature sequence in the step of extracting the preliminary lip information sequence is subjected to correlation calculation and feature weighting by using a time domain self-attention mechanism to obtain a second candidate word set sequence.

[0032] The second candidate word set sequence is decoded and classified by using a bidirectional GRU and a full connection layer to obtain the corresponding second candidate word set of the second lip information sequence.

[0033] Optionally, the step of comparing and fusing the first text information corresponding to the speech frame sequence and the second candidate word set corresponding to the lip information sequence by using a trained fusion neural model to output the final text information includes the following steps:

[0034] The first text information and the second candidate word set are subjected to weighting processing by using a deep neural network model and adding a residual connection between the hierarchical structures.

[0035] The preset fusion neural model is used to compare the weighted words, and the words in the corresponding first text information with a weight lower than the second candidate word set are replaced to output the final text information.

[0036] Compared with the prior art, the technical principles and beneficial effects of the present scheme are as follows:

[0037] The embodiment of the present application first acquires video information, extracts call audio and visual information from the video information, and uses the call audio and visual information to identify speech information from the video information according to the speech information features and the lip signal features.

[0038] Further, in the embodiment of the present application, the standard speech is subjected to frame processing and endpoint detection to obtain a speech frame sequence, thereby reducing the workload of the central processing unit when decoding the speech frame sequence and reducing the occurrence of errors.

[0039] Further, in the embodiment of the present application, the speech frame sequence is decoded to obtain the first text information corresponding to the speech frame sequence, so that the text information can be output when the video information is lost.

[0040] Further, in the embodiment of the present application, the visual information is subjected to multi-feature extraction to obtain the clear face feature sequence of the visual information, so as to improve the accuracy of local feature extraction of the face information and the interference of some background figures in the environment.

[0041] Further, in the embodiment of the present application, the face information sequence is subjected to extraction analysis to obtain the preliminary lip information sequence, so as to improve the accuracy and integrity of lip recognition.

[0042] Further, in the embodiment of the present application, the preliminary lip information sequence is optimized to obtain the second lip information sequence, so as to prevent the influence of some abnormal states such as lip information tilt and incompleteness on the recognition result.

[0043] Further, in the embodiment of the present application, the second feature sequence is subjected to decoding to obtain the second candidate word set corresponding to the second feature sequence, so as to reduce the workload and error rate when the first text information is modified subsequently.

[0044] Further, in the embodiment of the present application, the trained fusion neural model is used for comparison and fusion to output the final text information, so as to solve the bad influence of video call, noise or missing sound information on the call quality.

[0045] In the embodiment of the present application, the face living body detection is performed by using the following algorithm:

[0046]

[0047] Wherein, M is the number of the current key frame sequence, e is the natural logarithm, x is the number of frames in which the face changes in the key frame, and F(x, M) is the logarithmic dynamic value of face living body detection change.

[0048] Wherein, l is the value of the contrast of the current frame sequence, h is the exposure compensation value, g is the saturation, z is the color temperature of the photo, and n is the number of frames constituting the current word. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 A flowchart of a cross-modal speech recognition method provided by an embodiment of the present application is shown in the figure;

[0050] Figure 2 A flowchart of generating the first text information provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are merely exemplary and do not limit the application.

[0052] The cross-modal speech recognition method provided by the embodiments of the application can be executed by at least one of a server, a terminal, and the like, which can be configured to execute the method provided by the embodiments of the application.

[0053] In other words, the method can be executed by software or hardware installed in a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster, and the like. The server can be a stand-alone server, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, and the like.

[0054] Referring to Figure 1 and Figure 2 , a flowchart of a cross-modal speech semantic analysis and recognition method provided by an embodiment of the application is shown. In this embodiment, the cross-modal speech recognition method includes the following steps:

[0055] In step S1, video information to be analyzed is obtained, and the call audio in the video information is extracted. The call audio is subjected to a correction and filtering operation to obtain standard speech.

[0056] In the embodiments of the application, the video information refers to complete screen recording information in a user video call.

[0057] In the embodiments of the application, the correction and filtering operation on the call recording to obtain standard speech includes:

[0058] The call recording is converted into a sequence of speech frames, and a sequence of interference key frames is removed from the sequence of speech frames according to the sequence of speech frames and a preset sampling frequency configuration.

[0059] According to the sequence of speech frames and using a filter, interference waveband frequencies in the initial speech are removed and eliminated from the sequence of speech frames to obtain corresponding standard speech.

[0060] In detail, the interference key frame sequence refers to the process that the vibration signal data collected in the vibration test deviates from the baseline due to the zero drift of the amplifier with temperature change, the instability of the low frequency performance outside the frequency range of the sensor, and the environmental interference around the sensor, etc. Through the operation, the influence of noise in the environment on speech recognition can be reduced, thereby improving the quality of the output text.

[0061] In step S2, the video information is frame-processed to obtain visual information.

[0062] In detail, the video is divided into several overlapping segments, so that the features of each frame are extracted and weighted by using the Imagenet pre-trained VGG-16, so as to be sorted according to the average of the weights of all frames in each segment, and finally the frames with the largest weights are selected from the segments with higher rankings.

[0063] It should be understood that when the lips of a person are in a closed state for a long time, the text information generated by the decoding operation on the lip information has no actual meaning or has a great impact on the generated text fusion.

[0064] In step S4, the speech recognition model is used to decode the speech frame sequence to obtain first text information corresponding to the speech frame sequence, including:

[0065] In step S21, the self-attention network in the speech recognition model is used to decode the candidate word set sequence corresponding to the speech frame sequence.

[0066] In step S22, a plurality of candidate words with higher weights in the candidate word set sequence are selected to form a first candidate word set.

[0067] In step S22, the candidate word with the largest weight in the first candidate word set sequence is selected as a target candidate word, and the target candidate word is arranged in time sequence to form first text information.

[0068] In detail, each frame waveform is converted into a multi-dimensional vector containing sound information by using the linear predictive cepstrum coefficient (LPCC) and the Mel cepstrum coefficient (MFCC) in the self-attention network in the speech recognition model, so as to be matched with the dictionary of the speech recognition model, and the audio data after feature extraction is output in text.

[0069] It should be understood that directly outputting the candidate words to form the first text information can easily cause the situation that the beginning does not match the end. The coding by the self-attention network can greatly improve the relevance of the text before and after the sentence.

[0070] Step S5, the step of performing multi-feature extraction on the visual information to obtain a clear face feature sequence corresponding to the visual information in the embodiment of the application is specifically:

[0071] extracting a preliminary face feature sequence from the visual information according to a preset face feature set;

[0072] performing live face detection on the preliminary face feature sequence by using a face live detection algorithm, removing the interference face of the preliminary face feature sequence, and obtaining a preliminary live face feature sequence;

[0073] converting the preliminary live face feature sequence into a clear face feature sequence according to a preset image enhancement technology;

[0074] In detail, the image and label of the face image data are read, and grayscale processing and histogram equalization are performed to convert the two-dimensional image data information into a one-dimensional vector, and then the columns are combined into an original data matrix. The original matrix is normalized, and PCA or LDA algorithm is used to analyze and reduce the dimension of the original data matrix, so as to read the image to be recognized and convert it into the same vector representation as in the training set. The training set is traversed to find an image whose difference with the image to be recognized is less than a threshold (or the minimum difference), which is taken as the recognition result (i.e. the first feature sequence).

[0075] It can be understood that the live detection of the face can effectively prevent the interference of some abnormal lip information in the process of video call of billboards and the like;

[0076] In detail, the recognition result is processed by one or more combinations of data enhancement transformation modes such as cutting, rotation / reflection / flip transformation, scaling transformation, translation transformation, scale transformation, contrast transformation, noise disturbance, color transformation, etc. to obtain a clearer and more complete face image for the next step of processing.

[0077] Step S6, in the embodiment of the application, a preset lip recognition model is used to perform local feature extraction on the second feature sequence to obtain a preliminary lip information sequence.

[0078] The second feature sequence is encoded into a state feature encoding sequence by using an encoder of a preset self-attention model;

[0079] A decoder of the attention model decodes a lip state sequence from the state feature encoding sequence;

[0080] The lip state sequence is processed by a multi-layer perceptron of the attention model to obtain a corresponding preliminary lip information sequence.

[0081] Step S7, in the embodiment of the application, the step of using the preset image enhancement model to perform feature enhancement processing on the first lip sequence to obtain the corrected lip sequence includes:

[0082] The edge information, contour information and contrast of the first lip sequence are highlighted or enhanced using the preset image enhancement model, so that the corrected lip information sequence is obtained.

[0083] In detail, the spatial domain image enhancement technology is adopted, the histogram of the gray image is obtained by the function imhist(), and the gray value of the pixel point is changed point by point without changing the pixel position in the original image. At the same time, the function imadjust() can also analyze and supplement the incomplete lip sequence, so that the pixels in the lip region are more clear.

[0084] It can be understood that analyzing and supplementing the incomplete lip sequence means that the incomplete lip caused by human turning and the actual lip caused by shooting is supplemented through algorithm and advanced experience.

[0085] Step S8, in the embodiment of the application, the step of decoding the second lip information sequence to obtain the second candidate word set corresponding to the lip information sequence includes:

[0086] The time domain self-attention mechanism is used to calculate the correlation and weight the feature sequence in the step of extracting the preliminary lip information sequence, so as to obtain the second candidate word set sequence;

[0087] The bidirectional GRU and the full connection layer are used to decode and classify the second candidate word set sequence, so as to obtain the second candidate word set corresponding to the second lip information sequence.

[0088] It can be understood that the second lip information sequence does not directly generate the corresponding text information, which is beneficial to reduce the workload of text replacement in the later stage, but the step can still form the corresponding text information. In order to reduce the workload of the processor, the step is omitted.

[0089] Step S9, in the embodiment of the application, the step of using the trained fusion neural model, the first text information corresponding to the speech frame sequence and the second candidate word set corresponding to the lip information sequence to compare and fuse, and outputting the final text information includes:

[0090] The deep neural network model is used to add residual connection between the hierarchical structures to weight process the first text information and the second candidate word set.

[0091] The preset fusion neural model is used to compare the words subjected to the weighting processing, the words with a weight lower than the second candidate word set in the corresponding first text information are replaced, and finally the text information is output.

[0092] Further, in the embodiment of the present application, the lip information sequence is decoded by the preset lip recognition model to obtain a candidate word set corresponding to the lip sequence, and the acoustic model is combined to analyze the text, and the word with the highest accuracy among the language model, the acoustic model and the lip model is selected as the output word of the text information, so that the accuracy of speech recognition is improved.

[0093] It should be understood that when some words in the first text information are missing, spaces are used to replace them, at this time, the candidate words in the lip information sequence will become the replacement of the words, and when the first text information is largely missing, the candidate word set corresponding to the lip sequence will also be spliced by using the self-attention model; similarly, when the lip information sequence is missing, the first text information can also be output as the final text.

[0094] In the embodiment of the present application, face liveness detection is performed by using the following algorithm:

[0095]

[0096] Wherein, M is the number of the current key frame sequence, e is the natural logarithm, x is the number of frames in which the face changes in the key frame, and F(x,M) is the face liveness detection change logarithmic dynamic value;

[0097] It should be understood that a threshold value should be set for the face liveness detection change logarithmic dynamic value, when the face liveness detection change logarithmic dynamic value F(x,M) is understood as the face information that has an influence on lip recognition;

[0098] It should be understood that when the angle of the camera device changes, some parameters such as the proportion of static faces may change accordingly, which may have a corresponding influence on the subsequent lip information recognition and may have an influence on the final text information, thereby affecting the user's judgment of the current call content;

[0099] In the embodiment of the present application, the following formula is used for feature weighting:

[0100]

[0101] Wherein, l is the contrast value of the current frame sequence, h is the exposure compensation value, g is the saturation, z is the color temperature of the photo, and n is the number of frames constituting the current word.

[0102] It should be understood that sometimes in the conversion of the lip information of a single word and the text, some words may have an impact, and some unnecessary lip subtle movements may also have an impact on the output text, so that the weight processing of the lip information effectively increases the quality of the corresponding candidate word set.

[0103] The embodiment of the application can remove unstable factors in the call recording, reduce noise and noise in the call recording, improve the accuracy of speech semantic analysis, obtain a speech frame sequence through frame processing and endpoint detection on the standard speech, and conveniently split the standard speech into morphemes, thereby improving the accuracy during decoding, and further improving the correctness of the corresponding output text when the lip information is missing.

[0104] The embodiment of the application first obtains video information, extracts call audio and visual information from the video information, and then identifies speech information from the video information according to the speech information features and the lip signal features.

[0105] For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0106] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any additional reference signs in the claims should not be considered as limiting the claims involved.

[0107] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0108] Furthermore, the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural and vice-versa, unless the context clearly requires these exclusions. The conjunction "or" is used to link items in a list or a set of alternatives, and is not disjunctive, unless the context clearly requires it to be disjunctive. The conjunction "and" is used to link items in a list or a set of alternatives, and is not conjunctive, unless the context clearly requires it to be conjunctive. The prefix "first", "second", "third", etc. is used to identify similar entities of a different class or a different category, and is not a requirement of temporal priority, unless the context clearly requires it to be a requirement of temporal priority. The suffix "er", "or", "ian", etc. is used to identify an entity of a different class or a different category, and is not a requirement of membership of a particular group, unless the context clearly requires it to be a requirement of membership of a particular group.

[0109] Finally, it should be noted that the above-mentioned embodiments illustrate rather than limit the application, since the scope of the application is determined by the appended claims.

Claims

1. A cross-modal speech recognition method, characterized in that, The method includes: Step S1: Obtain the video information to be analyzed and extract the call audio from it. Perform correction and filtering operations on the call audio to obtain standard speech. Step S2: Perform frame segmentation on the video information to obtain visual information; Step S3: Perform frame segmentation and endpoint detection on the standard speech to obtain a speech frame sequence; Step S4: Use a speech recognition model to decode the speech frame sequence to obtain the first text information corresponding to the speech frame sequence; Step S5: Perform multiple feature extraction operations on the visual information to obtain a clear facial feature sequence of the visual information; the specific steps of performing multiple feature extraction operations on the visual information to obtain the clear facial feature sequence corresponding to the visual information are as follows: Preliminary facial feature sequences are extracted from the visual information based on a preset facial feature set; Using a face liveness detection algorithm, liveness detection is performed on the preliminary facial feature sequence to remove interfering faces from the preliminary facial feature sequence, thus obtaining a preliminary liveness facial feature sequence. According to the preset image enhancement technology, the preliminary live facial feature sequence is converted into a clear facial feature sequence; Step S6: Use a preset target detection model to extract and analyze the clear facial feature sequence to obtain a preliminary lip information sequence; Step S7: Use a preset image enhancement model to highlight or enhance the edge information, contour information and contrast of the preliminary lip information sequence, thereby obtaining the corrected lip information sequence; Step S8: Use the trained lip-reading conversion model to decode the corrected lip information sequence to obtain the second candidate word set corresponding to the corrected lip information sequence; Step S9: For the first text information corresponding to the speech frame sequence and the second candidate word set corresponding to the corrected lip information sequence, the trained fusion neural model is used to compare and fuse them to output the final text information; the steps of comparing and fusing the first text information corresponding to the speech frame sequence and the second candidate word set corresponding to the lip information sequence using the trained fusion neural model to output the final text information include: By using a deep neural network model, residual connections are added between hierarchical structures, and a fusion connection is established between the first text information and the second candidate word set. Using a pre-defined fusion neural model, the error between the text before and after fusion is calculated. Phrases with errors reaching a threshold are replaced accordingly to output the final text information. The following algorithm is used for face liveness detection: Where M is the number of current keyframe sequences, e is the natural logarithm, x is the number of keyframes that generate change frames, and F(x,M) is the dynamic value of face liveness detection change. Feature weighting is performed using the following formula: Where l is the contrast value of the current frame sequence, h is the exposure compensation value, g is the saturation, z is the color temperature of the photo, and n is the number of frames that make up the current word.

2. The cross-modal speech recognition method as described in claim 1, characterized in that, Step S1 is as follows: The call audio is converted into a column speech frame sequence, and interference key frame sequences are removed from the column speech frame sequence and the call audio according to the column speech frame sequence and the preset sampling frequency configuration. Based on the series of voice frames, the interference keyframe sequence using filters, and the removal of interference band frequencies from the call audio in the voice frame sequence, the corresponding standard voice is obtained.

3. The cross-modal speech recognition method as described in claim 1, characterized in that, The specific steps for decoding the speech frame sequence using a speech recognition model to obtain the first text information corresponding to the speech frame sequence are as follows: The candidate word set sequence corresponding to the speech frame sequence is decoded by the self-attention network in the speech recognition model; Select several candidate words with higher weights from the candidate word set sequence to form a first candidate word set; The candidate word with the highest weight in the first candidate word set sequence is selected as the target candidate word, and the target candidate words are arranged in chronological order to form the first text information.

4. The cross-modal speech recognition method as described in claim 1, characterized in that, The steps of extracting local features from the clear facial feature sequence using a pre-defined lip recognition model to obtain the preliminary lip information sequence include: The feature sequence of the second candidate word set is encoded into a state feature encoding sequence using an encoder based on a pre-defined self-attention model; The lip state sequence is decoded from the state feature encoding sequence using the decoder of the attention model; The attention model is used to apply weights and biases to the lip state sequence to obtain a preliminary lip information sequence.

5. The cross-modal speech recognition method as described in claim 1, characterized in that, The step of decoding the corrected lip information sequence to obtain the second candidate word set corresponding to the lip information sequence specifically includes: Using a temporal self-attention mechanism, the feature sequences in the step of extracting the corrected lip information sequence are relevance calculated and feature weighted to obtain the second candidate word set sequence; The second candidate word set sequence is decoded and classified using a bidirectional GRU and a fully connected layer, and word groups with weights below a threshold are removed from the second candidate word set sequence to obtain the second candidate word set corresponding to the corrected lip information sequence.

Images