Speech data processing method and device, model training method and device, and storage medium

Abstract

The application discloses a speech data processing method and device, a model training method and a storage medium. The speech data processing method and device, the model training method and the storage medium fuse phoneme features in speech data and visual features in video image data to perform speech recognition. The fusion of the visual feature mode enriches the features used for speech recognition, and effectively improves the accuracy of speech recognition. In addition, the application not only extracts each video frame and audio frame, but also fuses and splices the first phoneme feature information and the first visual feature information after feature classification by a first full connection layer and a second full connection layer, so that the phoneme feature information in the speech data and the visual feature in the video data can be more fully fused, and the accuracy of speech recognition is further improved. The speech data processing method and device, the model training method and the storage medium provided in the application can be applied to various scenes such as video playing, live broadcast, video conference, cloud technology, artificial intelligence, intelligent transportation and Internet of Vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio technology, and in particular to a speech data processing method and device, a model training method and device, and a storage medium.

[0002] BACKGROUND

[0003] Speech interaction is a very important way of communication in the process of people communicating with each other, and speech content contains a large amount of information in the process of people communicating with each other. Therefore, how to accurately extract speech content in speech has always been a popular research field. With the development of artificial intelligence neural networks and machine learning algorithms, acoustic models based on deep learning are gradually adopted in speech recognition. Speech recognition technology has made significant progress in recent years and has been widely used in industries, communications, medical treatment and other fields. However, the current speech recognition technology still has the problem of low recognition accuracy. SUMMARY

[0004] To solve at least one of the above technical problems, the present application provides a speech data processing method, a model training method, a text information extraction device, a model training device and a computer readable storage medium, which can improve the accuracy of speech recognition.

[0005] According to a first aspect of the present application, a speech data processing method is provided, comprising the following steps:

[0006] obtaining speech data and video image data of a sound producing object, wherein the speech data comprises a plurality of audio frames, and the video image data comprises a plurality of image frames;

[0007] extracting phoneme features from each audio frame in the speech data to obtain a plurality of speech phoneme features, and splicing the plurality of speech phoneme features to obtain first phoneme feature information;

[0008] inputting the first phoneme feature information into a first fully connected layer for feature classification to obtain second phoneme feature information;

[0009] extracting visual features from each image frame in the video image data to obtain a plurality of visual features, and splicing the plurality of visual features to obtain first visual feature information;

[0010] inputting the first visual feature information into a second fully connected layer for feature classification to obtain second visual feature information;

[0011] splicing the second phoneme feature information and the second visual feature information to obtain fusion feature information;

[0012] performing speech recognition on the fusion feature information to obtain speech content information corresponding to the speech data.

[0013] According to a second aspect of the present application, a model training method is provided, comprising the following steps:

[0014] obtaining training speech data, training video image data and target speech content information of the training speech data of a sound producing object, wherein the training speech data comprises a plurality of audio frames, and the training video image data comprises a plurality of image frames;

[0015] performing phoneme feature extraction on each audio frame in the training speech data to obtain a plurality of speech phoneme features, and splicing the plurality of speech phoneme features to obtain third phoneme feature information;

[0016] inputting the third phoneme feature information into a first fully connected layer for feature classification to obtain fourth phoneme feature information;

[0017] performing visual feature extraction on each image frame in the training video image data to obtain a plurality of visual features, and splicing the plurality of visual features to obtain third visual feature information;

[0018] inputting the third visual feature information into a second fully connected layer for feature classification to obtain fourth visual feature information;

[0019] splicing the fourth phoneme feature information and the fourth visual feature information to obtain training fusion feature information;

[0020] performing speech recognition on the training fusion feature information to obtain predicted speech content information corresponding to the training speech data;

[0021] correcting parameters of the model according to the predicted speech content information and the target speech content information.

[0022] According to a third aspect of the present application, a speech data processing apparatus is provided, comprising:

[0023] an acquisition unit configured to acquire speech data and video image data of a sound producing object, wherein the speech data comprises a plurality of audio frames, and the video image data comprises a plurality of image frames;

[0024] a phoneme feature extraction unit configured to perform phoneme feature extraction on each audio frame in the speech data to obtain a plurality of speech phoneme features, and splice the plurality of speech phoneme features to obtain first phoneme feature information;

[0025] a first processing unit configured to input the first phoneme feature information into a first fully connected layer for feature classification to obtain second phoneme feature information;

[0026] a visual feature extraction unit configured to perform visual feature extraction on each image frame in the video image data to obtain a plurality of visual features, and splice the plurality of visual features to obtain first visual feature information;

[0027] The second processing unit is configured to input the first visual feature information into a second full connection layer for feature classification to obtain second visual feature information.

[0028] The fusion unit is configured to splice the second phoneme feature information and the second visual feature information to obtain fusion feature information.

[0029] The speech recognition unit is configured to perform speech recognition on the fusion feature information to obtain speech content information corresponding to the speech data.

[0030] Further, the speech data processing apparatus further comprises:

[0031] The image framing unit is configured to perform framing processing on the video image data to obtain a plurality of image frames.

[0032] The speech framing unit is configured to perform framing processing on the speech data to obtain a plurality of audio frames.

[0033] Further, the phoneme feature extraction unit is further configured to splice the plurality of speech phoneme features in sequence according to the framing sequence of the speech data to obtain the first phoneme feature information.

[0034] The phoneme feature extraction unit is further configured to splice the plurality of visual features in sequence according to the framing sequence of the video image data to obtain the first visual feature information.

[0035] The visual feature extraction unit is further configured to splice the plurality of visual features in sequence according to the framing sequence of the video image data to obtain the first visual feature information.

[0036] Further, the acquisition unit is further configured to acquire video data of the sound-making object, separate the video data to obtain speech data of the sound-making object and original video image data, extract lip movement video image of the sound-making object in the original video image data to obtain the video image data.

[0037] Alternatively, the video data of the sound-making object is acquired, the original video image data is separated from the video data, the lip movement video image of the sound-making object in the original video image data is extracted to obtain the video image data, and the speech data corresponding to the video image data in time sequence is extracted from the video data.

[0038] Further, the phoneme feature extraction unit is further configured to input each audio frame in the speech data into an acoustic neural network to obtain speech phoneme features of each audio frame.

[0039] Further, the visual feature extraction unit is further configured to input each image frame in the video image data into a video feature extraction network for visual feature extraction to obtain visual features of each image frame.

[0040] Further, the voice recognition unit is further configured to input the fusion feature information into a third fully connected layer to perform feature classification, to obtain voice classification features; and input the voice classification features into a normalization layer to perform normalization processing, to obtain voice content information corresponding to the voice data.

[0041] According to a fourth aspect of the present application, a model training apparatus is provided, comprising:

[0042] a training data extraction unit configured to obtain training voice data, training video image data, and target voice content information of the training voice data of a sound producing object, wherein the training voice data comprises a plurality of audio frames, and the training video image data comprises a plurality of image frames;

[0043] a phoneme training feature extraction unit configured to perform phoneme feature extraction on each audio frame in the training voice data, to obtain a plurality of voice phoneme features, and splice the plurality of voice phoneme features, to obtain third phoneme feature information;

[0044] a third processing unit configured to input the third phoneme feature information into a first fully connected layer to perform feature classification, to obtain fourth phoneme feature information;

[0045] a visual training feature extraction unit configured to perform visual feature extraction on each image frame in the training video image data, to obtain a plurality of visual features, and splice the plurality of visual features, to obtain third visual feature information;

[0046] a fourth processing unit configured to input the third visual feature information into a second fully connected layer to perform feature classification, to obtain fourth visual feature information;

[0047] a training feature fusion unit configured to splice the fourth phoneme feature information and the fourth visual feature information, to obtain training fusion feature information;

[0048] a voice training data recognition unit configured to perform voice recognition on the training fusion feature information, to obtain predicted voice content information corresponding to the training voice data;

[0049] a correction unit configured to correct parameters of the model according to the predicted voice content information and the target voice content information.

[0050] Further, the training data extraction unit is further configured to obtain training video data of the sound producing object and target voice content information corresponding to the training video data, separate the training video data, to obtain training voice data and original training video image data of the sound producing object, extract a lip movement video image of the sound producing object in the original training video image data, and obtain the training video image data.

[0051] Or, the training video data of the sound object and the target speech content information corresponding to the training video data are acquired, the original training video image data is separated from the training video data, the lip action video image of the sound object in the original training video image data is extracted, the training video image data is obtained, and the training speech data corresponding to the training video image data in the time sequence is extracted from the training video data.

[0052] Further, the phoneme training feature extraction unit is further configured to: input each audio frame in the training speech data into an acoustic neural network to obtain speech phoneme features of each audio frame; the parameters of the model include parameters of the acoustic neural network; and the correction unit is further configured to: correct the parameters of the video feature extraction network according to the predicted speech content information and the target speech content information.

[0053] Further, the visual training feature extraction unit is further configured to: input each image frame in the training video image data into the video feature extraction network for visual feature extraction to obtain visual features in each image frame; the parameters of the model include parameters of the video feature extraction network; and the correction unit is further configured to: correct the parameters of the video feature extraction network according to the predicted speech content information and the target speech content information.

[0054] Further, the speech training data recognition unit is further configured to: input the training fusion feature information into a third fully connected layer for feature classification to obtain speech classification features; and input the speech classification features into a normalization layer to obtain the predicted speech content information corresponding to the speech data.

[0055] The parameters of the model include parameters of the first fully connected layer, parameters of the second fully connected layer, and parameters of the third fully connected layer; and the correction unit is further configured to: correct the parameters of the first fully connected layer, the parameters of the second fully connected layer, and the parameters of the third fully connected layer according to the predicted speech content information and the target speech content information.

[0056] According to a fifth aspect of the present application, a speech data processing apparatus is provided, comprising:

[0057] at least one memory for storing programs;

[0058] at least one processor for loading programs to execute the speech data processing method of the first aspect of the present application or execute the model training method of the second aspect of the present application.

[0059] According to a sixth aspect of the present application, a computer readable storage medium is provided, which stores computer executable instructions, and the computer executable instructions are configured to execute the speech data processing method of the first aspect of the present application or execute the model training method of the second aspect of the present application.

[0060] According to a seventh aspect of the present application, a computer program product comprises computer programs / instructions which, when executed by a processor, implement the speech data processing method of the first aspect of the present application or implement the model training method of the second aspect of the present application.

[0061] According to an eighth aspect of the present application, a computer program is provided, comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the speech data processing method of the first aspect of the present application or perform the model training method of the second aspect of the present application.

[0062] The embodiments of the present application at least have the following beneficial effects: by obtaining speech data and video image data of a sound producing object, extracting speech phoneme features of each audio frame in the speech data, and splicing each speech phoneme feature to obtain first phoneme feature information, performing feature classification on the first phoneme feature information through a first fully connected layer to obtain second phoneme feature information, wherein the second phoneme feature information contains speech phoneme features of each audio frame in the speech data, by extracting visual features of each image frame in the video image data, and splicing each visual feature to obtain first visual feature information, performing feature classification on the first visual feature information through a second fully connected layer to obtain second visual feature information, splicing the second phoneme feature information and the second visual feature information to obtain fusion feature information, and finally performing speech recognition on the fusion feature information to obtain speech content information, the embodiments of the present application fuse speech phoneme features in the speech data and visual features in the video image data for speech recognition, the fusion of the visual feature mode enriches the features used for speech recognition, which can effectively improve the accuracy of speech recognition, in addition, the embodiments of the present application not only extract each video frame and audio frame, but also fuse and splice after performing feature classification on the first phoneme feature information and the first visual feature information through the first fully connected layer and the second fully connected layer respectively, so that the speech phoneme features in the speech data and the visual features in the video data can be more fully fused, further improving the accuracy of speech recognition. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a system architecture diagram of the speech data processing method and the model training method in the embodiments of the present application;

[0064] Figure 2 is a flowchart of the speech data processing method provided by an embodiment of the present application;

[0065] Figure 3 is a structural schematic diagram of the image frame sequence provided by an embodiment of the present application;

[0066] Figure 4 is a schematic diagram of an original speech time domain signal provided by an embodiment of the present application;

[0067] Figure 5 is a specific method flowchart of step 210;

[0068] Figure 6 is a schematic diagram of frame processing of video image data provided by an embodiment of the present application;

[0069] Figure 7 is a specific method flowchart of step 270;

[0070] Figure 8 is a structural schematic diagram of a third full connection layer provided by an embodiment of the present application;

[0071] Figure 9 is an architecture diagram of a video speech recognition model provided by an embodiment of the present application;

[0072] Figure 10 is a flowchart of a model training method provided by an embodiment of the present application;

[0073] Figure 11 is a structural diagram of a speech data processing apparatus provided by an embodiment of the present application;

[0074] Figure 12 is a structural diagram of a model training apparatus provided by an embodiment of the present application. DETAILED DESCRIPTION

[0075] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0076] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0078] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0079] 1) Video data, video data is a continuous change of images, video usually refers to the dynamic demonstration of the actual scene, such as movies, television camera data, video or live video. Video data with video image data and synchronous audio data. If the video data presents the speech video of the sound object, it includes the speech data and video image data of the sound object during speaking, wherein the video image data can contain the facial action image of the sound object.

[0080] 2) Speech, refers to the material shell of language, is the external form of language, is the most direct symbol system to record human thinking activity. It is the sound emitted by the human vocal organs with certain social significance.

[0081] 3) Speech data, speech data is the product after speech data is converted, wherein the speech data can be divided into analog speech data and digital speech data according to different recording carriers; analog audio is recorded in the form of magnetic recording or mechanical scale after sound pickup processing, at this time the change of residual magnetism on the tape or the change of groove in the record groove is proportional to the change of sound signal; digital audio is first converted into audio file, then the level signal is converted into binary data for saving or further processing, so that the data can be converted into analog level signal when playing and sent to the loudspeaker for playing. Unless otherwise specified, the speech data in the embodiments of the present application refers to digital speech data.

[0082] 4) Frame sequence, represents the arrangement of a series of data with sequential relationship, wherein each frame represents a minimum unit of data, for example, an audio frame represents a unit length of audio data, and an image frame represents an image data. Since the video image data includes a plurality of continuous image frames, the video image data can also be regarded as a video image frame sequence. Similarly, audio data such as speech data can be regarded as a speech frame sequence.

[0083] 5) Phoneme, is the smallest unit of speech or the smallest segment of speech divided according to the natural properties of speech, which is analyzed according to the pronunciation action in a syllable. A phoneme is divided into two categories: vowels and consonants. For example, Chinese syllable ā (ah) has only one phoneme, ai (love) has two phonemes, and dāi (dull) has three phonemes. The method of marking speech with international phonetic symbols is called phonetic transcription, which has two types: wide and strict. The phonetic symbol of international phonetic symbols (established by International Phonetic Association, used to unify the symbols of speech) corresponds to the phonemes of all human languages.

[0084] 6) Visual features, visual features refer to features in images that can be recognized or distinguished by computers, in other words, computers can extract visual features from image data, such as color, texture, shape, structure and other visual features in images.

[0085] 7) Full-connected layer (FC layer), the full-connected layer can map the learned features to the sample label space, which mainly plays the role of "classifier" in the whole convolutional neural network, each node of the full-connected layer is connected with all the nodes outputted by the last layer (such as the down-sampling layer in the convolutional layer), wherein, one node of the full-connected layer is referred to as one neuron in the full-connected layer, the number of neurons in the full-connected layer can be determined according to the actual application requirement, for example, can be all set to 512, or can be all set to 128, etc.

[0086] 8) Deep residual network (ResNet), ResNet is a network structure obtained by stacking a series of residual network modules, the core of the ResNet model is to establish "short circuit connection" (shortcuts, skip connection) between the front layer and the rear layer, which is helpful for the back propagation of gradient in the training process, so that a deeper CNN network can be trained, the training error of ResNet will always decrease with the increase of network depth. Using ResNet structure can avoid the problem of model performance degradation. The number of hidden layers of ResNet can be set according to the requirement, for example, ResNet-101 represents a deep residual network containing 101 hidden layers.

[0087] 9) Recurrent neural network (RNN), RNN is a kind of recursive neural network which takes sequence data as input, performs recursion in the evolution direction of sequence, and all nodes (recurrent units) are connected in chain.

[0088] 10) Connectionist temporal classification (CTC), CTC network can be understood as a neural network-based time sequence classification, CTC network only needs an input sequence and an output sequence to train, without the need for data alignment and one-to-one labeling, so it can solve the problem of character alignment in training.

[0089] The voice data processing method and the model training method provided in the embodiments of the present application can be applied in artificial intelligence. Artificial intelligence (AI) is a theory, method, technology and application system for utilizing a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0090] Artificial intelligence (AI): is a theory, method, technology and application system for utilizing a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results.

[0091] It should be noted that artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0092] In addition, artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0093] With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields; for example, common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned vehicle, autonomous vehicle, unmanned aerial vehicle, robot, smart medical treatment and smart customer service, etc.; with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. In the embodiments of the present application, the application of artificial intelligence in the field of audio technology, especially voice recognition, will be described.

[0094] The research of speech recognition originated in the 1950s, and the early speech recognition system was a simple isolated word recognition system. In the 1970s, the substantial improvement of computer performance and the development of pattern recognition basic research, such as the emergence of codebook generation algorithm (LBG) and linear predictive coding (LPC), promoted the development of speech recognition. In the 1980s, speech recognition developed rapidly, and speech recognition began to develop from isolated word recognition system to large vocabulary continuous speech recognition system. With the development of artificial intelligence neural network and machine learning algorithm, a deep learning boom has been triggered, and the speech recognition field also gradually adopts the acoustic model based on deep learning, and remarkable progress has been made in recent years, and has been widely used in industry, communication, medical treatment and other fields. In the related technology of recognizing the speech of the sound object, most of them are simply extracting speech feature information from the audio frame, and then recognizing the speech based on the extracted speech feature information and converting it into speech text. In order to improve the accuracy of speech recognition, the main research direction at present is mainly how to optimize the model and algorithm. However, in such methods, only the speech modal information of the sound object is used for speech recognition, and the image modal information (such as the lip movement image features of the sound object) of the sound object is not considered. Since the video image data also contains the lip movement features of the sound object, which is important information that is helpful to speech recognition and is conducive to enhancing the accuracy of speech recognition.

[0095] Based on this, the embodiment of the application provides a speech data processing method, a model training method, an apparatus and a storage medium, which can improve the accuracy of speech recognition.

[0096] The specific embodiments of the application will be described below with reference to the drawings. The video speech recognition model is used to recognize and process the video data of the sound object, and the speech data processing method mentioned below is used to obtain the speech content information of the sound object. Of course, before using the video speech recognition model for speech recognition, the video speech recognition model needs to be trained in advance. In the embodiment of the application, the video speech recognition model is trained by the video speech recognition model training system, and the model training method mentioned below is used.

[0097] Referring to Figure 1 Fig. 1 is a use scenario diagram of the speech data processing method provided by the embodiment of the application. The system architecture can include a terminal 101, a server 102 and a network 103. The network 103 is used to provide a communication link medium between the terminal 101 and the server 102. The network 103 can include various connection types, such as wired, wireless communication link or optical fiber cable and the like.

[0098] Referring to Figure 1As shown, the embodiment of the present application provides a speech data processing method, which is used for speech recognition processing of video data of a sound object to obtain corresponding speech content information. The speech data processing method can be applied to a terminal 101, can be applied to a server 102, and can also be software running in the terminal or the server.

[0099] In some embodiments, the speech data processing method is executed by the terminal 101. The terminal 101 can be various electronic devices with a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, a wearable device, a virtual reality device, a smart home device, and the like. The terminal 101 can also be a live environment device requiring speech recognition, such as a teaching system terminal, an access control system terminal, a live broadcast system terminal, a conference system terminal, or a video playback system terminal. The terminal 101 stores an algorithm or a model for executing the speech data processing method.

[0100] In other embodiments, the speech data processing method is executed by the server 102. The server 102 can be a server providing various services, such as a background management server supporting an apparatus operated by a user using the terminal 101. The background management server can analyze and process received request data, and feed back the processing result to the terminal. The server 102 can be a server of one entity, or can be composed of multiple servers (for example, a distributed server). According to actual needs, the server 102 can have any number of terminal devices, networks, and servers. In addition, the multiple servers can be composed of a block chain, and the server is a node on the block chain.

[0101] In other embodiments, the speech data processing method can be executed by the server 102 in cooperation with the terminal 101. The terminal 101 is used to obtain speech data and video image data of a sound object. The server 102 is in communication connection with the terminal 101, and is used to receive the speech data and the video image data sent by the terminal 101. The server 102 stores an algorithm or a model for executing the speech data processing method. The server 102 executes the speech data processing method on the speech data and the video image data sent by the terminal 101 to obtain corresponding speech content information, and feeds back the speech content information to the terminal 101.

[0102] Reference Figure 2 As shown, the speech data processing method provided by the embodiment of the present application includes the following steps:

[0103] In step 210, speech data and video image data of a sound object are obtained. The speech data includes multiple audio frames, and the video image data includes multiple image frames.

[0104] In this step, the sound-emitting object is an object that can emit speech. In an embodiment, the sound-emitting object can be the user himself / herself. For example, when the user speaks, the speech signal emitted by the user is collected by a sound pickup device (e.g., a microphone of the terminal 101) to generate speech data, and the picture of the user when speaking is collected by a video recording device (e.g., a camera of the terminal 101) to generate video image data, wherein the speech data and the video image data are collected simultaneously during the user's speech. In another embodiment, the sound-emitting object can be a speaking object in a pre-recorded and stored video, for example, a teacher teaching in a teaching video. In another embodiment, the sound-emitting object can also be a speaking object in a live broadcast or a network meeting. In another embodiment, the sound-emitting object can be a user using voice input, for example, a user inputting voice instructions to a navigation system by speaking or a user using a cloud technology intelligent voice assistant. In another embodiment, the sound-emitting object can be an object simulating the sound emitted by the user, for example, a robot simulating human speech, a virtual object with a real human image synthesized by artificial intelligence, etc.

[0105] The acquired video image data and speech data are both composed of a plurality of data frames. The video image data is represented as an image frame sequence Vd = {vd_1, vd_2,..., vd_N}, and the image frame sequence Vd includes a plurality of image frames vd_i, where i is a natural number. Each image frame represents an image, which can be a speaking image of the sound-emitting object, for example, a facial image, a partial facial image, or a body image of the sound-emitting object when speaking, etc. Referring to Figure 3 As shown in FIG. 2, an image frame sequence diagram provided by an embodiment is shown, and each row of image frame sequences represents a sound-emitting object, for example, Figure 3 As shown in FIG. 2, an image frame sequence diagram provided by an embodiment is shown, and each row of image frame sequences represents a sound-emitting object, for example, Figure 4 As shown in FIG. 2, an image frame sequence diagram provided by an embodiment is shown, and each row of image frame sequences represents a sound-emitting object, for example,

[0106] In an embodiment, the voice data and the video image data can be acquired separately, for example, the voice data and the video image data can be two data files stored separately, in order to improve the effect of voice recognition, the audio frames in the voice data and the image frames in the video image data can be frame-aligned, so that the image frames are matched and corresponded with the audio frames. In another embodiment, the voice data and the video image data can be separated from the same video data. Since the audio and the image of the video are generally synchronized, the data frames of the separated voice data and the video image data are aligned in time sequence.

[0107] In step 220, phoneme feature extraction is performed on each audio frame in the voice data to obtain a plurality of voice phoneme features, and the plurality of voice phoneme features are spliced to obtain first phoneme feature information.

[0108] In this step, each audio frame of the voice data carries voice spectrum data of a corresponding voice segment. In order to extract voice content, voice phoneme features in the voice spectrum data need to be extracted. In an embodiment, the voice phoneme features can be extracted by a preset voice phoneme feature extraction algorithm. The voice phoneme feature extraction algorithm can use an MFCC (Mel-frequency Cepstrum Coefficients) algorithm or an FBank (Mel-scale Filter Bank) algorithm, which can correspond to extraction of MFCC features and FBank features of the voice data, respectively. In another embodiment, the voice phoneme features can also be extracted by a pre-trained voice phoneme feature extraction model. For example, for a voice frame sequence Vo={vo_1, vo_2,..., vo_N}, each audio frame is respectively subjected to feature extraction by the voice phoneme feature extraction algorithm or the voice phoneme feature extraction model to obtain a voice phoneme feature sequence Vo_v={vo_v_1, vo_v_2,..., vo_v_N}. The voice phoneme feature corresponding to each audio frame can be represented in the form of a feature vector or a feature matrix.

[0109] Then, all voice phoneme features in the voice phoneme feature sequence are spliced (concat) to obtain first phoneme feature information Vo_vec. If the voice phoneme features are represented by a feature vector, each voice phoneme feature forms a feature vector after splicing. If the voice phoneme features are represented in the form of a feature matrix, each voice phoneme feature forms a new feature matrix after splicing.

[0110] In step 230, the first phoneme feature information is input to a first fully connected layer for feature classification to obtain second phoneme feature information. In this step, the first fully connected layer is a pre-trained classification model, which acts as a classifier. By inputting the first phoneme feature Vo_vec to the first fully connected layer for feature classification, the first phoneme feature Vo_vec can be mapped to one-dimensional M-column second phoneme feature information, which contains the speech feature information of all audio frames in the speech data.

[0111] In step 240, visual features are extracted from each image frame in the video image data to obtain a plurality of visual features, and the plurality of visual features are spliced to obtain first visual feature information.

[0112] In this step, each image frame of the video image data carries image data of a corresponding video image segment. In order to improve the accuracy of speech recognition, visual features need to be extracted from the image data. In an embodiment, the visual features can be extracted by a pre-set visual feature extraction algorithm. The visual feature extraction algorithm can use SIFT (Scale-invariant feature transform), HOG (histogram of Oriented Gradient), DOG (Difference of Gaussian), etc. In another embodiment, the visual features can also be extracted by a pre-trained visual feature extraction model, such as a pre-trained face recognition model, a neural network model, etc. For example, for an image frame sequence Vd = {vd_1, vd_2,..., vd_N}, each image frame is respectively subjected to feature extraction by a visual feature extraction algorithm or a visual feature extraction model to obtain a visual feature sequence Vd_v = {vd_v_1, vd_v_2,..., vd_v_N}. Each image frame corresponding visual feature can be represented in the form of a feature vector or a feature matrix.

[0113] Then, all visual features in the visual feature sequence are spliced (concat) to obtain first visual feature information Vd_vec. If the visual features are represented by feature vectors, each visual feature is spliced to form a feature vector. If the visual features are represented in the form of a feature matrix, each visual feature is spliced to form a new feature matrix.

[0114] In step 250, the first visual feature information is input to a second fully connected layer for feature classification to obtain second visual feature information.

[0115] In this step, the second full connection layer is a pre-trained classification model, which acts as a classifier. By inputting the visual feature information Vd_vec into the second full connection layer for feature classification, the visual feature information Vd_vec can be mapped to a second visual feature information of one dimension and N columns. The second phoneme feature information contains the visual feature information of all image frames in the image data.

[0116] In step 260, the second phoneme feature information and the second visual feature information are spliced to obtain fusion feature information.

[0117] In this step, the second phoneme feature information and the second visual feature information are information of the same feature type. For example, the second phoneme feature information and the second visual feature information are both feature vectors, or the second phoneme feature information and the second visual feature information are both feature matrices. By splicing the second phoneme feature information and the second visual feature information, fusion feature information V_vec is obtained. The fusion feature information V_vec contains both the speech feature information of all audio frames in the speech data and the visual feature information of all image frames in the video image data.

[0118] In step 270, the fusion feature information is subjected to speech recognition to obtain speech content information corresponding to the speech data.

[0119] In an embodiment, the fusion feature information V_vec can be subjected to speech recognition by a pre-trained speech recognition model to obtain speech content information corresponding to the speech data. The speech content information can be speech text information or information capable of expressing speech content, such as classification information or label information of the speech data, or feature information facilitating artificial intelligence (AI) to understand the speech content.

[0120] The technical scheme provided in the application is characterized in that: speech data and video image data of a sound emitting object are acquired, speech phoneme features of each audio frame in the speech data are extracted, and first phoneme feature information is obtained by splicing the speech phoneme features; the first phoneme feature information is classified by a first full connection layer to obtain second phoneme feature information, wherein the second phoneme feature information contains the speech phoneme features of each audio frame in the speech data; visual features of each image frame in the video image data are extracted, and first visual feature information is obtained by splicing the visual features; the first visual feature information is classified by a second full connection layer to obtain second visual feature information; the second phoneme feature information and the second visual feature information are spliced to obtain fusion feature information; and finally, speech recognition is performed on the fusion feature information to obtain speech content information. The application fuses the phoneme features in the speech data and the visual features in the video image data for speech recognition, the fusion of the visual feature mode enriches the features used for speech recognition, and the accuracy of speech recognition can be effectively improved. In addition, the application not only extracts each video frame and audio frame, but also fuses and splices the first phoneme feature information and the first visual feature information after classification by the first full connection layer and the second full connection layer, so that the phoneme feature information in the speech data and the visual feature information in the video data can be more fully fused, and the accuracy of speech recognition is further improved.

[0121] In the application, the first phoneme feature information and the first visual feature information are not simply spliced and fused, but are spliced and fused after classification by the first full connection layer and the second full connection layer, and the fusion degree is higher. If the first phoneme feature information and the first visual feature information are simply spliced, only the data frames are simply spliced, and since the phoneme features and the visual features are two different modal features, the speech recognition algorithm or the speech recognition model cannot recognize different types of data frames, which easily leads to recognition failure, and even causes model training failure. In the application, each data is first classified to obtain a classification result, and then the features are fused, the fusion degree is higher and more sufficient, the accuracy of subsequent speech recognition is higher, and the model training is more likely to be successful.

[0122] In an embodiment, the structure of the first full connection layer and the second full connection layer can also be adjusted according to the characteristics of the acquired voice data and video image data, so that the voice data and video image data can be more fully fused. For example, the number of neuron nodes of the first full connection layer and the second full connection layer can be adjusted, and then the fusion degree of the fused feature information is adjusted. For example, if the number of neuron nodes of the first full connection layer is more than that of the second full connection layer, it means that the phoneme features in the voice data are more emphasized in the fused feature information. In an embodiment, the number of neuron nodes of the first full connection layer and the second full connection layer is the same. In another embodiment, the weights of the neuron nodes in the first full connection layer and the second full connection layer can be limited or preset, so that the voice data and video image data can be more fully fused. The adjustment of the structure of the first full connection layer and the second full connection layer in the embodiment of the application can also be regarded as the feature dimension processing of the first phoneme feature information and the first visual feature information. In an embodiment, the structure of the first full connection layer and the second full connection layer can also be adjusted according to the different voice emitting objects. For example, different structures of the first full connection layer and the second full connection layer can be set for adults and children. In another embodiment, the structure of the first full connection layer and the second full connection layer can also be adjusted according to the use scene of voice recognition. For example, different structures of the first full connection layer and the second full connection layer can be set for different scenes such as live broadcast and conference.

[0123] With reference to Figure 5 In an embodiment, after the step 210 acquires the voice data and video image data of the voice emitting object, the following steps are further included:

[0124] In step 510, the video image data is frame processed to obtain a plurality of image frames.

[0125] In an embodiment of the present step, the video image data can be divided into a plurality of image frames according to the frame number of the video image data itself to form an image frame sequence Vd = {vd_1, vd_2,..., vd_N}. In another embodiment, the video image data can be outputted and extracted according to a preset frame number to obtain an image frame sequence. For example, if the preset frame number is half of the frame number of the video image data itself, then one image frame can be extracted every other frame to join the image frame sequence. Similarly, the interval of extracting image frames from the video image data can be determined according to the ratio of the preset frame number to the frame number of the video image data itself.

[0126] In step 520, the voice data is frame processed to obtain a plurality of audio frames.

[0127] In one embodiment of the step, the voice data can be divided into a plurality of audio frames according to the number of frames of the voice data itself, to form a voice frame sequence Vo = {vo_1, vo_2,..., vo_N}. The number of frames of the voice data itself is related to the sampling frequency of the digital processing of the original audio data. In another embodiment, the voice data can be outputted by extraction according to a preset number of frames. For example, if the preset number of frames is half of the number of frames of the voice data itself, then one audio frame can be extracted every other frame to be added to the voice frame sequence. Similarly, the interval of the audio frames extracted from the voice data can be determined according to the ratio of the preset number of frames to the number of frames of the voice data itself.

[0128] In another embodiment, the voice data can be divided into a plurality of audio frames according to the number of frames of the voice data or the time sequence of the image frame sequence. For example, referring to FIG. 5B, the video image data is divided into a plurality of image frames by the step 510, to form an image frame sequence Vd = {vd_1, vd_2, vd_3, vd_4}, Figure 6 Figure 6 It is also shown in FIG. 5B that the voice data corresponding to the image frame sequence is divided into four audio frames vo_1, vo_2, vo_3, vo_4 according to the time sequence of the image frames of the image frame sequence Vd, to form a voice frame sequence Vo = {vo_1, vo_2, vo_3, vo_4}.

[0129] In one embodiment, the plurality of voice phoneme features are spliced in the step 220 to obtain the first phoneme feature information, which specifically includes:

[0130] The plurality of voice phoneme features are spliced in sequence according to the frame sequence of the voice frame sequence (i.e., the frame sequence of the voice data) to obtain the first phoneme feature information.

[0131] In this step, the plurality of voice phoneme features are spliced in sequence according to the frame sequence of the voice frame sequence (i.e., the frame sequence of the voice data) to obtain the first phoneme feature information. For example, for Vo = {vo_1, vo_2,..., vo_N}, the plurality of voice phoneme features are spliced in sequence according to vo_1, vo_2,..., vo_N to obtain the first phoneme feature information Vo_ve. Since the order of the phonemes is also important information in addition to the phonemes themselves in the voice recognition, the plurality of voice phoneme features are spliced in sequence according to the frame sequence of the voice data, so that the order relationship of the phonemes is not lost, and the accuracy of the voice recognition is improved.

[0132] In one embodiment, the plurality of visual features are spliced in the step 240 to obtain the first visual feature information, which specifically includes:

[0133] The plurality of visual features are spliced in sequence according to the frame sequence of the video image data to obtain the first visual feature information.​

[0134] In this step, the visual features are sequentially stitched according to the frame sequence of the image frame sequence (i.e., the frame sequence of the image data), to obtain first visual feature information. For example, for Vd = {vd_1, vd_2,..., vd_N}, the visual features are sequentially stitched in the order of vd_1, vd_2,..., vd_N to obtain first phoneme feature information Vd_ve. Since the visual features of the user when speaking include not only the visual information of each image frame, but also the arrangement order of the image frames, which reflects the action of the user when speaking, sequentially stitching the visual features according to the frame sequence of the image data does not lose the order relationship of the image action, and can improve the accuracy of speech recognition.

[0135] In an embodiment, the step 210 specifically includes:

[0136] The video data of the sound- producing object is obtained, the video data is separated to obtain speech data of the sound- producing object and original video image data, and lip action video image data of the sound- producing object in the original video image data is extracted to obtain video image data.

[0137] In this step, speech recognition is performed using all data in the video data. First, the video data is separated to obtain speech data of the sound- producing object and original video image data. The original video image data is processed to extract lip action video image data of the original video image data. In an embodiment, a face recognition model can be used to recognize the face features of the sound- producing object to determine the lip position of the sound- producing object. The image of the lip position is cropped to obtain the video image data. For example, refer to FIG. 6, which shows the video image data after cropping. Figure 3

[0138] In another embodiment, the step 210 specifically includes:

[0139] The video data of the sound- producing object is obtained, the original video image data is separated from the video data, the lip action video image data of the sound- producing object in the original video image data is extracted to obtain video image data, and the speech data corresponding to the video image data in the time sequence is extracted from the video data.

[0140] In this step, the original video image data in the video data is first separated, and the original video image data is processed to extract the lip action video image data of the original video image data. In an embodiment, a face recognition model can be used to recognize the face features of the sound- producing object to determine the lip position of the sound- producing object. The image of the lip position is cropped to obtain the video image data. For example, refer to FIG. 6, which shows the video image data after cropping. Figure 3 ​As shown, it is the video image data after clipping. Since not all data in the video data is valid content of the speech object speaking, in this embodiment, the video image data containing the lip movement of the speech object is first determined as the video image data, the image frame sequence containing the lip movement of the speech object is first determined, the valid audio data in the speech data is correspondingly determined, the speech frame sequence corresponding to the image frame sequence in the time sequence is extracted, and then the speech data is obtained. In this step, by identifying the valid image data, the noise removal effect can be achieved, and the accuracy of speech recognition can be improved.

[0141] In an embodiment, the step 220 specifically includes:

[0142] The audio frame in the speech data is input into the acoustic neural network to obtain the speech phoneme feature of each audio frame.

[0143] In an embodiment of this step, the acoustic neural network can use neural network models such as time delay neural network TDNN-LSTM, recurrent neural network RNN-T, and sequence short-time classification network CTC, which are used to predict the pronunciation sequence of the input audio. In an embodiment, the acoustic neural network uses the recurrent neural network RNN combined with the sequence short-time classification network CTC, that is, the output result of the RNN is input into the CTC, wherein the RNN can not only use historical speech information but also use future speech information when judging the current speech frame, so as to make more accurate decisions. The CTC makes the training process unnecessary to be labeled at the frame level, realizes effective “end-to-end” training, and finally realizes better speech recognition effect.

[0144] In an embodiment, the step 240 specifically includes:

[0145] Each image frame in the video image data is input into the video feature extraction network for visual feature extraction to obtain the visual feature of each image frame. In an embodiment, the video feature extraction network can use a residual network Resnet (for example, a residual network with 101 hidden layers can be used as the video feature extraction network in an embodiment). In addition, other face recognition models can also be used for visual feature extraction, which are not listed one by one here.

[0146] Referring to Figure 7 In an embodiment, the step 270 specifically includes:

[0147] In step 710, the fusion feature information is input into the third fully connected layer for feature classification to obtain the speech classification feature.

[0148] In this step, the third fully connected layer is a pre-trained classification model, which acts as a classifier. Referring to Figure 8As shown, the third full connection layer includes a plurality of nodes f, and the input of the full connection layer is the fusion feature information 801 obtained in step 260. The third full connection layer performs nonlinear transformation on the input fusion feature information and outputs speech classification features 802, wherein the calculation formula of the nonlinear transformation is as follows: wherein f is a node in the third full connection layer, that is, an activation function, W is a weight matrix, and b is a bias constant. Figure 8

[0149] In step 720, the speech classification features are input into a normalization layer for normalization processing to obtain speech content information corresponding to the speech data.

[0150] In this step, the speech classification features of the third full connection layer are normalized by the normalization layer to obtain the classification probability of the speech content information, and the calculation method is as follows:

[0151] wherein wherein Z j is the output of the full connection layer, and W and b are the parameters to be trained of the layer.

[0152] Referring to FIG. 8, Figure 8 As shown in FIG. 8, a video speech recognition model architecture is provided in an embodiment of the present application, and the flow of the speech data processing method is described below in combination with the video speech recognition model architecture.

[0153] First, video data of a sound object is obtained, and the video data is separated to obtain speech data and video image data. The speech data and the video image data are respectively frame processed to obtain an image frame sequence Vd={vd_1, vd_2,..., vd_N} and a speech frame sequence Vo={vo_1, vo_2,..., vo_N}.

[0154] For the speech frame sequence Vo, each audio frame in the speech frame sequence Vo is extracted for speech phoneme features by an acoustic neural network (a recurrent neural network RNN combined with a sequence short-time classification network CTC) to form a speech phoneme feature sequence Vo_v={vo_v_1, vo_v_2,..., vo_v_N}. All speech phoneme features in the speech phoneme feature sequence are concatenated to obtain first phoneme feature information Vo_vec. The first phoneme feature information is input into a first full connection layer for feature classification to obtain second phoneme feature information Vo1.

[0155] ​For the image frame sequence Vd, visual feature extraction is performed on each image frame in the image frame sequence Vd by a video feature extraction network (a residual network Resnet) to form a visual feature sequence Vd_v = {vd_v_1, vd_v_2,..., vd_v_N}, all visual features in the visual feature sequence are concatenated to obtain first visual feature information Vd_vec, and the first visual feature information is input into a second fully connected layer for feature classification to obtain second visual feature information Vd1.

[0156] Then, the second phoneme feature information Vo1 and the second visual feature information Vd1 are concatenated to obtain fusion feature information V_vec. The fusion feature information V_vec is input into a third fully connected layer for feature classification to obtain speech classification features, and the speech classification features are input into a normalization layer for normalization processing to obtain speech content information corresponding to the speech data.

[0157] The speech data processing method provided by any of the above embodiments fuses phoneme features in speech data and visual features in video image data for speech recognition, the fusion of visual feature modalities enriches the features used for speech recognition, and the accuracy of speech recognition can be effectively improved. In addition, the embodiments of the present application not only perform feature extraction on each video frame and audio frame, but also fuse and concatenate the first phoneme feature information and the first visual feature information after feature classification by the first fully connected layer and the second fully connected layer, respectively, so that the phoneme feature information in the speech data and the visual feature information in the video data can be more fully fused, and the accuracy of speech recognition is further improved. The speech data processing method provided by the embodiments of the present application can be applied to various scenarios with speech recognition needs or audio and video data processing needs, such as video playing, live streaming, video conferencing, cloud technology, artificial intelligence (AI), intelligent transportation, Internet of Vehicles, etc. For example, it can be used for subtitle translation of network / local video, live streaming / conference on-site speech translation, artificial intelligence understanding of human speech, voice navigation, voice hailing, etc.

[0158] Referring to Figure 10 FIG. 1 is a flowchart of a model training method provided by an embodiment of the present application. The model trained by the method can be applied to the speech data processing method shown in Figure 2 FIG. 1. In the embodiments of the present application, the server 102 shown in Figure 1 FIG. 1 is taken as an example for illustration. The server 102 is provided with a video speech recognition model, which includes a first fully connected layer and a second fully connected layer. The training method of the model includes the following steps:

[0159] At step 1010, training speech data, training video image data and target speech content information of the training speech data of the sound-making object are obtained, wherein the training speech data comprises a plurality of audio frames, and the training video image data comprises a plurality of image frames.

[0160] In this step, the obtained training video image data and the training speech data are both composed of a plurality of data frames. The training video image data is represented as a training image frame sequence Vd’ = {vd’_1, vd’_2, …, vd’_N}, and the training image frame sequence Vd’ comprises a plurality of image frames vd’_i, wherein i is a natural number. Each image frame represents an image, which can be a speaking image of the sound-making object, such as a facial image, a partial facial image or a limb image of the sound-making object when speaking. The training speech data is represented as a training speech frame sequence Vo’ = {vo’_1, vo’_2, …, vo’_N}, and the speech frame sequence Vi’ comprises a plurality of audio frames vo_i, wherein i is a natural number.

[0161] In an embodiment, the training speech data and the training video image data can be obtained separately, for example, the training speech data and the training video image data can be two data files stored separately. In order to improve the training effect of the model, the audio frames in the training speech data and the image frames in the training video image data can be frame-aligned, so that the image frames and the audio frames are matched and corresponded. In another embodiment, the training speech data and the training video image data can be obtained separately in the same training video data. Since the audio and the image of the video are generally synchronized, the data frames of the obtained speech data and video image data are aligned in time sequence.

[0162] In an embodiment, the training video data and the target speech content information corresponding to the training video data of the sound-making object are obtained, the training video data is separated to obtain the training speech data and the original training video image data of the sound-making object, the lip action video image of the sound-making object in the original training video image data is extracted to obtain the training video image data.

[0163] In another embodiment, the training video data of the sound object and the target speech content information corresponding to the training video data are acquired, the original training video image data is separated from the training video data, the lip movement video image of the sound object in the original training video image data is extracted to obtain the training video image data, and the training speech data corresponding to the training video image data in time sequence is extracted from the training video data. In this embodiment, the lip movement training video image containing the sound object is first determined as the training video image data, the training image frame sequence containing the lip movement of the sound object is first determined, the effective audio data in the corresponding determined speech data is extracted, the training speech frame sequence corresponding to the training image frame sequence in time sequence is extracted, and then the training speech data is obtained. In this step, by recognizing the effective training video image data, the denoising effect can be achieved, and the accuracy of speech recognition is improved.

[0164] In step 1020, phoneme feature extraction is performed on each audio frame in the training speech data to obtain a plurality of speech phoneme features, and the plurality of speech phoneme features are spliced to obtain third phoneme feature information.

[0165] In this step, each audio frame in the training speech data is input into an acoustic neural network to obtain the speech phoneme feature of each audio frame, wherein the acoustic neural network can use a time delay neural network TDNN-LSTM, a recurrent neural network RNN-T, a sequence short-time classification network CTC, and the like. For example, for a speech frame sequence Vo’ = {vo’_1, vo’_2,..., vo’_N}, each audio frame is extracted by a speech phoneme feature extraction algorithm or a speech phoneme feature extraction model to obtain a speech phoneme feature sequence Vo’_v = {vo’_v_1, vo’_v_2,..., vo’_v_N}. Each audio frame corresponding to the speech phoneme feature can be represented in the form of a feature vector or a feature matrix. Then, all speech phoneme features in the speech phoneme feature sequence are spliced (concat) to obtain third phoneme feature information Vo’_vec. In an embodiment, each speech phoneme feature is spliced in order according to the frame order of the speech phoneme feature sequence Vo’_v. Since the speech frame sequence Vo’ can be obtained by frame processing of the training speech data, the plurality of speech factor features can also be spliced in order according to the frame order of the training speech data to obtain the third phoneme feature information Vo’_vec.

[0166] In step 1030, the third phoneme feature information is input into a first fully connected layer for feature classification to obtain fourth phoneme feature information.

[0167] In this step, the first full connection layer functions as a classifier, and by inputting the third phoneme feature information Vo’_vec into the first full connection layer for feature classification, the third phoneme feature information Vo’_vec can be mapped into fourth phoneme feature information of one dimension and M columns, which contains the speech feature information of all audio frames in the training speech data.

[0168] In step 1040, visual features of each image frame in the training video image data are extracted to obtain a plurality of visual features, and the plurality of visual features are spliced to obtain third visual feature information.

[0169] In this step, each image frame in the training video image data is input into a video feature extraction network for visual feature extraction to obtain visual features in each image frame. The video feature extraction network can be a face recognition model, a neural network model, etc. to extract visual features from each image frame in the training video image data, obtain a plurality of visual features, and splice the plurality of visual features to obtain third visual feature information. For example, for an image frame sequence Vd’ = {vd’_1, vd’_2,..., vd’_N}, each image frame is respectively extracted by a visual feature extraction model to obtain a visual feature sequence Vd’_v = {vd’_v_1, vd’_v_2,..., vd’_v_N}. Each image frame corresponding visual feature can be represented in the form of a feature vector or a feature matrix.

[0170] Then, all visual features in the visual feature sequence are spliced (concat) to obtain third visual feature information Vd’_vec. If the visual features are represented by feature vectors, each visual feature is spliced to form a feature vector after splicing. If the visual features are represented in the form of a feature matrix, each visual feature is spliced to form a new feature matrix after splicing. In an embodiment, each visual feature is spliced according to the frame order of Vd’_v. Since the image frame sequence Vd’ can be obtained by frame processing of the training video image data, the plurality of visual features can also be spliced in sequence according to the frame order of the training video image feature to obtain the third visual feature information Vd’_vec.

[0171] In step 1050, the third visual feature information is input into a second full connection layer for feature classification to obtain fourth visual feature information.

[0172] In this step, the second full connection layer functions as a classifier, and the three visual feature information Vd'_vec is input into the second full connection layer for feature classification, so as to map the three visual feature information Vd'_vec into one-dimensional fourth visual feature information with N columns, which contains visual feature information of all image frames in the training image data.

[0173] In step 1060, the fourth phonetic feature information and the fourth visual feature information are spliced to obtain training fusion feature information.

[0174] In this step, the fourth phonetic feature information and the fourth visual feature information are information of the same feature type, for example, the fourth phonetic feature information and the fourth visual feature information are both feature vectors, or the fourth phonetic feature information and the fourth visual feature information are both feature matrices. By splicing the fourth phonetic feature information and the fourth visual feature information, training fusion feature information V_vec' is obtained, which contains both phonetic feature information of all audio frames in the training speech data and visual feature information of all image frames in the training video image data.

[0175] In step 1070, speech recognition is performed on the training fusion feature information to obtain predicted speech content information corresponding to the training speech data.

[0176] In an embodiment of this step, the training fusion feature information is input into a third full connection layer for feature classification to obtain speech classification features, and the speech classification features are input into a normalization layer to obtain predicted speech content information corresponding to the training speech data.

[0177] In step 1080, parameters of the model are corrected according to the predicted speech content information and the target speech content information.

[0178] In this step, by comparing the target speech content information with the predicted speech content information, the cross entropy is calculated, and then the parameters of the video speech recognition model are updated through the BP (Backpropagation, i.e. error back propagation) algorithm. The input-output relationship of the BP network is essentially a mapping relationship: the function completed by a BP neural network with m inputs and m outputs is a continuous mapping from a one-dimensional Euclidean space to a finite field in an m-dimensional Euclidean space, and this mapping has a high degree of nonlinearity. Its information processing capacity comes from the multiple composition of simple nonlinear functions, so it has strong function reproduction ability. This is the basis for the application of the BP algorithm. The back propagation algorithm mainly consists of two loops (excitation propagation and weight update) which are repeated iteratively until the network's response to the input reaches the predetermined target range. In the model learning stage, if the expected output value is not obtained at the output layer, the square sum of the error between the output and the expectation is taken as the objective function, and the back propagation is turned on, the partial derivatives of the objective function with respect to the weights of each neuron are calculated layer by layer, the gradient of the objective function with respect to the weight vector is constructed, and the weight modification is used as the basis for modification. The network learning is completed in the weight modification process, and when the error reaches the expected value, the network learning is completed.

[0179] In an embodiment, the parameters of the first fully connected layer and the second fully connected layer in the video speech recognition model can be jointly trained to modify the parameters of the first fully connected layer and the second fully connected layer; or the parameters of the acoustic neural network can be modified; or the parameters of the video feature extraction network can be modified; or the parameters of the third fully connected layer can be modified. In an embodiment, all network models in the video speech recognition model can be jointly trained, i.e. the parameters of the acoustic neural network, the video speech recognition model, the first fully connected layer, the second fully connected layer and the third fully connected layer are modified at the same time.

[0180] The model training method provided in any of the above embodiments trains the video speech recognition model by fusing the phoneme features in the training speech data and the visual features in the training video image data, the fusion of the visual feature modalities enriches the training feature data used for model training, and can effectively improve the speech recognition accuracy of the video speech recognition model. In addition, the embodiments of the present application not only perform feature extraction on each video frame and audio frame, but also fuse and splice the third phoneme feature information and the third visual feature information after feature classification by the first full connection layer and the second full connection layer, so that the phoneme feature information in the training speech data and the visual feature in the training video image data can be more fully fused, further improving the speech recognition accuracy of the video speech recognition model. The video speech recognition model trained by the model training method provided in the embodiments of the present application can be applied to various scenarios with speech recognition needs or audio and video data processing needs, such as video playing, live broadcast, video conference, cloud technology, artificial intelligence (AI), intelligent transportation, Internet of Vehicles, etc. For example, it can be used for subtitle translation of network / local video, live broadcast / conference on-site speech translation, artificial intelligence understanding of human speech, voice navigation, voice call, etc.

[0181] Referring to Figure 11 The embodiments of the present application provide a speech data processing apparatus, as shown in the accompanying drawings, comprising:

[0182] The acquisition unit 1110 is configured to acquire speech data and video image data of a sound producing object, wherein the speech data comprises a plurality of audio frames, and the video image data comprises a plurality of image frames;

[0183] The phoneme feature extraction unit 1120 is configured to perform phoneme feature extraction on each audio frame in the speech data to obtain a plurality of speech phoneme features, and splice the plurality of speech phoneme features to obtain first phoneme feature information;

[0184] The first processing unit 1130 is configured to input the first phoneme feature information to the first full connection layer for feature classification to obtain second phoneme feature information;

[0185] The visual feature extraction unit 1140 is configured to perform visual feature extraction on each image frame in the video image data to obtain a plurality of visual features, and splice the plurality of visual features to obtain first visual feature information;

[0186] The second processing unit 1150 is configured to input the first visual feature information to the second full connection layer for feature classification to obtain second visual feature information;

[0187] The fusion unit 1160 is configured to splice the second phoneme feature information and the second visual feature information to obtain fusion feature information;

[0188] The speech recognition unit 1170 is configured to perform speech recognition on the fused feature information to obtain speech content information corresponding to the speech data.

[0189] Further, the speech data processing apparatus further comprises:

[0190] An image framing unit is configured to perform framing processing on the video image data to obtain a plurality of image frames.

[0191] A speech framing unit is configured to perform framing processing on the speech data to obtain a plurality of audio frames.

[0192] Further, the phoneme feature extraction unit is further configured to sequentially splice the plurality of speech phoneme features according to the framing sequence of the speech data to obtain the first phoneme feature information.

[0193] The phoneme feature extraction unit is further configured to sequentially splice the plurality of visual features according to the framing sequence of the video image data to obtain the first visual feature information.

[0194] The visual feature extraction unit is further configured to sequentially splice the plurality of visual features according to the framing sequence of the video image data to obtain the first visual feature information.

[0195] Further, the obtaining unit is further configured to obtain video data of the sound- producing object, separate the video data to obtain speech data of the sound-producing object and original video image data, extract lip movement video images of the sound-producing object in the original video image data to obtain the video image data.

[0196] Alternatively, the video data of the sound-producing object is obtained, the original video image data is separated from the video data, the lip movement video images of the sound-producing object in the original video image data are extracted to obtain the video image data, and the speech data corresponding to the video image data in the time sequence is extracted from the video data.

[0197] Further, the phoneme feature extraction unit is further configured to input each audio frame in the speech data into an acoustic neural network to obtain speech phoneme features of each audio frame.

[0198] Further, the visual feature extraction unit is further configured to input each image frame in the video image data into a video feature extraction network for visual feature extraction to obtain visual features of each image frame.

[0199] Further, the speech recognition unit is further configured to input the fused feature information into a third fully connected layer for feature classification to obtain speech classification features, and input the speech classification features into a normalization layer for normalization processing to obtain the speech content information corresponding to the speech data.

[0200] Referring to Figure 11As shown, the embodiment of the present application provides a model training device, comprising:

[0201] The training data extraction unit 1210 is configured to acquire training speech data, training video image data and target speech content information of the training speech data of the sound emitting object, wherein the training speech data comprises a plurality of audio frames, and the training video image data comprises a plurality of image frames.

[0202] The phoneme training feature extraction unit 1220 is configured to perform phoneme feature extraction on each audio frame in the training speech data to obtain a plurality of speech phoneme features, and splice the plurality of speech phoneme features to obtain third phoneme feature information.

[0203] The third processing unit 1230 is configured to input the third phoneme feature information into the first fully connected layer for feature classification to obtain fourth phoneme feature information.

[0204] The visual training feature extraction unit 1240 is configured to perform visual feature extraction on each image frame in the training video image data to obtain a plurality of visual features, and splice the plurality of visual features to obtain third visual feature information.

[0205] The fourth processing unit 1250 is configured to input the third visual feature information into the second fully connected layer for feature classification to obtain fourth visual feature information.

[0206] The training feature fusion unit 1260 is configured to splice the fourth phoneme feature information and the fourth visual feature information to obtain training fusion feature information.

[0207] The speech training data recognition unit 1270 is configured to perform speech recognition on the training fusion feature information to obtain predicted speech content information corresponding to the training speech data.

[0208] The correction unit 1280 is configured to correct parameters of the model according to the predicted speech content information and the target speech content information.

[0209] Further, the training data extraction unit is further configured to: acquire training video data of the sound emitting object and target speech content information corresponding to the training video data, separate the training video data to obtain training speech data and original training video image data of the sound emitting object, extract a lip movement video image of the sound emitting object in the original training video image data to obtain the training video image data.

[0210] Or, the training video data of the sound object and the target speech content information corresponding to the training video data are acquired, the original training video image data is separated from the training video data, the lip movement video image of the sound object in the original training video image data is extracted, the training video image data is obtained, and the training speech data corresponding to the training video image data in the time sequence is extracted from the training video data.

[0211] Further, the phoneme training feature extraction unit is further configured to: input each audio frame in the training speech data into an acoustic neural network to obtain speech phoneme features of each audio frame; the parameters of the model include parameters of the acoustic neural network; and the correction unit is further configured to: correct the parameters of the video feature extraction network according to the predicted speech content information and the target speech content information.

[0212] Further, the visual training feature extraction unit is further configured to: input each image frame in the training video image data into the video feature extraction network for visual feature extraction to obtain visual features in each image frame; the parameters of the model include parameters of the video feature extraction network; and the correction unit is further configured to: correct the parameters of the video feature extraction network according to the predicted speech content information and the target speech content information.

[0213] Further, the speech training data recognition unit is further configured to: input the training fusion feature information into a third fully connected layer for feature classification to obtain speech classification features; and input the speech classification features into a normalization layer to obtain the predicted speech content information corresponding to the speech data.

[0214] The parameters of the model include parameters of the first fully connected layer, parameters of the second fully connected layer, and parameters of the third fully connected layer; and the correction unit is further configured to: correct the parameters of the first fully connected layer, the parameters of the second fully connected layer, and the parameters of the third fully connected layer according to the predicted speech content information and the target speech content information.

[0215] Embodiments of the present application provide a speech data processing apparatus, comprising:

[0216] at least one memory for storing programs;

[0217] at least one processor for loading programs to execute the speech data processing method or model training method provided by any of the above embodiments.

[0218] Embodiments of the present application disclose a storage medium storing a program, which is executed by a processor to implement the speech data processing method or model training method provided by any of the above embodiments.

[0219] The embodiment of the present application discloses a computer program product, comprising computer program / instruction, which realizes the voice data processing method provided by any of the above embodiments or the model training method provided by any of the above embodiments when executed by a processor.

[0220] The embodiment of the present application discloses a computer program, which comprises computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the voice data processing method or the model training method provided by any of the above embodiments.

[0221] The terms "first", "second", "third", "fourth" and the like in the specification of the present application and in the above drawings, if any, are used to distinguish similar objects, and are not necessarily used to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented, for example, in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0222] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0223] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0224] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0225] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0226] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that makes a contribution to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0227] For the step numbers in the above method embodiments, they are set only for the convenience of description and explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0228] The above describes the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A voice data processing method characterized by comprising: The method comprises the following steps: obtaining video data of a sound object, separating the video data to obtain speech data and original video image data of the sound object, extracting lip movement video images of the sound object in the original video image data to obtain video image data; or, obtaining video data of a sound object, separating original video image data from the video data, extracting lip movement video images of the sound object in the original video image data to obtain video image data, and extracting speech data corresponding to the video image data in time sequence from the video data, wherein the speech data comprises a plurality of audio frames, and the video image data comprises a plurality of image frames; extracting phoneme features of each audio frame in the speech data to obtain a plurality of speech phoneme features, and splicing the plurality of speech phoneme features to obtain first phoneme feature information; inputting the first phoneme feature information into a first full connection layer for feature classification to obtain second phoneme feature information; extracting visual features of each image frame in the video image data to obtain a plurality of visual features, and splicing the plurality of visual features to obtain first visual feature information; inputting the first visual feature information into a second full connection layer for feature classification to obtain second visual feature information; splicing the second phoneme feature information and the second visual feature information to obtain fusion feature information; performing speech recognition on the fusion feature information to obtain speech content information corresponding to the speech data.

2. The voice data processing method of claim 1, wherein, After obtaining the speech data and the video image data of the sound object, the method further comprises: performing frame processing on the video image data to obtain a plurality of image frames; performing frame processing on the speech data to obtain a plurality of audio frames.

3. The voice data processing method of claim 1, wherein, The splicing of the plurality of speech phoneme features to obtain the first phoneme feature information comprises: sequentially splicing the plurality of speech phoneme features according to the frame processing order of the speech data to obtain the first phoneme feature information. The splicing of the plurality of visual features to obtain the first visual feature information comprises: sequentially splicing the plurality of visual features according to the frame processing order of the video image data to obtain the first visual feature information.

4. The voice data processing method of claim 1, wherein, The extraction of the phoneme features of each audio frame in the speech data to obtain the plurality of speech phoneme features comprises: inputting each audio frame in the speech data into an acoustic neural network to obtain speech phoneme features of each audio frame.

5. The voice data processing method of claim 1, wherein, The extraction of the visual features of each image frame in the video image data comprises: inputting each image frame in the video image data into a video feature extraction network for visual feature extraction to obtain visual features of each image frame.

6. The voice data processing method of claim 1, wherein, The speech recognition on the fusion feature information to obtain the speech content information corresponding to the speech data comprises: inputting the fusion feature information into a third full connection layer for feature classification to obtain speech classification features; The speech classification features are input into a normalization layer for normalization processing to obtain speech content information corresponding to the speech data.

7. A model training method, comprising: The method comprises the following steps: training video data of a sounder and target speech content information corresponding to the training video data are obtained, the training video data is separated to obtain training speech data of the sounder and original training video image data, lip movement video images of the sounder in the original training video image data are extracted to obtain training video image data; Alternatively, training video data of a sounder and target speech content information corresponding to the training video data are obtained, original training video image data is separated from the training video data, lip movement video images of the sounder in the original training video image data are extracted to obtain training video image data, and training speech data corresponding to the training video image data in time sequence is extracted from the training video data, wherein the training speech data comprises a plurality of audio frames, and the training video image data comprises a plurality of image frames; phoneme features of each of the audio frames in the training speech data are extracted to obtain a plurality of speech phoneme features, and the plurality of speech phoneme features are spliced to obtain third phoneme feature information; the third phoneme feature information is input into a first full connection layer for feature classification to obtain fourth phoneme feature information; visual features of each of the image frames in the training video image data are extracted to obtain a plurality of visual features, and the plurality of visual features are spliced to obtain third visual feature information; the third visual feature information is input into a second full connection layer for feature classification to obtain fourth visual feature information; the fourth phoneme feature information and the fourth visual feature information are spliced to obtain training fusion feature information; speech recognition is performed on the training fusion feature information to obtain predicted speech content information corresponding to the training speech data; parameters of the model are corrected according to the predicted speech content information and the target speech content information. 8.The method of Claim 7, wherein, The method comprises the following steps: each of the audio frames in the training speech data is input into an acoustic neural network to obtain speech phoneme features of each of the audio frames; The method comprises the following steps: each of the image frames in the training video image data is input into a video feature extraction network for visual feature extraction to obtain visual features in each of the image frames; The parameters of the model comprise parameters of the acoustic neural network and parameters of the video feature extraction network, and the correction of the parameters of the model according to the predicted speech content information and the target speech content information comprises correction of at least one of the parameters of the acoustic neural network and the parameters of the video feature extraction network according to the predicted speech content information and the target speech content information. 9.The method of Claim 7, wherein, The speech recognition on the training fusion feature information obtains predicted speech content information corresponding to the training speech data, and the speech recognition on the fusion feature information obtains speech content information corresponding to the speech data. The training fusion feature information is input into a third fully connected layer for feature classification to obtain speech classification features. The speech classification features are input into a normalization layer to obtain predicted speech content information corresponding to the speech data. The parameters of the model include the parameters of the first fully connected layer, the parameters of the second fully connected layer, and the parameters of the third fully connected layer, and the correction of the parameters of the model according to the predicted speech content information and the target speech content information includes correction of the parameters of the first fully connected layer, the parameters of the second fully connected layer, and the parameters of the third fully connected layer according to the predicted speech content information and the target speech content information.

10. A voice data processing apparatus characterized by comprising: It comprises: An acquisition unit is configured to acquire video data of a sound-producing object, separate the video data to obtain speech data and original video image data of the sound-producing object, extract lip movement video images of the sound-producing object in the original video image data to obtain video image data, or acquire video data of a sound-producing object, separate original video image data from the video data, extract lip movement video images of the sound-producing object in the original video image data to obtain video image data, and extract speech data corresponding in time sequence to the video image data from the video data, wherein the speech data comprises a plurality of audio frames, and the video image data comprises a plurality of image frames; A phoneme feature extraction unit is configured to perform phoneme feature extraction on each of the audio frames in the speech data to obtain a plurality of speech phoneme features, and splice the plurality of speech phoneme features to obtain first phoneme feature information; A first processing unit is configured to input the first phoneme feature information into a first fully connected layer for feature classification to obtain second phoneme feature information; A visual feature extraction unit is configured to perform visual feature extraction on each of the image frames in the video image data to obtain a plurality of visual features, and splice the plurality of visual features to obtain first visual feature information; A second processing unit is configured to input the first visual feature information into a second fully connected layer for feature classification to obtain second visual feature information; A fusion unit is configured to splice the second phoneme feature information and the second visual feature information to obtain fusion feature information; A speech recognition unit is configured to perform speech recognition on the fusion feature information to obtain speech content information corresponding to the speech data.

11. A voice data processing apparatus characterized by comprising: It comprises: At least one memory for storing a program; At least one processor for loading the program to execute the speech data processing method of any one of claims 1 to 6 or the model training method of any one of claims 7 to 9. 12.A computer readable storage medium storing computer executable instructions, and the computer executable instructions comprise the steps of: The computer executable instructions are used to execute the speech data processing method of any one of claims 1 to 6 or the model training method of any one of claims 7 to 9.

13. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the voice data processing method of any one of claims 1 to 6 or implement the model training method of any one of claims 7 to 9.

Images