Voice processing method, voice processing model training method, device and equipment

Abstract

The invention provides a speech processing method, a speech processing model training method, a speech processing model training device and equipment, and relates to the technical field of computers, in particular to the technical field of artificial intelligence such as speech enhancement and deep learning. According to the specific implementation scheme, the method comprises the steps of obtaining a lip movement video corresponding to a target sound area in a vehicle and mixed voice in the vehicle; performing voice enhancement on the mixed voice to obtain initial voice corresponding to the target voice area; performing feature extraction on the lip movement video to obtain lip movement features; performing feature extraction on the initial voice to obtain audio features; and determining a target voice corresponding to the target voice area according to the audio features and the lip movement features.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of artificial intelligence technology such as speech enhancement and deep learning, and specifically to a speech processing method, a training method, a device and equipment for a speech processing model. Background Art

[0002] In-vehicle voice interaction can convert the user's voice commands into commands that the car computer can understand, and then convey the system's feedback information to the user in the form of voice through speech synthesis technology. In-vehicle voice interaction technology is widely used in in-vehicle scenarios such as in-vehicle navigation, in-vehicle entertainment, in-vehicle communication, vehicle control, and intelligent assistants.

[0003] However, in scenarios such as mobile phone navigation or watching short videos on the phone, the inability to block the sound of the phone will affect the accuracy of the user's voice recognition, making it impossible to accurately judge the user's intentions, affecting the interactive experience. There is even the possibility that the mobile phone interferes with the interaction with the car computer, causing the car computer to make incorrect feedback. Summary of the invention

[0004] The present invention provides a speech processing method, a training method, a device and equipment for a speech processing model.

[0005] According to a first aspect of the present disclosure, there is provided a speech processing method, comprising:

[0006] Acquire a lip movement video corresponding to a target sound zone in a vehicle and a mixed voice in the vehicle;

[0007] Performing speech enhancement on the mixed speech to obtain an initial speech corresponding to the target voice range;

[0008] Performing feature extraction on the lip movement video to obtain lip movement features;

[0009] Performing feature extraction on the initial speech to obtain audio features;

[0010] The target speech corresponding to the target sound range is determined according to the audio feature and the lip movement feature.

[0011] According to a second aspect of the present disclosure, a method for training a speech processing model is provided, comprising:

[0012] Obtaining a sample voice, a first lip movement video, and a first voice tag corresponding to the sample voice zone;

[0013] Inputting the first lip movement video into a lip movement encoder in an initial speech processing model to obtain sample lip movement features;

[0014] Inputting the sample speech into the initial audio code in the initial speech processing model to obtain sample audio features,

[0015] Inputting the sample lip movement features and the sample audio features into the initial enhancement network in the initial speech processing model to obtain a first predicted speech corresponding to the sample sound range;

[0016] According to the difference between the first predicted speech and the first speech label, the initial audio encoder and the initial enhancement network are modified to obtain a speech processing model, and the speech processing model is used to implement the speech processing method as described in the first aspect.

[0017] According to a third aspect of the present disclosure, there is provided a speech processing device, comprising:

[0018] A first acquisition module is used to acquire a lip movement video corresponding to a target sound zone in a vehicle and a mixed voice in the vehicle;

[0019] A second acquisition module is used to perform speech enhancement on the mixed speech to obtain an initial speech corresponding to the target voice range;

[0020] A first extraction module, used for performing feature extraction on the lip movement video to obtain lip movement features;

[0021] A second extraction module is used to extract features from the initial speech to obtain audio features;

[0022] A determination module is used to determine the target speech corresponding to the target sound area according to the audio features and the lip movement features.

[0023] According to a fourth aspect of the present disclosure, there is provided a training device for a speech processing model, comprising:

[0024] An acquisition module, used to acquire a sample voice, a first lip movement video and a first voice tag corresponding to a sample voice zone;

[0025] A first extraction module, configured to input the first lip movement video into a lip movement encoder in an initial speech processing model to obtain sample lip movement features;

[0026] The second extraction module is used to input the sample speech into the initial audio code in the initial speech processing model to obtain the sample audio feature.

[0027] A prediction module, used for inputting the sample lip movement features and the sample audio features into an initial enhancement network in the initial speech processing model to obtain a first predicted speech corresponding to the sample sound range;

[0028] A correction module is used to correct the initial audio encoder and the initial enhancement network according to the difference between the first predicted speech and the first speech label to obtain a speech processing model, and the speech processing model is used to implement the speech processing method as described in the first aspect.

[0029] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0030] at least one processor; and

[0031] a memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the speech processing method as described in the first aspect, or execute the training method of the speech processing model as described in the second aspect.

[0033] According to the sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the speech processing method as described in the first aspect, or to execute the speech processing model training method as described in the second aspect.

[0034] According to the seventh aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the speech processing method as described in the first aspect, or implement the steps of the speech processing model training method as described in the second aspect.

[0035] The speech processing method, speech processing model training method, device and equipment provided by the present disclosure have the following

[0036] Beneficial effects:

[0037] In the disclosed embodiment, the lip movement video corresponding to the target sound zone in the vehicle and the mixed speech in the vehicle are first obtained, and then the mixed speech is voice enhanced to obtain the initial speech corresponding to the target sound zone, and then the lip movement video is feature extracted to obtain the lip movement features, and the initial speech is feature extracted to obtain the audio features, and finally the target speech corresponding to the target sound zone is determined based on the audio features and the lip movement features. Thus, the mixed speech can first be subjected to pure speech enhancement to obtain the initial speech corresponding to the target sound zone after pure speech enhancement, and then combined with the lip movement video corresponding to the target sound zone, the initial speech after pure speech enhancement is further multimodally enhanced, thereby improving the accuracy of user speech recognition, and then accurately judging the user's intention, so that the vehicle computer can make correct feedback and improve the user experience.

[0038] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0040] Figure 1 is a flow chart of a speech processing method provided according to an embodiment of the present disclosure;

[0041] Figure 2 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure;

[0042] Figure 3 is a structural schematic diagram of a speech processing model provided according to another embodiment of the present disclosure;

[0043] Figure 4 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure;

[0044] Figure 5 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure;

[0045] Figure 6 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure;

[0046] Figure 7 is a flow chart of a method for training a speech processing model provided according to another embodiment of the present disclosure;

[0047] Figure 8 is a structural schematic diagram of a speech processing device provided according to an embodiment of the present disclosure;

[0048] Fig. 9 is a structural schematic diagram of a training device for a speech processing model provided according to an embodiment of the present disclosure;

[0049] Fig.10 The block diagram is a block diagram of an electronic device used to implement the speech processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0051] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0052] The following describes the speech processing method, speech processing model training method, device and equipment of the embodiments of the present disclosure with reference to the accompanying drawings.

[0053] It should be noted that the executor of the speech processing method of this embodiment is a speech processing device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include but is not limited to a terminal, a server, etc.

[0054] Figure 1 It is a flowchart of a speech processing method provided according to an embodiment of the present disclosure.

[0055] like Figure 1 As shown, the speech processing method includes:

[0056] S101, obtaining a lip movement video corresponding to a target sound zone in a vehicle and a mixed voice in the vehicle.

[0057] In some embodiments, the sound zones can be divided according to the number of seats in the vehicle. For example, a car can be divided into four sound zones: driver's seat, co-driver's seat, left rear, and right rear.

[0058] In some embodiments, the target sound zone may be a sound zone in which a user is present in the vehicle. The number of target sound zones may be one or more, and the present disclosure does not limit this. For example, if there are users in the main and co-driver seats, the target sound zones are determined to be the main and co-driver seats. If there is only a user in the main driver seat, the target sound zone is determined to be the main driver seat.

[0059] The lip movement video may be a video containing the lips of the user.

[0060] The mixed voice may be the voice collected by the microphone array of the vehicle.

[0061] S102: Perform speech enhancement on the mixed speech to obtain an initial speech corresponding to the target voice range.

[0062] In some embodiments, operations such as sound source localization and noise reduction processing may be performed on the mixed speech to determine the initial speech corresponding to the target sound range.

[0063] In some embodiments, beamforming can be performed through complex neural network modules such as complex convolutional layers and complex linear layers to extract complex features containing directional phase information from the mixed speech, and then the initial speech corresponding to the target sound area is obtained through a series of neural network structures, such as convolutional neural networks (CNN), long short-term memory networks (LSTM), Transformer, Conformer, etc.

[0064] It should be noted that the pure speech enhancement technology can separate the speech signal of each sound zone, but it cannot suppress the interference of the same sound zone. For example, in one sound zone, there are both user speech and speech produced by the user's mobile phone playback, and there may also be interference speech from other sound zones. Therefore, in the embodiment of the present disclosure, after obtaining the initial speech corresponding to each sound zone, the lip movement video corresponding to the target sound zone can be further combined to determine the user speech in the target sound zone, so that the user speech can be accurately recognized.

[0065] S103, extracting features from the lip movement video to obtain lip movement features.

[0066] In some embodiments, the lip movement video can be input into an image feature extraction network to obtain lip movement features.

[0067] In some embodiments, the lip shape change trajectory can also be extracted from the lip movement video to obtain the lip movement features.

[0068] S104: extract features from the initial speech to obtain audio features.

[0069] In some embodiments, the initial speech may be sent to a speech feature extraction network to obtain speech features.

[0070] In some embodiments, when there are at least two initial speech, at least two initial speech can be input into the speech feature extraction network together to obtain the audio features corresponding to each initial speech. Alternatively, each initial speech can be input into the speech feature extraction network separately to obtain the audio features corresponding to each initial speech.

[0071] S105, determining the target speech corresponding to the target pitch range according to the audio features and the lip movement features.

[0072] The target voice may be the voice generated by a user speaking in the target voice zone.

[0073] In some embodiments, the audio features and lip movement features may be fused to obtain fused features, and then the fused features may be recognized to obtain the target speech corresponding to the target pitch range.

[0074] In some embodiments, when the number of target sound zones is at least two, the lip movement features corresponding to each target sound zone may be directly fused with the audio features to obtain a fused feature corresponding to each target sound zone. Alternatively, the lip movement features corresponding to at least two target sound zones may be fused with the audio features to obtain a fused feature. This disclosure is not limited to this.

[0075] In the disclosed embodiment, the lip movement video corresponding to the target sound zone in the vehicle and the mixed speech in the vehicle are first obtained, and then the mixed speech is voice enhanced to obtain the initial speech corresponding to the target sound zone, and then the lip movement video is feature extracted to obtain the lip movement features, and the initial speech is feature extracted to obtain the audio features, and finally the target speech corresponding to the target sound zone is determined based on the audio features and the lip movement features. Thus, the mixed speech can first be subjected to pure speech enhancement to obtain the initial speech corresponding to the target sound zone after pure speech enhancement, and then combined with the lip movement video corresponding to the target sound zone, the initial speech after pure speech enhancement is further multimodally enhanced, thereby improving the accuracy of user speech recognition, and then accurately judging the user's intention, so that the vehicle computer can make correct feedback and improve the user experience.

[0076] Figure 2 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure;

[0077] like Figure 2 As shown, the speech processing method includes:

[0078] S201, obtaining a lip movement video corresponding to a target sound zone in a vehicle and a mixed voice in the vehicle.

[0079] S202: Perform speech enhancement on the mixed speech to obtain an initial speech corresponding to the target voice range.

[0080] The specific implementation forms of step S201 and step S202 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0081] S203, when the number of target sound zones is at least two, based on the lip movement encoder in the speech processing model, feature extraction is performed on the lip movement video corresponding to each target sound zone to obtain the lip movement features corresponding to each target sound zone.

[0082] The speech processing model includes a lip movement encoder, an audio encoder, and an enhanced network, and the enhanced network includes a fusion network and a recognition network.

[0083] Figure 3 A schematic diagram of a speech processing model provided by an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the lip movement encoder may include a 3D convolutional network (conv3d), a 3D pooling network (Pool3d), a deep residual network (ResidualNetwork, ResNet), and a fully connected network (Full Connected, FC). The lip movement encoder may be used to track the lip movement trajectory of the lip shape in the time dimension.

[0084] The parameters of the lip movement encoder are the same as those of the lip movement encoder in the trained lip reading recognition model. It should be noted that the lip movement encoder in the lip reading recognition model can learn certain semantic expression capabilities during training. Therefore, by using the same lip movement encoder as that in the lip reading recognition model in the speech processing model, lip movement features containing semantic information can be extracted from the lip movement video, thereby more accurately determining the target speech corresponding to the target tone range.

[0085] In some embodiments, the specific implementation forms of training the speech processing model and the lip reading recognition model can refer to the detailed description in the subsequent embodiments of this disclosure, and will not be described in detail here.

[0086] S204, based on the audio encoder in the speech processing model, jointly encode the initial speech corresponding to at least two target sound areas respectively, so as to obtain the audio feature corresponding to each target sound area.

[0087] Among them, it should be noted that since the sound zones in the vehicle are small and adjacent, the initial speech of the target adjacent zone may contain part of the speech of the adjacent sound zone. For example, when the user of the front passenger seat speaks, the target sound zone corresponding to the main driver will contain a small amount of the front passenger's speech. Therefore, the initial speech corresponding to at least two target sound zones is jointly encoded, and the same speech in different sound zones can be uniformly encoded, which can further improve the accuracy of the target speech corresponding to each target sound zone.

[0088] like Figure 3 As shown, the audio encoder may include a filter bank (Filter Bank, FBank), a linear layer (Linear), a gated recurrent unit (Gated Recurrent Unit, GRU), etc. This disclosure does not limit this.

[0089] S205, inputting the speech features and lip movement features into a fusion network in the speech processing model to obtain fusion features.

[0090] In some embodiments, since the audio source of the mixed speech and the video source of the lip movement video are different, the sampling rates of the two may be inconsistent, so the lip movement features and the audio features need to be aligned in the time dimension. For example, upsampling or downsampling can be used to align the lip movement features and the audio features in the time dimension.

[0091] In some embodiments, the fusion network can be composed of a temporal convolutional network (TCN), a GRU, etc. Figure 3 As shown, the fusion network can be composed of a SUM network and a TCN network. This disclosure does not limit this.

[0092] S206, inputting the fused features into the recognition network in the speech processing model to obtain the target speech corresponding to the target pitch range.

[0093] like Figure 3 As shown, the identification network may be a fully connected network (Fully Connected, FC), which is not limited in the present disclosure.

[0094] In some embodiments, the fusion network is used to model the correlation between speech features and lip movement features, enhance signals with high correlation, and suppress signals with low correlation, so that the speech of the target speaker can be accurately separated in ultra-low signal-to-noise ratio scenarios, greatly improving the enhancement effect in complex scenarios.

[0095] In the disclosed embodiment, the lip movement video corresponding to the target sound zone in the vehicle and the mixed speech in the vehicle are first obtained, and then the mixed speech is enhanced to obtain the initial speech corresponding to the target sound zone, and then based on the lip movement encoder in the speech processing model, the lip movement video corresponding to each target sound zone is respectively extracted to obtain the lip movement features corresponding to each target sound zone, and based on the audio encoder in the speech processing model, the initial speech corresponding to at least two target sound zones is jointly encoded to obtain the audio features corresponding to each target sound zone, and finally the speech features and lip movement features are input into the fusion network in the speech processing model to obtain the fusion features, and the fusion features are input into the recognition network in the speech processing model to obtain the target speech corresponding to the target sound zone. Thus, the mixed speech can be firstly enhanced in pure speech to obtain the initial speech corresponding to the target sound zone after pure speech enhancement, and then the initial speech and lip movement video are input into the speech processing model, and the initial speech after pure speech enhancement is multimodally enhanced, thereby not only improving the accuracy of user speech recognition, but also improving the efficiency of user speech recognition based on the speech processing model.

[0096] In some embodiments, when the number of target sound zones is one, the lip movement video corresponding to the target sound zone can also be input into a lip movement encoder to obtain lip movement features, the initial speech can be input into an audio encoder to obtain audio features, the audio features and lip movement features can be input into a fusion network in a speech processing model to obtain fusion features, and the fusion features can be input into a recognition network in the speech processing model to obtain the target speech corresponding to the target sound zone.

[0097] In some embodiments, when the number of target sound zones is at least two, the lip movement video and initial speech corresponding to each target sound zone may be respectively input into the speech processing model to obtain the target speech corresponding to each target sound zone.

[0098] Figure 4 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure;

[0099] like Figure 4 As shown, the speech processing method includes:

[0100] S401, obtaining video data corresponding to each sound zone in the vehicle.

[0101] The video data may be collected by one or more cameras deployed in the vehicle. The present disclosure is not limited to this. In some embodiments, only one camera may be deployed in the vehicle to collect videos of all audio zones, and then the video may be processed based on the position of each audio zone to obtain video data of each audio zone. Alternatively, a camera may be deployed in each audio zone to obtain video data corresponding to each audio zone.

[0102] S402: Perform object detection on the video data corresponding to each sound zone to obtain a first detection result corresponding to each sound zone.

[0103] The target detection may be face detection or body detection, which is not limited in the present disclosure.

[0104] The first detection result is used to indicate whether a preset object is detected. The preset object may be a person or a face.

[0105] S403: Determine the sound zone containing the preset object in the first detection result as the target sound zone.

[0106] Thus, target detection is performed on each sound zone, and the sound zone where the user exists is determined as the target sound zone, so that the target sound zone where the user is located can be accurately determined.

[0107] S404, performing lip region recognition on each frame of image in the video data corresponding to the target sound area, and determining a second detection result corresponding to each frame of image.

[0108] The second detection result may be used to indicate that the lip region is detected and the position of the lip region in the image when the lip region is detected; or may be used to indicate that the lip region is not detected.

[0109] In some embodiments, a target detection model can be trained based on sample data containing lips and position labels of lips in sample images, and then the trained target detection model can be used to identify the lip area of ​​each frame image in the video data to determine the second detection result corresponding to each frame image.

[0110] S405: Extract the lip area of ​​the video data according to the second detection result corresponding to each frame of the image to obtain a lip movement video.

[0111] In the disclosed embodiment, extracting a lip movement video containing only the lip area from the video data can reduce the amount of computational data for extracting lip movement features, thereby improving the efficiency of determining the target voice.

[0112] In some embodiments, when the second detection result corresponding to the image indicates that the lip area is detected, the lip image can be extracted based on the position of the lip area in the image to obtain a lip movement video.

[0113] In some embodiments, when the second detection result corresponding to any frame image indicates that the lip area is not detected, the image corresponding to any frame image in the lip movement video is determined to be a black image. Thus, by using a black image to replace the image of the missing lip area, the target speech when the lip vision is occasionally lost can be reverted to the result after pure speech enhancement, thereby ensuring that the speech is enhanced by multimodal speech when there is lip vision, and the speech is enhanced by pure speech when the lip vision disappears occasionally, thereby ensuring the quality of the determined target speech.

[0114] S406, obtaining mixed voice in the vehicle.

[0115] S407: Perform speech enhancement on the mixed speech to obtain an initial speech corresponding to the target voice range.

[0116] S408: extract features from the lip movement video to obtain lip movement features.

[0117] S409: extract features from the initial speech to obtain audio features.

[0118] S410, determining a target speech corresponding to a target pitch range according to audio features and lip movement features.

[0119] The specific implementation forms of step S406 to step S410 may refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0120] In the disclosed embodiment, the video data corresponding to each sound zone in the vehicle is first obtained, and then the target detection is performed on the video data corresponding to each sound zone to obtain the first detection result corresponding to each sound zone, and the sound zone containing the preset object in the first detection result is determined as the target sound zone, and then the lip area recognition is performed on each frame image in the video data corresponding to the target sound zone to determine the second detection result corresponding to each frame image, and the lip area is extracted from the video data according to the second detection result corresponding to each frame image to obtain the lip movement video. In this way, not only can the target sound zone where the user is located be accurately determined, but also the lip movement video containing only the lip area can be extracted from the video data corresponding to the target sound zone, which reduces the amount of calculation data for extracting lip movement features, and thus can improve the efficiency of determining the target voice.

[0121] Figure 5 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure;

[0122] like Figure 5 As shown, the speech processing method includes:

[0123] S501, obtaining a lip movement video corresponding to a target sound zone in a vehicle and a mixed voice in the vehicle.

[0124] S502, sequentially directing a main lobe of a beam formed by a microphone array in a vehicle to a target sound area, and directing a side lobe of the beam to other sound areas except the target sound area, so as to extract a first speech corresponding to each target sound area from the mixed speech.

[0125] Among them, the microphone array is used to collect mixed speech.

[0126] For example, there are two target sound zones, namely the driver's seat and the co-driver's seat. The main lobe of the beam formed by the microphone array in the vehicle is directed to the driver's seat, and the side lobe of the beam is directed to other sound zones except the driver's seat, so that the voices in other sound zones except the driver's seat can be suppressed and the first voice of the driver's seat can be obtained. The main lobe of the beam formed by the microphone array in the vehicle is directed to the co-driver's seat, and the side lobe of the beam is directed to other sound zones except the co-driver's seat, so that the voices in other sound zones except the co-driver's seat can be suppressed and the first voice of the co-driver's seat can be obtained.

[0127] S503: Perform at least one operation of noise elimination, echo elimination, interference suppression, and howling suppression on the first speech to obtain an initial speech.

[0128] It should be noted that since the beamforming technology cannot completely suppress the noise outside the beam pointing, and noise will be mixed into the main lobe of the beam, there will be residual noise in the extracted first speech. Post-processing filtering technology is required to continue to eliminate the noise of the extracted first speech to improve the signal-to-noise ratio.

[0129] The echo cancellation may be used to eliminate the echo contained in the first speech.

[0130] Among them, interference suppression can be used to suppress the interference of other voice ranges on the speech of the target voice range.

[0131] S504: extract features from the lip movement video to obtain lip movement features.

[0132] S505: extract features from the initial speech to obtain audio features.

[0133] S506: Determine the target speech corresponding to the target pitch range according to the audio features and the lip movement features.

[0134] The specific implementation forms of steps S504 to S506 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0135] In the disclosed embodiment, after obtaining the mixed speech, the main lobe of the beam formed by the microphone array in the vehicle is sequentially directed to a target sound area, and the side lobe of the beam is directed to other sound areas except the target sound area, so as to extract the first speech corresponding to each target sound area from the mixed speech, and perform at least one operation of noise elimination, echo elimination, interference suppression, and howling suppression on the first speech to obtain the initial speech. Thus, the mixed speech can be enhanced by using pure speech enhancement means such as beamforming, noise elimination, echo elimination, interference suppression, and howling suppression, which provides a relatively clean initial speech for the subsequent multimodal speech enhancement combined with lip movement video, thereby improving the accuracy of the determined target speech.

[0136] Figure 6 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure;

[0137] like Figure 6 As shown, the speech processing method includes:

[0138] S601, performing lip region recognition on each frame of image in the video data corresponding to the target sound area, and determining a second detection result corresponding to each frame of image.

[0139] The second detection result may be used to indicate that the lip region is detected and the position of the lip region in the image when the lip region is detected; or may be used to indicate that the lip region is not detected.

[0140] In some embodiments, a target detection model can be trained based on sample data containing lips and position labels of lips in sample images, and then the trained target detection model can be used to identify the lip area of ​​each frame image in the video data to determine the second detection result corresponding to each frame image.

[0141] S602: When the second detection result corresponding to any frame image indicates that the lip area is not detected, determine that the image corresponding to any frame image in the lip movement video is a black image.

[0142] S603, obtaining mixed voice in the vehicle.

[0143] S604: Perform speech enhancement on the mixed speech to obtain an initial speech corresponding to the target voice range.

[0144] The specific implementation forms of step S603 and step S604 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0145] S605, when the number of frames of continuous black images in the lip movement video corresponding to any target sound zone is greater than a threshold, the initial speech corresponding to any target sound zone is determined as the target speech corresponding to any target sound zone.

[0146] The threshold may be 100 frames, 200 frames, etc., which is not limited in the present disclosure.

[0147] In the disclosed embodiment, in a scenario where the lip vision of any target sound zone is lost for a long time, the initial speech corresponding to any target sound zone can be directly determined as the target speech without further executing steps S103 to S105, thereby saving the amount of calculation while ensuring the speech recognition results of the mixed speech.

[0148] In the disclosed embodiment, lip area recognition is performed on each frame of image in the video data corresponding to the target sound zone, and the second detection result corresponding to each frame of image is determined. According to the second detection result corresponding to each frame of image, when the second detection result corresponding to any frame of image indicates that the lip area is not detected, the image corresponding to any frame of image in the lip movement video is determined to be a black image, and then the mixed speech in the vehicle is obtained, and the mixed speech is speech enhanced to obtain the initial speech corresponding to the target sound zone. Finally, when the number of consecutive black image frames in the lip movement video corresponding to any target sound zone is greater than a threshold, the initial speech corresponding to any target sound zone is determined as the target speech corresponding to any target sound zone. Thus, in the scenario where the lip vision is lost for a long time in any target sound zone, the initial speech after pure speech enhancement can be directly determined as the target speech, which ensures that the user's speech can still be obtained more accurately in the scenario where the user's lip vision is lost, and the ability of human-vehicle interaction is guaranteed.

[0149] Figure 7 is a flow chart of a speech processing method provided according to another embodiment of the present disclosure, such as Figure 7 As shown, the training of the speech processing model includes:

[0150] S701, obtaining a sample voice, a first lip movement video and a first voice tag corresponding to a sample voice zone.

[0151] The sample sound zone may be a sound zone where there is a user in the vehicle. The number of the sample sound zones may be one or more, and the present disclosure does not limit this. For example, if there are users in the main and co-driver seats, the sample sound zones are determined to be the main and co-driver seats. If there is only a user in the main driver seat, the sample sound zone is determined to be the main driver seat.

[0152] The first lip movement video may be a video corresponding to the lips of the user in the sample sound area. In some embodiments, lip area detection may be performed on each frame of the video data corresponding to the sample sound area, and the lip area may be captured from each frame to obtain a lip movement video.

[0153] In some embodiments, when the lip detection result corresponding to any frame image indicates that the lip area is not detected, the image corresponding to any frame image in the first lip movement video is determined to be a black image.

[0154] The first voice tag may be the voice produced by a user speaking in the sample voice zone.

[0155] In some embodiments, when any frame image in the first lip movement video is a black image, based on the timestamp corresponding to any frame image, the sample voice segment corresponding to the timestamp is determined from the sample voice, and the voice segment corresponding to the timestamp in the first voice tag is determined to be the sample voice segment. Thus, in the scenario where lip vision is lost, the initial voice processing model can be trained based on the same input voice and output voice, so that the target voice output by the trained language processing model is the same as the initial voice after pure voice enhancement, so that the target voice obtained when lip vision is occasionally lost can be reverted to the result after pure voice enhancement, which not only ensures that when there is lip vision, the voice after multimodal voice enhancement can be used, but also ensures that when the lip vision disappears occasionally, the voice after pure voice enhancement can be used, thereby ensuring the quality of the determined target voice.

[0156] S702, input the first lip movement video into the lip movement encoder in the initial speech processing model to obtain sample lip movement features.

[0157] It should be noted that the structure of the initial speech processing model is the same as the structure of the trained speech processing model, such as Figure 3 As shown, no further details are given here.

[0158] Among them, the lip movement encoder is used to track the lip movement trajectory in the time dimension and extract the lip movement features.

[0159] The lip movement encoder in the initial speech processing model is pre-trained and does not need further training. In some embodiments, the parameters of the lip movement encoder are the same as the parameters of the lip movement encoder in the trained lip reading recognition model. That is, the lip reading recognition model can be trained to obtain the lip movement encoder.

[0160] In some embodiments, a second lip movement video and a corresponding second voice tag are obtained, and the second lip movement video is input into an initial lip reading recognition model to obtain a second predicted voice, wherein the initial lip reading recognition model includes an initial lip movement encoder, and then the initial lip movement encoder is corrected according to the difference between the second predicted voice and the second voice tag to obtain a lip movement encoder. Thus, the lip movement encoder is obtained by training the initial lip reading recognition model, so that the lip movement encoder can learn certain semantic expression capabilities during training, thereby obtaining lip movement features with voice information.

[0161] The initial lip reading recognition model also includes a recognition encoder and a recognition decoder.

[0162] The recognition encoder is used to transform the lip movement features into recognition features containing semantic information. The recognition encoder can be composed of one or more Conformer networks, or can also include LSTM, GRU and other structures.

[0163] The recognition decoder is used to decode the recognition features containing semantic information to obtain the lip reading recognition results. It is composed of one or more Transformer networks, and can also contain LSTM, GRU and other structures.

[0164] In some embodiments, the difference between the second predicted speech and the second speech label may be calculated using a connectionist temporal classification loss function or the like.

[0165] In the disclosed embodiment, after obtaining the trained lip reading recognition model, the lip movement encoder in the initial speech processing model can be initialized based on the lip movement encoder in the lip reading recognition model, and the parameters can be frozen so that it does not participate in the training of the initial speech processing model.

[0166] S703, input the sample speech into the initial audio code in the initial speech processing model to obtain the sample audio feature.

[0167] In some embodiments, since the sound zones in the vehicle are small and adjacent, the initial speech of the target adjacent zone may contain part of the speech of the adjacent zone. For example, when the user of the co-pilot is speaking, the target sound zone corresponding to the driver may contain a small amount of the co-pilot's speech. Therefore, when the number of sample sound zones is at least two, the sample speech of at least two sample sound zones can be input into the initial audio encoding for joint encoding. Alternatively, the sample speech corresponding to each sample sound zone can be input into the initial audio encoder in sequence.

[0168] S704: Input the sample lip movement features and the sample audio features into the initial enhancement network in the initial speech processing model to obtain a first predicted speech corresponding to the sample sound range.

[0169] Among them, the initial enhancement network includes an initial fusion network and an initial recognition network.

[0170] In some embodiments, the sample lip movement features and the sample audio features are input into the initial fusion network in the initial enhancement network to obtain the sample fusion features corresponding to the sample sound area, and the sample fusion features are input into the initial recognition network in the initial enhancement network to obtain the first predicted speech corresponding to the sample sound area.

[0171] For example, when the number of sample sound zones is at least two, the sample speech and the first lip movement video corresponding to at least two sample sound zones can be input into the initial speech processing model together. Alternatively, the sample speech and the first lip movement video corresponding to each sample sound zone can be input into the initial speech processing model to obtain the first predicted speech corresponding to each sample sound zone. For example, the sample speech and the first lip movement video corresponding to the first sample sound zone are first input into the initial speech processing model to obtain the first predicted speech corresponding to the first sample sound zone, and then the sample speech and the first lip movement video corresponding to the second sample sound zone are input into the initial speech processing model to obtain the first predicted speech corresponding to the second sample sound zone.

[0172] S705: According to the difference between the first predicted speech and the first speech label, the initial audio encoder and the initial enhancement network are modified to obtain a speech processing model.

[0173] Among them, the speech processing model is used to implement the speech processing method in the embodiment of the present disclosure.

[0174] In some embodiments, a mean squared error loss function (MSEloss) or a signal-to-noise ratio loss (SNR loss) may be used to calculate the difference between the first predicted speech and the first speech label.

[0175] In the disclosed embodiment, a sample speech, a first lip movement video, and a first speech label corresponding to the sample sound zone are obtained, the first lip movement video is input into the lip movement encoder in the initial speech processing model to obtain sample lip movement features, the sample speech is input into the initial audio encoder in the initial speech processing model to obtain sample audio features, the sample lip movement features and the sample audio features are input into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample sound zone, and finally, according to the difference between the first predicted speech and the first speech label, the initial audio encoder and the initial enhancement network are corrected to obtain the speech processing model. Thus, the initial speech processing model is trained based on the sample speech, the first lip movement video, and the first speech label corresponding to the sample sound zone to obtain a speech processing model that can perform multimodal speech enhancement, which provides support for combining the lip movement video to perform multimodal enhancement on the initial speech after pure speech enhancement, and further improves the efficiency of determining the target speech.

[0176] Figure 8 is a structural diagram of a speech processing device provided according to an embodiment of the present disclosure; Figure 8 As shown, the speech processing device 800 includes:

[0177] The first acquisition module 801 is used to acquire the lip movement video corresponding to the target sound zone in the vehicle and the mixed speech in the vehicle;

[0178] The second acquisition module 802 is used to perform speech enhancement on the mixed speech to obtain the initial speech corresponding to the target voice range;

[0179] The first extraction module 803 is used to extract features from the lip movement video to obtain lip movement features;

[0180] The second extraction module 804 is used to extract features from the initial speech to obtain audio features;

[0181] The determination module 805 is used to determine the target speech corresponding to the target pitch range according to the audio features and lip movement features.

[0182] In some embodiments of the present disclosure, the first extraction module 803 is used to:

[0183] When the number of target sound zones is at least two, feature extraction is performed on the lip movement video corresponding to each target sound zone based on the lip movement encoder in the speech processing model to obtain the lip movement features corresponding to each target sound zone, wherein the parameters of the lip movement encoder are the same as those of the lip movement encoder in the trained lip reading recognition model.

[0184] In some embodiments of the present disclosure, the second extraction module 804 is used to:

[0185] Based on the audio encoder in the speech processing model, the initial speech corresponding to at least two target sound areas is jointly encoded to obtain the audio features corresponding to each target sound area.

[0186] In some embodiments of the present disclosure, the determination module 805 is used to:

[0187] Input the speech features and lip movement features into the fusion network in the speech processing model to obtain fusion features;

[0188] The fused features are input into the recognition network in the speech processing model to obtain the target speech corresponding to the target pitch range.

[0189] In some embodiments of the present disclosure, a processing module is further included, which is used to:

[0190] Obtain video data corresponding to each sound zone in the vehicle;

[0191] Performing target detection on the video data corresponding to each sound zone to obtain a first detection result corresponding to each sound zone;

[0192] The sound zone containing the preset object in the first detection result is determined as the target sound zone.

[0193] In some embodiments of the present disclosure, the first acquisition module 801 is used to:

[0194] Perform lip region recognition on each frame of image in the video data corresponding to the target sound area, and determine a second detection result corresponding to each frame of image;

[0195] According to the second detection result corresponding to each frame of image, the lip area is extracted from the video data to obtain the lip movement video.

[0196] In some embodiments of the present disclosure, the first acquisition module 801 is used to:

[0197] When the second detection result corresponding to any frame image indicates that the lip area is not detected, it is determined that the image corresponding to any frame image in the lip movement video is a black image.

[0198] In some embodiments of the present disclosure, a second processing module is further included, which is used to:

[0199] When the number of frames of continuous black images in the lip movement video corresponding to any target sound zone is greater than a threshold, the initial speech corresponding to any target sound zone is determined as the target speech corresponding to any target sound zone.

[0200] In some embodiments of the present disclosure, the second acquisition module 802 is used to:

[0201] sequentially directing a main lobe of a beam formed by a microphone array in a vehicle toward a target sound area, and directing a side lobe of the beam toward other sound areas except the target sound area, so as to extract a first speech corresponding to each target sound area from a mixed speech, wherein the microphone array is used to collect the mixed speech;

[0202] At least one of noise elimination, echo elimination, interference suppression, and howling suppression is performed on the first speech to obtain an initial speech.

[0203] It should be noted that the above explanation of the speech processing method is also applicable to the speech processing device of this embodiment and will not be repeated here.

[0204] In the disclosed embodiment, the lip movement video corresponding to the target sound zone in the vehicle and the mixed speech in the vehicle are first obtained, and then the mixed speech is voice enhanced to obtain the initial speech corresponding to the target sound zone, and then the lip movement video is feature extracted to obtain the lip movement features, and the initial speech is feature extracted to obtain the audio features, and finally the target speech corresponding to the target sound zone is determined based on the audio features and the lip movement features. Thus, the mixed speech can first be subjected to pure speech enhancement to obtain the initial speech corresponding to the target sound zone after pure speech enhancement, and then combined with the lip movement video corresponding to the target sound zone, the initial speech after pure speech enhancement is further multimodally enhanced, thereby improving the accuracy of user speech recognition, and then accurately judging the user's intention, so that the vehicle computer can make correct feedback and improve the user experience.

[0205] Fig. 9 is a structural diagram of a training device for a speech processing model provided according to an embodiment of the present disclosure; Fig. 9 As shown, the training device 900 of the speech processing model includes:

[0206] An acquisition module 901 is used to acquire a sample voice, a first lip movement video and a first voice tag corresponding to a sample voice zone;

[0207] A first extraction module 902 is used to input the first lip movement video into a lip movement encoder in an initial speech processing model to obtain sample lip movement features;

[0208] The second extraction module 903 is used to input the sample speech into the initial audio code in the initial speech processing model to obtain the sample audio features.

[0209] Prediction module 904, used to input sample lip movement features and sample audio features into an initial enhancement network in the initial speech processing model to obtain a first predicted speech corresponding to the sample sound range;

[0210] The correction module 905 is used to correct the initial audio encoder and the initial enhancement network according to the difference between the first predicted speech and the first speech label to obtain a speech processing model, and the speech processing model is used to implement the speech processing method as claimed in any one of claims 1-9.

[0211] In some embodiments of the present disclosure, a processing module is further included, which is used to:

[0212] Obtain a second lip movement video and a corresponding second voice tag;

[0213] Inputting the second lip movement video into an initial lip reading recognition model to obtain a second predicted speech, wherein the initial lip reading recognition model includes an initial lip movement encoder;

[0214] According to the difference between the second predicted speech and the second speech label, the initial lip movement encoder is modified to obtain a lip movement encoder.

[0215] In some embodiments of the present disclosure, the prediction module 904 is used to:

[0216] Input the sample lip movement features and the sample audio features into the initial fusion network in the initial enhancement network to obtain the sample fusion features corresponding to the sample sound area;

[0217] The sample fusion features are input into the initial recognition network in the initial enhancement network to obtain the first predicted speech corresponding to the sample sound area.

[0218] In some embodiments of the present disclosure, the first acquisition module 901 is used to:

[0219] When any frame image in the first lip movement video is a black image, based on a timestamp corresponding to any frame image, determine a sample speech segment corresponding to the timestamp from the sample speech;

[0220] A speech segment corresponding to the timestamp in the first speech tag is determined to be a sample speech segment.

[0221] It should be noted that the above explanation of the speech processing method is also applicable to the speech processing device of this embodiment and will not be repeated here.

[0222] In the disclosed embodiment, a sample speech, a first lip movement video, and a first speech label corresponding to the sample sound zone are obtained, the first lip movement video is input into the lip movement encoder in the initial speech processing model to obtain sample lip movement features, the sample speech is input into the initial audio encoder in the initial speech processing model to obtain sample audio features, the sample lip movement features and the sample audio features are input into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample sound zone, and finally, according to the difference between the first predicted speech and the first speech label, the initial audio encoder and the initial enhancement network are corrected to obtain the speech processing model. Thus, the initial speech processing model is trained based on the sample speech, the first lip movement video, and the first speech label corresponding to the sample sound zone to obtain a speech processing model that can perform multimodal speech enhancement, which provides support for combining the lip movement video to perform multimodal enhancement on the initial speech after pure speech enhancement, and further improves the efficiency of determining the target speech.

[0223] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0224] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0225] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0226] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0227] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a speech processing method, or a training method for a speech processing model. For example, in some embodiments, a speech processing method, or a training method for a speech processing model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM 1002 and / or a communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the speech processing method described above, or the training method for a speech processing model may be executed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a speech processing method, or a training method for a speech processing model, in any other appropriate manner (for example, by means of firmware).

[0228] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0229] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0230] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0231] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0232] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0233] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0234] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0235] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the description of the present disclosure, the words "if" and "if" used can be interpreted as "at the time of" or "when" or "in response to determination" or "under the circumstances of".

[0236] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A speech processing method, comprising: Acquire a lip movement video corresponding to a target sound zone in a vehicle and a mixed voice in the vehicle; Performing speech enhancement on the mixed speech to obtain an initial speech corresponding to the target voice range; Performing feature extraction on the lip movement video to obtain lip movement features; Performing feature extraction on the initial speech to obtain audio features; The target speech corresponding to the target sound range is determined according to the audio feature and the lip movement feature.

2. The method according to claim 1, wherein: The step of extracting features from the lip movement video to obtain lip movement features includes: When the number of the target sound zones is at least two, based on the lip movement encoder in the speech processing model, feature extraction is performed on the lip movement video corresponding to each of the target sound zones to obtain the lip movement features corresponding to each of the target sound zones, wherein the parameters of the lip movement encoder are the same as the parameters of the lip movement encoder in the trained lip reading recognition model.

3. The method according to claim 2, wherein: The extracting features of the initial speech to obtain audio features includes: Based on the audio encoder in the speech processing model, the initial speech corresponding to at least two of the target sound areas is jointly encoded to obtain the audio feature corresponding to each of the target sound areas.

4. The method according to claim 1 or 3, wherein: The step of determining the target speech corresponding to the target sound range according to the audio feature and the lip movement feature includes: Inputting the speech feature and the lip movement feature into a fusion network in a speech processing model to obtain fusion features; The fused features are input into the recognition network in the speech processing model to obtain the target speech corresponding to the target sound range.

5. The method according to claim 1, wherein: Before obtaining the lip movement video corresponding to the target sound zone in the vehicle and the mixed voice in the vehicle, the method further includes: Acquire video data corresponding to each sound zone in the vehicle; Performing target detection on the video data corresponding to each of the sound zones to obtain a first detection result corresponding to each of the sound zones; The sound zone containing the preset object in the first detection result is determined as the target sound zone.

6. The method according to claim 5, wherein: The step of obtaining the lip movement video corresponding to the target sound zone in the vehicle includes: Perform lip region recognition on each frame of image in the video data corresponding to the target sound area, and determine a second detection result corresponding to each frame of image; According to the second detection result corresponding to each frame of image, lip area is extracted from the video data to obtain the lip movement video.

7. The method according to claim 6, wherein: The step of extracting the lip region from the video data according to the second detection result corresponding to each frame of the image to obtain the lip movement video includes: When the second detection result corresponding to any frame image indicates that the lip area is not detected, it is determined that the image corresponding to the any frame image in the lip movement video is a black image.

8. The method according to claim 7, wherein: Also includes: When the number of frames of continuous black images in the lip movement video corresponding to any target sound zone is greater than a threshold, the initial speech corresponding to any target sound zone is determined as the target speech corresponding to any target sound zone.

9. The method according to claim 1, wherein: The performing speech enhancement on the mixed speech to obtain the initial speech corresponding to the target sound range includes: sequentially directing a main lobe of a beam formed by a microphone array in the vehicle to a target sound area, and directing a side lobe of a beam to other sound areas except the target sound area, so as to extract a first speech corresponding to each target sound area from the mixed speech, wherein the microphone array is used to collect the mixed speech; At least one of noise elimination, echo elimination, interference suppression, and howling suppression is performed on the first speech to obtain the initial speech.

10. A method for training a speech processing model, comprising: Obtaining a sample voice, a first lip movement video, and a first voice tag corresponding to the sample voice zone; Inputting the first lip movement video into a lip movement encoder in an initial speech processing model to obtain sample lip movement features; Inputting the sample speech into the initial audio code in the initial speech processing model to obtain sample audio features; Inputting the sample lip movement features and the sample audio features into the initial enhancement network in the initial speech processing model to obtain a first predicted speech corresponding to the sample sound range; According to the difference between the first predicted speech and the first speech label, the initial audio encoder and the initial enhancement network are modified to obtain a speech processing model, and the speech processing model is used to implement the speech processing method as described in any one of claims 1-9.

11. The method according to claim 10, characterized in that Also includes: Obtain a second lip movement video and a corresponding second voice tag; Inputting the second lip movement video into an initial lip reading recognition model to obtain a second predicted speech, wherein the initial lip reading recognition model includes an initial lip movement encoder; The initial lip movement encoder is modified according to the difference between the second predicted speech and the second speech label to obtain the lip movement encoder.

12. The method according to claim 10, wherein: The step of inputting the sample lip movement features and the sample audio features into the initial enhancement network in the initial speech processing model to obtain a first predicted speech corresponding to the sample sound range includes: Inputting the sample lip movement features and the sample audio features into the initial fusion network in the initial enhancement network to obtain sample fusion features corresponding to the sample sound range; The sample fusion features are input into the initial recognition network in the initial enhancement network to obtain a first predicted speech corresponding to the sample sound range.

13. The method according to claim 10, wherein: Also includes: When any frame image in the first lip movement video is a black image, based on a timestamp corresponding to the any frame image, determine a sample voice segment corresponding to the timestamp from the sample voice; A speech segment corresponding to the timestamp in the first speech tag is determined to be the sample speech segment.

14. A speech processing device, comprising: A first acquisition module is used to acquire a lip movement video corresponding to a target sound zone in a vehicle and a mixed voice in the vehicle; A second acquisition module is used to perform speech enhancement on the mixed speech to obtain an initial speech corresponding to the target voice range; A first extraction module, used for performing feature extraction on the lip movement video to obtain lip movement features; A second extraction module is used to extract features from the initial speech to obtain audio features; A determination module is used to determine the target speech corresponding to the target sound area according to the audio features and the lip movement features.

15. The device according to claim 14, wherein: The first extraction module is used to: When the number of the target sound zones is at least two, based on the lip movement encoder in the speech processing model, feature extraction is performed on the lip movement video corresponding to each of the target sound zones to obtain the lip movement features corresponding to each of the target sound zones, wherein the parameters of the lip movement encoder are the same as the parameters of the lip movement encoder in the trained lip reading recognition model.

16. The device according to claim 15, wherein: The second extraction module is used to: Based on the audio encoder in the speech processing model, the initial speech corresponding to at least two of the target sound areas is jointly encoded to obtain the audio feature corresponding to each of the target sound areas.

17. The device according to claim 14 or 16, wherein: The determining module is used to: Inputting the speech feature and the lip movement feature into a fusion network in a speech processing model to obtain fusion features; The fused features are input into the recognition network in the speech processing model to obtain the target speech corresponding to the target sound range.

18. The device according to claim 14, wherein: Also included is a first processing module, configured to: Acquire video data corresponding to each sound zone in the vehicle; Performing target detection on the video data corresponding to each of the sound zones to obtain a first detection result corresponding to each of the sound zones; The sound zone containing the preset object in the first detection result is determined as the target sound zone.

19. The device according to claim 18, wherein: The first acquisition module is used to: Perform lip region recognition on each frame of image in the video data corresponding to the target sound area, and determine a second detection result corresponding to each frame of image; According to the second detection result corresponding to each frame of image, lip area is extracted from the video data to obtain the lip movement video.

20. The device according to claim 19, wherein The first acquisition module is used to: When the second detection result corresponding to any frame image indicates that the lip area is not detected, it is determined that the image corresponding to the any frame image in the lip movement video is a black image.

21. The device according to claim 20, wherein: Also included is a second processing module, configured to: When the number of frames of continuous black images in the lip movement video corresponding to any target sound zone is greater than a threshold, the initial speech corresponding to any target sound zone is determined as the target speech corresponding to any target sound zone.

22. The device according to claim 14, wherein: The second acquisition module is used to: sequentially directing a main lobe of a beam formed by a microphone array in the vehicle to a target sound area, and directing a side lobe of a beam to other sound areas except the target sound area, so as to extract a first speech corresponding to each target sound area from the mixed speech, wherein the microphone array is used to collect the mixed speech; At least one of noise elimination, echo elimination, interference suppression, and howling suppression is performed on the first speech to obtain the initial speech.

23. A training device for a speech processing model, comprising: An acquisition module, used to acquire a sample voice, a first lip movement video and a first voice tag corresponding to a sample voice zone; A first extraction module, configured to input the first lip movement video into a lip movement encoder in an initial speech processing model to obtain sample lip movement features; A second extraction module is used to input the sample speech into the initial audio code in the initial speech processing model to obtain the sample audio feature; A prediction module, used for inputting the sample lip movement features and the sample audio features into an initial enhancement network in the initial speech processing model to obtain a first predicted speech corresponding to the sample sound range; A correction module is used to correct the initial audio encoder and the initial enhancement network according to the difference between the first predicted speech and the first speech label to obtain a speech processing model, and the speech processing model is used to implement the speech processing method as described in any one of claims 1-9.

24. The device according to claim 23, characterized in that Also included is a processing module for: Obtain a second lip movement video and a corresponding second voice tag; Inputting the second lip movement video into an initial lip reading recognition model to obtain a second predicted speech, wherein the initial lip reading recognition model includes an initial lip movement encoder; The initial lip movement encoder is modified according to the difference between the second predicted speech and the second speech label to obtain the lip movement encoder.

25. The device according to claim 23, wherein: The prediction module is used to: Inputting the sample lip movement features and the sample audio features into the initial fusion network in the initial enhancement network to obtain sample fusion features corresponding to the sample sound range; The sample fusion features are input into the initial recognition network in the initial enhancement network to obtain a first predicted speech corresponding to the sample sound range.

26. The device according to claim 23, wherein The first acquisition module is used to: When any frame image in the first lip movement video is a black image, based on a timestamp corresponding to the any frame image, determine a sample voice segment corresponding to the timestamp from the sample voice; A speech segment corresponding to the timestamp in the first speech tag is determined to be the sample speech segment.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-9, or execute the method of any one of claims 10-13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 9, or to execute the method according to any one of claims 10 to 13.

29. A computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 9, or implement the steps of the method of any one of claims 10 to 13.

Images