Speech processing method, training method, device and equipment of speech processing model

Abstract

The present disclosure provides a speech processing method, a training method, device and equipment of a speech processing model, relates to the technical field of computers, and particularly relates to the technical field of artificial intelligence such as speech enhancement and deep learning. The specific implementation scheme is as follows: a lip movement video corresponding to a target sound area in a vehicle and mixed speech in the vehicle are obtained; speech enhancement is performed on the mixed speech to obtain initial speech corresponding to the target sound area; feature extraction is performed on the lip movement video to obtain lip movement features; feature extraction is performed on the initial speech to obtain audio features; and target speech corresponding to the target sound area is determined according to the audio features and the lip movement features.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence technology such as speech enhancement and deep learning. Specifically, it relates to a speech processing method, a training method for a speech processing model, an apparatus, and a device. Background Technology

[0002] In-vehicle voice interaction can convert user voice commands into commands that the vehicle's infotainment system can understand, and then use speech synthesis technology to convey the system's feedback information to the user in voice form. In-vehicle voice interaction technology is widely used in in-vehicle navigation, in-vehicle entertainment, in-vehicle communication, vehicle control, intelligent assistants, and other in-vehicle scenarios.

[0003] However, in scenarios such as mobile phone navigation or watching short videos, the inability to block out the phone's sound can affect the accuracy of voice recognition, making it difficult to accurately determine the user's intent, impacting the interactive experience, and even potentially causing the phone to interfere with the interaction between the phone and the vehicle's infotainment system, resulting in incorrect feedback from the system. Summary of the Invention

[0004] This disclosure provides a speech processing method, a speech processing model training method, an apparatus, and a device.

[0005] According to a first aspect of this disclosure, a speech processing method is provided, comprising:

[0006] Acquire lip movement video corresponding to the target sound region inside the vehicle and the mixed speech inside the vehicle;

[0007] Speech enhancement is performed on the mixed speech to obtain the initial speech corresponding to the target speech region;

[0008] Feature extraction is performed on the lip movement video to obtain lip movement features;

[0009] Feature extraction is performed on the initial speech to obtain audio features;

[0010] Based on the audio features and the lip movement features, the target speech corresponding to the target vocal region is determined.

[0011] According to a second aspect of this disclosure, a method for training a speech processing model is provided, comprising:

[0012] Obtain the sample speech, the first lip movement video, and the first speech tag corresponding to the sample speech region;

[0013] The first lip movement video is input into the lip movement encoder in the initial speech processing model to obtain sample lip movement features;

[0014] The sample speech is input into the initial audio encoding in the initial speech processing model to obtain the sample audio features.

[0015] The sample lip movement features and the sample audio features are input into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample vocal region;

[0016] Based on the difference between the first predicted speech and the first speech label, the initial audio encoder and the initial enhancement network are modified to obtain a speech processing model, which is used to implement the speech processing method as described in the first aspect.

[0017] According to a third aspect of this disclosure, a voice processing apparatus is provided, comprising:

[0018] The first acquisition module is used to acquire the lip movement video corresponding to the target sound region inside the vehicle and the mixed speech inside the vehicle.

[0019] The second acquisition module is used to perform speech enhancement on the mixed speech to obtain the initial speech corresponding to the target speech region;

[0020] The first extraction module is used to extract features from the lip movement video to obtain lip movement features;

[0021] The second extraction module is used to extract features from the initial speech to obtain audio features;

[0022] The determination module is used to determine the target speech corresponding to the target vocal region based on the audio features and the lip movement features.

[0023] According to a fourth aspect of this disclosure, a training apparatus for a speech processing model is provided, comprising:

[0024] The acquisition module is used to acquire the sample speech, the first lip movement video, and the first speech tag corresponding to the sample speech region;

[0025] The first extraction module is used to input the first lip movement video into the lip movement encoder in the initial speech processing model to obtain sample lip movement features;

[0026] The second extraction module is used to input the sample speech into the initial audio encoding in the initial speech processing model to obtain the sample audio features.

[0027] The prediction module is used to input the sample lip movement features and the sample audio features into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample vocal region.

[0028] A correction module is configured to correct the initial audio encoder and the initial enhancement network based on the difference between the first predicted speech and the first speech label, so as to obtain a speech processing model, the speech processing model being used to implement the speech processing method as described in the first aspect.

[0029] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0030] At least one processor; and

[0031] A memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the speech processing method as described in the first aspect, or to perform the training method for the speech processing model as described in the second aspect.

[0033] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing the computer to perform a speech processing method as described in the first aspect, or to perform a training method for a speech processing model as described in the second aspect.

[0034] According to a seventh aspect of this disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the speech processing method as described in the first aspect, or the steps of the training method for the speech processing model as described in the second aspect.

[0035] The speech processing methods, speech processing model training methods, devices, and equipment disclosed herein have the following characteristics:

[0036] Beneficial effects:

[0037] In this embodiment, lip movement video corresponding to the target audio region within the vehicle and mixed speech within the vehicle are first acquired. Then, speech enhancement is performed on the mixed speech to obtain the initial speech corresponding to the target audio region. Next, feature extraction is performed on the lip movement video to obtain lip movement features, and feature extraction is performed on the initial speech to obtain audio features. Finally, based on the audio features and lip movement features, the target speech corresponding to the target audio region is determined. Thus, pure speech enhancement can be performed on the mixed speech to obtain the initial speech corresponding to the target audio region after pure speech enhancement. Then, combined with the lip movement video corresponding to the target audio region, multimodal enhancement is further performed on the initial speech after pure speech enhancement, thereby improving the accuracy of user speech recognition. This allows for accurate judgment of user intent, enabling the vehicle system to provide correct feedback and improving the user experience.

[0038] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0039] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0040] Figure 1 This is a schematic flowchart of a speech processing method provided according to an embodiment of the present disclosure;

[0041] Figure 2 This is a schematic flowchart of a speech processing method provided according to yet another embodiment of the present disclosure;

[0042] Figure 3 This is a schematic diagram of the structure of a speech processing model provided according to yet another embodiment of the present disclosure;

[0043] Figure 4 This is a schematic flowchart of a speech processing method according to another embodiment of the present disclosure;

[0044] Figure 5 This is a schematic flowchart of a speech processing method according to another embodiment of the present disclosure;

[0045] Figure 6 This is a schematic flowchart of a speech processing method according to another embodiment of the present disclosure;

[0046] Figure 7 This is a flowchart illustrating a training method for a speech processing model according to yet another embodiment of the present disclosure;

[0047] Figure 8 This is a schematic diagram of the structure of a voice processing device according to an embodiment of the present disclosure;

[0048] Figure 9 This is a schematic diagram of the structure of a training device for a speech processing model according to an embodiment of the present disclosure;

[0049] Figure 10 This is a block diagram of an electronic device used to implement the speech processing method of the embodiments of this disclosure. Detailed Implementation

[0050] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0051] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0052] The speech processing method, speech processing model training method, apparatus, and device of the present disclosure are described below with reference to the accompanying drawings.

[0053] It should be noted that the voice processing method in this embodiment is executed by a voice processing device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include, but is not limited to, a terminal or a server.

[0054] Figure 1 This is a schematic flowchart of a speech processing method provided according to an embodiment of the present disclosure.

[0055] like Figure 1 As shown, the speech processing method includes:

[0056] S101, acquire the lip movement video corresponding to the target sound zone inside the vehicle and the mixed speech inside the vehicle.

[0057] In some embodiments, the sound zones can be divided according to the number of seats in the vehicle. For example, a car can be divided into four sound zones: driver's seat, passenger seat, left rear seat, and right rear seat.

[0058] In some embodiments, the target audio zone can be an audio zone in the vehicle where users are present. The number of target audio zones can be one or more, and this disclosure does not limit this. For example, if there are users in the driver's and passenger's seats, the target audio zone is determined to be for the driver and passenger's seats. If there are users only in the driver's seat, the target audio zone is determined to be for the driver's seat.

[0059] Among them, lip-movement videos can be videos that include the user's lips.

[0060] Among them, mixed speech can be the speech collected by the vehicle's microphone array.

[0061] S102, perform speech enhancement on the mixed speech to obtain the initial speech corresponding to the target speech region.

[0062] In some embodiments, sound source localization, noise reduction, and other operations can be performed on the mixed speech to determine the initial speech corresponding to the target speech region.

[0063] In some embodiments, beamforming can be performed using complex neural network modules such as complex convolutional layers and complex linear layers to extract complex features containing directional and phase information from mixed speech. Then, the initial speech corresponding to the target speech region is obtained by passing it through a series of neural network structures, such as convolutional neural networks (CNN), long short-term memory networks (LSTM), Transformers, and Conformers.

[0064] It should be noted that while pure speech enhancement techniques can separate the speech signal of each vocal region, they cannot suppress interference from other vocal regions. For example, a vocal region may contain both the user's own speech and speech played from the user's phone, as well as interfering speech from other vocal regions. Therefore, in this embodiment, after obtaining the initial speech corresponding to each vocal region, the lip movement video corresponding to the target vocal region can be further combined to determine the user's speech within the target vocal region, thereby accurately identifying the user's speech.

[0065] S103, extract features from the lip movement video to obtain lip movement features.

[0066] In some embodiments, lip movement video can be input into an image feature extraction network to obtain lip movement features.

[0067] In some embodiments, the trajectory of lip shape changes can be extracted from the lip movement video to obtain lip movement features.

[0068] S104, extract features from the initial speech to obtain audio features.

[0069] In some embodiments, the initial speech can be incorporated into a speech feature extraction network to obtain speech features.

[0070] In some embodiments, when there are at least two initial speech samples, the at least two initial speech samples can be input together into the speech feature extraction network to obtain the audio features corresponding to each initial speech sample. Alternatively, each initial speech sample can be input separately into the speech feature extraction network to obtain the audio features corresponding to each initial speech sample.

[0071] S105, based on audio features and lip movement features, determine the target speech corresponding to the target vocal region.

[0072] Among them, the target speech can be the speech generated by the user speaking in the target voice region.

[0073] In some embodiments, audio features and lip movement features can be fused to obtain fused features, and then the fused features can be identified to obtain the target speech corresponding to the target vocal region.

[0074] In some embodiments, when there are at least two target vocal regions, the lip movement features corresponding to each target vocal region can be directly fused with the audio features to obtain a fused feature for each target vocal region. Alternatively, the lip movement features corresponding to at least two target vocal regions can be fused with the audio features together to obtain a single fused feature. This disclosure does not limit this approach.

[0075] In this embodiment, lip movement video corresponding to the target audio region within the vehicle and mixed speech within the vehicle are first acquired. Then, speech enhancement is performed on the mixed speech to obtain the initial speech corresponding to the target audio region. Next, feature extraction is performed on the lip movement video to obtain lip movement features, and feature extraction is performed on the initial speech to obtain audio features. Finally, based on the audio features and lip movement features, the target speech corresponding to the target audio region is determined. Thus, pure speech enhancement can be performed on the mixed speech to obtain the initial speech corresponding to the target audio region after pure speech enhancement. Then, combined with the lip movement video corresponding to the target audio region, multimodal enhancement is further performed on the initial speech after pure speech enhancement, thereby improving the accuracy of user speech recognition. This allows for accurate judgment of user intent, enabling the vehicle system to provide correct feedback and improving the user experience.

[0076] Figure 2 This is a schematic flowchart of a speech processing method provided according to yet another embodiment of the present disclosure;

[0077] like Figure 2 As shown, the speech processing method includes:

[0078] S201, acquire the lip movement video corresponding to the target sound zone inside the vehicle and the mixed speech inside the vehicle.

[0079] S202, perform speech enhancement on the mixed speech to obtain the initial speech corresponding to the target speech region.

[0080] The specific implementation of steps S201 and S202 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0081] S203, when there are at least two target vocal regions, based on the lip movement encoder in the speech processing model, feature extraction is performed on the lip movement video corresponding to each target vocal region to obtain the lip movement features corresponding to each target vocal region.

[0082] The speech processing model includes a lip movement encoder, an audio encoder, and an enhancement network. The enhancement network includes a fusion network and a recognition network.

[0083] Figure 3 This is a schematic diagram of the structure of a speech processing model provided in an embodiment of the present disclosure, as shown below. Figure 3 As shown, a lip movement encoder can include a 3D convolutional network (conv3d), a 3D pooling network (Pool3d), a deep residual network (ResNet), or a fully connected network (FC). A lip movement encoder can be used to track the lip movement trajectory in the time dimension.

[0084] The parameters of the lip movement encoder are the same as those in the trained lip-reading recognition model. It should be noted that the lip movement encoder in the lip-reading recognition model can learn certain semantic representation capabilities during training. Therefore, using the same lip movement encoder in the speech processing model allows for the extraction of lip movement features containing semantic information from lip movement videos, thus enabling a more accurate determination of the target speech corresponding to the target vocal region.

[0085] In some embodiments, the specific implementation of training the speech processing model and the lip reading recognition model can be referred to in the detailed description of the subsequent embodiments in this disclosure, and will not be repeated here.

[0086] S204, based on the audio encoder in the speech processing model, jointly encodes the initial speech corresponding to at least two target speech regions to obtain the audio features corresponding to each target speech region.

[0087] It should be noted that, since the sound zones inside the vehicle are small and adjacent, the initial speech of the target neighboring zone may contain part of the speech of the adjacent sound zone. For example, when the passenger speaks, the target sound zone corresponding to the driver will contain a small amount of the passenger's speech. Therefore, jointly encoding the initial speech corresponding to at least two target sound zones can unify the encoding of the same speech in different sound zones, which can further improve the accuracy of the target speech corresponding to each target sound zone.

[0088] like Figure 3 As shown, an audio encoder may include a filter bank (FBank), a linear layer, a gated recurrent unit (GRU), etc. This disclosure does not limit the scope of the encoder.

[0089] S205. Input the speech features and lip movement features into the fusion network in the speech processing model to obtain fused features.

[0090] In some embodiments, since the audio source of the mixed speech and the video source of the lip movement video are different, their sampling rates may be inconsistent. Therefore, it is necessary to align the lip movement features and audio features in the time dimension. For example, upsampling or downsampling can be used to align the lip movement features and audio features in the time dimension.

[0091] In some embodiments, the fusion network may be composed of a Temporal Convolutional Network (TCN), a GRU, etc. Figure 3 As shown, the fusion network can be composed of a sum network and a TCN network. This disclosure does not limit this.

[0092] S206, the fused features are input into the recognition network in the speech processing model to obtain the target speech corresponding to the target voice region.

[0093] like Figure 3 As shown, the identified network can be a fully connected (FC) network. This disclosure does not limit this.

[0094] In some embodiments, the fusion network is used to model the correlation between speech features and lip movement features, enhance signals with high correlation, and suppress signals with low correlation, thereby accurately separating the speech of the target speaker in ultra-low signal-to-noise ratio scenarios and significantly improving the enhancement effect in complex scenarios.

[0095] In this embodiment, lip movement video corresponding to the target audio region inside the vehicle and mixed speech inside the vehicle are first acquired. Then, speech enhancement is performed on the mixed speech to obtain the initial speech corresponding to the target audio region. Next, based on the lip movement encoder in the speech processing model, feature extraction is performed on the lip movement video corresponding to each target audio region to obtain lip movement features for each target audio region. Based on the audio encoder in the speech processing model, the initial speech corresponding to at least two target audio regions is jointly encoded to obtain audio features for each target audio region. Finally, the speech features and lip movement features are input into the fusion network in the speech processing model to obtain fused features. The fused features are then input into the recognition network in the speech processing model to obtain the target speech corresponding to the target audio region. Therefore, pure speech enhancement can be performed on the mixed speech to obtain the initial speech corresponding to the target audio region after pure speech enhancement. Then, the initial speech and lip movement video are input into the speech processing model to perform multimodal enhancement on the initial speech after pure speech enhancement. This not only improves the accuracy of user speech recognition but also improves the efficiency of user speech recognition based on the speech processing model.

[0096] In some embodiments, when there is only one target vocal region, the lip movement video corresponding to the target vocal region can be input into the lip movement encoder to obtain lip movement features, the initial speech can be input into the audio encoder to obtain audio features, the audio features and lip movement features can be input into the fusion network in the speech processing model to obtain fusion features, and the fusion features can be input into the recognition network in the speech processing model to obtain the target speech corresponding to the target vocal region.

[0097] In some embodiments, when there are at least two target speech regions, the lip movement video and the initial speech corresponding to each target speech region can be input into the speech processing model to obtain the target speech corresponding to each target speech region.

[0098] Figure 4 This is a schematic flowchart of a speech processing method provided according to yet another embodiment of the present disclosure;

[0099] like Figure 4 As shown, the speech processing method includes:

[0100] S401, acquire video data corresponding to each audio zone inside the vehicle.

[0101] The video data can be collected by one or more cameras deployed inside the vehicle. This disclosure does not limit this. In some embodiments, only one camera can be deployed inside the vehicle to collect video from all audio zones, and then the video can be processed based on the location of each audio zone to obtain video data for each audio zone. Alternatively, a camera can be deployed for each audio zone to obtain video data corresponding to each audio zone.

[0102] S402, perform target detection on the video data corresponding to each audio region to obtain the first detection result for each audio region.

[0103] The object detection can be either face detection or human body detection. This disclosure does not limit it.

[0104] The first detection result indicates whether a preset object has been detected. The preset object can be a person or a face.

[0105] S403, the sound region containing the preset object in the first detection result is determined as the target sound region.

[0106] Therefore, by performing target detection on each sound region, the sound region where the user is located is identified as the target sound region, thus accurately determining the target sound region where the user is located.

[0107] S404, perform lip region recognition on each frame of the video data corresponding to the target audio region, and determine the second detection result corresponding to each frame.

[0108] The second detection result can be used to indicate the location of the lip region in the image when the lip region is detected, or to indicate when the lip region is not detected.

[0109] In some embodiments, a target detection model can be trained based on sample data including the lips and the position labels of the lips in the sample images. The trained target detection model can then be used to identify the lip region in each frame of the video data to determine the second detection result corresponding to each frame.

[0110] S405, based on the second detection result corresponding to each frame image, extract the lip region from the video data to obtain lip movement video.

[0111] In this embodiment of the disclosure, extracting lip movement videos containing only the lip region from video data can reduce the amount of computational data required to extract lip movement features, thereby improving the efficiency of determining the target speech.

[0112] In some embodiments, if the second detection result corresponding to the image indicates that a lip region has been detected, the lip image can be extracted based on the position of the lip region in the image, thereby obtaining a lip movement video.

[0113] In some embodiments, if the second detection result corresponding to any frame image indicates that no lip region was detected, the image in the lip movement video corresponding to any frame image is determined to be a black image. Therefore, by using a black image to replace the image with the missing lip region, the target speech when lip vision is occasionally lost can be reverted to the result after pure speech enhancement. This ensures both the quality of the target speech when lip vision is present (using multimodal speech enhancement) and the quality of the speech when lip vision is occasionally lost (using pure speech enhancement), thereby guaranteeing the quality of the determined target speech.

[0114] S406, acquire mixed voice inside the vehicle.

[0115] S407 performs speech enhancement on the mixed speech to obtain the initial speech corresponding to the target speech region.

[0116] S408 extracts features from the lip movement video to obtain lip movement features.

[0117] S409, extract features from the initial speech to obtain audio features.

[0118] S410 determines the target speech corresponding to the target speech region based on audio features and lip movement features.

[0119] The specific implementation of steps S406 to S410 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0120] In this embodiment, video data corresponding to each sound zone within the vehicle is first acquired. Then, target detection is performed on the video data corresponding to each sound zone to obtain a first detection result for each sound zone. Sound zones whose first detection results contain a preset object are identified as target sound zones. Next, lip region recognition is performed on each frame of the video data corresponding to the target sound zone to determine a second detection result for each frame. Based on the second detection result for each frame, lip region extraction is performed on the video data to obtain lip movement video. Therefore, not only can the target sound zone of the user be accurately determined, but also lip movement video containing only the lip region can be extracted from the video data corresponding to the target sound zone, reducing the amount of computational data required for extracting lip movement features and thus improving the efficiency of target speech determination.

[0121] Figure 5 This is a schematic flowchart of a speech processing method provided according to yet another embodiment of the present disclosure;

[0122] like Figure 5 As shown, the speech processing method includes:

[0123] S501, acquire the lip movement video corresponding to the target sound zone inside the vehicle and the mixed speech inside the vehicle.

[0124] S502, the main lobe of the beam formed by the microphone array in the vehicle is sequentially pointed to a target sound region, and the side lobes of the beam are pointed to other sound regions except the target sound region, so as to extract the first speech corresponding to each target sound region from the mixed speech.

[0125] The microphone array is used to capture mixed speech.

[0126] For example, there are two target sound zones, namely the driver's seat and the passenger's seat. By pointing the main lobe of the microphone array in the vehicle towards the driver's seat and the side lobes towards other sound zones besides the driver's seat, the speech in other sound zones besides the driver's seat can be suppressed, and the first speech of the driver's seat can be obtained. Similarly, by pointing the main lobe of the microphone array in the vehicle towards the passenger's seat and the side lobes towards other sound zones besides the passenger's seat, the speech in other sound zones besides the passenger's seat can be suppressed, and the first speech of the passenger's seat can be obtained.

[0127] S503 performs at least one of the following operations on the first speech: noise cancellation, echo cancellation, interference suppression, and howling suppression, to obtain the initial speech.

[0128] It should be noted that, since beamforming technology cannot completely suppress noise outside the beam's direction and noise will be mixed into the main lobe of the beam, noise will remain in the extracted first speech. Therefore, post-processing filtering technology is needed to further eliminate noise in the extracted first speech in order to improve the signal-to-noise ratio.

[0129] Echo cancellation can be used to eliminate echoes contained in the first speech.

[0130] Interference suppression can be used to suppress interference from other speech regions on the speech of the target speech region.

[0131] S504 extracts features from the lip movement video to obtain lip movement features.

[0132] S505 performs feature extraction on the initial speech to obtain audio features.

[0133] S506, based on audio features and lip movement features, determines the target speech corresponding to the target vocal region.

[0134] The specific implementation of steps S504 to S506 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0135] In this embodiment, after acquiring the mixed speech, the main lobe of the microphone array in the vehicle is sequentially pointed to a target sound region, and the side lobes are pointed to other sound regions besides the target sound region. This is to extract the first speech corresponding to each target sound region from the mixed speech. At least one of the following operations—noise cancellation, echo cancellation, interference suppression, and howling suppression—is then performed on the first speech to obtain the initial speech. Therefore, pure speech enhancement techniques such as beamforming, noise cancellation, echo cancellation, interference suppression, and howling suppression can be used to enhance the mixed speech, providing a relatively clean initial speech for subsequent multimodal speech enhancement combined with lip-movement video, thereby improving the accuracy of the determined target speech.

[0136] Figure 6 This is a schematic flowchart of a speech processing method provided according to yet another embodiment of the present disclosure;

[0137] like Figure 6 As shown, the speech processing method includes:

[0138] S601, perform lip region recognition on each frame of the video data corresponding to the target audio region, and determine the second detection result corresponding to each frame.

[0139] The second detection result can be used to indicate the location of the lip region in the image when the lip region is detected, or to indicate when the lip region is not detected.

[0140] In some embodiments, a target detection model can be trained based on sample data including the lips and the position labels of the lips in the sample images. The trained target detection model can then be used to identify the lip region in each frame of the video data to determine the second detection result corresponding to each frame.

[0141] S602, if the second detection result corresponding to any frame image indicates that no lip region has been detected, determine that the image in the lip movement video corresponding to any frame image is a black image.

[0142] S603, acquires mixed voices within the vehicle.

[0143] S604 performs speech enhancement on the mixed speech to obtain the initial speech corresponding to the target speech region.

[0144] The specific implementation of steps S603 and S604 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0145] S605, if the number of consecutive black frames in the lip movement video corresponding to any target speech region is greater than a threshold, the initial speech corresponding to any target speech region is determined as the target speech corresponding to any target speech region.

[0146] The threshold can be 100 frames, 200 frames, etc., and this disclosure does not limit it.

[0147] In this embodiment of the disclosure, in the scenario where lip vision is lost for a long time in any target speech region, the initial speech corresponding to any target speech region can be directly determined as the target speech without further executing steps S103 to S105, thereby saving computational load while ensuring the speech recognition result of mixed speech.

[0148] In this embodiment, lip region recognition is performed on each frame of the video data corresponding to the target audio region to determine the second detection result corresponding to each frame. Based on the second detection result, if the second detection result for any frame indicates that no lip region is detected, the image in the lip movement video corresponding to that frame is determined to be a black image. Then, mixed speech within the vehicle is acquired, and speech enhancement is performed on the mixed speech to obtain the initial speech corresponding to the target audio region. Finally, if the number of consecutive black images in the lip movement video corresponding to any target audio region exceeds a threshold, the initial speech corresponding to any target audio region is determined as the target speech for that target audio region. Therefore, even in scenarios where lip vision is lost for an extended period in any target audio region, the initial speech after pure speech enhancement can be directly determined as the target speech, ensuring that user speech can still be obtained relatively accurately even in scenarios where user lip vision is lost, thus guaranteeing the ability of human-vehicle interaction.

[0149] Figure 7 This is a flowchart illustrating a speech processing method according to yet another embodiment of the present disclosure, as shown below. Figure 7 As shown, the training of this speech processing model includes:

[0150] S701, acquire the sample speech, the first lip movement video, and the first speech tag corresponding to the sample speech region.

[0151] The sample audio zone can be any audio zone in the vehicle where users are present. The number of sample audio zones can be one or more, and this disclosure does not limit this. For example, if there are users in the driver's and passenger's seats, then the sample audio zones are determined to be for the driver and passenger's seats. If only the driver's seat has a user, then the sample audio zone is determined to be for the driver's seat.

[0152] The first lip movement video can be a video of the user's lips within the sample vocal region. In some embodiments, lip region detection can be performed on each frame of the video data corresponding to the sample vocal region, and the lip region can be extracted from each frame to obtain the lip movement video.

[0153] In some embodiments, if the lip detection result corresponding to any frame image indicates that no lip region was detected, the image in the first lip movement video corresponding to any frame image is determined to be a black image.

[0154] The first voice tag can be the voice generated by a user speaking in the sample voice region.

[0155] In some embodiments, when any frame in the first lip movement video is a black image, based on the timestamp corresponding to any frame, a sample speech segment corresponding to the timestamp is determined from the sample speech, and the speech segment corresponding to the timestamp in the first speech tag is determined as the sample speech segment. Thus, in scenarios where lip vision is lost, the initial speech processing model can be trained based on the same input and output speech, ensuring that the target speech output by the trained language processing model is the same as the initial speech after pure speech enhancement. This allows the target speech obtained when lip vision is occasionally lost to be reverted to the result after pure speech enhancement, ensuring that multimodal speech enhancement can be used when lip vision is present, and also ensuring that pure speech enhancement can be used when lip vision is occasionally lost, thereby guaranteeing the quality of the determined target speech.

[0156] S702, input the first lip movement video into the lip movement encoder in the initial speech processing model to obtain sample lip movement features.

[0157] It should be noted that the structure of the initial speech processing model is the same as the structure of the trained speech processing model, such as... Figure 3 As shown, further details will not be elaborated here.

[0158] Among them, the lip movement encoder is used to track the lip movement trajectory in the time dimension and extract lip movement features.

[0159] In this model, the lip movement encoder is pre-trained and requires no further training. In some embodiments, the parameters of the lip movement encoder are the same as those of the lip movement encoder in the trained lip-reading recognition model. That is, the lip-reading recognition model can be trained to obtain the lip movement encoder.

[0160] In some embodiments, a second lip movement video and its corresponding second speech tag are acquired. The second lip movement video is then input into an initial lip-reading recognition model to obtain a second predicted speech. The initial lip-reading recognition model includes an initial lip movement encoder. Subsequently, based on the difference between the second predicted speech and the second speech tag, the initial lip movement encoder is corrected to obtain a lip movement encoder. Thus, by training the initial lip-reading recognition model, a lip movement encoder is obtained. This lip movement encoder can learn certain semantic expression capabilities during training, thereby obtaining lip movement features with speech information.

[0161] The initial lip reading recognition model also includes a recognition encoder and a recognition decoder.

[0162] A recognition encoder is used to transform lip movement features into recognition features that contain semantic information. A recognition encoder can consist of one or more Conformer networks, or it can include structures such as LSTM and GRU.

[0163] The recognition decoder is used to decode the recognition features containing semantic information to obtain the lip-reading result. It consists of one or more Transformer networks, and may also include structures such as LSTM and GRU.

[0164] In some embodiments, the difference between the second predicted speech and the second speech label can be calculated using a connectionist temporal classification loss function, etc.

[0165] In this embodiment of the disclosure, after obtaining the trained lip-reading recognition model, the lip-motion encoder in the initial speech processing model can be initialized based on the lip-motion encoder in the lip-reading recognition model, and the parameters can be frozen and not participated in the training of the initial speech processing model.

[0166] S703, input the sample speech into the initial audio encoding in the initial speech processing model to obtain the sample audio features.

[0167] In some embodiments, because the sound zones within the vehicle are small and adjacent, the initial speech in the target neighboring zone may contain partial speech from adjacent sound zones. For example, when the passenger speaks, the target sound zone corresponding to the driver may contain a small amount of the passenger's speech. Therefore, when there are at least two sample sound zones, sample speech from at least two sample sound zones can be input into the initial audio encoder for joint encoding. Alternatively, the sample speech corresponding to each sample sound zone can be input into the initial audio encoder sequentially.

[0168] S704. Input the sample lip movement features and sample audio features into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample vocal region.

[0169] The initial enhancement network includes an initial fusion network and an initial identification network.

[0170] In some embodiments, the sample lip movement features and sample audio features are input into the initial fusion network in the initial enhancement network to obtain the sample fusion features corresponding to the sample sound region, and the sample fusion features are input into the initial recognition network in the initial enhancement network to obtain the first predicted speech corresponding to the sample sound region.

[0171] For example, when there are at least two sample speech regions, the sample speech and the first lip movement video corresponding to at least two sample speech regions can be input together into the initial speech processing model. Alternatively, the sample speech and the first lip movement video corresponding to each sample speech region can be input into the initial speech processing model to obtain the first predicted speech corresponding to each sample speech region. For example, first input the sample speech and the first lip movement video corresponding to the first sample speech region into the initial speech processing model to obtain the first predicted speech corresponding to the first sample speech region, and then input the sample speech and the first lip movement video corresponding to the second sample speech region into the initial speech processing model to obtain the first predicted speech corresponding to the second sample speech region.

[0172] S705, based on the difference between the first predicted speech and the first speech label, the initial audio encoder and the initial enhancement network are modified to obtain a speech processing model.

[0173] The speech processing model is used to implement the speech processing method in the embodiments of this disclosure.

[0174] In some embodiments, the mean squared error loss function (MSE loss) or the signal-to-noise ratio loss (SNR loss) can be used to calculate the difference between the first predicted speech and the first speech label.

[0175] In this embodiment, sample speech, a first lip movement video, and a first speech tag corresponding to a sample speech region are obtained. The first lip movement video is input into the lip movement encoder in the initial speech processing model to obtain sample lip movement features. The sample speech is input into the initial audio encoder in the initial speech processing model to obtain sample audio features. The sample lip movement features and sample audio features are input into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample speech region. Finally, based on the difference between the first predicted speech and the first speech tag, the initial audio encoder and the initial enhancement network are corrected to obtain the speech processing model. Thus, by training the initial speech processing model based on the sample speech, the first lip movement video, and the first speech tag corresponding to the sample speech region, a speech processing model capable of multimodal speech enhancement is obtained. This provides support for multimodal enhancement of the initial speech after pure speech enhancement by combining lip movement video, further improving the efficiency of determining the target speech.

[0176] Figure 8 This is a schematic diagram of the structure of a voice processing device according to an embodiment of the present disclosure; as shown below. Figure 8 As shown, the voice processing device 800 includes:

[0177] The first acquisition module 801 is used to acquire the lip movement video corresponding to the target sound area inside the vehicle and the mixed speech inside the vehicle.

[0178] The second acquisition module 802 is used to perform speech enhancement on the mixed speech in order to obtain the initial speech corresponding to the target speech region;

[0179] The first extraction module 803 is used to extract features from the lip movement video to obtain lip movement features;

[0180] The second extraction module 804 is used to extract features from the initial speech to obtain audio features;

[0181] The determination module 805 is used to determine the target speech corresponding to the target speech region based on audio features and lip movement features.

[0182] In some embodiments of this disclosure, the first extraction module 803 is configured to:

[0183] When there are at least two target vocal regions, based on the lip movement encoder in the speech processing model, feature extraction is performed on the lip movement video corresponding to each target vocal region to obtain the lip movement features corresponding to each target vocal region. The parameters of the lip movement encoder are the same as the parameters of the lip movement encoder in the trained lip reading recognition model.

[0184] In some embodiments of this disclosure, the second extraction module 804 is used for:

[0185] Based on the audio encoder in the speech processing model, the initial speech corresponding to at least two target speech regions is jointly encoded to obtain the audio features corresponding to each target speech region.

[0186] In some embodiments of this disclosure, the determining module 805 is configured to:

[0187] The speech features and lip movement features are input into the fusion network in the speech processing model to obtain fused features.

[0188] The fused features are input into the recognition network in the speech processing model to obtain the target speech corresponding to the target voice region.

[0189] In some embodiments of this disclosure, a processing module is further included, for:

[0190] Acquire video data corresponding to each audio zone inside the vehicle;

[0191] Perform object detection on the video data corresponding to each audio region to obtain the first detection result for each audio region;

[0192] The first detection result containing the sound region of the preset object is determined as the target sound region.

[0193] In some embodiments of this disclosure, the first acquisition module 801 is configured to:

[0194] Lip region recognition is performed on each frame of the video data corresponding to the target audio region to determine the second detection result corresponding to each frame.

[0195] Based on the second detection result corresponding to each frame image, the lip region is extracted from the video data to obtain lip movement video.

[0196] In some embodiments of this disclosure, the first acquisition module 801 is configured to:

[0197] If the second detection result corresponding to any frame image indicates that no lip region was detected, the image in the lip movement video corresponding to any frame image is determined to be a black image.

[0198] In some embodiments of this disclosure, a second processing module is further included, for:

[0199] If the number of consecutive black frames in the lip movement video corresponding to any target speech region is greater than a threshold, the initial speech corresponding to any target speech region is determined as the target speech corresponding to any target speech region.

[0200] In some embodiments of this disclosure, the second acquisition module 802 is configured to:

[0201] The main lobe of the beam formed by the microphone array in the vehicle is sequentially pointed to a target sound region, and the side lobes of the beam are pointed to other sound regions except the target sound region, so as to extract the first speech corresponding to each target sound region from the mixed speech. The microphone array is used to collect the mixed speech.

[0202] Perform at least one of the following operations on the first speech: noise cancellation, echo cancellation, interference suppression, and howling suppression, to obtain the initial speech.

[0203] It should be noted that the foregoing explanation of the speech processing method also applies to the speech processing device of this embodiment, and will not be repeated here.

[0204] In this embodiment, lip movement video corresponding to the target audio region within the vehicle and mixed speech within the vehicle are first acquired. Then, speech enhancement is performed on the mixed speech to obtain the initial speech corresponding to the target audio region. Next, feature extraction is performed on the lip movement video to obtain lip movement features, and feature extraction is performed on the initial speech to obtain audio features. Finally, based on the audio features and lip movement features, the target speech corresponding to the target audio region is determined. Thus, pure speech enhancement can be performed on the mixed speech to obtain the initial speech corresponding to the target audio region after pure speech enhancement. Then, combined with the lip movement video corresponding to the target audio region, multimodal enhancement is further performed on the initial speech after pure speech enhancement, thereby improving the accuracy of user speech recognition. This allows for accurate judgment of user intent, enabling the vehicle system to provide correct feedback and improving the user experience.

[0205] Figure 9 This is a schematic diagram of the structure of a training device for a speech processing model according to an embodiment of this disclosure; as shown below. Figure 9 As shown, the training device 900 for the speech processing model includes:

[0206] The acquisition module 901 is used to acquire the sample speech, the first lip movement video and the first speech tag corresponding to the sample speech region;

[0207] The first extraction module 902 is used to input the first lip movement video into the lip movement encoder in the initial speech processing model to obtain sample lip movement features;

[0208] The second extraction module 903 is used to input the sample speech into the initial audio encoding in the initial speech processing model to obtain the sample audio features.

[0209] Prediction module 904 is used to input the sample lip movement features and sample audio features into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample sound region.

[0210] The correction module 905 is used to correct the initial audio encoder and the initial enhancement network based on the difference between the first predicted speech and the first speech label, so as to obtain a speech processing model, which is used to implement the speech processing method as claimed in any one of claims 1-9.

[0211] In some embodiments of this disclosure, a processing module is further included, for:

[0212] Obtain the second lip movement video and the corresponding second voice tag;

[0213] The second lip movement video is input into the initial lip reading recognition model to obtain the second predicted speech, wherein the initial lip reading recognition model includes an initial lip movement encoder;

[0214] The initial lip encoder is corrected based on the difference between the second predicted speech and the second speech label to obtain the lip encoder.

[0215] In some embodiments of this disclosure, the prediction module 904 is configured to:

[0216] The lip movement features and audio features of the samples are input into the initial fusion network in the initial enhancement network to obtain the sample fusion features corresponding to the sample vocal region.

[0217] The sample fusion features are input into the initial recognition network in the initial enhancement network to obtain the first predicted speech corresponding to the sample speech region.

[0218] In some embodiments of this disclosure, the first acquisition module 901 is configured to:

[0219] If any frame in the first lip movement video is a black image, the sample speech segment corresponding to the timestamp is determined from the sample speech based on the timestamp corresponding to any frame image.

[0220] The speech segment corresponding to the timestamp in the first speech tag is identified as the sample speech segment.

[0221] It should be noted that the foregoing explanation of the speech processing method also applies to the speech processing device of this embodiment, and will not be repeated here.

[0222] In this embodiment, sample speech, a first lip movement video, and a first speech tag corresponding to a sample speech region are obtained. The first lip movement video is input into the lip movement encoder in the initial speech processing model to obtain sample lip movement features. The sample speech is input into the initial audio encoder in the initial speech processing model to obtain sample audio features. The sample lip movement features and sample audio features are input into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample speech region. Finally, based on the difference between the first predicted speech and the first speech tag, the initial audio encoder and the initial enhancement network are corrected to obtain the speech processing model. Thus, by training the initial speech processing model based on the sample speech, the first lip movement video, and the first speech tag corresponding to the sample speech region, a speech processing model capable of multimodal speech enhancement is obtained. This provides support for multimodal enhancement of the initial speech after pure speech enhancement by combining lip movement video, further improving the efficiency of determining the target speech.

[0223] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0224] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0225] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0226] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0227] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as speech processing methods or speech processing model training methods. For example, in some embodiments, the speech processing method or speech processing model training method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the speech processing method or speech processing model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform a speech processing method or a speech processing model training method by any other suitable means (e.g., by means of firmware).

[0228] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0229] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0230] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0231] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0232] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0233] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0234] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0235] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this disclosure, the words "if" and "suppose" as used may be interpreted as "when," "when," "in response to determination," or "in the circumstances."

[0236] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A speech processing method, comprising: Acquire lip movement video corresponding to the target sound region inside the vehicle and mixed speech inside the vehicle, wherein the target sound region is the sound region inside the vehicle where a user is present; Speech enhancement is performed on the mixed speech to obtain the initial speech corresponding to the target speech region; When there are at least two target vocal regions, based on the lip movement encoder in the speech processing model, feature extraction is performed on the lip movement video corresponding to each target vocal region to obtain the lip movement features corresponding to each target vocal region. The parameters of the lip movement encoder are the same as the parameters of the lip movement encoder in the trained lip reading recognition model. Feature extraction is performed on the initial speech to obtain audio features; Based on the audio features and the lip movement features, the target speech corresponding to the target vocal region is determined; wherein, the target speech region is determined; The audio features and the lip movement features are input into the fusion network in the speech processing model to obtain fusion features. The fusion network is used to model the correlation between the audio features and the lip movement features in order to separate the speech of the target speaker. The fused features are input into the recognition network in the speech processing model to obtain the target speech corresponding to the target voice region.

2. The method according to claim 1, wherein, The step of extracting features from the initial speech to obtain audio features includes: Based on the audio encoder in the speech processing model, the initial speech corresponding to at least two target speech regions is jointly encoded to obtain the audio features corresponding to each target speech region.

3. The method according to claim 1, wherein, Before acquiring the lip movement video corresponding to the target sound region inside the vehicle and the mixed speech inside the vehicle, the method further includes: Acquire video data corresponding to each audio zone within the vehicle; Target detection is performed on the video data corresponding to each of the audio regions to obtain a first detection result for each of the audio regions; The sound region containing the preset object in the first detection result is determined as the target sound region.

4. The method according to claim 3, wherein, The acquisition of the lip movement video corresponding to the target sound region inside the vehicle includes: Lip region recognition is performed on each frame of the video data corresponding to the target audio region to determine the second detection result corresponding to each frame of the image; Based on the second detection result corresponding to each frame image, the lip region is extracted from the video data to obtain the lip movement video.

5. The method according to claim 4, wherein, The step of extracting the lip region from the video data based on the second detection result corresponding to each frame image to obtain the lip movement video includes: If the second detection result corresponding to any frame image indicates that no lip region was detected, the image in the lip movement video corresponding to that frame image is determined to be a black image.

6. The method according to claim 5, wherein, Also includes: If the number of consecutive black frames in the lip movement video corresponding to any target sound region is greater than a threshold, the initial speech corresponding to the target sound region is determined as the target speech corresponding to the target sound region.

7. The method according to claim 1, wherein, The step of enhancing the mixed speech to obtain the initial speech corresponding to the target speech region includes: The main lobe of the beam formed by the microphone array in the vehicle is sequentially pointed to a target sound region, and the side lobes of the beam are pointed to other sound regions other than the target sound region, so as to extract the first speech corresponding to each target sound region from the mixed speech, wherein the microphone array is used to collect the mixed speech; Perform at least one of the following operations on the first speech: noise cancellation, echo cancellation, interference suppression, and howling suppression, to obtain the initial speech.

8. A method for training a speech processing model, comprising: Obtain the sample speech, the first lip movement video, and the first speech tag corresponding to the sample speech region; The first lip movement video is input into the lip movement encoder in the initial speech processing model to obtain sample lip movement features; The sample speech is input into the initial audio encoder in the initial speech processing model to obtain sample audio features; The sample lip movement features and the sample audio features are input into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample vocal region; Based on the difference between the first predicted speech and the first speech label, the initial audio encoder and the initial enhancement network are modified to obtain a speech processing model, which is used to implement the speech processing method as described in any one of claims 1-7.

9. The method according to claim 8, characterized in that, Also includes: Obtain the second lip movement video and the corresponding second voice tag; The second lip movement video is input into the initial lip reading recognition model to obtain the second predicted speech, wherein the initial lip reading recognition model includes an initial lip movement encoder; The initial lip encoder is corrected based on the difference between the second predicted speech and the second speech label to obtain the lip encoder.

10. The method according to claim 8, wherein, The step of inputting the sample lip movement features and the sample audio features into the initial enhancement network of the initial speech processing model to obtain the first predicted speech corresponding to the sample vocal region includes: The sample lip movement features and the sample audio features are input into the initial fusion network in the initial enhancement network to obtain the sample fusion features corresponding to the sample vocal region; The sample fusion features are input into the initial recognition network in the initial enhancement network to obtain the first predicted speech corresponding to the sample voice region.

11. The method according to claim 8, wherein, Also includes: If any frame in the first lip movement video is a black image, based on the timestamp corresponding to the frame, a sample speech segment corresponding to the timestamp is determined from the sample speech. The voice segment corresponding to the timestamp in the first voice tag is identified as the sample voice segment.

12. A voice processing device, comprising: The first acquisition module is used to acquire lip movement video corresponding to the target sound zone inside the vehicle and mixed speech inside the vehicle, wherein the target sound zone is the sound zone inside the vehicle where a user is present. The second acquisition module is used to perform speech enhancement on the mixed speech to obtain the initial speech corresponding to the target speech region; The first extraction module is used to extract features from the lip movement video corresponding to each target sound region based on the lip movement encoder in the speech processing model when there are at least two target sound regions, so as to obtain the lip movement features corresponding to each target sound region. The parameters of the lip movement encoder are the same as the parameters of the lip movement encoder in the trained lip reading recognition model. The second extraction module is used to extract features from the initial speech to obtain audio features; The determination module is used to determine the target speech corresponding to the target vocal region based on the audio features and the lip movement features. The determining module is used for: The audio features and the lip movement features are input into the fusion network in the speech processing model to obtain fusion features. The fusion network is used to model the correlation between the audio features and the lip movement features in order to separate the speech of the target speaker. The fused features are input into the recognition network in the speech processing model to obtain the target speech corresponding to the target voice region.

13. The apparatus according to claim 12, wherein, The second extraction module is used for: Based on the audio encoder in the speech processing model, the initial speech corresponding to at least two target speech regions is jointly encoded to obtain the audio features corresponding to each target speech region.

14. The apparatus according to claim 12, wherein, It also includes a first processing module, used for: Acquire video data corresponding to each audio zone within the vehicle; Target detection is performed on the video data corresponding to each of the audio regions to obtain a first detection result for each of the audio regions; The sound region containing the preset object in the first detection result is determined as the target sound region.

15. The apparatus according to claim 14, wherein, The first acquisition module is used for: Lip region recognition is performed on each frame of the video data corresponding to the target audio region to determine the second detection result corresponding to each frame of the image; Based on the second detection result corresponding to each frame image, the lip region is extracted from the video data to obtain the lip movement video.

16. The apparatus according to claim 15, wherein, The first acquisition module is used for: If the second detection result corresponding to any frame image indicates that no lip region was detected, the image in the lip movement video corresponding to that frame image is determined to be a black image.

17. The apparatus according to claim 16, wherein, It also includes a second processing module for: If the number of consecutive black frames in the lip movement video corresponding to any target sound region is greater than a threshold, the initial speech corresponding to the target sound region is determined as the target speech corresponding to the target sound region.

18. The apparatus according to claim 12, wherein, The second acquisition module is used for: The main lobe of the beam formed by the microphone array in the vehicle is sequentially pointed to a target sound region, and the side lobes of the beam are pointed to other sound regions other than the target sound region, so as to extract the first speech corresponding to each target sound region from the mixed speech, wherein the microphone array is used to collect the mixed speech; Perform at least one of the following operations on the first speech: noise cancellation, echo cancellation, interference suppression, and howling suppression, to obtain the initial speech.

19. A training device for a speech processing model, comprising: The acquisition module is used to acquire the sample speech, the first lip movement video, and the first speech tag corresponding to the sample speech region; The first extraction module is used to input the first lip movement video into the lip movement encoder in the initial speech processing model to obtain sample lip movement features; The second extraction module is used to input the sample speech into the initial audio encoder in the initial speech processing model to obtain sample audio features; The prediction module is used to input the sample lip movement features and the sample audio features into the initial enhancement network in the initial speech processing model to obtain the first predicted speech corresponding to the sample vocal region. A correction module is used to correct the initial audio encoder and the initial enhancement network based on the difference between the first predicted speech and the first speech label, so as to obtain a speech processing model, wherein the speech processing model is used to implement the speech processing method as described in any one of claims 1-7.

20. The apparatus according to claim 19, characterized in that, It also includes a processing module for: Obtain the second lip movement video and the corresponding second voice tag; The second lip movement video is input into the initial lip reading recognition model to obtain the second predicted speech, wherein the initial lip reading recognition model includes an initial lip movement encoder; The initial lip encoder is corrected based on the difference between the second predicted speech and the second speech label to obtain the lip encoder.

21. The apparatus according to claim 19, wherein, The prediction module is used for: The sample lip movement features and the sample audio features are input into the initial fusion network in the initial enhancement network to obtain the sample fusion features corresponding to the sample vocal region; The sample fusion features are input into the initial recognition network in the initial enhancement network to obtain the first predicted speech corresponding to the sample voice region.

22. The apparatus according to claim 19, wherein, The first acquisition module is used for: If any frame in the first lip movement video is a black image, based on the timestamp corresponding to the frame, a sample speech segment corresponding to the timestamp is determined from the sample speech. The voice segment corresponding to the timestamp in the first voice tag is identified as the sample voice segment.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-7, or the method of any one of claims 8-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method of any one of claims 1-7, or to perform the method of any one of claims 8-11.

Images