Vehicle-mounted voice interaction method and device, computer equipment and vehicle machine

Abstract

The invention relates to the technical field of vehicle control, in particular to a vehicle-mounted voice interaction method and device, computer equipment and a vehicle inforam.The vehicle-mounted voice interaction method comprises the steps that initial audio data are preprocessed, and audio data corresponding to personnel in each cabin are obtained; and inputting the audio data corresponding to the personnel in each cabin into a preset multi-mode voice activity detection algorithm network for processing, and determining whether the audio data corresponding to the personnel in each cabin belongs to real user voice activities or not, so that the initial audio data can be processed to generate an interaction instruction before the initial audio data is processed to generate the interaction instruction. And determining whether the audio data corresponding to the personnel in each cabin belongs to the real user voice activity or not, thereby reducing the false wake-up rate.

Description

Technical Field

[0001] The present invention relates to the field of vehicle control technology, and in particular to a vehicle-mounted voice interaction method, device, computer equipment and vehicle computer. Background Art

[0002] With the development of artificial intelligence and other scientific technologies, the level of automobile intelligence is getting higher and higher, and intelligent voice interaction has become the main interactive entrance to the car's intelligent cockpit. However, there are still many cars whose intelligent voice interaction needs to be awakened before use, that is, first say a wake-up word such as "Hello, Xiaoan" to wake up the voice assistant before voice interaction can be carried out. This interaction method is cumbersome and unnatural, which brings great inconvenience to the user experience.

[0003] In addition, even though some cars have implemented the wake-up-free function, there is still a problem of high false wake-up rate. For example, users may inadvertently receive the voice broadcast output of the voice assistant during a chat, or when users want to interact freely with the voice assistant, the voice assistant does not respond in some situations. Summary of the invention

[0004] In view of this, the present invention provides an in-vehicle voice interaction method, device, computer equipment and vehicle computer to solve the problem of high false wake-up rate in in-vehicle voice interaction.

[0005] In a first aspect, the present invention provides an in-vehicle voice interaction method, comprising the following steps: acquiring initial audio data; pre-processing the initial audio data to obtain audio data corresponding to each cabin occupant; inputting the audio data corresponding to each cabin occupant into a preset multi-mode voice activity detection algorithm network for processing to determine whether the audio data corresponding to each cabin occupant belongs to real user voice activity; and generating interaction instructions based on the audio data belonging to real user voice activity.

[0006] The in-vehicle voice interaction method provided by the present invention obtains audio data corresponding to each cabin occupant by pre-processing the initial audio data; the audio data corresponding to each cabin occupant is input into a preset multi-mode voice activity detection algorithm network for processing to determine whether the audio data corresponding to each cabin occupant belongs to a true user voice activity, thereby determining whether the audio data corresponding to each cabin occupant belongs to a true user voice activity before processing the initial audio data to generate an interaction instruction, thereby reducing the false awakening rate; and the in-vehicle voice interaction does not require awakening, the natural interaction level is high, and the user experience is good.

[0007] In an optional embodiment, pre-processing the initial audio data to obtain audio data corresponding to each cabin occupant includes: when the initial facial image data can be obtained, performing feature extraction and fusion on the initial facial image data to obtain an image feature vector; performing feature extraction on the initial audio data to obtain a voiceprint feature vector and an audio coding feature vector; fusing the voiceprint feature vector and the audio coding feature vector to obtain an audio feature vector; fusing the image feature vector and the audio feature vector to obtain a multi-mode feature vector; obtaining a first mask set based on the multi-mode feature vector, wherein the first mask set includes an activated voice mask corresponding to each cabin occupant; and obtaining audio data corresponding to each cabin occupant based on the first mask set.

[0008] That is to say, when the camera detects a face, it performs multi-mode data fusion based on face-related features such as age, gender, Face ID, and related features such as voiceprint and frequency domain after processing the initial audio data, and obtains the separated audio data of different cabin occupants. If a face is detected, the separated audio can correspond to the speaker's face and position in the cabin. Multi-mode adaptive speech separation processing can improve the accuracy of speech separation, solve the problem of incomplete and inaccurate separation when there are multiple passengers in the cabin, and obtain the identity, location and other information of each cabin occupant after separation, providing more possibilities for intelligent interaction in the cabin.

[0009] In an optional embodiment, the initial audio data is pre-processed to obtain audio data corresponding to each cabin occupant, including: when the initial facial image data cannot be obtained, the initial audio data is processed to obtain a voiceprint feature vector and an audio coding feature vector; the voiceprint feature vector and the audio coding feature vector are fused to obtain an audio feature vector; a second mask set is obtained based on the audio feature vector, wherein the second mask set includes an activated voice mask corresponding to each cabin occupant; and audio data corresponding to each cabin occupant is obtained based on the second mask set.

[0010] That is to say, if no face is detected, the audio of different users is separated by superimposing relevant features such as voiceprint and audio frequency domain.

[0011] In an optional embodiment, when the initial facial image data can be obtained, it also includes: inputting the initial facial image data into a face and key point detection model to obtain multiple facial frames, and key points corresponding to each facial frame; and obtaining lip image data corresponding to each facial frame based on the key points corresponding to each facial frame.

[0012] That is to say, when the initial facial image data can be obtained, the lip image data can be obtained by processing the initial facial image data, so that it can be determined based on the lip image data whether the audio data corresponding to each cabin occupant belongs to real user voice activity.

[0013] In an optional implementation, the audio data corresponding to each cabin occupant is input into a preset multi-mode voice activity detection algorithm network for processing, and determining whether the audio data corresponding to each cabin occupant belongs to a real user voice activity includes: when the initial face image data can be obtained, a section of lip image data in the image data queue is input into the multi-mode voice activity detection algorithm network for feature extraction, feature enhancement and offset embedding fusion processing of ordered different frames in the channel space to obtain first-stage image feature data; a section of audio data in the audio data queue is input into the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain first-stage audio feature data; the first-stage image feature data and the first-stage audio feature data are spliced ​​and fused to obtain multi-mode output data; and determining whether the audio data corresponding to each cabin occupant belongs to a real user voice activity according to the multi-mode output data; or;

[0014] The above method can filter out some bad cases where the user does not speak or the user chats but triggers feedback, thereby reducing the false wake-up rate; it also supports image and audio dual-modal data input and outputs multi-modal fusion reasoning results.

[0015] In an optional implementation, the audio data corresponding to each cabin occupant is input into a preset multi-mode voice activity detection algorithm network for processing, and determining whether the audio data corresponding to each cabin occupant belongs to a true user voice activity includes: when the initial facial image data cannot be obtained, inputting a segment of audio data in the audio data queue corresponding to each cabin occupant into the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain single-mode output data; and determining whether the audio data corresponding to each cabin occupant belongs to a true user voice activity based on the single-mode output data.

[0016] The above method can filter out some bad cases where the user does not speak or the user chats but triggers feedback, thereby reducing the false wake-up rate; it also supports audio single-modal data input and outputs single-modal reasoning results, and can adapt to situations where there is no camera.

[0017] In an optional embodiment, before the audio data corresponding to each cabin occupant is input into a preset multi-mode voice activity detection algorithm network for processing and it is determined whether the audio data corresponding to each cabin occupant belongs to a true user voice activity, it also includes: when the initial facial image data can be obtained, the lip image data is stored in an image data queue; the audio data corresponding to each cabin occupant is stored in an audio data queue; when the length of the audio data queue is greater than a preset threshold, the audio data to be processed is selected from the audio data queue, and the start time and end time of the audio data to be processed are determined; the image data to be processed is selected from the image data queue according to the start time and end time of the audio data to be processed; when the number of frames of the image data to be processed is less than a preset length, the last image frame is used to pad the number of frames of the image data to be processed to the preset length; when the image data to be processed is greater than the preset length, some image frames are screened out from the image to be processed, and the number of frames of the selected some image frames is the preset length.

[0018] That is to say, before the audio data corresponding to each cabin occupant is input into the preset multi-mode voice activity detection algorithm network for processing, the audio data to be processed and the image data to be processed are aligned, and the image frame data length is aligned, thereby ensuring that whether the audio data corresponding to each cabin occupant belongs to the normal progress of the real user voice activity.

[0019] In an optional embodiment, generating interaction instructions based on audio data belonging to real user voice activities includes: converting audio data with real user voice activities into text information; determining the type of text information, wherein the types of text information include: task-based strong intent, task-based weak intent, task-based unprocessed, non-task-based chat, non-task-based dialogue, and other non-task types; processing the text information according to the type of text information to generate interaction instructions.

[0020] Therefore, different processing methods can be used for different types of text information, which can improve interaction efficiency and reduce operating costs.

[0021] In an optional implementation, the text information is processed according to the type of the text information, and the generation of interaction instructions includes: when the type of the text information is task-based strong intent or task-based weak intent, determining whether the device side can process the text information; when the device side can process the text information, processing the text information on the device side and generating interaction instructions; otherwise, sending the text information to the cloud, and accepting the interaction instructions generated by the cloud based on the text information; when the type of the text information is task-based not processing, non-task-based chat, non-task-based dialogue, or other non-task-based, not responding to the text information.

[0022] That is to say, those that do not need to be processed are directly discarded, those that can be processed on the device side are directly called to output the results of the small model on the device side, and those that cannot be processed are sent to the cloud to use an algorithm model with larger parameters and better effects for higher accuracy processing. This dynamic adjustment makes full use of the respective advantages of algorithms on different platforms, not only reducing communication links to a certain extent and speeding up interactive touch responses, but also reducing cloud traffic consumption and concurrency pressure while ensuring high accuracy of output results.

[0023] In an optional embodiment, determining the type of text information includes: generating pinyin information corresponding to the text information; converting the text information into a text vector, and converting the pinyin information into a pinyin vector; encoding the text vector and the pinyin vector respectively, and concatenating and fusing the encoded text vector and pinyin vector to obtain a fused feature; performing feature enhancement processing on the fused feature, and inputting the fused feature into a preset fully connected network to obtain the type of the text information.

[0024] In actual application, it is found that there are often errors in the recognition of text information. For example, the recognized text often contains homophones and wrong characters, which has a great impact on the subsequent semantic understanding. In order to reduce this impact and improve the accuracy of semantic understanding, the present invention generates pinyin information corresponding to the text information, converts the text information into a text vector, converts the pinyin information into a pinyin vector, and determines the type of text information according to the text vector and the pinyin vector.

[0025] In a second aspect, the present invention also provides an in-vehicle voice interaction device, comprising an acquisition module, a pre-processing module, a true voice determination module and a true voice determination module; the acquisition module is used to acquire initial audio data; the pre-processing module is used to pre-process the initial audio data to obtain audio data corresponding to each cabin occupant; the true voice determination module is used to input the audio data corresponding to each cabin occupant into a preset multi-mode voice activity detection algorithm network for processing, and determine whether the audio data corresponding to each cabin occupant belongs to true user voice activity; the true voice processing module is used to generate interaction instructions based on the audio data belonging to true user voice activity.

[0026] In a third aspect, the present invention further provides a computer device, comprising a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the in-vehicle voice interaction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0027] In a fourth aspect, the present invention further provides a vehicle, comprising the computer device of the third aspect.

[0028] In a fifth aspect, the present invention further provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the in-vehicle voice interaction method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0029] In a sixth aspect, the present invention further provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the in-vehicle voice interaction method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 is a flow chart of a vehicle-mounted voice interaction method according to an embodiment of the present invention;

[0032] Figure 2 is a flow chart of another in-vehicle voice interaction method according to an embodiment of the present invention;

[0033] Figure 3 is a block diagram of multi-mode adaptive speech separation;

[0034] Figure 4 is a flow chart of another in-vehicle voice interaction method according to an embodiment of the present invention;

[0035] Figure 5 It is a schematic diagram of the image data and audio data alignment processing flow;

[0036] Figure 6 It is a processing flow diagram of multi-mode VAD;

[0037] Figure 7 It is a semantic diversion block diagram;

[0038] Figure 8 is a flowchart of an example of an in-vehicle voice interaction method;

[0039] Fig. 9 is a structural block diagram of a vehicle-mounted voice interaction device according to an embodiment of the present invention;

[0040] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0042] According to an embodiment of the present invention, an embodiment of an in-vehicle voice interaction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0043] In this embodiment, a vehicle-mounted voice interaction method is provided, which can be used in a computer device. Figure 1 is a flow chart of a vehicle voice interaction method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0044] Step S101: Acquire initial audio data.

[0045] The initial audio data is the audio data acquired by a microphone. After acquiring the initial audio data, it is necessary to perform noise suppression, echo cancellation, and speech separation on the initial audio data, output clean audio data of different users, and perform pre-emphasis, framing, windowing, Fourier transform, and filter processing on the audio data to calculate the audio frequency domain features. Common audio frequency domain features include MFCC (MFCC, Mel Frequency Cepstral Coefficients), FBANK (Filter Bank Features), etc. The present invention is not limited to these, and can also be applied to other feature calculation representations.

[0046] Step S102: pre-processing the initial audio data to obtain audio data corresponding to each cabin occupant.

[0047] It should be noted that the voice assistants in the related art cannot solve the problem of voice separation of multiple people speaking within a sound range very well; however, this embodiment can accurately and effectively separate the audio data of different cabin occupants based on the data of multiple people speaking in the car under various complex environmental conditions by pre-processing the initial audio data.

[0048] Step S103: input the audio data corresponding to each cabin occupant into a preset multi-mode voice activity detection algorithm network for processing to determine whether the audio data corresponding to each cabin occupant belongs to a real user voice activity.

[0049] Among them, the multi-mode voice activity detection algorithm network can be referred to as a multi-mode VAD algorithm network.

[0050] In the application scenario of in-vehicle voice interaction, the initial audio data obtained is not the sound made by the user, but the environmental sound, such as the broadcast being played. Therefore, it is necessary to determine whether the audio data corresponding to each cabin occupant belongs to the real user voice activity.

[0051] Step S104: Generate interaction instructions based on audio data belonging to real user voice activity.

[0052] The in-vehicle voice interaction method provided in this embodiment obtains audio data corresponding to each cabin occupant by pre-processing the initial audio data; the audio data corresponding to each cabin occupant is input into a preset multi-mode voice activity detection algorithm network for processing to determine whether the audio data corresponding to each cabin occupant belongs to a real user voice activity. Therefore, before the initial audio data is processed to generate an interaction instruction, it can be determined whether the audio data corresponding to each cabin occupant belongs to a real user voice activity, thereby reducing the false wake-up rate.

[0053] In this embodiment, a vehicle-mounted voice interaction method is provided, which can be used in a computer device. Figure 2 is a flow chart of another in-vehicle voice interaction method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0054] Step S201: Acquire initial audio data.

[0055] Step S202: pre-process the initial audio data to obtain audio data corresponding to each cabin occupant.

[0056] Figure 3 It is a block diagram of multi-mode adaptive speech separation, such as Figure 3 As shown, in an optional implementation, pre-processing the initial audio data to obtain audio data corresponding to each cabin occupant includes the following steps a1 to a5.

[0057] Step a1: When the initial face image data can be obtained, feature extraction and fusion are performed on the initial face image data to obtain an image feature vector.

[0058] In an optional embodiment, feature extraction and fusion of initial facial image data includes: extracting features from the initial facial image data to obtain a facial region feature vector, an identity feature vector, an age feature vector, and a gender feature vector; and fusing the facial region feature vector, the identity feature vector, the age feature vector, and the gender feature vector to obtain an image feature vector.

[0059] The input of the multi-mode adaptive speech separation algorithm network model has two branches, corresponding to image data and audio data respectively. The image branch part has a backbone network, upsampling and downsampling multi-layer feature fusion modules, and multiple output branches. For the backbone network, a model similar to MobileNet, ShuffleNet, etc. can be used to perform preliminary network layer feature extraction, and the fourth layer feature L d 4. The sixth layer feature L d 6. The eighth layer feature L d 8. The tenth layer feature L d 10. For upsampling and downsampling multi-layer feature fusion modules, the upsampling of specific layers of features with different intervals can be obtained by fusion L u 8. L u 6. L u 4. The calculation formula is:

[0060] L u 8=upsampler(L d 8,L d 10), L u 6=upsampler(L d 6,L u 8), L u 4=upsampler(L d 4,L u 6)

[0061] Next, L u 8. L u 6. L u 4 Perform feature enhancement to obtain LE u 8.LE u 6.LE u 4.

[0062] Set the image output branches to BoxHead, FaceIDHead, AgeHead, and GenderHead, and use BoxHead to generate LE u 8.LE u 6.LE u 4 After processing, the face area feature vector feat is obtained BH , through FaceIDHead to LE u 8.LEu 6.LE u 4 After processing, the identity feature vectors feat of different faces are obtained FH , through AgeHead to LE u 8.LE u 6.LE u 4 After processing, the age feature vectors feat of different faces are obtained AH , through GenderHead to LE u 8.LE u 6.LE u 4 After processing, the gender feature vectors feat of different faces are obtained GH .

[0063] Will feature BH 、feat FH 、feat AH 、feat GH Align them to the same dimension, concatenate and fuse them to get the image feature vector vf. The calculation formula is as follows.

[0064] vf = cat (feat BH ,feat FH ,feat AH ,feat GH )

[0065] Step a2: extract features from the initial audio data to obtain a voiceprint feature vector and an audio coding feature vector; fuse the voiceprint feature vector and the audio coding feature vector to obtain an audio feature vector.

[0066] The audio branch part includes audio frequency domain feature calculation, voiceprint recognition module, feature extraction module, and feature fusion module. The audio frequency domain feature calculation includes STFT (Short-Time Fourier Transform), Fbank (Filter Bank, filter bank, etc. This embodiment performs pre-emphasis, windowing, framing, and FFT calculation on the audio data to obtain the feat STFT , then after Mel filtering and power logarithmization, we get feat Fbank . Set the feat within the set time STFT Input into the time-delay neural network to obtain the voiceprint feature vector feat of the set dimension SV . Set the feat within the time period Fbank Input to the backbone network, upsampling and downsampling multi-layer feature fusion module to obtain the audio coding feature vector feat of the set dimension AE .

[0067] Set the voiceprint feature vector feat of the set dimension SV and the audio encoding feature vector feat of set dimension AE Align them to the same dimension and concatenate them to get the audio feature vector af. The calculation formula is as follows:

[0068] af=cat(feat SV ,feat AE )

[0069] Step a3: Fuse the image feature vector and the audio feature vector to obtain a multi-mode feature vector.

[0070] In the case of image data input, the image feature vector vf and the audio feature vector af are concatenated and fused to obtain the multi-mode feature vector avf in the multi-mode adaptive speech separation algorithm network model. The calculation formula is:

[0071] avf=cat(vf,af)

[0072] Step a4: obtaining a first mask set according to the multi-mode feature vector, wherein the first mask set includes an activated speech mask corresponding to each cabin occupant.

[0073] After the multi-mode feature vector avf passes through the feature enhancement network module, it outputs av_mask (Active Voice Mask) corresponding to the number of different cabin personnel. Next, the feature vector of the audio STFT is multiplied by the av_mask vector one by one to obtain the spectrum data AVS of different cabin personnel after voice separation. The calculation formula is as follows:

[0074] AVS i =feat STFT ☉av_mask i , i∈(1,n speaker )

[0075] Step a5: Obtain audio data corresponding to each cabin occupant according to the first mask set.

[0076] After performing ISTFT (Inverse Short-Time Fourier Transform) on the spectrum data AVS of different cabin personnel, the original audio data AVA of different cabin personnel separated at last is obtained. The calculation formula is as follows:

[0077] AVA i =ISTFT(AVS i ), i∈(1, n speaker )

[0078] That is to say, when the camera detects a face, multi-mode data fusion is performed based on face-related features such as age, gender, Face ID, and voiceprint, frequency domain and other related features after initial audio data processing. If a face is detected, the separated audio can correspond to the speaker's face and the speaker's position in the cabin. Multi-mode adaptive speech separation processing can improve the accuracy of speech separation, solve the problem of incomplete and inaccurate separation when there are multiple passengers in the cabin, and obtain the identity, location and other information of each cabin occupant after separation, providing more possibilities for intelligent interaction in the cabin.

[0079] In another optional implementation, pre-processing the initial audio data to obtain audio data corresponding to each cabin occupant includes the following steps b1 to b5.

[0080] Step b1: When the initial facial image data cannot be obtained, the initial audio data is processed to obtain a voiceprint feature vector and an audio coding feature vector; the voiceprint feature vector and the audio coding feature vector are fused to obtain an audio feature vector.

[0081] As mentioned above, the audio branch part includes audio frequency domain feature calculation, voiceprint recognition module, feature extraction module, and feature fusion module, wherein the audio frequency domain feature calculation includes STFT (Short-Time Fourier Transform), Fbank (Filter Bank, filter bank, etc. This embodiment performs pre-emphasis, windowing, framing, and FFT calculation on the audio data to obtain the feat STFT , then after Mel filtering and power logarithmization, we get feat Fbank . Set the feat within the set time STFT Input into the time-delay neural network to obtain the voiceprint feature vector feat of the set dimension SV . Set the feat within the time period Fbank Input to the backbone network, upsampling and downsampling multi-layer feature fusion module to obtain the audio coding feature vector feat of the set dimension AE .

[0082] Set the voiceprint feature vector feat of the set dimension SV and the audio encoding feature vector feat of set dimension AE Align them to the same dimension and concatenate them to get the audio feature vector AF. The calculation formula is as follows:

[0083] AF = cat (feat SV ,feat. AE )

[0084] Step b2: obtaining a second mask set according to the audio feature vector, wherein the second mask set includes activation voice masks corresponding to different cabin occupants.

[0085] In the case where only audio data is input, the audio feature vector AF is passed through the feature enhancement network module to output a_mask corresponding to the number of different cabin occupants. The feature vector of the audio STFT is multiplied point by point by the a_mask vector to obtain the spectrum data AS of different cabin occupants after speech separation. The calculation formula is as follows:

[0086] AS i =feat STFT ☉a_mask i , i∈(1,n speaker )

[0087] Step b3: Obtain audio data corresponding to each cabin occupant according to the second mask set.

[0088] After performing ISTFT on the spectrum data AS of different cabin personnel, the final separated original audio data AA of different cabin personnel are obtained. The calculation formula is as follows:

[0089] AA i =ISTFT(AS i ), i∈(1,n speaker )

[0090] That is to say, if no face is detected, the audio of different users is separated by superimposing relevant features such as voiceprint and audio frequency domain, and then performing audio spectrum separation and ISTFT (Inverse Short-Time Fourier Transform) processing.

[0091] The output of the multi-mode adaptive speech separation algorithm network model has two branches: multi-mode and single-mode. The multi-mode branch corresponds to the case where valid audio and image are input at the same time, and the output data is AVA; the single-mode branch corresponds to the case where only audio data is input, and the output data is AA. In practical applications, the corresponding output can be flexibly selected for subsequent processing according to the input situation. In other words, it supports two modes: image and audio dual-modal data input and audio single-modal data input. The output also has multi-modal fusion reasoning results and single-modal reasoning results, which can adapt to the situation where there is no camera.

[0092] In a further optional embodiment, when the initial facial image data can be obtained, it also includes: inputting the initial facial image data into a face and key point detection model to obtain multiple facial frames, and key points corresponding to each facial frame; and obtaining lip image data corresponding to each facial frame based on the key points corresponding to each facial frame.

[0093] Specifically, the original image data returned by the camera is in YUV format. This module needs to generate image data in RGB or GRAY format that meets the requirements through scaling and conversion of the YUV format image data. Initialize the structure of the current frame image and set the timestamp of the current frame image data. Input the RGB or GRAY image data into the face and key point detection model to infer and calculate and output face frames and corresponding face key points at different positions. Based on the key points of each face, the lip image data of each person can be calculated and extracted, and the lip image data of the current frame is set in the structure of the current frame image, and the structure is added to the image data queue. Common face and key point detection methods include YOLO, Retinaface and other algorithms, which are not limited in the present invention.

[0094] After obtaining the original audio data, echo cancellation, noise suppression, and voice separation are performed to obtain audio data of different users. Initialize the structure of the current audio frame, set the timestamp of the current audio frame, and add the structure to the audio data queue.

[0095] Step S203: input the audio data corresponding to each cabin occupant into a preset multi-mode voice activity detection algorithm network for processing to determine whether the audio data corresponding to each cabin occupant belongs to a real user voice activity.

[0096] In an optional implementation, the audio data corresponding to each cabin occupant is input into a preset multi-mode voice activity detection algorithm network for processing, and determining whether the audio data corresponding to each cabin occupant belongs to a true user voice activity includes steps c1 to c4.

[0097] Step c1: When the initial face image data can be obtained, a section of lip image data in the image data queue is input into the multi-mode voice activity detection algorithm network for feature extraction, feature enhancement and ordered different frame offset embedding fusion processing in the channel space to obtain the first stage image feature data.

[0098] In an optional implementation, when the initial face image data can be obtained, a section of lip image data in the image data queue is input into the image network branch of the multi-mode voice activity detection algorithm for feature extraction, feature enhancement, and offset embedding fusion processing of ordered different frames in the channel space to obtain the first stage image feature data. The details are as follows:

[0099] The multi-modal VAD algorithm network includes an image network branch, an audio network branch, and a fusion network. The inputs are two modalities of image data and audio data, denoted as image_input and audio_input respectively, and the outputs are two results of multi-modal and single-modal, denoted as multi_output and audio_output respectively.

[0100] The image network branch mainly includes a feature extraction network layer, a feature embedding network layer, and a linear network layer. The feature extraction network layer consists of a certain number of network layers, and each network layer mainly includes convolution, normalization, activation function, and attention module. Exemplarily, the first layer network Layer1 to the seventeenth layer network Layer17 are the feature extraction network layer, which calculates the input image data, and extracts the features of the image data through the feature extraction network layer to obtain the contour features, edge features, texture features, shape features of the low-level image and the semantic features of the high-level image. The calculation methods of the features FL of different network layers are as follows:

[0101] FL1 = fe(image_input), the feature calculation of the first layer network.

[0102] FL i+1 = fe(FL i ), 0 < i < 17, the feature calculation of the first layer to the seventeenth layer network.

[0103] The feature embedding network layer mainly includes the offset stitching fusion calculation between different image frames. It is set that the second layer network Layer2 to the fifteenth layer network Layer15 are the feature extraction network layers adding the feature embedding network layer. The feature embedding network layer is added to the preset feature extraction network layer to perform offset embedding fusion processing on different image frames in order in the channel space dimension, and obtain the fusion feature FS of a certain sequence of image frames in the spatio-temporal dimension.

[0104] FS i = shift(x i , frame_length, div), 2 ≤ i ≤ 15

[0105] x is the image sequence feature of different network layers, frame_length is the length of the image sequence, div is the offset processing length, and i is the serial number of the set network layer.

[0106] The linear network layer mainly includes linear processing and activation function, which are used to map and transform the fusion feature into a feature of a specific dimension to obtain the first-stage image feature data VF.

[0107] VF = linear(FL)

[0108] Step c2: Input a segment of audio data in the audio data queue into the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain first-stage audio feature data.

[0109] In an optional implementation, a segment of audio data in the audio data queue corresponding to each cabin occupant is input into the audio network branch of the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain first-stage audio feature data. Specifically, as follows:

[0110] The audio network branch mainly includes feature extraction network layer and linear network layer. The feature extraction network layer is composed of a certain number of network layers, each of which mainly includes convolution, normalization, and activation functions. For example, the first layer of the network Layer1 to the 14th layer of the network Layer14 are feature extraction network layers, which calculate the input audio data, extract features of the audio data through the feature extraction network layer, and obtain the audio features AFL of different network layers. The calculation method of audio features of different network layers is as follows:

[0111] AFL i+1 =afe(audio_input)

[0112] The linear network layer mainly includes linear processing and activation functions, which are used to convert the fused feature map into features of a specific dimension to obtain the first-stage audio feature data AF.

[0113] AF=linear(AFL)

[0114] Step c3: concatenate and fuse the first-stage image feature data and the first-stage audio feature data to obtain multi-mode output data.

[0115] In an optional implementation, the first-stage image feature data and the first-stage audio feature data are concatenated and fused in a fusion network of a multi-mode voice activity detection algorithm to obtain multi-mode output data, as follows:

[0116] The fusion network mainly includes splicing fusion and linear network layers. The first-stage image feature data VF calculated by the image network branch and the first-stage audio feature data AF calculated by the audio network branch are spliced ​​and fused to obtain the multi-mode fusion feature AVF. After calculation by the image linear network layer, the multi-mode output multi_output is obtained.

[0117] AVF=cat(VF,AF)

[0118] multi_output = linear(AVF)

[0119] Step c4: Determine whether the audio data corresponding to each cabin occupant belongs to a real user voice activity based on the multi-mode output data.

[0120] In another optional implementation, the audio data corresponding to each cabin occupant is input into a preset multi-mode voice activity detection algorithm network for processing, and determining whether the audio data corresponding to each cabin occupant belongs to a true user voice activity includes steps d1 to d2.

[0121] Step d1: When the initial facial image data cannot be obtained, a segment of audio data in the audio data queue corresponding to each cabin occupant is input into the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain single-mode output data.

[0122] Specifically, the first-stage audio feature data AF is calculated by the audio linear network layer to obtain the single-mode output audio_output.

[0123] audio_output = linear(AF)

[0124] Step d2: Determine whether the audio data corresponding to each cabin occupant belongs to real user voice activity based on the single-mode output data.

[0125] Step S204: Generate interaction instructions based on the audio data belonging to the real user voice activity.

[0126] That is to say, the multi-mode VAD network of this embodiment is a streaming adaptive multi-mode VAD, which can infer the input audio and image information in real time, and make adaptive output according to the modality of the input data, that is, if the input is valid audio and image data, the output is the multi-mode VAD result; if the input is valid audio data, the output is the single-mode VAD result. If there is no audio data, no detection is performed and -1 is directly returned, which means that there is no user voice activity.

[0127] Compared with the single-mode audio VAD processing method, the streaming adaptive multi-mode VAD designed in this embodiment can reduce the false wake-up rate in scenarios such as high noise. Compared with the traditional multi-mode VAD processing method, it can improve the detection accuracy and automatically switch modes according to actual input data to provide accurate results and convenient interaction.

[0128] In this embodiment, a vehicle-mounted voice interaction method is provided, which can be used in a computer device. Figure 4 is a flow chart of another in-vehicle voice interaction method according to an embodiment of the present invention. In this embodiment, both initial audio data and initial facial image data can be obtained; Figure 8 FIG. 1 is a flowchart of an example of an in-vehicle voice interaction method. Figure 4 and Figure 8 As shown, the process includes the following steps:

[0129] Step S401: Acquire initial audio data and initial face image data.

[0130] Step S402: pre-process the initial audio data to obtain audio data corresponding to each cabin occupant.

[0131] Specifically, pre-processing the initial audio data to obtain audio data corresponding to each cabin occupant includes: extracting features from the initial facial image data to obtain a facial region feature vector, an identity feature vector, an age feature vector, and a gender feature vector; fusing the facial region feature vector, the identity feature vector, the age feature vector, and the gender feature vector to obtain an image feature vector; extracting features from the initial audio data to obtain a voiceprint feature vector and an audio coding feature vector; fusing the voiceprint feature vector and the audio coding feature vector to obtain an audio feature vector; fusing the image feature vector and the audio feature vector to obtain a multi-mode feature vector; obtaining a first mask set based on the multi-mode feature vector, wherein the first mask set includes an activated voice mask corresponding to each cabin occupant; and obtaining audio data corresponding to each cabin occupant based on the first mask set.

[0132] Step S403: input the initial face image data into the face and key point detection model to obtain multiple face frames and key points corresponding to each face frame; obtain the lip image data corresponding to each face frame according to the key points corresponding to each face frame.

[0133] Step S404: storing the lip image data corresponding to each face frame into an image data queue.

[0134] Step S405: storing the audio data corresponding to each cabin occupant into an audio data queue.

[0135] Step S406: When the length of the audio data queue is greater than a preset threshold, select the audio data to be processed from the audio data queue, and determine the start time and end time of the audio data to be processed.

[0136] The audio data to be processed is the audio data input into the multi-mode voice activity detection algorithm network as mentioned above.

[0137] Step S407: Select the image data to be processed from the image data queue according to the start time and end time of the audio data to be processed.

[0138] The image data to be processed is the image data input into the multi-modal voice activity detection algorithm network as mentioned above.

[0139] Step S408: When the number of frames of the image data to be processed is less than the preset length, the last image frame is used to pad the number of frames of the image data to be processed to the preset length; when the image data to be processed is greater than the preset length, some image frames are filtered out from the image to be processed, and the number of frames of the filtered out image frames is the preset length.

[0140] For example, Figure 5 This is a schematic diagram of the image data and audio data alignment process. Figure 6 This is a flowchart of the processing flow of multi-mode VAD, such as Figure 5 and Figure 6 As shown, audio data and image data are acquired in real time, and the data are stored in the audio data queue and the image data queue respectively. The length of the audio data queue is judged each time. If the length is greater than the preset value, a specific amount of audio data is selected from the audio data queue, and the audio frequency domain features, such as FBANK, MFCC, etc., are calculated according to the set frame length, window length and other parameters. At the same time, the timestamp begin_timestamp corresponding to the first selected audio data and the timestamp end_timestamp corresponding to the last audio data are taken out, and the image data between these two timestamps are filtered out in the image data queue in chronological order to obtain the image data queue image_list to be processed, and the queue length is image_list_size. The screening strategy is related to the algorithm network structure.

[0141] Furthermore, when the number of frames of the image data to be processed is less than a preset length, the last image frame is used to pad the number of frames of the image data to be processed to the preset length; when the image data to be processed is greater than the preset length, some image frames are filtered out from the image to be processed, and the number of frames of the selected some image frames is the preset length.

[0142] For example, for image data in RGB format, set the number of image frames frame_count to 4, and for image data in GRAY format, set the number of image frames frame_count to 12. If the number of image frames obtained from the two timestamps of begin_timestamp and end_timestamp is greater than 0 and less than frame_count, the last image frame is used to pad the total number of image frames to frame_count. The calculation formula is as follows:

[0143]

[0144] If the number of image frames obtained from the two timestamps begin_timestamp and end_timestamp is greater than frame_count, the equal-step algorithm is used to calculate and filter out frame_count image frames. The calculation formula is as follows:

[0145] index=1+(n-1)d, n∈(1,frame_count), d=image_list_size / frame_count

[0146] Select the corresponding image frame data from image_list according to index to generate an image frame sequence for subsequent calculations.

[0147] If the number of image frames calculated from begin_timestamp and end_timestamp is equal to 0, the audio frequency domain feature data is directly input into the multi-mode VAD algorithm network to obtain the VAD result; if the calculated number of image frames is greater than 0, the image frame data and the corresponding audio frequency domain feature data are input into the multi-mode VAD algorithm network to obtain the multi-mode VAD result.

[0148] Step S409: input the image data to be processed and the audio data to be processed into a preset multi-mode voice activity detection algorithm network for processing, and determine whether the audio data corresponding to each cabin occupant belongs to a real user voice activity.

[0149] Step S410: Generate interaction instructions based on audio data belonging to real user voice activity.

[0150] In an optional implementation, generating an interaction instruction based on audio data belonging to a real user voice activity includes steps S4101 to S4103.

[0151] Step S4101: Convert the audio data containing real user voice activity into text information.

[0152] Step S4102: Determine the type of text information, where the types of text information include: task-based strong intent, task-based weak intent, task-based unprocessed, non-task-based chat, non-task-based dialogue, and other non-task-based.

[0153] Among them, other non-task types are non-task types other than non-task chat and non-task conversation.

[0154] Specifically, determining the type of text information includes the following steps e1 to e4.

[0155] Step e1: Generate pinyin information corresponding to the text information.

[0156] Step e2: Convert text information into text vectors, and convert pinyin information into pinyin vectors.

[0157] Step e3: Encode the text vector and the phonetic vector respectively, and concatenate and fuse the encoded text vector and phonetic vector to obtain a fusion feature.

[0158] Step e4: After feature enhancement processing, the fused features are input into a preset fully connected network to obtain the type of text information.

[0159] This is because there are often cases of errors in the recognition of text information. For example, the recognized text often contains homophones and wrong characters, which have a great impact on subsequent semantic understanding. In order to reduce this impact and improve the accuracy of semantic understanding, the present invention generates pinyin information corresponding to the text information, converts the text information into text vectors, converts the pinyin information into pinyin vectors, and determines the type of text information based on the text vectors and the pinyin vectors.

[0160] After obtaining the text information, it is converted into pinyin, and the pinyin and text information are input into the semantic diversion model for diversion detection.

[0161] Among them, the semantic diversion model of text and pinyin fusion mainly includes modules such as text to pinyin, word list construction, encoding calculation, text and pinyin fusion, etc.

[0162] The text-to-pinyin module mainly uses the pinyin library to convert the input text input_text into pinyin to get pinyin_text.

[0163] The word list construction part mainly designs Chinese word lists and Pinyin word lists that can cover specific tasks. The Chinese word list includes the actual Chinese text, special symbols and their corresponding IDs, and the Pinyin word list includes the Pinyin converted from Chinese, special symbols and their corresponding IDs. The Chinese and Pinyin word lists must be consistent.

[0164] The encoding vector part first vectorizes the input and converts it into a vector of a specific dimension. The calculation formula is as follows:

[0165] f input =embed0ing(input_text),f pinyin =embedding(pinyin_tex)

[0166] Finput and fpinyin are encoded separately. A simple CNN can be used, or a network with a transformer structure can be used to process finput and fpinyin. The calculation formula is as follows:

[0167] einput =encoder(f input ), e pinyin =encoder(f pinyin )

[0168] E input and e pinyin Unify them to the same dimension, perform splicing and fusion processing on them, and obtain the fused feature CF. The calculation formula is as follows:

[0169] CF = concat(e input , e pinyin ) 。

[0170] After feature enhancement processing on the fused features, ECF (Equalization-Cancellation Filter) is obtained, and ECF is input into a specific fully connected network to obtain the final classification results of task-based strong intent, task-based weak intent, task-based unprocessed, non-task-based chat, non-task-based dialogue, and other non-task-based.

[0171] Step S4103: Process the text information according to the type of the text information and generate an interactive instruction.

[0172] Specifically, the text information is processed according to the type of the text information, and the interactive instructions are generated, including: when the type of the text information is a task-based strong intent or a task-based weak intent, determine whether the device can process the text information; when the device can process the text information, process the text information on the device and generate interactive instructions; otherwise, send the text information to the cloud, and accept the interactive instructions generated by the cloud based on the text information; when the type of the text information is task-based not processing, non-task-based small talk, non-task-based dialogue, or other non-task-based, do not respond to the text information. Figure 7 As shown, in this embodiment, text information is processed according to the type of text information, and interactive instructions are generated including offline ASR and online ASR. The offline ASR algorithm model is adapted and enhanced for some corpora of cockpit vehicle control and vehicle equipment. Therefore, this part of the corpus is recognized offline first, while the online ASR recognition is more difficult for navigation, telephone, media and other corpora.

[0173] Furthermore, the in-vehicle voice interaction method of this embodiment also includes natural language processing. Natural language processing mainly includes semantic understanding and dialogue management. Semantic understanding covers corpus rewriting, rejection distribution, field classification, intent classification and other parts. Corpus rewriting is to rewrite the colloquial corpus in a standardized format that is better understood by the machine, so that it can be easily understood. Rejection distribution is to detect and analyze the corpus to see whether it needs further understanding. If it is analyzed that it needs to be processed, it will be classified into a specific business category according to the set logic; if it does not need to be processed, it will be returned directly. That is, the input corpus is sentence-wise detected, simple corpus is processed using a small model, and multi-intention complex corpus is rewritten by a large model. The rewritten corpus is input into the distribution rejection module, and rejection or corresponding tasks are distributed for different corpora.

[0174] If it is a task-based conversation, domain and intent recognition are performed. Domain classification and intent classification are to process the corpus in more detail according to the business definition and output preliminary semantics. If a large model is required for implementation, the relevant task large model is called. If it is a specific skill, the relevant information source is called for processing. The implementation based on traditional machine learning includes algorithms such as SVM, and the implementation based on deep learning includes algorithm models such as CNN, RNN, and Bert. Now, large language models can also be used for related enhancements.

[0175] Dialogue management covers context management, task flow management and other parts. It combines the previously parsed preliminary semantics with the context content and task information, and outputs complete semantics after comprehensive processing.

[0176] It should be noted that steps 301 to S404 in this embodiment can be completed in a voice assistant. The voice assistant integrates different algorithm modules, performs noise reduction, echo cancellation, voice separation, speech detection, diversion detection and other processing based on audio data and image data, recognizes the voice as text, discards it according to the results, or converts it into interactive instructions and sends it to the ecological application for operation control.

[0177] In addition, this embodiment also includes a cloud platform service. The cloud platform service serves as a connector between the voice assistant and other cloud services. It mainly processes requests and responses from the voice assistant, online ASR services, large model services, etc. It is also responsible for the arbitration and distribution of services to ensure the normal completion of voice interaction.

[0178] In summary, the in-vehicle voice interaction method of this embodiment includes the following steps: after the microphone picks up the sound, the audio data is transmitted to the system, and after being processed at the system HAL (Hardware Abstraction Layer) layer, it will be transparently transmitted to the upper layer. The voice assistant application can obtain the real-time audio stream through the relevant interface, and obtain the clean audio stream data of different users through noise suppression, echo cancellation, voice separation and other algorithms. At this time, the lip images of different users are extracted, combined with the audio data of the corresponding users, and input into the multi-mode VAD algorithm model for sound activity detection. If the user is not detected speaking, it will not be processed. If the user is detected speaking, the audio data between the start and end of VAD is input into the offline ASR for speech-to-text processing. The text recognized by the offline ASR is input into the offline semantic diversion algorithm model for detection. If it is a valid and processable instruction, it is input into the offline NLU for semantic understanding and output of semantic instructions. If it does not need to be processed, it is directly discarded. If it needs to be processed online, it is input into the cloud platform for processing. After receiving the audio data and offline ASR text transmitted by the client-side application, the cloud platform inputs the audio data into the online ASR for more accurate recognition. After obtaining the online ASR text, it inputs it into the online NLU for semantic understanding and dialogue management or into the large language model for reasoning and understanding, and finally returns the result to the client-side application.

[0179] In summary, the in-vehicle voice interaction method of this embodiment can avoid waking up the voice interaction, separate the voices of different users in the cockpit, flexibly detect whether the user is speaking according to the different hardware configurations of the in-vehicle cockpit, reduce the false awakening rate, filter out non-task instructions such as person-to-person dialogue and TTS broadcast, and dynamically schedule the device-side and cloud platform-side algorithm network models for processing. This dynamic adjustment makes full use of the respective advantages of different platform algorithms, not only reducing the communication link to a certain extent, speeding up the interactive touch response, but also ensuring that the output results are highly accurate while reducing cloud traffic consumption and concurrent pressure. The multi-mode full-time voice interaction system of the present invention is more stable and reliable, which can make users use voice more conveniently, improve interaction efficiency, improve the natural friendliness and intelligence level of in-vehicle cockpit voice interaction, and reduce operating costs.

[0180] In this embodiment, a vehicle-mounted voice interaction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0181] This embodiment provides a vehicle-mounted voice interaction device, such as Fig. 9As shown, including:

[0182] The acquisition module 901 is used to acquire initial audio data.

[0183] The pre-processing module 902 is used to pre-process the initial audio data to obtain audio data corresponding to each cabin occupant.

[0184] The real voice determination module 903 is used to input the audio data corresponding to each cabin occupant into a preset multi-mode voice activity detection algorithm network for processing, and determine whether the audio data corresponding to each cabin occupant belongs to real user voice activity.

[0185] The real voice processing module 904 is used to generate interaction instructions based on the audio data belonging to the real user voice activity.

[0186] In some optional embodiments, the pre-processing module 902 is specifically used for: when the initial facial image data can be obtained, performing feature extraction and fusion on the initial facial image data to obtain an image feature vector; performing feature extraction on the initial audio data to obtain a voiceprint feature vector and an audio coding feature vector; fusing the voiceprint feature vector and the audio coding feature vector to obtain an audio feature vector; fusing the image feature vector and the audio feature vector to obtain a multi-mode feature vector; obtaining a first mask set based on the multi-mode feature vector, wherein the first mask set includes an activated voice mask corresponding to each cabin occupant; and obtaining audio data corresponding to each cabin occupant based on the first mask set.

[0187] In some optional embodiments, the pre-processing module 902 is specifically used for: when the initial facial image data cannot be obtained, processing the initial audio data to obtain a voiceprint feature vector and an audio coding feature vector; fusing the voiceprint feature vector and the audio coding feature vector to obtain an audio feature vector; obtaining a second mask set based on the audio feature vector, wherein the second mask set includes an activated voice mask corresponding to each cabin occupant; and obtaining audio data corresponding to each cabin occupant based on the second mask set.

[0188] In some optional embodiments, the pre-processing module 902 is also used to: input the initial facial image data into the face and key point detection model to obtain multiple facial frames and key points corresponding to each facial frame; and obtain lip image data corresponding to each facial frame based on the key points corresponding to each facial frame.

[0189] In some optional embodiments, the true voice determination module 903 is specifically used for: when the initial facial image data can be obtained, inputting a section of lip image data in the image data queue into the multi-mode voice activity detection algorithm network for feature extraction, feature enhancement and ordered different frames in the channel space offset embedding fusion processing to obtain the first stage image feature data; inputting a section of audio data in the audio data queue corresponding to each cabin occupant into the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain the first stage audio feature data; splicing and fusing the first stage image feature data and the first stage audio feature data to obtain multi-mode output data; and determining whether the audio data corresponding to each cabin occupant belongs to true user voice activity based on the multi-mode output data.

[0190] In some optional embodiments, the true voice determination module 903 is specifically used for: when the initial facial image data cannot be obtained, inputting a segment of audio data in the audio data queue corresponding to each of the cabin occupants into the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain single-mode output data; and determining whether the audio data corresponding to each cabin occupant belongs to true user voice activity based on the single-mode output data.

[0191] In some optional embodiments, before the audio data corresponding to each cabin occupant is input into a preset multi-mode voice activity detection algorithm network for processing and determining whether the audio data corresponding to each cabin occupant belongs to a true user voice activity, the true voice determination module 903 is also used to: when the initial facial image data can be obtained, store the lip image data corresponding to each facial frame into an image data queue; store the audio data corresponding to each cabin occupant into an audio data queue; when the length of the audio data queue is greater than a preset threshold, select the audio data to be processed in the audio data queue, and determine the start time and end time of the audio data to be processed; select the image data to be processed in the image data queue according to the start time and end time of the audio data to be processed; when the number of frames of the image data to be processed is less than a preset length, use the last image frame to pad the number of frames of the image data to be processed to the preset length; when the image data to be processed is greater than the preset length, filter out some image frames in the image to be processed, and the number of frames of the filtered out some image frames is the preset length.

[0192] In some optional implementations, the real voice processing module 904 includes a text information generation unit, a text type determination unit, and a text processing unit. The text information generation unit is used to convert audio data with real user voice activity into text information; the text type determination unit is used to determine the type of text information, where the types of text information include: task-based strong intention, task-based weak intention, task-based unprocessed, non-task-based small talk, non-task-based dialogue, and other non-task-based; the text processing unit is used to process the text information according to the type of the text information and generate an interactive instruction.

[0193] In some optional embodiments, the text processing unit is used to determine whether the device can process the text information when the type of the text information is task-based strong intent or task-based weak intent; when the device can process the text information, the text information is processed on the device to generate interaction instructions; otherwise, the text information is sent to the cloud, and the interaction instructions generated by the cloud based on the text information are accepted; when the type of the text information is task-based not processing, non-task-based chat, non-task-based dialogue, or other non-task-based, the text information is not responded to.

[0194] In some optional embodiments, the text information generation unit is used to: generate pinyin information corresponding to the text information; convert the text information into a text vector, and convert the pinyin information into a pinyin vector; encode the text vector and the pinyin vector respectively, and concatenate and fuse the encoded text vector and pinyin vector to obtain a fused feature; perform feature enhancement processing on the fused feature, and input it into a preset fully connected network to obtain the type of the text information.

[0195] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0196] The in-vehicle voice interaction device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0197] The embodiment of the present invention also provides a computer device having the above Fig. 9 The in-vehicle voice interaction device shown.

[0198] An embodiment of the present invention further provides a vehicle, comprising the above-mentioned computer device.

[0199] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0200] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0201] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0202] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0204] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0205] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0206] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0207] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0208] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A vehicle-mounted voice interaction method, characterized in that: include: Get initial audio data; Pre-processing the initial audio data to obtain audio data corresponding to each cabin occupant; Inputting the audio data corresponding to each of the cabin personnel into a preset multi-mode voice activity detection algorithm network for processing to determine whether the audio data corresponding to each of the cabin personnel belongs to a true user voice activity; Interaction instructions are generated based on audio data belonging to real user voice activity.

2. The method according to claim 1, characterized in that The pre-processing of the initial audio data to obtain audio data corresponding to each cabin occupant includes: When the initial face image data can be obtained, feature extraction and fusion are performed on the initial face image data to obtain an image feature vector; Extracting features from the initial audio data to obtain a voiceprint feature vector and an audio coding feature vector; fusing the voiceprint feature vector and the audio coding feature vector to obtain an audio feature vector; Fusing the image feature vector and the audio feature vector to obtain a multi-mode feature vector; Obtaining a first mask set according to the multi-mode feature vector, wherein the first mask set includes an activated voice mask corresponding to each cabin occupant; Audio data corresponding to each of the cabin occupants is obtained according to the first mask set.

3. The method according to claim 1, characterized in that The pre-processing of the initial audio data to obtain audio data corresponding to each cabin occupant includes: When the initial face image data cannot be obtained, the initial audio data is processed to obtain a voiceprint feature vector and an audio coding feature vector; the voiceprint feature vector and the audio coding feature vector are fused to obtain an audio feature vector; Obtaining a second mask set according to the audio feature vector, wherein the second mask set includes an activated voice mask corresponding to each cabin occupant; The audio data corresponding to each of the cabin occupants is obtained according to the second mask set.

4. The method according to claim 2, characterized in that: When the initial face image data can be obtained, it also includes: Inputting the initial face image data into a face and key point detection model to obtain a plurality of face frames and key points corresponding to each face frame; The lip image data corresponding to each face frame is obtained according to the key points corresponding to each face frame.

5. The method according to claim 4, characterized in that Inputting the audio data corresponding to each of the cabin personnel into a preset multi-mode voice activity detection algorithm network for processing, and determining whether the audio data corresponding to each of the cabin personnel belongs to a true user voice activity comprises: When the initial face image data can be obtained, a section of lip image data in the image data queue is input into the multi-mode voice activity detection algorithm network to perform feature extraction, feature enhancement and offset embedding fusion processing of ordered different frames in channel space to obtain first-stage image feature data; Inputting a segment of audio data in the audio data queue into the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain first-stage audio feature data; The first-stage image feature data and the first-stage audio feature data are concatenated and fused to obtain multi-mode output data; Determine whether the audio data corresponding to each of the cabin occupants belongs to real user voice activity based on the multi-mode output data.

6. The method according to claim 4, characterized in that Inputting the audio data corresponding to each of the cabin personnel into a preset multi-mode voice activity detection algorithm network for processing, and determining whether the audio data corresponding to each of the cabin personnel belongs to a true user voice activity comprises: When the initial face image data cannot be obtained, a segment of the audio data in the audio data queue corresponding to each of the cabin occupants is input into the multi-mode voice activity detection algorithm network for feature extraction and feature enhancement processing to obtain single-mode output data; Whether the audio data corresponding to each of the cabin occupants belongs to real user voice activity is determined according to the single-mode output data.

7. The method according to claim 5, characterized in that Before inputting the audio data corresponding to each of the cabin personnel into a preset multi-mode voice activity detection algorithm network for processing and determining whether the audio data corresponding to each of the cabin personnel belongs to a real user voice activity, the method further includes: When the initial face image data can be acquired, the lip image data corresponding to each face frame is stored in the image data queue; Storing audio data corresponding to each of the cabin occupants into the audio data queue; When the length of the audio data queue is greater than a preset threshold, selecting audio data to be processed from the audio data queue, and determining the start time and end time of the audio data to be processed; Selecting the image data to be processed from the image data queue according to the start time and end time of the audio data to be processed; When the number of frames of the image data to be processed is less than a preset length, the last image frame is used to fill the number of frames of the image data to be processed to the preset length; When the to-be-processed image data is greater than the preset length, some image frames are screened out from the to-be-processed image, and the number of frames of the screened out image frames is the preset length.

8. The method according to claim 1, characterized in that The generating of the interaction instruction based on the audio data belonging to the real user voice activity comprises: Converting audio data containing real user voice activity into text information; Determine the type of the text information, wherein the type of the text information includes: task-based strong intention, task-based weak intention, task-based unprocessed, non-task-based small talk, non-task-based dialogue, and other non-task-based; The text information is processed according to the type of the text information to generate an interaction instruction.

9. The method according to claim 8, characterized in that The processing of the text information according to the type of the text information to generate an interaction instruction includes: When the type of the text information is a task-based strong intent or a task-based weak intent, determining whether the device can process the text information; when the device can process the text information, processing the text information on the device to generate the interaction instruction; otherwise, sending the text information to the cloud, and receiving the interaction instruction generated by the cloud based on the text information; When the type of the text message is task-based unprocessed, non-task-based small talk, non-task-based dialogue, or other non-task-based types, the text message is not responded to.

10. The method according to claim 9, characterized in that Determining the type of the text information includes: Generating pinyin information corresponding to the text information; Convert the text information into a text vector, and convert the phonetic information into a phonetic vector; Encoding the text vector and the phonetic vector respectively, and concatenating and fusing the encoded text vector and phonetic vector to obtain a fusion feature; After the fusion features are subjected to feature enhancement processing, they are input into a preset fully connected network to obtain the type of the text information.

11. A vehicle-mounted voice interaction device, characterized in that: include: An acquisition module, used to acquire initial audio data; A pre-processing module, used to pre-process the initial audio data to obtain audio data corresponding to each cabin occupant; a real voice determination module, used to input the audio data corresponding to each of the cabin personnel into a preset multi-mode voice activity detection algorithm network for processing, and determine whether the audio data corresponding to each of the cabin personnel belongs to a real user voice activity; A real voice processing module is used to generate interaction instructions based on audio data belonging to real user voice activities.

12. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the in-vehicle voice interaction method according to any one of claims 1 to 10 by executing the computer instructions.

13. A vehicle, characterized in that: A computer device comprising the method of claim 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the in-vehicle voice interaction method according to any one of claims 1 to 10.

15. A computer program product, characterized in that It includes computer instructions, which are used to enable a computer to execute the in-vehicle voice interaction method according to any one of claims 1 to 10.

Images