Vehicle-mounted voice interaction method and device, equipment and storage medium

Abstract

The invention discloses a vehicle-mounted voice interaction method, device and equipment and a storage medium, and relates to the technical field of voice recognition, and the vehicle-mounted voice interaction method comprises the steps: obtaining a microphone audio signal, a vehicle machine recovery signal, a camera visual signal and a seat sensor signal; processing the microphone audio signal, the vehicle machine recovery signal, the camera visual signal and the seat sensor signal to obtain audio characteristics, visual information and seat sensor position information; inputting the audio features, the visual information and the seat sensor position information into a target voice signal separation model to obtain voice signals corresponding to the positions of the seats; and responding to the voice signal to complete vehicle-mounted voice interaction. According to the invention, the instruction of each passenger can be accurately responded, simultaneous interaction between the vehicle machine and a plurality of speakers is realized, and the accuracy of voice interaction and the user experience are improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to an in-vehicle speech interaction method, device, equipment and storage medium. Background Art

[0002] With the advancement of intelligent technology, in-vehicle voice interaction systems have gradually become an important configuration of modern cars. Users hope to use voice commands to achieve functions such as navigation, phone calls, and music playback to improve driving safety and convenience. This demand has promoted the development of in-vehicle voice interaction technology, enabling it to provide a more natural and smooth human-computer interaction experience during driving.

[0003] At present, the in-vehicle voice interaction system mainly relies on single-channel voice recognition technology, and usually performs command parsing through a central processing unit. Users can activate the voice assistant by pressing a button and complete the operation through simple commands. Although this system can meet basic interaction needs, the recognition accuracy and response speed are still limited when processing complex commands. The in-vehicle environment is complex, including music, navigation sound, air conditioning noise, tire noise, etc. These factors seriously affect the success rate of in-vehicle voice interaction. Advanced in-vehicle voice interaction systems need to support independent interaction in multiple positions such as the main driver, co-driver, and back row, so as to achieve a higher response rate for vehicle control commands. Patent CN110459234A provides a multi-step processing solution for echo cancellation, blind source separation, sound zone positioning and noise reduction through microphone array signals. However, this method has problems of inaccurate signal separation and influence between steps when dealing with high volume and complex noise in the in-vehicle environment. Patent CN117789748A uses a microphone array for echo cancellation and sound zone separation, but its method of judging the sound zone by energy is prone to errors when people in the car lean or turn their heads, and traditional algorithms are difficult to handle complex background noise. Patent CN115019810A combines a microphone array and seat sensors to determine the authority to issue voice commands, but there are still problems with misjudgment of sound zones and noise interference when multiple people use the in-car system at the same time.

[0004] Existing single-channel speech recognition technology often encounters recognition confusion when facing multiple speakers, and it is difficult to accurately distinguish different sound sources. There are problems such as serial processing steps, low signal separation accuracy, and large noise interference. In addition, when the ambient noise in the car is large or multiple commands are issued at the same time, the accuracy and response speed of the system are greatly affected. Especially when there are multiple people speaking at the same time in the car, it is difficult for the system to effectively process multiple voice inputs. Therefore, how to achieve simultaneous interaction between the car computer and multiple speakers has become an urgent problem to be solved.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The purpose of this application is to provide a vehicle-mounted voice interaction method, device, equipment and storage medium, aiming to solve the technical problem of how to achieve simultaneous interaction between a vehicle computer and multiple speakers.

[0007] To achieve the above objectives, the present application proposes a vehicle-mounted voice interaction method, the method comprising:

[0008] Obtain microphone audio signals, vehicle computer feedback signals, camera visual signals, and seat sensor signals;

[0009] Processing the microphone audio signal, the vehicle computer return signal, the camera visual signal and the seat sensor signal to obtain audio features, visual information and seat sensor position information;

[0010] Inputting the audio features, the visual information and the seat sensor position information into a target speech signal separation model to obtain speech signals corresponding to the positions of the respective seats;

[0011] Respond to the voice signal to complete the in-vehicle voice interaction.

[0012] In one embodiment, the step of processing the microphone audio signal, the vehicle machine return signal, the camera visual signal and the seat sensor signal to obtain audio features, visual information and seat sensor position information includes:

[0013] Performing short-time Fourier transform on the microphone audio signal and the vehicle-computer collected signal to obtain audio features;

[0014] Extracting visual information from the camera visual signal;

[0015] The seat sensor signal is analyzed to obtain seat sensor position information.

[0016] In one embodiment, the audio feature includes a first audio feature, a second audio feature, and a third audio feature, and the step of performing short-time Fourier transform on the microphone audio signal and the vehicle-computer return signal to obtain the audio feature includes:

[0017] Performing short-time Fourier transform on the vehicle-computer collected signal to obtain a third audio feature;

[0018] Framing the microphone audio signal to obtain audio frames;

[0019] Processing the audio frame by using a preset window function to obtain a weighted audio frame;

[0020] Processing the weighted audio frame by discrete Fourier transform to obtain a frequency domain signal;

[0021] A first audio feature and a second audio feature are obtained according to the frequency domain signal.

[0022] In one embodiment, the frequency domain signal includes amplitude information and phase information, and the step of obtaining the first audio feature and the second audio feature according to the frequency domain signal includes:

[0023] Extracting the amplitude information to obtain a first audio feature;

[0024] According to the phase information, a phase value of each frequency point is obtained;

[0025] Subtract the phase values ​​of each pair of microphones at the same frequency point to obtain the phase difference at the same frequency point;

[0026] The phase difference at the same frequency point is processed to obtain a second audio feature.

[0027] In one embodiment, the step of extracting visual information from the camera visual signal includes:

[0028] extracting facial information from the camera visual signal;

[0029] When the facial information does not exist, extracting seat information;

[0030] When the facial information exists, extracting lip movement information from the facial information;

[0031] The facial information, the lip movement information and the seat information are integrated to obtain visual information.

[0032] In one embodiment, before the step of inputting the audio feature, the visual information and the seat sensor position information into the target speech signal separation model, the method further includes:

[0033] Obtaining an initial neural network model, historical speech signals at different positions, and historical noise signals;

[0034] Mixing the historical speech signal and the historical noise signal to obtain a historical mixed signal;

[0035] The initial neural network model is trained using the historical mixed signal, the historical speech signal and the historical noise signal to obtain a target speech signal separation model.

[0036] In one embodiment, the step of responding to the voice signal to complete the in-vehicle voice interaction includes:

[0037] Parsing the voice signal to obtain a voice command and position information corresponding to the voice command;

[0038] Respond according to the voice command and the location information to complete the in-vehicle voice interaction.

[0039] In addition, to achieve the above purpose, the present application also proposes a vehicle-mounted voice interaction device, the device comprising:

[0040] The signal acquisition module is used to obtain microphone audio signals, vehicle computer feedback signals, camera visual signals and seat sensor signals;

[0041] A signal processing module, used for processing the microphone audio signal, the vehicle machine return signal, the camera visual signal and the seat sensor signal to obtain audio features, visual information and seat sensor position information;

[0042] A signal separation module, used for inputting the audio features, the visual information and the seat sensor position information into a target voice signal separation model to obtain a voice signal corresponding to the position of each seat;

[0043] The response module is used to respond to the voice signal and complete the in-vehicle voice interaction.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes an in-vehicle voice interaction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the in-vehicle voice interaction method described above.

[0045] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the in-vehicle voice interaction method described above are implemented.

[0046] One or more technical solutions proposed in this application have at least the following technical effects:

[0047] Acquire microphone audio signals, vehicle-computer acquisition signals, camera visual signals, and seat sensor signals; process the microphone audio signals, vehicle-computer acquisition signals, camera visual signals, and seat sensor signals to obtain audio features, visual information, and seat sensor position information; input the audio features, visual information, and seat sensor position information into the target voice signal separation model to obtain voice signals corresponding to the positions of each seat; respond to the voice signals to complete the in-vehicle voice interaction. First, by acquiring microphone audio signals, vehicle-computer acquisition signals, camera visual signals, and seat sensor signals, the system comprehensively captures the sound, visual information, and passenger seat position data in the vehicle, and establishes a multi-dimensional in-vehicle environment data foundation. The acquisition of these signals provides a complete information source for subsequent processing. Next, these signals are processed, including extracting audio features, analyzing vehicle-computer acquisition signals to remove interference, extracting visual information from the camera, and parsing seat sensor data. This step of processing improves the accuracy and practicality of the data and provides clear input for subsequent voice separation. Subsequently, the processed audio features, visual information, and seat position data are input into the trained target speech signal separation model, which uses the comprehensive information to separate the independent speech signals of each seat position, thereby effectively identifying and distinguishing the speech sources in the car. Finally, the separated speech signals are parsed, specific voice commands are identified, and corresponding operations are performed according to the instructions of each seat position, such as adjusting the volume or setting the navigation. Through this series of steps, the vehicle system can accurately respond to the instructions of each passenger, realize the simultaneous interaction between the vehicle computer and multiple speakers, and improve the accuracy of voice interaction and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 A flowchart of the first embodiment of the vehicle voice interaction method of the present application is provided;

[0051] Figure 2 A flowchart of the second embodiment of the vehicle voice interaction method of the present application is provided;

[0052] Figure 3A schematic diagram of a signal feature extraction and processing architecture of an in-vehicle voice interaction method provided in Embodiment 2 of the present application;

[0053] Figure 4 This is a schematic diagram of the module structure of the vehicle-mounted voice interaction device according to an embodiment of the present application;

[0054] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the in-vehicle voice interaction method in the embodiment of the present application.

[0055] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0057] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0058] With the development of intelligent technology, in-vehicle voice interaction systems have become a key configuration to improve driving safety and convenience, allowing users to control functions such as navigation, phone calls and music playback through voice commands. Although the current system is mainly based on single-channel speech recognition technology and parses commands through the central processing unit to meet basic interaction needs, it still has limitations in processing complex commands, distinguishing the voices of multiple speakers, and accuracy and response speed in noisy environments. Especially when multiple people are speaking at the same time, existing systems have difficulty in effectively managing and recognizing multiple voice inputs, which limits the performance and user experience of in-vehicle voice interaction technology.

[0059] The main solution of the embodiment of the present application is: the system first establishes a comprehensive vehicle environment data foundation by acquiring microphone audio signals, vehicle computer feedback signals, camera visual signals and seat sensor signals. Subsequently, these signals are processed to extract audio features, remove interference, analyze visual information and parse seat data to improve the accuracy and practicality of the data. The processed information is input into the trained voice signal separation model to accurately separate the voice signals of each seat position and identify specific voice commands. Finally, corresponding operations are performed according to the instructions of each seat, such as adjusting the volume or setting the navigation, to improve the accuracy and functionality of the in-vehicle voice interaction.

[0060] It should be noted that the execution subject of the embodiment of the present application may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a vehicle system, a voice interaction system, etc. that can realize the above functions. The following takes the voice interaction system as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the embodiment of the present application provides a vehicle-mounted voice interaction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the in-vehicle voice interaction method of the present application.

[0062] In this embodiment, the vehicle-mounted voice interaction method includes steps S10 to S40:

[0063] Step S10, obtaining microphone audio signals, vehicle computer feedback signals, camera visual signals and seat sensor signals;

[0064] It should be noted that the microphone audio signal refers to the audio data captured by multiple microphone arrays in the car. These data include the speaking sounds and background noise in the car, which are used to extract voice features and perform source separation and positioning. The car-computer feedback signal refers to the audio signals such as music and navigation sounds played by the car-computer system that are collected by the car-mounted system. These signals help identify and remove audio interference emitted by the car-computer to ensure the accuracy of voice interaction. The camera visual signal includes video data captured by the camera in the car, which is mainly used to extract the facial information or seat position of the passenger, and to assist in locating the position and direction of the speaker. The seat sensor signal refers to the data captured by the pressure sensor inside the seat. These data provide the seat occupancy status of the passenger, assist in confirming the speaker's position and further improve the separation accuracy of the voice signal.

[0065] It can be understood that, first, audio signals are obtained through the microphone array in the car. These signals record the ambient noise in the car and the voice of the speaker for subsequent sound source separation and positioning processing. Secondly, the car-machine recollection signal refers to the music played by the car-machine system, navigation sound, and other sounds collected by the car-mounted system. These signals help to identify and remove the background noise generated by the car-machine, thereby improving the clarity of voice commands. Next, the camera visual signal refers to the video image captured by the camera in the car. These images are used to extract the facial features or seat status of the passengers to help determine the specific position and direction of the speaker. Finally, the seat sensor signal comes from the pressure sensor inside the seat. These signals provide the occupancy status of each seat, which is used to further confirm the position of the speaker and assist in the precise separation of audio signals. By comprehensively processing these signals, accurate recognition and response to voice commands in the car can be achieved.

[0066] Step S20, processing the microphone audio signal, the vehicle machine return signal, the camera visual signal and the seat sensor signal to obtain audio features, visual information and seat sensor position information;

[0067] It should be noted that audio features refer to key information extracted from microphone audio signals. These features help identify and separate speech sounds and background noise in the car, and support sound source localization and separation. Visual information refers to data extracted from camera visual signals, such as seat status. This information is used to assist in identifying the speaker's position and movements, such as seat occupancy, thereby improving the accuracy of voice signal separation. Seat sensor position information refers to data obtained from seat sensor signals, providing the occupancy status and position of each seat. This information is used to confirm the specific position of passengers and further optimize the positioning and processing of audio signals.

[0068] It is understandable that the microphone audio signal is processed and audio features are extracted through methods such as Fourier transform. These features include the spectrum and direction information of the sound, which are used to separate the speaking voice and background noise in the car. The car computer's collected signal is analyzed to extract audio features to help remove the interference sound played by the car computer. The camera visual signal is processed to extract visual information, such as the passenger's facial features or seat status, which is used to determine the speaker's position and movement. The seat sensor signal is analyzed to obtain seat position information, which provides the occupancy status of each seat and helps to accurately confirm the specific position of the speaker. Combining these processing results, the sound source in the car can be accurately identified and located.

[0069] Step S30, inputting the audio features, the visual information and the seat sensor position information into a target voice signal separation model to obtain a voice signal corresponding to the position of each seat;

[0070] It should be noted that the target speech signal separation model is a model trained by a neural network (such as DNN, CNN, LSTM, CRN, Transformer, etc.), which is designed to process and integrate audio features, visual information and seat sensor position information from different sensors. The model can perform complex speech signal processing based on these input information and separate independent speech signals from each seat position in the car. The speech signal refers to the sound signal extracted from the audio data in the car, which has been separated and processed by the model. These signals correspond to the voice of a specific speaker at each seat position in the car, and are used to accurately identify and respond to voice commands at each position.

[0071] It can be understood that the processed audio features, visual information, and seat sensor position information (such as the occupancy status of each seat) are input into the trained target speech signal separation model. The model uses these comprehensive input data and performs complex analysis and processing to distinguish the independent speech signals of each seat position in the car. In this way, the model can accurately separate the voices of multiple speakers in the car and provide corresponding clear speech signals for each seat position, thereby supporting accurate speech recognition and response.

[0072] As an example, before the step of inputting the audio features, the visual information and the seat sensor position information into the target speech signal separation model, it also includes: obtaining an initial neural network model, historical speech signals at different positions and historical noise signals; mixing the historical speech signals and the historical noise signals to obtain a historical mixed signal; and using the historical mixed signal, the historical speech signal and the historical noise signal to train the initial neural network model to obtain a target speech signal separation model.

[0073] The initial neural network model refers to a preliminary constructed neural network framework for processing and separating speech signals. The model has not been specifically optimized for the in-vehicle environment before training. The historical voice signal refers to previously recorded data collected from the vehicle that represents the speech of speakers in different seat positions. These data are used to train the model on how to recognize and separate speech signals from different sources. The historical noise signal refers to various background noise data recorded in the in-vehicle environment, such as music, navigation sound, air conditioning noise, etc. These data help the model learn how to remove interference from mixed signals. The historical mixed signal refers to the signal obtained by mixing the historical voice signal and the historical noise signal according to the characteristics of the actual in-vehicle environment. It is used to train the model to simulate the actual speech and noise mixing in use and improve the model's separation ability in the real environment.

[0074] First, an initial neural network model is obtained. This model is a basic model that has not been optimized for a specific application environment. Then, historical speech signals and historical noise signals at different locations are collected. These data are used to simulate the speech and noise conditions in the actual vehicle environment. The historical speech signal is mixed with the historical noise signal to generate a historical mixed signal, which simulates the actual speech and noise mixing state in the vehicle environment. Then, the historical mixed signal, historical speech signal, and historical noise signal are used to train the initial neural network model, thereby optimizing the model to accurately separate the speech signals in the vehicle. The trained model is the target speech signal separation model, which can accurately separate the speech signals of each seat position in actual applications based on the input audio features, visual information, and seat sensor position information.

[0075] Step S40, responding to the voice signal to complete the in-vehicle voice interaction.

[0076] It should be noted that response refers to the system performing appropriate operations or feedback based on the voice signals from each seat position obtained through processing and separation. This includes identifying and parsing voice commands issued from each position, and then executing corresponding vehicle control commands (such as adjusting volume, navigation instructions, air conditioning settings, etc.), or providing appropriate voice feedback (such as confirming information or requesting further instructions). The response process ensures that the in-vehicle voice system can provide accurate and timely interactions based on the positions and instructions of different passengers, thereby improving the in-vehicle user experience.

[0077] It is understandable that the system first parses and understands the independent voice signals of each seat position obtained from the target voice signal separation model, and then the system performs corresponding operations based on the recognized voice signals, such as adjusting the in-vehicle settings (such as volume, navigation path, air conditioning temperature, etc.), or providing feedback information based on the voice signals. The entire process includes the recognition, processing and execution of voice commands, ensuring that passengers in each seat position can effectively interact with the in-vehicle system, achieve precise in-vehicle function control and enhance user experience.

[0078] As an example, the steps of responding to the voice signal and completing the in-vehicle voice interaction include: parsing the voice signal to obtain the voice command and the location information corresponding to the voice command; responding according to the voice command and the location information to complete the in-vehicle voice interaction.

[0079] Voice commands refer to specific instructions or requests identified from processed and separated voice signals, such as "adjust the air conditioning temperature" or "navigate home", which reflect the actions that the passenger wants the in-vehicle system to perform. Position information refers to the specific seat position data associated with each voice command, such as the main driver, the front passenger, or the left rear seat, which helps the system determine the position of the passenger issuing the command, thereby ensuring that the command is executed correctly to the corresponding position.

[0080] In this process step, the system first parses the processed voice signal to identify and extract the specific voice commands and the location information corresponding to each command. Then, the system performs corresponding operations based on the extracted voice commands and location information, and correctly applies the commands to the specified location. For example, if the main driver's position issues an "adjust the air conditioning temperature" command, the system will adjust the air conditioning settings in the main driver's area accordingly. In this way, the system can effectively complete in-vehicle voice interaction, achieve precise function control and enhance user experience.

[0081] This embodiment provides a method for in-vehicle voice interaction, which obtains microphone audio signals, vehicle-machine acquisition signals, camera visual signals and seat sensor signals; processes the microphone audio signals, vehicle-machine acquisition signals, camera visual signals and seat sensor signals to obtain audio features, visual information and seat sensor position information; inputs the audio features, visual information and seat sensor position information into the target voice signal separation model to obtain voice signals corresponding to the positions of each seat; responds to the voice signals to complete the in-vehicle voice interaction. First, by obtaining microphone audio signals, vehicle-machine acquisition signals, camera visual signals and seat sensor signals, the system comprehensively captures the sound, visual information and passenger seat position data in the vehicle, and establishes a multi-dimensional vehicle-mounted environment data foundation. The acquisition of these signals provides a complete information source for subsequent processing. Next, these signals are processed, including extracting audio features, analyzing vehicle-machine acquisition signals to remove interference, extracting visual information from the camera, and parsing seat sensor data. This step of processing improves the accuracy and practicality of the data and provides clear input for subsequent voice separation. Subsequently, the processed audio features, visual information, and seat position data are input into the trained target speech signal separation model, which uses the comprehensive information to separate the independent speech signals of each seat position, thereby effectively identifying and distinguishing the speech sources in the car. Finally, the separated speech signals are parsed, specific voice commands are identified, and corresponding operations are performed according to the instructions of each seat position, such as adjusting the volume or setting the navigation. Through this series of steps, the vehicle system can accurately respond to the instructions of each passenger, realize the simultaneous interaction between the vehicle computer and multiple speakers, and improve the accuracy of voice interaction and user experience.

[0082] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the in-vehicle voice interaction method of the present application. Step S20 of the in-vehicle voice interaction method includes steps S21 to S23:

[0083] Step S21, performing short-time Fourier transform on the microphone audio signal and the vehicle-mounted audio signal to obtain audio features;

[0084] It should be noted that the Short-Time Fourier Transform (STFT) is a signal processing technology used to analyze the spectral characteristics of non-stationary signals. It divides the signal into multiple short-term segments and applies the Fourier transform to each segment to obtain the frequency domain information of the signal in each time window. This transform combines time and frequency information, making it possible to observe the time changes and frequency characteristics of the signal at the same time. In vehicle-mounted systems, STFT is used to extract the spectral characteristics of audio signals to help identify and separate different sound sources, such as the echo signal played by the car computer and the ambient noise in the car.

[0085] It can be understood that these signals are first divided into several short time windows, and the signal data in each window is Fourier transformed. Through this process, the system converts the signal from the time domain to the frequency domain, thereby obtaining the spectral characteristics in each time window. These spectral characteristics reflect the intensity and changes of the signal at different frequencies, making it possible to analyze and distinguish the various components in the audio signal in detail, such as the background noise in the car and the echo sound played by the car computer. The frequency domain information provided by the short-time Fourier transform helps to further process and separate different sound sources, thereby improving the accuracy of speech recognition and processing.

[0086] As an example, the audio features include a first audio feature, a second audio feature and a third audio feature, and the step of performing short-time Fourier transform on the microphone audio signal and the vehicle-computer re-collected signal to obtain the audio features includes: performing short-time Fourier transform on the vehicle-computer re-collected signal to obtain a third audio feature; framing the microphone audio signal to obtain an audio frame; processing the audio frame through a preset window function to obtain a weighted audio frame; processing the weighted audio frame through a discrete Fourier transform to obtain a frequency domain signal; and obtaining the first audio feature and the second audio feature according to the frequency domain signal.

[0087] The first audio feature refers to the short-time Fourier spectrum, which is a spectrogram obtained by short-time Fourier transform (STFT). It shows the frequency distribution and intensity of the signal in different time windows. Short-time Fourier transform converts the time domain signal into the frequency domain to obtain a spectrogram in each time window. The spectrogram shows the energy distribution of the signal on each frequency component, which helps to identify the frequency characteristics of the sound, such as the fundamental frequency and harmonic components of speech. The first audio feature is used to analyze the frequency components of the microphone audio signal, so as to help identify and separate the sounds from different sources in the car, such as the voice of the passengers and the air conditioning noise, tire noise, road noise, wind noise, etc. in the car. The second audio feature refers to the short-time Fourier phase difference, which is the phase difference between the signals obtained by short-time Fourier transform between different microphones. The phase difference represents the time delay or phase shift of the signal sampled at different positions, which is very critical for locating the sound source, because the signals captured by microphones at different positions are different in time, thus reflecting the spatial position of the sound source. By analyzing the phase difference, it can help determine the location of the sound source in different sound zones in the car, and assist in accurately separating and identifying the speech signals at different positions. The third audio feature is the spectrum information obtained by performing short-time Fourier transform on the vehicle-machine recollection signal. The vehicle-machine recollection signal refers to the signal of music or navigation sound played by the vehicle-machine. The short-time Fourier transform converts it into frequency domain information to obtain the intensity distribution of the signal at different frequencies, which helps to remove the interference of the vehicle-machine audio from the noise in the car. The third audio feature is used to identify and separate the audio signals played by the vehicle-machine to avoid the interference of these signals on voice interaction and ensure that the system can accurately process and respond to the voice instructions of passengers. An audio frame is a small segment of a fixed length that is divided into a continuous audio signal according to time, and each segment contains audio data within a certain time. When processing, the audio signal is divided into multiple small segments, each of which is called an audio frame. This is done so that a short-time Fourier transform can be performed on each small segment to analyze the spectral characteristics of the signal within the time period. The audio frame enables the short-time Fourier transform to perform frequency domain analysis within a local time range, thereby extracting the audio features within each time window. The preset window function is a mathematical function used to weight each audio frame to reduce spectrum leakage and boundary effects. At the boundary of the audio frame, the sudden change of the signal will cause spectrum leakage. The preset window function (such as Hanning window or Hamming window) is weighted on each frame, so that the signal changes more smoothly at the boundary of the window function, thereby improving the accuracy of frequency domain analysis. By applying the window function to the audio frame, the quality of spectrum analysis of the short-time Fourier transform can be improved, the error in the analysis process can be reduced, and the obtained audio features can be ensured to be more accurate. The weighted audio frame is an audio frame processed by the preset window function. The window function weights the data of each frame to reduce the boundary effect. After the audio frame is subjected to the window function, the boundary area of ​​the signal value is smoothed, reducing the spectrum leakage phenomenon. In this way, the weighted audio frame can better reflect the real signal characteristics during frequency domain analysis.The weighted audio frame improves the accuracy of the short-time Fourier transform by reducing the boundary effect, making the extracted frequency domain signal more accurate, thereby improving the recognition and separation of speech signals. Discrete Fourier Transform (DFT) is a mathematical method used to convert a discrete time signal into its frequency domain representation and calculate the amplitude and phase of the signal at each frequency. DFT processes the weighted audio frame, converts it from the time domain to the frequency domain, generates a frequency domain signal, reveals the frequency components of the signal and their intensity, and the result of DFT is a spectrogram, which describes the energy distribution of the signal at different frequencies. DFT is used to obtain the frequency domain characteristics of the audio signal, providing data support for extracting the first audio feature (spectrum) and the second audio feature (phase difference), thereby effectively identifying and separating the sound sources in the car. The frequency domain signal is a signal representation obtained by discrete Fourier transform, showing the amplitude and phase of the signal at different frequencies. The frequency domain signal reflects the energy distribution and phase relationship of the signal at different frequencies, making it possible to analyze the spectral characteristics of the signal, which helps to identify and separate different components in the audio signal. The frequency domain signal provides detailed spectral information for calculating the first and second audio features, thereby helping to accurately extract and process different sound signals in the car and improve the accuracy of voice interaction.

[0088] First, the short-time Fourier transform is performed on the vehicle-mounted signal to obtain the third audio feature, which helps identify the audio components played by the vehicle-mounted signal, such as music and navigation sound. Next, the microphone audio signal is divided into frames, each frame represents the audio data of a short period of time, so that the local characteristics of the signal can be analyzed. A preset window function is applied to these audio frames to obtain weighted audio frames, with the purpose of reducing spectrum leakage and boundary effects and improving analysis accuracy. Then, the weighted audio frames are processed using a discrete Fourier transform to obtain a frequency domain signal. This conversion converts the time domain signal into the frequency domain to reveal the components of different frequencies. Finally, the first audio feature (short-time Fourier spectrum) is extracted from the frequency domain signal to show the intensity distribution of the signal at each frequency, and the second audio feature (short-time Fourier phase difference) provides phase information between different microphones. These features are used together to accurately identify and separate the voice signal in the car.

[0089] As an example, the frequency domain signal includes amplitude information and phase information, and the step of obtaining the first audio feature and the second audio feature based on the frequency domain signal includes: extracting the amplitude information to obtain the first audio feature; obtaining the phase value of each frequency point based on the phase information; subtracting the phase values ​​of each pair of microphones at the same frequency point to obtain the phase difference at the same frequency point; processing the phase difference at the same frequency point to obtain the second audio feature.

[0090] Amplitude information is a numerical value describing the intensity of each frequency component in the frequency domain signal, reflecting the energy of the signal at a specific frequency. Amplitude information is obtained by taking the modulus of the discrete Fourier transform result of the signal, usually expressed as the amplitude part of the spectrum diagram, which can help identify and distinguish the main frequency components in the audio signal. Phase information is a numerical value describing the phase angle of each frequency component in the frequency domain signal, indicating the relative time position of the signal at that frequency. Phase information is obtained by taking the angle of the discrete Fourier transform result of the signal, which helps to understand the waveform characteristics of the signal and the relative timing relationship between different audio sources. Phase value refers to the specific angle value provided by the phase information at a specific frequency point. The phase value of each frequency point represents the phase angle of the frequency component, which can be used to analyze the waveform changes of the audio signal. The phase difference at the same frequency point refers to the difference between the phase values ​​measured at the same frequency point in the frequency domain signals of two different microphones. By calculating the difference in the phase values ​​at the same frequency point, the phase difference at this frequency point can be obtained, reflecting the relative position relationship of the sound source in space and its impact on different microphones.

[0091] First, the amplitude information is extracted from the frequency domain signal. This part of the data shows the intensity distribution of the signal at each frequency point, and then the first audio feature (short-time Fourier spectrum) is obtained to identify the frequency components of the signal. Secondly, based on the phase information, the phase values ​​of each frequency point are calculated. These values ​​reflect the phase angle of the signal at different frequency points. Then, the phase values ​​measured by each pair of microphones at the same frequency point are subtracted to obtain the phase difference at the same frequency point. This difference reflects the difference in phase between the sounds received by the two microphones, which can help analyze the spatial position of the sound source. Finally, these phase differences are further processed to obtain the second audio feature (short-time Fourier phase difference), which is used to accurately distinguish different sound sources and their locations, and improve the separation and recognition accuracy of speech signals.

[0092] Step S22, extracting visual information from the camera visual signal;

[0093] It should be noted that visual information refers to feature data extracted from the image data captured by the camera for analysis. These feature data can include object recognition, scene analysis, and detection of dynamic changes in the image. By processing these image data, the spatial layout and possible interference sources in the car can be identified, and combined with audio signals for comprehensive analysis. This can more accurately locate the source of the sound and improve the accuracy and reliability of the in-vehicle voice interaction system.

[0094] It is understood that the step of extracting visual information from the camera visual signal includes analyzing the image data captured by the camera to extract key information that helps understand the in-vehicle environment and passenger positions. This visual information may include the spatial layout of the vehicle, the location of objects, and other dynamic changes. By processing this image data, key features in the vehicle, such as the structure of the vehicle compartment and environmental changes, can be identified, which helps to assist in the analysis of audio signals. Combining visual information with audio signals can more accurately determine the source and impact of the sound, improving the overall performance and accuracy of the in-vehicle voice interaction system.

[0095] As an example, the step of extracting visual information from the camera visual signal includes: extracting facial information from the camera visual signal; when the facial information does not exist, extracting seat information; when the facial information exists, extracting lip movement information from the facial information; integrating the facial information, the lip movement information and the seat information to obtain visual information.

[0096] Facial information refers to the facial features of passengers identified and extracted from the image data captured by the camera. This includes the position, shape, expression, etc. of the face, which is used to identify the identity or location of the passenger. Seat information refers to the position information and configuration of the seats in the car extracted from the camera image, which includes the existence, position and possible adjustment of the seats, which is used to determine the specific seat position of the passenger. Lip movement information refers to the passenger lip movement data extracted from the facial information, which includes the movement and shape changes of the lips, which is used to assist in identifying the speaker's voice, especially to help enhance the accuracy of speech recognition in noisy environments.

[0097] First, facial information is extracted from the image to identify the identity and location of the passenger. If no facial information is detected in the image, seat information is extracted to determine which seat positions are empty, thereby eliminating interference from these positions. If facial information exists in the image, lip movement information, i.e., the passenger's lip movement data, is further extracted from the facial information to assist in speech recognition, especially to improve recognition accuracy in noisy environments. Finally, the extracted facial information, lip movement information, and seat information are integrated to form comprehensive visual information, which helps to more accurately analyze and process in-vehicle voice signals.

[0098] Step S23, analyzing the seat sensor signal to obtain seat sensor position information.

[0099] It should be noted that the seat sensor position information refers to specific data related to the seats in the car, including: Seat position: The sensor can detect the exact position of the seat in the car, such as the front row, back row, left or right side, which helps determine the specific seat of the passenger. Seat occupancy status: The sensor can determine whether the seat is occupied, that is, whether there is a passenger sitting on the seat, or whether the seat is free, including detected pressure or weight changes. Seat adjustment: The sensor can also detect whether the seat is adjusted, such as tilting, moving forward and backward, etc., to provide the current status of the seat.

[0100] It is understood that the step of parsing the seat sensor signal to obtain the seat sensor position information includes first reading the data captured by the seat sensor, which includes the weight distribution, pressure change or position adjustment information of the seat. By analyzing these signals, the system can determine the specific position of each seat, including its coordinates and direction in the car. At the same time, these sensor data can also show the occupancy status of the seat, that is, whether there are passengers sitting on it. If the seat is not occupied, the sensor will send a corresponding signal to mark these seats as empty. Combining this information, the system can accurately identify the specific position and status of the seats in the car, providing necessary data support for further voice signal processing and passenger identification.

[0101] In this embodiment, the microphone audio signal and the vehicle-mounted return signal are subjected to short-time Fourier transform to obtain audio features; the vehicle-mounted return signal is subjected to the short-time Fourier transform to obtain the third audio feature; visual information is extracted from the camera visual signal; and the seat sensor signal is analyzed to obtain the seat sensor position information. First, by performing short-time Fourier transform on the vehicle-mounted microphone audio signal and the vehicle-mounted return signal, these signals can be converted from the time domain to the frequency domain, and audio features such as amplitude information and phase information can be extracted. These audio features help to effectively distinguish and identify different sound sources in the car, such as voice, music and noise, thereby improving the accuracy of voice recognition. Next, the third audio feature obtained by short-time Fourier transform processing of the vehicle-mounted return signal further helps to identify and suppress the background sound played by the vehicle-mounted signal, and optimizes the recognition process of voice commands. At the same time, the visual information extracted from the visual signal captured by the camera can be used to identify the position and status of passengers in the car, so that the system can provide more accurate personalized services. In addition, the seat sensor position information obtained by analyzing the seat sensor signal can determine the occupancy status and position of each seat, providing the system with key information about the distribution of passengers. These steps work together to ensure that the in-vehicle voice system can accurately recognize and respond to voice commands in a complex in-vehicle environment, thereby improving the overall effect and user experience of in-vehicle voice interaction.

[0102] For example, in order to help understand the implementation process of the vehicle voice interaction method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 3 , Figure 3 A schematic diagram of the signal feature extraction and processing architecture of the vehicle voice interaction method provided in the second embodiment of the present application, specifically:

[0103] The figure shows the signal processing flow in an in-vehicle voice interaction system. The system first obtains audio signals through the in-vehicle microphone array, and collects vehicle-mounted feedback signals, camera visual signals, and seat sensor signals at the same time. These signals are sent to the signal processing module for short-time Fourier transform to extract audio features, including spectrum information (first audio feature, i.e., audio feature 1 in the figure) and phase difference information (second audio feature, i.e., audio feature 2 in the figure). In addition, the vehicle-mounted feedback signal is short-time Fourier transformed to obtain the third audio feature (i.e., audio feature 3 in the figure) to distinguish and remove audio interference played by the vehicle-mounted computer. The camera visual signal is processed to extract visual information, such as facial and lip movement information, while the seat sensor signal is parsed to obtain seat sensor position information. All of this feature information is then input into a target speech signal separation model, which is a trained neural network used to separate and recognize speech signals from different seat positions (driver, co-driver, behind the driver, behind the co-driver, etc.). Finally, the separated voice signal is sent to the response module, which performs corresponding vehicle control operations according to the recognized voice command and location information, such as adjusting the volume or setting the navigation, thereby achieving accurate in-vehicle voice interaction. This process not only improves the accuracy of voice recognition, but also enhances the user experience, especially in complex in-vehicle environments with multiple passengers and high noise.

[0104] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the in-vehicle voice interaction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0105] This application also provides a vehicle-mounted voice interaction device, please refer to Figure 4 , the vehicle-mounted voice interaction device comprises:

[0106] The signal acquisition module 10 is used to acquire microphone audio signals, vehicle computer feedback signals, camera visual signals and seat sensor signals;

[0107] The signal processing module 20 is used to process the microphone audio signal, the vehicle machine return signal, the camera visual signal and the seat sensor signal to obtain audio features, visual information and seat sensor position information;

[0108] A signal separation module 30, configured to input the audio features, the visual information and the seat sensor position information into a target voice signal separation model to obtain a voice signal corresponding to the position of each seat;

[0109] The response module 40 is used to respond to the voice signal to complete the in-vehicle voice interaction.

[0110] In one embodiment, the audio features include a first audio feature, a second audio feature and a third audio feature. The signal processing module 20 is further used to perform short-time Fourier transform on the microphone audio signal and the vehicle-computer feedback signal to obtain the audio features; perform short-time Fourier transform on the vehicle-computer feedback signal to obtain the third audio feature; extract visual information from the camera visual signal; and analyze the seat sensor signal to obtain seat sensor position information.

[0111] In one embodiment, the signal processing module 20 is also used to perform short-time Fourier transform on the vehicle-computer feedback signal to obtain a third audio feature; frame the microphone audio signal to obtain an audio frame; process the audio frame through a preset window function to obtain a weighted audio frame; process the weighted audio frame through a discrete Fourier transform to obtain a frequency domain signal; and obtain the first audio feature and the second audio feature based on the frequency domain signal.

[0112] In one embodiment, the signal processing module 20 is further used to extract the amplitude information to obtain the first audio feature; obtain the phase value of each frequency point according to the phase information; subtract the phase values ​​of each pair of microphones at the same frequency point to obtain the phase difference at the same frequency point; and process the phase difference at the same frequency point to obtain the second audio feature.

[0113] In one embodiment, the signal processing module 20 is also used to extract facial information from the camera visual signal; when the facial information does not exist, extract seat information; when the facial information exists, extract lip movement information from the facial information; and integrate the facial information, the lip movement information and the seat information to obtain visual information.

[0114] In one embodiment, the signal separation module 30 is also used to obtain an initial neural network model, historical speech signals at different locations, and historical noise signals; mix the historical speech signals and the historical noise signals to obtain a historical mixed signal; and use the historical mixed signal, the historical speech signal, and the historical noise signal to train the initial neural network model to obtain a target speech signal separation model.

[0115] In one embodiment, the response module 40 is further used to parse the voice signal to obtain a voice command and location information corresponding to the voice command; respond according to the voice command and the location information to complete the in-vehicle voice interaction.

[0116] The vehicle-mounted voice interaction device provided by the present application adopts the vehicle-mounted voice interaction method in the above embodiment, which can solve the technical problem of how to realize the simultaneous interaction between the vehicle machine and multiple speakers. Compared with the prior art, the beneficial effects of the vehicle-mounted voice interaction device provided by the present application are the same as the beneficial effects of the vehicle-mounted voice interaction method provided by the above embodiment, and the other technical features in the vehicle-mounted voice interaction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0117] The present application provides an in-vehicle voice interaction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the in-vehicle voice interaction method in the above-mentioned embodiment one.

[0118] Reference below Figure 5 , which shows a schematic diagram of the structure of a vehicle-mounted voice interaction device suitable for implementing an embodiment of the present application. The vehicle-mounted voice interaction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The in-vehicle voice interaction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0119] like Figure 5As shown, the vehicle-mounted voice interaction device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the vehicle-mounted voice interaction device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the vehicle-mounted voice interaction device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a vehicle-mounted voice interaction device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0120] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0121] The in-vehicle voice interaction device provided by the present application adopts the in-vehicle voice interaction method in the above embodiment, which can solve the technical problem of how to realize the simultaneous interaction between the vehicle and multiple speakers. Compared with the prior art, the beneficial effects of the in-vehicle voice interaction device provided by the present application are the same as the beneficial effects of the in-vehicle voice interaction method provided by the above embodiment, and the other technical features in the in-vehicle voice interaction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0122] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0124] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the in-vehicle voice interaction method in the above-mentioned embodiment.

[0125] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0126] The above-mentioned computer-readable storage medium may be included in the vehicle-mounted voice interaction device; or it may exist independently without being assembled into the vehicle-mounted voice interaction device.

[0127] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the in-vehicle voice interaction device, the in-vehicle voice interaction device: obtains microphone audio signals, vehicle computer feedback signals, camera visual signals and seat sensor signals; processes the microphone audio signals, the vehicle computer feedback signals, the camera visual signals and the seat sensor signals to obtain audio features, visual information and seat sensor position information; inputs the audio features, the visual information and the seat sensor position information into the target voice signal separation model to obtain voice signals corresponding to the positions of each seat; responds to the voice signals to complete the in-vehicle voice interaction.

[0128] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0131] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned vehicle-mounted voice interaction method, and can solve the technical problem of how to achieve simultaneous interaction between a vehicle machine and multiple speakers. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the vehicle-mounted voice interaction method provided in the above-mentioned embodiment, and will not be repeated here.

[0132] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned in-vehicle voice interaction method when executed by a processor.

[0133] The computer program product provided by this application can solve the technical problem of how to achieve simultaneous interaction between a vehicle computer and multiple speakers. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the vehicle-mounted voice interaction method provided by the above embodiment, which will not be repeated here.

[0134] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A vehicle-mounted voice interaction method, It is characterized in that The method comprises: Obtain microphone audio signals, vehicle computer feedback signals, camera visual signals, and seat sensor signals; Processing the microphone audio signal, the vehicle computer return signal, the camera visual signal and the seat sensor signal to obtain audio features, visual information and seat sensor position information; Inputting the audio features, the visual information and the seat sensor position information into a target speech signal separation model to obtain speech signals corresponding to the positions of the respective seats; Respond to the voice signal to complete the in-vehicle voice interaction.

2. The method according to claim 1, It is characterized in that The step of processing the microphone audio signal, the vehicle computer return signal, the camera visual signal and the seat sensor signal to obtain audio features, visual information and seat sensor position information includes: Performing short-time Fourier transform on the microphone audio signal and the vehicle-computer collected signal to obtain audio features; Extracting visual information from the camera visual signal; The seat sensor signal is analyzed to obtain seat sensor position information.

3. The method according to claim 2, It is characterized in that The audio features include a first audio feature, a second audio feature, and a third audio feature. The step of performing short-time Fourier transform on the microphone audio signal and the vehicle-computer return signal to obtain the audio features includes: Performing short-time Fourier transform on the vehicle-computer collected signal to obtain a third audio feature; Framing the microphone audio signal to obtain audio frames; Processing the audio frame by using a preset window function to obtain a weighted audio frame; Processing the weighted audio frame by discrete Fourier transform to obtain a frequency domain signal; A first audio feature and a second audio feature are obtained according to the frequency domain signal.

4. The method according to claim 3, It is characterized in that The frequency domain signal includes amplitude information and phase information, and the step of obtaining the first audio feature and the second audio feature according to the frequency domain signal includes: Extracting the amplitude information to obtain a first audio feature; According to the phase information, a phase value of each frequency point is obtained; Subtract the phase values ​​of each pair of microphones at the same frequency point to obtain the phase difference at the same frequency point; The phase difference at the same frequency point is processed to obtain a second audio feature.

5. The method according to claim 2, It is characterized in that The step of extracting visual information from the camera visual signal comprises: extracting facial information from the camera visual signal; When the facial information does not exist, extracting seat information; When the facial information exists, extracting lip movement information from the facial information; The facial information, the lip movement information and the seat information are integrated to obtain visual information.

6. The method according to claim 1, It is characterized in that Before the step of inputting the audio feature, the visual information and the seat sensor position information into the target speech signal separation model, the method further includes: Obtaining an initial neural network model, historical speech signals at different positions, and historical noise signals; Mixing the historical speech signal and the historical noise signal to obtain a historical mixed signal; The initial neural network model is trained using the historical mixed signal, the historical speech signal and the historical noise signal to obtain a target speech signal separation model.

7. The method according to any one of claims 1 to 6, It is characterized in that The step of responding to the voice signal and completing the in-vehicle voice interaction includes: Parsing the voice signal to obtain a voice command and position information corresponding to the voice command; Respond according to the voice command and the location information to complete the in-vehicle voice interaction.

8. A vehicle-mounted voice interaction device, It is characterized in that The device comprises: The signal acquisition module is used to obtain microphone audio signals, vehicle computer feedback signals, camera visual signals and seat sensor signals; A signal processing module, used for processing the microphone audio signal, the vehicle machine return signal, the camera visual signal and the seat sensor signal to obtain audio features, visual information and seat sensor position information; A signal separation module, used for inputting the audio features, the visual information and the seat sensor position information into a target voice signal separation model to obtain a voice signal corresponding to the position of each seat; The response module is used to respond to the voice signal and complete the in-vehicle voice interaction.

9. A vehicle-mounted voice interaction device, It is characterized in that The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the in-vehicle voice interaction method as described in any one of claims 1 to 7.

10. A storage medium, It is characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the in-vehicle voice interaction method as described in any one of claims 1 to 7 are implemented.

Images