Vehicle control method, device and equipment based on voice recognition and vehicle

Abstract

The invention discloses a vehicle control method, device and equipment based on voice recognition and a vehicle, and relates to the technical field of vehicles, and the vehicle control method based on voice recognition comprises the steps: obtaining voice information and lip video image information of a driver in the vehicle; and then, information fusion is carried out based on the voice information and the lip video image information to obtain a fusion result, the fusion result comprises a feature-level fusion result and a decision-level fusion result, then, the intention of a driver is determined through the feature-level fusion result and the decision-level fusion result, and the vehicle is controlled to execute corresponding operation according to the intention of the driver. According to the vehicle control method based on voice recognition provided by the embodiment of the invention, information fusion is performed on the basis of the voice information and the lip video image information, and the vehicle is further controlled according to the feature level fusion result and the decision level fusion result, so that the accuracy of voice recognition can be improved, and the user experience is improved. And therefore, the vehicle can be effectively controlled.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a vehicle control method, device, equipment and vehicle based on speech recognition. Background Art

[0002] With the development of intelligent driving technology, vehicle voice control systems, as an important part of modern automobiles, have been widely used. Especially in vehicle control systems, voice recognition technology has become an important way of human-computer interaction.

[0003] The in-vehicle voice recognition control technology in the related technology mainly realizes control by recognizing the driver's voice information. When the vehicle is driving at high speed, there is noise interference or the user's voice is unclear, the accuracy of voice recognition is low, resulting in the inability to effectively control the vehicle, which further has a certain impact on driving safety. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a vehicle control method, device, equipment and vehicle based on voice recognition, aiming to improve the accuracy of voice recognition to achieve effective control of the vehicle.

[0005] To achieve the above object, one aspect of an embodiment of the present application provides a vehicle control method based on speech recognition, comprising the following steps:

[0006] Acquire voice information and lip video image information of the driver in the vehicle;

[0007] Performing information fusion based on the voice information and the lip video image information to obtain a fusion result, wherein the fusion result includes a feature-level fusion result and a decision-level fusion result;

[0008] Determining the driver's intention through the feature-level fusion result and the decision-level fusion result;

[0009] The vehicle is controlled to perform corresponding operations according to the driver's intention.

[0010] In some embodiments, the information fusion includes feature-level fusion, and the information fusion is performed based on the voice information and the lip video image information to obtain a fusion result, including:

[0011] Extracting speech features through the speech information;

[0012] Extracting lip reading features through the lip video image information;

[0013] The speech feature and the lip reading feature are subjected to feature fusion to obtain the feature-level fusion result.

[0014] In some embodiments, extracting speech features through the speech information includes:

[0015] The acoustic features in the speech information are extracted to obtain the speech features.

[0016] In some embodiments, extracting lip reading features through the lip video image information includes:

[0017] The visual features of the lip movements in the lip video image information are extracted to obtain the lip reading features.

[0018] In some embodiments, the information fusion includes decision-level fusion, and the information fusion is performed based on the voice information and the lip video image information to obtain a fusion result, including:

[0019] Processing and analyzing the voice information to obtain voice text instructions;

[0020] Inputting the lip video image information into a lip reading recognition model to obtain a lip reading text instruction, wherein the lip reading recognition model is obtained through training;

[0021] The voice text instruction and the lip reading text instruction are fused at a decision level to obtain a decision level fusion result.

[0022] In some embodiments, performing decision-level fusion on the speech text instruction and the lip reading text instruction to obtain a decision-level fusion result includes:

[0023] According to the speech text instruction, the confidence corresponding to the speech text instruction, the lip reading text instruction and the confidence corresponding to the lip reading text instruction, decision-level fusion is performed through weighted voting or Bayesian fusion to obtain a decision-level fusion result.

[0024] In some embodiments, determining the driver's intention by using the feature-level fusion result and the decision-level fusion result includes:

[0025] The feature-level fusion result and the decision-level fusion result are decided by a voting mechanism or a weighted average to determine the driver's intention.

[0026] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a vehicle control device based on speech recognition, and the vehicle control device based on speech recognition includes:

[0027] An acquisition module, used to acquire voice information and lip video image information of the driver in the vehicle;

[0028] An information fusion module, used for performing information fusion based on the voice information and the lip video image information to obtain a fusion result, wherein the fusion result includes a feature-level fusion result and a decision-level fusion result;

[0029] A determination module, used to determine the driver's intention through the feature-level fusion result and the decision-level fusion result;

[0030] The control module is used to control the vehicle to perform corresponding operations according to the driver's intention.

[0031] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the vehicle control method based on voice recognition of the above-mentioned embodiment is implemented.

[0032] To achieve the above objectives, another aspect of an embodiment of the present application provides a vehicle, which includes a vehicle control device based on voice recognition according to the above embodiment or an electronic device according to the above embodiment.

[0033] The embodiments of the present application include at least the following beneficial effects:

[0034] The present application provides a vehicle control method, device, equipment and vehicle based on speech recognition. In an embodiment of the present application, first, the voice information and lip video image information of the driver in the vehicle are obtained; then, information fusion is performed based on the voice information and the lip video image information to obtain a fusion result, and the fusion result includes a feature-level fusion result and a decision-level fusion result; then, the driver's intention is determined by the feature-level fusion result and the decision-level fusion result; finally, the vehicle is controlled to perform corresponding operations according to the driver's intention. The vehicle control method based on speech recognition provided in the embodiment of the present application controls the vehicle based on information fusion of voice information and lip video image information, and further determines the driver's intention according to the feature-level fusion result and the decision-level fusion result. Then, the vehicle is controlled according to the driver's intention. Therefore, compared with the related art that only controls the vehicle by recognizing the driver's voice information, the embodiment of the present application can improve the accuracy of speech recognition, thereby achieving effective control of the vehicle.

[0035] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0037] Figure 1 is a flow chart of a vehicle control method based on speech recognition provided by some embodiments of the present application;

[0038] Figure 2 is a schematic block diagram of a vehicle control system based on speech recognition provided by some embodiments of the present application;

[0039] Figure 3 is a module schematic block diagram of a vehicle control device based on speech recognition provided by some embodiments of the present application;

[0040] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the reference to "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0042] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0043] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0046] In order to make the inventive concept of the present application easy to understand, before describing the embodiments of the present application in detail, the English abbreviations (terms) / related concepts involved in the embodiments of the present application are first described. The English abbreviations (terms) / related concepts involved in the embodiments of the present application are subject to the following explanations.

[0047] In-vehicle voice control system: An in-vehicle voice control system is a system that uses voice recognition technology to allow the driver to control various functions of the vehicle through voice commands. The purpose of this system is to reduce driver distraction and improve driving safety and convenience.

[0048] Information fusion: Information fusion refers to the process of comprehensively processing information from multiple sources and types to obtain a more accurate and comprehensive understanding of information. Information fusion includes data layer fusion, feature layer fusion and decision layer fusion.

[0049] With the development of intelligent driving technology, vehicle voice control systems, as an important part of modern automobiles, have been widely used. Especially in vehicle control systems, voice recognition technology has become an important way of human-computer interaction.

[0050] The in-vehicle voice recognition control technology in the related art mainly realizes control by recognizing the driver's voice information. The driver can send voice to the in-vehicle voice engine, and the voice engine analyzes the text spoken by the user through voice recognition and semantic understanding, obtains the vehicle control command, and controls the vehicle according to the control command. This method has problems such as misrecognition and recognition delay when the vehicle is driving at high speed, there is noise interference, or the user's voice is unclear, resulting in low accuracy and reliability of voice recognition, which makes it impossible to effectively control the vehicle, and further affects driving safety.

[0051] In view of this, the present application proposes a vehicle control method, device, equipment and vehicle based on speech recognition. The scheme obtains the voice information and lip video image information of the driver in the vehicle, and then performs information fusion based on the voice information and the lip video image information to obtain a fusion result, which includes a feature-level fusion result and a decision-level fusion result. Then, the driver's intention is determined by the feature-level fusion result and the decision-level fusion result. Finally, the vehicle is controlled to perform corresponding operations according to the driver's intention. In the vehicle control method based on speech recognition provided in the embodiment of the present application, the control of the vehicle is based on information fusion of voice information and lip video image information, and the driver's intention is further determined according to the feature-level fusion result and the decision-level fusion result. Then, the control of the vehicle is achieved according to the driver's intention. Therefore, compared with the related art that only controls the vehicle by recognizing the driver's voice information, the embodiment of the present application can improve the accuracy of speech recognition, thereby achieving effective control of the vehicle.

[0052] The method provided in the embodiment of the present application can be applied to the electronic device provided in the embodiment of the present application, wherein the electronic device can be a vehicle controller, and the vehicle controller can be a domain controller of the vehicle smart cockpit or a master controller in the vehicle smart cockpit, or the electronic device can also be integrated into the domain controller in the vehicle smart cockpit or the master controller in the vehicle smart cockpit, and the embodiment of the present application does not specifically limit this.

[0053] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0054] The following will describe in detail the implementation steps of a vehicle control method based on voice recognition provided by an embodiment of the present application in conjunction with the accompanying drawings.

[0055] Please refer to Figure 1 , Figure 1A flow chart of a vehicle control method based on speech recognition provided for some embodiments of the present application. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and, although a logical order is shown in the flow chart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0056] The method of the embodiment of the present application comprises the following steps:

[0057] Step 101: Acquire voice information and lip video image information of a driver in a vehicle;

[0058] Step 102: performing information fusion based on the voice information and the lip video image information to obtain a fusion result, where the fusion result includes a feature-level fusion result and a decision-level fusion result;

[0059] Step 103: determining the driver's intention through the feature-level fusion result and the decision-level fusion result;

[0060] Step 104: Control the vehicle to perform corresponding operations according to the driver's intention.

[0061] In steps 101 to 104 shown in the embodiment of the present application, information fusion is performed through voice information and lip video image information, and the driver's intention is further determined based on the feature-level fusion results and the decision-level fusion results. Thereafter, the vehicle is controlled based on the driver's intention. Therefore, compared with the related art in which the vehicle is controlled only by recognizing the driver's voice information, the embodiment of the present application can improve the accuracy of voice recognition, thereby achieving effective control of the vehicle.

[0062] The specific implementation methods of the above steps are introduced below.

[0063] In step 101, voice information and lip video image information of a driver in a vehicle are obtained.

[0064] The driver's voice information can be collected through a microphone (audio collection module), and the audio collection module can convert the sound signal in the car into a digital signal, which includes the driver's voice information.

[0065] The driver's lip video image information can be collected by a vehicle-mounted camera, and the driver's video image can be collected by the vehicle-mounted camera, and the driver's lip video image information can be segmented from the video image.

[0066] It should be understood that the acquisition of the driver's voice information and lip video image information can be started by a wake-up command, that is, after receiving the wake-up command, the acquisition of the driver's voice information and lip video image information in the vehicle is started. The wake-up command can be customized by the user, for example, it can be set to "Xiaolan Xiaolan", "Hi, Romi" or "Xiaoka Guanjia".

[0067] For the acquisition of the driver's voice information, it can be determined by positioning that the sound information is emitted by the driver. For example, the source position of the sound can be determined by using an array composed of multiple microphones. Specifically, the specific position of the sound source can be calculated by the time difference (TDOA) or intensity difference between the sound reaching different microphones. If the sound source position is consistent with the driver's seat, it can be preliminarily determined that the voice is emitted by the driver. It is also possible to obtain the sound information emitted by the driver by means of sound feature recognition. For example, the driver's voice can be feature extracted in advance, and a personal voice model can be established. The real-time collected voice can be compared with the driver's voice model to determine whether it matches. The features may include timbre, pitch, spectral characteristics, etc. It can also be determined by an authorization mechanism. For example, before the vehicle is started or used, the driver can be required to perform voice authorization and establish an initial voice model. When the system confirms that the voice matches the authorized voice model, it is determined that the sound information emitted by the driver is obtained.

[0068] In order to cope with noise interference in complex environments and obtain clearer driver voice information, adaptive audio enhancement and noise suppression technologies can be integrated into the audio acquisition module. Adaptive audio enhancement technology aims to improve the quality of audio signals, making them clearer and more natural. It usually involves technologies such as frequency equalization, dynamic range compression, loudness control, and reverberation elimination. Noise suppression technology is mainly aimed at reducing or eliminating background noise in audio signals. It can be achieved by estimating the power spectrum of the noise, subtracting the noise component from the power spectrum of the noisy speech, and reconstructing the speech signal.

[0069] To obtain the driver's lip video image information, you can install a camera inside the vehicle and point it directly at the driver's face. These cameras can be: a front-facing camera mounted on the dashboard or steering wheel, a rearview mirror camera integrated in the rearview mirror, or a surround-view camera. Adjust the camera angle to ensure that the camera is correctly aimed at the driver's head and lips so that the lip movements can be clearly captured. Further adjust the camera's exposure, contrast, and brightness settings to adapt to different lighting conditions. By using computer vision algorithms to identify and track lip movements and using image segmentation technology, the lip area is extracted from the video frame to obtain the driver's lip video image information.

[0070] The embodiment of the present application obtains the voice information and lip video image information of the driver in the vehicle, thereby providing data support for the subsequent integration of the two information. By integrating the driver's voice information and lip video image information, the driver's intention can be understood more accurately, thereby realizing intelligent vehicle control, voice assistant services and rapid response in emergency situations, further improving driving safety and convenience.

[0071] In step 102, information fusion is performed based on the voice information and the lip video image information to obtain a fusion result, which includes a feature-level fusion result and a decision-level fusion result.

[0072] For example, the driver's voice signal can be collected by a high-precision microphone array, and the driver's lip movement video image can be captured by a high-resolution camera installed in the car. Subsequently, these two types of information are subjected to noise elimination, feature extraction, and synchronization alignment in the preprocessing stage to ensure the consistency and availability of the data. In the information fusion stage, it can be divided into feature-level fusion, decision-level fusion, and attention-based encoder-decoder.

[0073] Feature-level fusion is to fuse speech features and lip reading features in the feature extraction stage to form a unified feature vector, which is then used in the subsequent recognition process. This method requires ensuring the consistency of the two features in time and space.

[0074] Decision-level fusion is to fuse the results of speech recognition and lip reading recognition after the recognition of each information (driver's voice information and lip video image information) is completed. Common methods include weighted voting, Bayesian fusion, fuzzy logic, etc., which assign weights according to their respective confidence or reliability.

[0075] Attention-based encoder-decoder, this method is particularly common in deep learning. The encoder extracts features from the input data of different modalities (driver’s voice data and lip video image data), and the decoder performs attention calculations based on these features, automatically adjusting the weights of different modalities to achieve effective information fusion.

[0076] This combination of information from different sources (here, speech and lip reading) can improve the overall recognition performance. By comprehensively utilizing the complementarity of speech and lip reading, lip reading information can compensate when speech signals are disturbed, and vice versa.

[0077] In some embodiments, information fusion includes feature-level fusion, and speech features can be extracted through speech information, lip reading features can be extracted through lip video image information, and speech features and lip reading features can be fused to obtain feature-level fusion results.

[0078] Exemplarily, the acquired original speech signal can be subjected to noise reduction, pre-emphasis, framing and windowing operations to reduce the impact of environmental noise and enhance the time domain and frequency domain characteristics of the signal. Afterwards, the acoustic features in the speech information are extracted, such as extracting Mel frequency cepstral coefficients (MFCC), linear prediction coefficients (LPC), perceptual linear prediction (PLP), etc. These features can reflect the spectral characteristics and vocal tract shape of the speech. The dynamic features in the speech information are extracted, that is, the rate of change of the features over time, such as the first-order difference and the second-order difference, are calculated to capture the dynamic characteristics of the speech; the pitch features in the speech information are extracted, that is, the fundamental frequency (F0) or the pitch trajectory is extracted to characterize the pitch changes of the speech. In this way, the speech features are extracted through the speech information.

[0079] Exemplarily, in the lip video image information processing stage, the following method is used to extract lip reading features. Image processing techniques, such as skin color model, edge detection, active contour model, etc., can be used to detect and segment the lip area from the video frame. After that, feature extraction is performed, which includes extracting geometric features, that is, calculating geometric parameters such as lip contour point coordinates, lip area, lip angle distance, etc., which can describe the shape and movement of the lips; extracting appearance features, that is, extracting lip texture information, such as local binary pattern (LBP), scale-invariant feature transform (SIFT), etc., to characterize the surface details of the lips; extracting motion features, that is, analyzing the lip motion trajectory and speed, and obtaining the dynamic information of the lips through optical flow method or other motion estimation techniques.

[0080] After that, the speech features and lip reading features are fused to obtain the feature-level fusion result. Since the sampling rates of speech and lip videos may be different, the time axes of the two features need to be aligned to ensure that they correspond at the same time point.

[0081] Feature fusion can adopt the method of feature concatenation, that is, simply connecting the speech feature vector and the lip reading feature vector to form a new, longer feature vector; it can also adopt the method of feature selection, that is, using machine learning algorithms such as principal component analysis (PCA) and feature selection algorithms to select the feature combination that is most beneficial for classification; it can also use deep learning models such as convolutional neural networks (CNN) or recurrent neural networks (RNN) through feature transformation to learn the joint representation between speech and lip features.

[0082] By fusing speech features with lip reading features, a feature-level fusion result is obtained. This result can comprehensively reflect the information of speech and lip movements, and provide rich feature representation for further decision-level processing.

[0083] In some implementations, lip reading features may be obtained by extracting visual features of lip movements in lip video image information.

[0084] For example, a video sequence containing lip movements can be extracted from the lip video image information, and then the video sequence is preprocessed. The preprocessing may include: adjusting the frame rate, grayscale conversion and noise removal of the video frame, etc. A facial feature point detection algorithm (such as 68-point facial feature point detection) can be used to locate the key points of the lips to achieve lip positioning. After lip positioning, the visual features of lip movement can be extracted.

[0085] For example, the visual features of lip movement can be realized by extracting geometric features, including extracting the coordinates of the lip contour points, calculating the lip height, lip width, lip angle distance, etc., and then tracking the changes of the lip key points over time, extracting the length, direction, speed, etc. of the motion trajectory to realize the extraction of the visual features of lip movement; it is also possible to calculate the optical flow field of the lip area to describe the speed and direction of the lip movement, and then generate a motion history image by accumulating the optical flow field to represent the movement pattern of the lips, thereby obtaining the visual features of the lip movement. The extracted features are represented as feature vectors, each of which corresponds to a time point or a lip movement state in the video sequence.

[0086] By extracting the visual features of lip movement from the lip video image information, lip reading features are obtained. These features can be used to fuse with speech features to obtain feature-level fusion results.

[0087] In some embodiments, information fusion includes decision-level fusion, which can process and analyze voice information to obtain voice text instructions, input lip video image information into a lip reading recognition model to obtain lip reading text instructions, the lip reading recognition model is obtained through training, and the voice text instructions and lip reading text instructions are fused at the decision level to obtain a decision-level fusion result.

[0088] The method provided in the embodiment of the present application can be used to extract the features of the speech signal, and then the extracted speech signal features are converted into text instructions. Exemplarily, the speech signal features can be converted into text instructions based on the method of deep learning, and the deep learning method can be a recurrent neural network (RNN), especially a long short-term memory network (LSTM) and a gated recurrent unit (GRU), or a convolutional neural network, or an end-to-end system, such as a neural network using a CTC (Connectionist Temporal Classification) loss function, which can directly map from speech waveform to text. Exemplarily, the conversion of speech signal features into text instructions can also be achieved through speech decoding. Speech decoding is to evaluate the probability of a given input speech signal feature and a specific pronunciation, evaluate the probability of a given pronunciation sequence forming a specific sentence, and combine the acoustic model and the language model to find the most likely word sequence.

[0089] The lip video image information is input into the lip reading recognition model to obtain the lip reading text instruction, wherein the lip reading recognition model is established in advance. Before receiving the wake-up instruction, or when the vehicle controller is started, or when the vehicle voice recognition control function is started, the pre-established lip reading recognition model is loaded. The lip reading recognition model is learned and established through a large number of lip reading images and corresponding wake-up words to improve the accuracy and efficiency of lip reading recognition. The lip reading recognition model can automatically analyze the lip video image information, extract the lip image feature value, and further predict the text instruction corresponding to the lip image.

[0090] In some implementations, decision-level fusion may be performed through weighted voting or Bayesian fusion based on the speech-to-text instructions, the confidence corresponding to the speech-to-text instructions, the lip-reading-to-text instructions, and the confidence corresponding to the lip-reading-to-text instructions to obtain a decision-level fusion result.

[0091] After obtaining the voice-text instruction and the lip-reading text instruction, the voice-text instruction, the confidence corresponding to the voice-text instruction, the lip-reading text instruction and the confidence corresponding to the lip-reading text instruction can be used to perform decision-level fusion to obtain the decision-level fusion result. This fusion process can be achieved through two different decision-level fusion methods, namely weighted voting or Bayesian fusion.

[0092] In the weighted voting fusion approach, voice-to-text instructions and lip-to-text instructions can be assigned weights based on their respective performance and reliability. These weights are multiplied by their respective confidences to reflect the relative importance of each instruction in the final decision. In this way, a weighted average confidence score can be calculated, which is then used to select the most likely correct instruction from multiple possible instructions.

[0093] In the Bayesian fusion method, Bayes' theorem can be used to calculate the posterior probability of speech text instructions and lip reading text instructions, which involves the estimation of prior probability and the calculation of likelihood probability. This method can take into account the interdependence between different information sources and make an optimal decision based on all available data.

[0094] The embodiments provided in the present application obtain a fusion result by fusing information based on voice information and lip video image information, which can complement the deficiencies of a single modality and improve the recognition accuracy of voice and lip reading. In a noisy environment or when some information is lost, fusing information from other modalities can help the system better recognize instructions and improve robustness in complex environments.

[0095] In step 103, the driver's intention is determined through the feature-level fusion results and the decision-level fusion results.

[0096] By comprehensively analyzing the feature-level fusion results and the decision-level fusion results, the driver's real-time intention can be more accurately identified and understood.

[0097] In some embodiments, at the feature-level fusion stage, key features of speech information and lip video image information, such as acoustic features of speech and visual features of lip movement, are extracted and integrated. These features are effectively fused to obtain feature-level fusion results to form a comprehensive description of the driver's intention. Subsequently, at the decision-level fusion stage, these comprehensive features are used to perform higher-level information processing and pattern recognition to obtain decision-level fusion results, thereby inferring the driver's specific intentions, including but not limited to operations such as steering, changing lanes, and parking.

[0098] Optionally, the acoustic features of speech and the visual features of lip movement may be combined with the behavioral features of vehicle operation and / or the body movement features of the driver, and these features may be effectively integrated to form a comprehensive description of the driver's intention.

[0099] The behavioral characteristics of vehicle operation refer to the various behavioral patterns exhibited by the driver in the process of operating the vehicle, such as the driver's acceleration behavior, deceleration behavior, steering behavior, lane keeping behavior, etc. These behavioral characteristics can be collected through the vehicle's sensors (such as accelerometers, steering angle sensors, GPS, etc.) and processed through data analysis technology to form a description of the driver's intentions.

[0100] The driver's body movement characteristics refer to the movement patterns when the driver uses hands, feet, head and other parts of the body to operate and observe during driving. For example, the driver's head movement includes the frequency of the driver's observation of the surrounding environment through the rearview mirror, side mirror and direct line of sight, and the angle and range of the driver's head rotation when observing; the driver's line of sight direction, such as the duration of the driver's gaze at the road ahead, the dashboard or other specific points inside and outside the car, and the driver's line of sight movement patterns during driving, such as scanning, gazing and searching. The driver's body movement characteristics can be monitored and analyzed in a variety of ways, such as camera monitoring, motion capture technology, biomechanical analysis, etc. The driver's intentions can be understood through the driver's body movement characteristics.

[0101] For example, during the driver's stable driving process, the behavioral characteristics of the vehicle operation indicate that the driver is decelerating; the characteristics of the driver's body movements indicate that the driver is looking at the left rearview mirror; and the acoustic characteristics of the speech and the visual characteristics of the lip movements indicate that the driver's voice command is to turn left. At this time, the vehicle can be controlled to turn left.

[0102] In some implementations, the feature-level fusion results and the decision-level fusion results may also be decided through a voting mechanism or weighted average to determine the driver's intention.

[0103] Optionally, after obtaining the feature-level fusion result and the decision-level fusion result, a decision can be made between the two through a voting mechanism to obtain the driver's intention.

[0104] It should be understood that the voting mechanism is a commonly used decision fusion method, which can "vote" on a decision through multiple models or sensors, and then make the final decision, that is, the driver's intention, based on these votes.

[0105] Optionally, a decision may be made between the feature-level fusion result and the decision-level fusion result by weighted averaging to obtain the driver's intention.

[0106] It should be understood that weighted average is a statistical method that calculates the average by assigning different weights to each data item. In decision fusion, different weights can be assigned to each data item (feature-level fusion results and decision-level fusion), and the weights can be fixed or dynamically adjusted. After that, each data item is multiplied by its corresponding weight, and then all weighted values ​​are added together and finally divided by the sum of the weights to output the probability distribution of the driver's intention, thereby determining the driver's intention.

[0107] The embodiment provided in the present application determines the driver's intention through feature-level fusion results and decision-level fusion results, which helps to more accurately identify the driver's intention and ensure driving safety.

[0108] In step 104 , the vehicle is controlled to perform corresponding operations according to the driver's intention.

[0109] After determining the driver's intention, a corresponding vehicle control strategy can be formulated based on the determined driver's intention, and ensure that the control strategy complies with traffic regulations and driving safety. Afterwards, the vehicle control strategy is converted into specific vehicle control instructions, such as engine power output, brake pressure, steering angle, etc., and the instructions are sent to the vehicle's actuators (such as engine control module, braking system, steering system, etc.) through the electronic control unit (ECU), and the control instructions are dynamically adjusted according to the actual response of the vehicle and environmental changes to optimize performance and safety.

[0110] The following is a detailed description and explanation of the solution of the embodiment of the present invention in conjunction with a specific application example:

[0111] In an embodiment of the present application, a vehicle control method based on speech recognition is provided to achieve effective control of a vehicle. The vehicle control method based on speech recognition can be applied to a vehicle control system based on speech recognition.

[0112] See also Figure 2 , Figure 2is a schematic block diagram of a vehicle control system based on voice recognition provided by some embodiments of the present application, Figure 2 The system includes a camera and a microphone. The video and audio signals collected by the camera and the microphone are respectively input into a lip reading collection module and a voice collection module, and then the output content of the lip reading collection module is input into a lip reading recognition module; the output content of the voice collection module is input into a voice recognition module, and the outputs of the lip reading recognition module and the voice recognition module are input into a control decision module to obtain a control instruction of the vehicle, and then the instruction is sent to an execution module to realize control of the vehicle.

[0113] Specifically, when the vehicle control system based on speech recognition is initialized, the speech recognition module, the lip reading recognition module and the control decision module are started, and the pre-established lip reading training model is loaded. The driver's speech signal is collected through the vehicle microphone, and the driver's lip video image is collected through the vehicle camera. The collected speech signal is processed and analyzed by the speech recognition module, and converted into speech features and speech text instructions. The collected lip video image is analyzed by the lip reading recognition module, and the lip reading features are extracted and the lip reading text instructions are generated by the lip reading recognition model. The control decision module performs feature fusion on the speech features and lip reading features to obtain feature-level fusion results; performs decision-level fusion on the speech text instructions and the lip reading text instructions to obtain decision-level fusion results, and makes decisions on the feature-level fusion results and the decision-level fusion results through a voting mechanism or weighted average, generates a final control instruction, and controls the vehicle's actuator to perform corresponding operations according to the final control instruction.

[0114] The embodiment of the present application greatly enhances the security of the voice recognition control system by combining voice recognition and lip reading recognition technology, and can achieve high-accuracy vehicle control in complex environments. When the voice recognition module fails to recognize or the recognition result is unreliable, the lip reading recognition module can be used to ensure the accuracy and safety of vehicle control.

[0115] The implementation of the vehicle control device based on voice recognition provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0116] For the vehicle control method based on speech recognition provided in the above embodiment, the embodiment of the present application also provides a vehicle control device based on speech recognition, which is used to implement the above method, such as Figure 3 As shown, Figure 3 This is a module schematic block diagram of a vehicle control device based on speech recognition according to an embodiment of the present application. The vehicle control device based on speech recognition 300 includes:

[0117] An acquisition module 301 is used to acquire voice information and lip video image information of a driver in a vehicle;

[0118] An information fusion module 302 is used to perform information fusion based on the voice information and the lip video image information to obtain a fusion result, wherein the fusion result includes a feature-level fusion result and a decision-level fusion result;

[0119] A determination module 303, used to determine the driver's intention through the feature level fusion result and the decision level fusion result;

[0120] The control module 304 is used to control the vehicle to perform corresponding operations according to the driver's intention.

[0121] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0122] like Figure 4 As shown, the embodiment of the present application further provides an electronic device, the electronic device 400 includes a memory 401, one or more processors 402 ( Figure 4 Only one is shown) and a computer program stored in the memory 401 and executable on the processor 402. Among them: the memory 401 is used to store software programs and units, and the processor 402 executes various functional applications and data processing by running the software programs and units stored in the memory 401 to obtain the resources corresponding to the above-mentioned preset events. Optionally, the processor 402 implements the above-mentioned vehicle control method based on voice recognition by running the above-mentioned computer program stored in the memory 401.

[0123] The memory 401, as a non-transitory computer-readable medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 401 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 401 may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor 402 via a network.

[0124] It can be understood that the contents of the above method embodiments are all applicable to the electronic device embodiment, the functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0125] The embodiment of the present application also provides a vehicle, which includes the above-mentioned vehicle control device based on voice recognition or the electric drive assembly of the above-mentioned electronic device. Specifically, the vehicle can be a private car, such as a sedan, SUV, MPV or pickup truck. The vehicle can also be an operating vehicle, such as a van, bus, small truck or large trailer. The vehicle can be a gasoline vehicle or a new energy vehicle. When the vehicle is a new energy vehicle, it can be a hybrid vehicle or a pure electric vehicle.

[0126] It can be understood that the contents of the above method embodiments are all applicable to the present vehicle embodiment, the functions specifically implemented by the present vehicle embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0127] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it can implement the steps of the vehicle control method based on voice recognition as described above.

[0128] It can be understood that the contents of the above method embodiments are all applicable to the present computer program product, the functions specifically implemented by the present computer program product embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0129] The vehicle control method, device, electronic device, vehicle and computer program product based on speech recognition provided by the embodiment of the present application obtain the voice information and lip video image information of the driver in the vehicle; then, information fusion is performed based on the voice information and lip video image information to obtain a fusion result, and the fusion result includes a feature-level fusion result and a decision-level fusion result; then, the driver's intention is determined by the feature-level fusion result and the decision-level fusion result; finally, the vehicle is controlled to perform corresponding operations according to the driver's intention. The vehicle control method based on speech recognition provided by the embodiment of the present application is based on information fusion of voice information and lip video image information, and further determines the driver's intention according to the feature-level fusion result and the decision-level fusion result. Then, the vehicle is controlled according to the driver's intention. Therefore, compared with the related art that only controls the vehicle by recognizing the driver's voice information, the embodiment of the present application can improve the accuracy of speech recognition, enhance the security of the speech recognition control system, and can realize high-accuracy vehicle control in complex environments. When the speech recognition module fails to recognize or the recognition result is unreliable, the lip recognition module can be used to ensure the accuracy and safety of vehicle control.

[0130] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0131] Although specific embodiments are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are also within the scope of the present disclosure. For example, any of the functions and / or processing capabilities described in conjunction with a particular device or component may be performed by any other device or component. In addition, although various exemplary implementations and architectures have been described according to embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the exemplary implementations and architectures described herein are also within the scope of the present disclosure.

[0132] Some aspects of the present disclosure are described above with reference to the block diagram and flow chart of the system, method, system and / or computer program product according to the exemplary embodiments.It should be understood that the combination of one or more blocks in the block diagram and the flow chart and the blocks in the block diagram and the flow chart can be realized by executing computer executable program instructions respectively.Equally, according to some embodiments, some blocks in the block diagram and the flow chart may not need to be executed in the order shown, or may not need to be executed in full.In addition, additional components and / or operations beyond those components and / or operations shown in the blocks in the block diagram and the flow chart may be present in certain embodiments.

[0133] Therefore, the blocks in the block diagrams and flow charts support the combination of devices for performing the specified functions, the combination of elements or steps for performing the specified functions, and the program instruction devices for performing the specified functions. It should also be understood that each block in the block diagrams and flow charts and the combination of blocks in the block diagrams and flow charts can be implemented by a special-purpose hardware computer system or a combination of special-purpose hardware and computer instructions that performs a specific function, element or step.

[0134] The program modules, applications, etc. described herein may include one or more software components, including, for example, software objects, methods, data structures, etc. Each such software component may include computer-executable instructions that, in response to execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein) to be performed.

[0135] Software components can be encoded with any of various programming languages. An exemplary programming language can be a low-level programming language, such as an assembly language associated with a specific hardware architecture and / or an operating system platform. The software component comprising assembly language instructions may need to be converted to executable machine code by an assembler before being executed by hardware architecture and / or platform. Another exemplary programming language can be a higher-level programming language, which can be transplanted across multiple architectures. The software component comprising a higher-level programming language may need to be converted to an intermediate representation by an interpreter or a compiler before execution. Other examples of programming languages ​​include but are not limited to macro languages, shells or command languages, job control languages, scripting languages, database queries or search languages ​​or report writing languages. In one or more exemplary embodiments, the software component comprising the instruction of one of the above-mentioned programming language examples can be directly executed by an operating system or other software components, without first being converted into another form.

[0136] Software components may be stored as files or other data storage structures. Software components of similar type or related functions may be stored together, such as in a specific directory, folder, or library. Software components may be static (e.g., preset or fixed) or dynamic (e.g., created or modified at execution time).

[0137] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A vehicle control method based on speech recognition, characterized in that: The following steps are involved: Acquire voice information and lip video image information of the driver in the vehicle; Performing information fusion based on the voice information and the lip video image information to obtain a fusion result, wherein the fusion result includes a feature-level fusion result and a decision-level fusion result; Determining the driver's intention through the feature-level fusion result and the decision-level fusion result; The vehicle is controlled to perform corresponding operations according to the driver's intention.

2. The vehicle control method based on speech recognition according to claim 1, characterized in that: The information fusion includes feature-level fusion, and the information fusion is performed based on the voice information and the lip video image information to obtain a fusion result, including: Extracting speech features through the speech information; Extracting lip reading features through the lip video image information; The speech feature and the lip reading feature are subjected to feature fusion to obtain the feature-level fusion result.

3. The vehicle control method based on speech recognition according to claim 2, characterized in that: The extracting speech features through the speech information comprises: The acoustic features in the speech information are extracted to obtain the speech features.

4. The vehicle control method based on speech recognition according to claim 2, characterized in that: The step of extracting lip reading features through the lip video image information includes: The visual features of the lip movements in the lip video image information are extracted to obtain the lip reading features.

5. The vehicle control method based on speech recognition according to claim 1, characterized in that: The information fusion includes decision-level fusion, and the information fusion is performed based on the voice information and the lip video image information to obtain a fusion result, including: Processing and analyzing the voice information to obtain voice text instructions; Inputting the lip video image information into a lip reading recognition model to obtain a lip reading text instruction, wherein the lip reading recognition model is obtained through training; The voice text instruction and the lip reading text instruction are fused at a decision level to obtain a decision level fusion result.

6. The vehicle control method based on speech recognition according to claim 5, characterized in that: The performing decision-level fusion on the speech text instruction and the lip reading text instruction to obtain a decision-level fusion result includes: According to the speech text instruction, the confidence corresponding to the speech text instruction, the lip reading text instruction and the confidence corresponding to the lip reading text instruction, decision-level fusion is performed through weighted voting or Bayesian fusion to obtain a decision-level fusion result.

7. The vehicle control method based on speech recognition according to claim 1, characterized in that: The determining the driver's intention by using the feature-level fusion result and the decision-level fusion result includes: The feature-level fusion result and the decision-level fusion result are decided by a voting mechanism or a weighted average to determine the driver's intention.

8. A vehicle control device based on voice recognition, characterized in that: The device comprises: An acquisition module, used to acquire voice information and lip video image information of the driver in the vehicle; An information fusion module, used for performing information fusion based on the voice information and the lip video image information to obtain a fusion result, wherein the fusion result includes a feature-level fusion result and a decision-level fusion result; A determination module, used to determine the driver's intention through the feature-level fusion result and the decision-level fusion result; The control module is used to control the vehicle to perform corresponding operations according to the driver's intention.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the vehicle control method based on voice recognition as described in any one of claims 1 to 7 when executing the computer program.

10. A vehicle, characterized in that: The vehicle includes the vehicle control device based on voice recognition as claimed in claim 8 or the electronic device as claimed in claim 9.

Images