Speech recognition method and device of vehicle, vehicle and storage medium

Abstract

The invention relates to a voice recognition method and device of a vehicle, the vehicle and a storage medium, and the method comprises the steps: collecting a voice instruction of the vehicle; in response to the voice instruction, acquiring video data corresponding to the voice instruction, and determining weights of the voice instruction and the video data according to the internal noise level of the vehicle; and based on the voice instruction and the weight of the video data, fusing the video data and the voice instruction to identify an actual control intention of the user, and controlling the vehicle to execute a control action corresponding to the actual control intention. According to the embodiment of the invention, the complementarity of the voice data and the video data can be utilized, more accurate voice recognition is realized, the limitation on the pronunciation standard of the user is reduced, the voice recognition is more humanized, the probability of wrong recognition is reduced, and the interaction experience of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, vehicle, and storage medium for a vehicle. Background Art

[0002] To improve the accuracy of user voice input, related technologies employ a hotword activation approach. Users or system administrators register specific hotwords or phrases in the voice recognition system. These registered hotwords are then integrated into the system's acoustic and language models. The acoustic model converts the input voice signal into a feature vector, while the language model calculates the probability of the voice signal corresponding to a possible phrase sequence based on linguistic theory. For registered hotwords, the system adjusts the relevant parameters of the acoustic and language models to make their recognition more sensitive and accurate, thereby improving the recognition accuracy of registered hotwords. However, this approach has significant limitations. It only achieves higher recognition rates for registered hotwords; recognition does not improve for unregistered or non-dictionary words. Furthermore, registering hotwords typically requires adjusting the parameters of the acoustic, language, or decoding algorithms, which can increase system resource usage, such as computing resources and storage space. Similarly, related technologies also have certain requirements for the sound environment and the standardization of the user's voice. If the environment is noisy or the user's pronunciation is not standard, it will be difficult to recognize registered hotwords, resulting in a high probability of misrecognition, which urgently needs improvement. Summary of the Invention

[0003] The present application provides a vehicle voice recognition method, device, vehicle and storage medium to solve the technical problem in related technologies that the accuracy of voice recognition is limited by the sound environment and the user's pronunciation standard, the probability of misrecognition is high, and it is difficult to meet user needs.

[0004] The first aspect of the present application provides a method for speech recognition of a vehicle, comprising the following steps: collecting voice commands of the vehicle; in response to the voice commands, acquiring video data corresponding to the voice commands, and determining the weights of the voice commands and the video data according to the internal noise level of the vehicle; fusing the video data and the voice commands based on the weights of the voice commands and the video data to identify the actual control intention of the user, and controlling the vehicle to perform a control action corresponding to the actual control intention.

[0005] According to the above technical means, the embodiments of the present application can utilize the complementarity of voice data and video data to achieve more accurate voice recognition, reduce the restrictions on user pronunciation standards, make voice recognition more humane, and reduce the probability of incorrect recognition, thereby improving the user's interactive experience.

[0006] Optionally, in one embodiment of the present application, the obtaining of video data corresponding to the voice command includes: collecting initial video data of the vehicle; time-aligning the initial video data with the voice command to obtain time-corresponding video data; determining whether the context in the time-corresponding video data is consistent with the voice command; if the context is consistent with the voice command, determining that the time-corresponding video data is the video data.

[0007] According to the above technical means, the embodiment of the present application can process the initial video data to ensure that the video data and voice commands are in the same scene at the same time, thereby increasing the probability of recognizing the accurate intention.

[0008] Optionally, in one embodiment of the present application, after determining the weights of the voice command and the video data based on the internal noise level of the vehicle, it also includes: obtaining the internal environment data of the vehicle; judging whether the video data meets a preset confidence condition based on the internal environment data; if the preset confidence condition is not met, adjusting the weight based on the internal environment data.

[0009] According to the above technical means, the embodiment of the present application can adjust the weight of video data according to the internal visual environment to avoid unreliable environmental data affecting the accuracy of intent recognition.

[0010] Optionally, in one embodiment of the present application, after identifying the user's actual control intention, it also includes: generating a corresponding vehicle control inquiry instruction based on the actual control intention; receiving the inquiry result of the vehicle control inquiry instruction within a preset time period; verifying the accuracy of the actual control intention based on the inquiry result, and adjusting the weight based on the accuracy if the accuracy does not meet the preset verification qualification conditions.

[0011] According to the above technical means, the embodiment of the present application can initiate an inquiry to the user after identifying the control intention to confirm whether the identification is correct, and then adjust the weight according to the verification result to achieve continuous optimization and evolution of the identification process.

[0012] Optionally, in one embodiment of the present application, it also includes: obtaining at least one control instruction from the actual control intention; determining the execution priority of each control instruction based on the voice source of each control instruction, the current driving status and status parameters of the vehicle, so as to control the vehicle to perform the control action based on the execution priority.

[0013] According to the above technical means, when multiple control instructions are identified, the embodiment of the present application can perform priority determination based on parameters such as the instruction issuer and vehicle status to avoid instruction conflicts.

[0014] The second aspect of the present application provides a speech recognition device for a vehicle, comprising: an acquisition module for acquiring voice commands of the vehicle; a determination module for acquiring video data corresponding to the voice commands in response to the voice commands, and determining the weights of the voice commands and the video data according to the internal noise level of the vehicle; and an identification module for fusing the video data and the voice commands based on the weights of the voice commands and the video data to identify the actual control intention of the user, and controlling the vehicle to perform a control action corresponding to the actual control intention.

[0015] Optionally, in one embodiment of the present application, the determination module includes: an acquisition unit for acquiring initial video data of the vehicle; an alignment unit for time-aligning the initial video data with the voice instruction to obtain time-corresponding video data; a judgment unit for judging whether the context in the time-corresponding video data is consistent with the voice instruction; and a determination unit for determining that the time-corresponding video data is the video data when the context is consistent with the voice instruction.

[0016] Optionally, in one embodiment of the present application, it also includes: a first acquisition module for acquiring the internal environment data of the vehicle; a judgment module for judging whether the video data meets a preset credibility condition based on the internal environment data; and a first adjustment module for adjusting the weight based on the internal environment data when the preset credibility condition is not met.

[0017] Optionally, in one embodiment of the present application, it also includes: a generation module for generating a corresponding vehicle control inquiry instruction based on the actual control intention; a receiving module for receiving the inquiry result of the vehicle control inquiry instruction within a preset time period; a second adjustment module for verifying the accuracy of the actual control intention based on the inquiry result, and adjusting the weight based on the accuracy if the accuracy does not meet the preset verification qualification conditions.

[0018] Optionally, in one embodiment of the present application, it also includes: a second acquisition module, used to obtain at least one control instruction from the actual control intention; an execution module, used to determine the execution priority of each control instruction based on the voice source of each control instruction, the current driving status and status parameters of the vehicle, so as to control the vehicle to perform the control action based on the execution priority.

[0019] The third aspect of the present application provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle speech recognition method as described in the above embodiment.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the vehicle voice recognition method as described in the above embodiment.

[0021] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned vehicle voice recognition method.

[0022] The embodiments of the present application can respond to voice commands, obtain video data corresponding to the voice commands, and perform multimodal data fusion based on the weights of the voice commands and video data determined by the vehicle's internal noise level. This allows the use of the complementarity of voice and video data to identify the user's actual control intent, achieve more accurate voice recognition, reduce restrictions on the user's voice standards, make voice recognition more humane, and control the vehicle to perform control actions corresponding to the actual control intent, reducing the probability of misidentification and improving the user's interactive experience. This solves the technical problem in related technologies where the accuracy of voice recognition is limited by the sound environment and the user's pronunciation standards, resulting in a high probability of misidentification and difficulty in meeting user needs.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0025] Figure 1 A flowchart of a vehicle speech recognition method provided according to an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the principle of a vehicle speech recognition method provided according to one embodiment of the present application;

[0027] Figure 3 A schematic diagram of the effect of a vehicle voice recognition method provided according to one embodiment of the present application;

[0028] Figure 4 A schematic structural diagram of a vehicle voice recognition device provided according to an embodiment of the present application;

[0029] Figure 5 A schematic structural diagram of a vehicle provided according to an embodiment of the present application.

[0030] Among them, 10 is a vehicle voice recognition device, 100 is a collection module, 200 is a determination module, 300 is a recognition module; 501 is a memory, 502 is a processor, and 503 is a communication interface. DETAILED DESCRIPTION

[0031] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0032] The following describes a vehicle voice recognition method, device, vehicle, and storage medium according to an embodiment of the present application with reference to the accompanying drawings. In response to the technical problem mentioned in the above background technology, the accuracy of voice recognition is limited by the sound environment and the user's pronunciation standard, the probability of misidentification is high, and it is difficult to meet user needs. The present application provides a vehicle voice recognition method, in which, in response to a voice command, video data corresponding to the voice command is obtained, and multimodal data fusion is performed based on the weight of the voice command and the video data determined according to the vehicle's internal noise level. The method utilizes the complementarity of the voice data and the video data to identify the user's actual control intention, achieve more accurate voice recognition, reduce the restrictions on the user's voice standard, make the voice recognition more humane, and control the vehicle to perform control actions corresponding to the actual control intention, reduce the probability of misidentification, and improve the user's interactive experience. Thus, the technical problem of the related art, in which the accuracy of voice recognition is limited by the sound environment and the user's pronunciation standard, the probability of misidentification is high, and it is difficult to meet user needs, is solved.

[0033] With the rapid development of artificial intelligence (AI), speech recognition technology has made significant progress and has been widely used in a wide range of fields, including smart homes, intelligent customer service, and voice assistants. However, traditional speech recognition systems rely primarily on a single audio signal for recognition, which faces numerous challenges in practical applications, such as background noise, accent differences, and unclear pronunciation, which seriously affect recognition accuracy.

[0034] In related technologies, specific keywords, i.e. registered hot words, can be triggered to determine associated voice commands. By adjusting the relevant parameters of the acoustic model and language model, the recognition of these hot words can be made more sensitive and accurate, thereby improving the recognition accuracy of registered hot words.

[0035] Traditional acoustic and language models rely solely on speech data, which is less informative and has limited contextual awareness. They can be inadequate when processing complex contexts or implicit information. For example, these approaches are susceptible to background noise and accents, resulting in reduced recognition accuracy. For example, when performing speech recognition on noisy streets or in areas with strong accents, traditional models may not be able to accurately identify the speech content.

[0036] Secondly, acoustic and language models have limited capabilities for processing multilingual and diverse speech signals. For example, when recognizing speech with varying speech speeds, pitches, or accents, traditional models may not be able to accurately recognize all speech content. Furthermore, adjusting the parameters of acoustic and language models to accommodate the varying pronunciation habits of different users requires significant time and resources.

[0037] To address the above issues, the embodiments of the present application can utilize the complementarity of multimodal information to better adapt to speech input in different scenarios and languages, thereby improving generalization capabilities. Traditional language models mainly focus on the statistical relationship between words, while multimodal information can be combined with technologies such as deep learning and computer vision to more deeply understand the complex semantics and emotions in speech. This enables large multimodal models to demonstrate higher accuracy when recognizing speech containing complex semantics such as metaphors and similes.

[0038] Furthermore, through continuous learning and online updates, speech recognition technology that integrates multimodal information can adapt to new data and changes, thereby maintaining its high performance and accuracy. This reduces the difficulty and cost of model updates and maintenance, ensuring continuous provision of high-quality services to users.

[0039] Specifically, Figure 1 A flow chart of a vehicle speech recognition method provided in an embodiment of the present application.

[0040] like Figure 1 As shown, the vehicle voice recognition method includes the following steps:

[0041] In step S101 , a voice command of the vehicle is collected.

[0042] During the actual implementation process, the embodiment of the present application can use the microphone array in the car to collect audio data in the car, perform feature extraction on the audio data in the car, and preliminarily determine the voice commands of the vehicle. When confirming the voice commands, feature analysis or semantic segmentation can be used to preliminarily determine the voice intention, that is, to collect the voice commands of the vehicle.

[0043] After acquiring the audio data, embodiments of the present application may preprocess the data to extract relevant features and then obtain the voice command. During preprocessing, embodiments of the present application may perform operations such as sampling, quantization, framing, and windowing on the input voice signal. The sampling rate and quantization bit number may be determined based on application requirements and the quality of the voice signal. Framing may be used to ensure the continuity and stability of the voice signal, and windowing may be used to smooth the voice signal and reduce spectral leakage.

[0044] In step S102 , in response to a voice command, video data corresponding to the voice command is acquired, and weights of the voice command and the video data are determined according to an interior noise level of the vehicle.

[0045] Since there is a probability of misrecognition of the voice instructions obtained based solely on audio data at this time, in order to further determine the actual language control intention, the embodiment of the present application can also combine visual data to confirm the confidence of the intention to improve the accuracy of voice recognition.

[0046] The embodiment of the present application can reuse the camera inside the car to collect video data inside the car, and intercept data of the corresponding time period from the video data based on the timing data corresponding to the voice command to achieve fusion judgment of the two modal data of auditory and visual.

[0047] In order to ensure the accuracy of the fusion judgment result, the embodiment of the present application can also allocate weights for the auditory and visual aspects according to the internal noise level of the vehicle.

[0048] For example, when the noise inside the car is loud, it will make it difficult to accurately identify the command intent from the collected audio data. In this case, the weight of the auditory aspect, that is, the weight of the voice command, can be appropriately reduced, and the weight of the visual aspect, that is, the weight of the video data, can be increased to focus on the voice control intent derived from the user's expression, lip shape, etc. extracted from the video data.

[0049] The interior noise level can be obtained based on the acquired audio data inside the vehicle. The specific level classification can be set accordingly by those skilled in the art based on actual conditions and is not specifically limited here.

[0050] Optionally, in one embodiment of the present application, obtaining video data corresponding to a voice command includes: collecting initial video data of a vehicle; time-aligning the initial video data with the voice command to obtain time-corresponding video data; determining whether the context in the time-corresponding video data is consistent with the voice command; if the context is consistent with the voice command, determining that the time-corresponding video data is video data.

[0051] As a possible implementation method, the embodiment of the present application can perform data preprocessing after the initial video data is collected to ensure that the initial video data and audio data are synchronized in time, to ensure that the voice signal and visual data are strictly synchronized in time, and to have data consistency in scenes and contexts, to ensure that data sources of different modalities have the same context or background, to ensure the correlation between data.

[0052] For video data that is time-aligned and context-aligned, the embodiments of the present application may use corresponding feature extraction methods, such as convolutional neural networks, to extract visual features from images or videos.

[0053] Optionally, in one embodiment of the present application, after determining the weights of voice commands and video data based on the internal noise level of the vehicle, it also includes: obtaining the internal environment data of the vehicle; judging whether the video data meets a preset confidence condition based on the internal environment data; if the preset confidence condition is not met, adjusting the weight based on the internal environment data.

[0054] In addition to the impact of noise on recognition results, the visual environment will also affect the confidence of video data. The embodiment of the present application can obtain the internal environment data of the vehicle, including light data, etc., and then determine whether the image captured by the video data is blurred, that is, whether the confidence is affected by the internal environment, such as whether the light is insufficient to extract the facial contours, expression information and lip shape of the character from the video data, and whether the user who issues the voice command is obscured, such as when the user wears a mask or sunglasses. At this time, the embodiment of the present application can reduce the weight of the video data to reduce the probability of incorrect recognition.

[0055] In step S103 , the video data and the voice command are fused based on the weights of the voice command and the video data to identify the actual control intention of the user, and the vehicle is controlled to perform a control action corresponding to the actual control intention.

[0056] Furthermore, after obtaining the weight of the voice command and the weight of the video data, the embodiment of the present application can fuse the two to calculate the credibility of each voice command. For example, a credibility interval that can be used is pre-set, and the voice command falling within the credibility interval is identified as the actual control intention.

[0057] Optionally, in one embodiment of the present application, after identifying the user's actual control intention, it also includes: generating a corresponding vehicle control inquiry instruction based on the actual control intention; receiving the inquiry result of the vehicle control inquiry instruction within a preset time period; verifying the accuracy of the actual control intention based on the inquiry result, and adjusting the weight based on the accuracy if the accuracy does not meet the preset verification qualification conditions.

[0058] The embodiment of the present application can continuously optimize the adjustment of weights. After identifying the actual control intention, the embodiment of the present application can send a vehicle control inquiry instruction to the vehicle central control, and obtain the user's response to the vehicle control inquiry instruction within a certain period of time. If the user does not agree with the identification, he or she can reject the inquiry. If he or she agrees with the identification, he or she can agree or ignore the inquiry within a preset period of time. Through the response results of the vehicle control inquiry instruction, the embodiment of the present application can determine the accuracy of the actual control intention. When the accuracy is insufficient, the weight is adjusted.

[0059] In addition, the embodiment of the present application can also confirm the real-time performance of voice recognition. After the user issues a command, the timing starts until the vehicle control inquiry command is sent, and the time required for the entire recognition process is confirmed. If the time required is too long, it means that there are certain problems in the recognition process. At this time, a maintenance reminder can be generated to avoid affecting the user experience.

[0060] Optionally, in one embodiment of the present application, it also includes: obtaining at least one control instruction from the actual control intention; determining the execution priority of each control instruction based on the voice source of each control instruction, the current driving status and status parameters of the vehicle, so as to control the vehicle to perform control actions based on the execution priority.

[0061] It is understandable that when there are multiple people in the car, multiple people may issue voice commands at the same time. In this case, the embodiment of the present application can prioritize the control commands based on the voice source (the user who issued the voice command), the current driving state and state parameters of the vehicle. For example, a higher priority weight can be assigned to the driver, and priority weights can be assigned based on the urgency of the response, thereby avoiding the simultaneous execution of multiple tasks, which may cause the vehicle's processing system to malfunction. Figure 2 and Figure 3 The working principle of the vehicle speech recognition method of the embodiment of the present application is described in detail using an embodiment as shown.

[0062] In the actual implementation process of the embodiment of the present application, the fusion model can be relied upon to achieve multimodal data fusion and speech recognition. Taking the fusion model as an example, the embodiment of the present application may include the following steps:

[0063] Step S201: Collect audio data and multimodal data.

[0064] For example, in this embodiment, a MEMS microphone array (6 channels) can be used to collect audio at a 16kHz sampling rate, and visual information can be collected through an RGB camera, infrared thermal imager, etc., with a synchronized timestamp accuracy of ≤1ms. The audio signal sampling rate can be set to 8kHz, 16kHz, or even higher, depending on the application requirements and the quality of the voice signal.

[0065] Step S202: audio data preprocessing.

[0066] The embodiment of the present application can perform operations such as sampling, quantization, framing, and windowing on the input audio data, and output 768-dimensional time series features (acoustic features + rhythmic features), that is, convert them into digital signals suitable for model processing.

[0067] Among them, the quantization bit number can be 8 bits, 16 bits and 32 bits, which affects the data range and accuracy of the audio signal; when framing, each frame can be 20-30ms, and the frame shift is 10ms to ensure the continuity and stability of the speech signal; the window function used for windowing can be a rectangular window, Hamming window, etc., which is used to smooth the speech signal and reduce spectral leakage.

[0068] Step S203: speech model analysis.

[0069] Extract features that can describe speech characteristics from the preprocessed audio signal.

[0070] Step S204: multimodal data preprocessing.

[0071] Based on the preprocessed speech data obtained in step S202, the embodiment of the present application can synchronize the speech frame and the visual frame through timestamps, and output 1024-dimensional spatial features (facial action units + micro-expression features) through visual key frame extraction and face alignment.

[0072] When performing multimodal data preprocessing, data synchronization needs to be guaranteed: ensuring that the voice signal and visual data are strictly synchronized in time to facilitate subsequent multimodal data fusion; data consistency: data sources of different modalities need to have the same context or background to ensure the relevance between the data.

[0073] For multimodal data, such as video data, the embodiments of the present application can reduce computational complexity by extracting visual key frames. For example, the frequency of key frame extraction is determined based on the face detection confidence level: the face detection confidence threshold is ≥ 0.7, the ROI ratio of the lip movement area is: for example, the mouth occupies 20% of the height of the face, and the lip movement sequence frame sampling interval is: features are extracted every 5 frames.

[0074] Step S205: multimodal model analysis.

[0075] The embodiment of the present application can obtain multimodal features based on the preprocessed multimodal data, such as image features (facial features, lip features, etc.) in the video data.

[0076] Step S206: fusion model data fusion and decision making.

[0077] It should be noted that fusion data relies on the input of multi-source data (speech, vision), and data acquisition is the physical basis. MEMS microphone arrays and RGB / infrared cameras provide raw data, and timestamp synchronization ensures modal alignment to avoid spatiotemporal misalignment. The temporal features of speech (such as MFCC and rhythm) and the spatial features of vision (such as facial action units) are key inputs for fusion. If effective features are not extracted, the fusion model cannot capture cross-modal associations, resulting in reduced recognition performance. Preprocessing before data fusion (such as framing and windowing) can reduce noise interference, visual keyframe extraction can reduce computational complexity, and face alignment can improve feature stability. These are the basis of the system's anti-interference ability.

[0078] The fusion arbitration model fuses extracted speech and visual features to form a richer feature representation. Furthermore, through a dynamic weight calculation model and a hybrid arbitration mechanism, the prediction results of multiple models are combined to improve the final recognition accuracy.

[0079] First, the embodiment of the present application needs to confirm the time alignment error between modalities (such as ≤50ms) and the consistency of feature dimensions (needs to be projected into the same space).

[0080] In the settings of the fusion model, the number of Transformer attention heads (4-8 heads) and the hidden layer dimension (512-1024); the discriminator loss weight (such as 0.5) and the generator loss weight (such as 0.5) in adversarial training.

[0081] When making decision fusion, the confidence calculation cycle of each modality (such as updating every 100ms) and the environmental noise level (such as high / medium / low) are considered.

[0082] The key parameters involved are, for example, the weights of the reinforcement learning reward function: speech confidence weight (e.g., 0.6), visual confidence weight (e.g., 0.4); PPO algorithm parameters: discount factor (γ=0.99), PPO clipping parameter (ε=0.3), and learning rate (3e-5).

[0083] Furthermore, embodiments of the present application can dynamically integrate speech and visual features through the weighting of a fusion model, achieving environmentally adaptive decision-making. The fusion model adjusts weights through reinforcement learning (e.g., reducing speech weight during noise reduction), significantly improving recognition rates in noisy environments. Experiments have shown that the accuracy rate can be increased by over 30% after fusion.

[0084] In addition, the embodiments of the present application may also adopt a hybrid arbitration mechanism (such as weighted averaging, voting) to integrate the outputs of multiple models, avoid the bias of a single model, and enhance the credibility of the results.

[0085] Among them, the fusion model is based on the multimodal Transformer architecture, integrating multi-source data of speech and vision, and can adopt the pre-training-fine-tuning paradigm to achieve cross-modal semantic alignment.

[0086] The embodiments of the present application can fuse the extracted speech and visual features in a fusion model. Within the fusion model, a dynamic weighted adaptive fusion mechanism is adopted based on the fusion strategy required by different scenarios and tasks. The model automatically assigns dynamic weights to the speech and visual features, thereby outputting a fusion result with a high degree of confidence.

[0087] For example, the dynamic weight calculation process can be as follows:

[0088] Input the real-time noise level: N (db), and decompose the noise level for weight distribution.

[0089]

[0090] We can set the constraint condition: α∈[0.3,0.8] (to prevent extreme weight imbalance).

[0091] Based on dynamic weight allocation, the embodiment of the present application can realize multi-modal assisted correction, taking vision as an example.

[0092] Calculate the speech signal-to-noise ratio (SNR) (calculation formula: SNR = 10*log10(Ps / Pn)), and obtain the visual confidence level (Vconf) (the maximum probability of the expression category output by the ViT-H classifier). If the confidence level is greater than a certain value, the data is usable.

[0093] During visually assisted correction, such as noise suppression, when SNR < 10dB, the weight of the lip shape feature is increased.

[0094] When there are pronunciation errors: When a pronunciation error (such as dialect confusion) is detected, the confidence level is adjusted based on the gaze direction (whether the person is looking at the microphone or not).

[0095] In summary, in a noisy environment, the following judgment formula can be set in the embodiment of the present application:

[0096] D=α×S+(1-α)×E,

[0097] Here, D represents the final fusion judgment result; S is the output of the speech model, and E is the output of the expression model. α is the weight of the speech data, whose value is dynamically adjusted by the agent based on the ambient noise level N and a preset function, which can be expressed as α = f(N). When the noise level N increases and the speech features become unstable, the fusion model uses a reinforcement learning mechanism to adjust the function f(N) based on the reward feedback obtained from interacting with the environment, reducing α and correspondingly increasing the weight of the expression data (1-α). This effectively integrates information from multiple models and makes precise decisions.

[0098] During the initial setting, empirical values can be assigned based on the importance of voice and expression to the final judgment in different scenarios in historical data. For example, in general quiet communication scenarios, voice contributes more, and α can be set to 0.6-0.8.

[0099] In a noisy environment, when the noise level N reaches a certain threshold, such as exceeding 60 decibels, the fusion model uses reinforcement learning to adjust α based on the reward feedback obtained from interacting with the environment. If, over multiple interactions, adjusting α to 0.3-0.5 results in a closer match between the fusion judgment and the real situation, resulting in higher rewards, the fusion model will tend to maintain α in this range, thereby increasing the weight of expression data. As the model continues to learn and the scene changes, α will continue to dynamically adjust to meet the needs of multimodal data fusion judgment in different scenarios.

[0100] The fusion model performs data fusion and result arbitration on the collected multimodal data. Leveraging reinforcement learning, the model utilizes a large amount of annotated data from specialized fields such as intelligent cockpits, intelligent customer service, and health consultations during its training and enhancement phases. This data includes speech transcription, expression categories or feature descriptions, and ultimately, accurate recognition results, including the speaker's intent and expressed emotion.

[0101] When dealing with multi-source data, such as speech and visual data, the system utilizes a dynamic weighted adaptive reciprocal mechanism to achieve data fusion. When speech features are unstable in noisy environments, the fusion model automatically increases the weight of facial expression data in the fusion judgment. By continuously interacting with the environment and obtaining reward feedback, it optimizes fusion decisions, achieving comprehensive utilization of two or more data sources and making accurate decisions.

[0102] Step S207: output the speech recognition result.

[0103] During the actual implementation process, the embodiment of the present application can perform real-time speech recognition, process the input speech signal in real time, output the recognition result, and then perform visual assisted recognition. During the recognition process, the recognition result is corrected and optimized in combination with additional visual information.

[0104] The probability distribution output by the fusion model is then converted into the final recognition result, such as Viterbi decoding, beam search, etc. A confidence threshold interval is pre-set to confirm that the recognition results falling within the confidence threshold interval are credible, and the recognition results below the threshold interval may be rejected or marked as low confidence.

[0105] It should be noted that voice recognition requires instant feedback (such as smart assistants and real-time translation). A delay of more than 200ms will affect the user experience. At this time, an early warning can be issued to ensure timely maintenance.

[0106] It is understandable that in the embodiment of the present application, visual correction (such as lip synchronization) can improve the accuracy of specific scenarios (such as noisy environments), but if the visual module fails (such as occlusion), the embodiment of the present application can still output the results through voice alone.

[0107] Finally, the embodiment of the present application can output the recognition results to the user in text or other forms.

[0108] In addition, the embodiments of the present application can also perform continuous training and optimization of the model. By collecting new data, the model is continuously trained and optimized to improve the generalization ability and adaptability of the model; the performance of the model is regularly evaluated, and the poorly performing model is adjusted or replaced to ensure the long-term stable operation of the system; the diversity of training data is increased through data enhancement techniques (such as noise addition, speed perturbation, etc.); regularization methods such as L1 / L2 regularization, Dropout, etc. are used to prevent model overfitting.

[0109] Over time, the fusion model may encounter new data or scenarios. To maintain its accuracy and robustness, it needs to be regularly optimized and updated. Through cross-modal transfer learning, the fusion model builds a flexible learning framework, deeply mines common knowledge from multimodal data, and quickly adapts to new application scenarios and tasks.

[0110] It's important to note that while continuous learning can improve generalization in dynamic environments (e.g., multiple speakers, varying dialects), it can be redundant in static scenarios (e.g., with a fixed instruction set). Therefore, continuous learning requires additional computing resources, and a cost-benefit trade-off must be made.

[0111] The core of cross-modal transfer learning is to break down the barriers between different modal data and realize cross-domain transfer of knowledge. For example, the training model finds that the acoustic features expressing hunger in speech and the facial muscle movement features corresponding to the hungry expression in the image both represent "hunger" semantically, thus establishing a cross-modal association. The fusion data established based on this association enhances the understanding ability of the fusion model. When the user says "I want to buy apples", it can be judged based on the fusion data features whether he wants to buy fruit or an electronic brand. Similarly, in situations where the pronunciation is unclear, such as Figure 3 As shown, in the single recognition model in the related art, when the user says a pronunciation that is easily confused (such as the tenth song), it is very easy to be recognized as other instructions (such as the fourth song). However, after combining the multimodal data for judgment, the correct recognition result can be obtained (such as combining the lip shape to confirm it as the tenth song).

[0112] In addition to the vehicle field, the multimodal recognition of the embodiments of the present application can also be applied to its special technical fields, such as smart home, smart customer service, education and medical care, etc., to provide users with a better voice interaction experience.

[0113] In summary, the embodiments of the present application can improve speech recognition accuracy and provide richer and more comprehensive contextual information for speech recognition, thereby improving recognition accuracy. Whether dealing with complex environments, different accents, or changes in speech rate, they can demonstrate higher recognition capabilities and better cope with various challenges such as background noise, signal distortion, and accent differences. This robustness makes the system more reliable and stable in practical applications, reducing the possibility of misidentification.

[0114] Accurate speech recognition reduces friction between users and devices, improving the user experience. Users no longer need to repeatedly confirm or revise recognition results, saving time and effort. Furthermore, multimodal data fusion offers a richer range of interaction options, such as combining speech with gestures and linking speech with images, further enhancing the user experience.

[0115] During speech recognition, through deep learning and multimodal data fusion, the embodiments of the present application can more deeply understand the user's intentions and needs, thereby providing more intelligent services. At the same time, it also reduces dependence on human intervention, thereby reducing related development and maintenance costs.

[0116] According to the vehicle voice recognition method proposed in the embodiment of the present application, it is possible to respond to voice commands, obtain video data corresponding to the voice commands, and perform multimodal data fusion based on the weights of the voice commands and video data determined by the vehicle's internal noise level, so as to utilize the complementarity of voice data and video data to identify the user's actual control intention, achieve more accurate voice recognition, reduce restrictions on the user's pronunciation standards, make voice recognition more humane, and control the vehicle to perform control actions corresponding to the actual control intention, reduce the probability of misrecognition, and improve the user's interactive experience. This solves the technical problem in the related art that the accuracy of voice recognition is limited by the sound environment and the user's pronunciation standards, the probability of misrecognition is high, and it is difficult to meet user needs.

[0117] Next, the speech recognition device for a vehicle according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0118] Figure 4 It is a block diagram of a speech recognition device for a vehicle according to an embodiment of the present application.

[0119] like Figure 4 As shown, the vehicle speech recognition device 10 includes: a collection module 100 , a determination module 200 and a recognition module 300 .

[0120] Specifically, the collection module 100 is used to collect voice commands of the vehicle.

[0121] The determination module 200 is configured to obtain video data corresponding to the voice command in response to the voice command, and determine a weight between the voice command and the video data according to the interior noise level of the vehicle.

[0122] The recognition module 300 is used to fuse the video data and the voice command based on the weight of the voice command and the video data to recognize the user's actual control intention and control the vehicle to perform a control action corresponding to the actual control intention.

[0123] Optionally, in one embodiment of the present application, the determination module 200 includes: a collection unit, an alignment unit, a judgment unit and a determination unit.

[0124] The acquisition unit is used to acquire initial video data of the vehicle.

[0125] The alignment unit is used to time-align the initial video data with the voice command to obtain time-series corresponding video data.

[0126] The judgment unit is used to judge whether the context in the time-series corresponding video data is consistent with the voice command.

[0127] The determining unit is used to determine that the time-series corresponding video data is the video data when the context is consistent with the voice instruction.

[0128] Optionally, in one embodiment of the present application, the vehicle voice recognition device 10 further includes: a first acquisition module, a judgment module and a first adjustment module.

[0129] The first acquisition module is used to acquire the internal environment data of the vehicle.

[0130] The judgment module is used to judge whether the video data meets the preset reliability conditions based on the internal environment data.

[0131] The first adjustment module is used to adjust the weight based on the internal environment data when a preset confidence condition is not met.

[0132] Optionally, in one embodiment of the present application, the vehicle voice recognition device 10 further includes: a generating module, a receiving module and a second adjusting module.

[0133] Among them, the generation module is used to generate corresponding vehicle control query instructions based on the actual control intention.

[0134] The receiving module is used to receive the query result of the vehicle control query instruction within a preset time period.

[0135] The second adjustment module is used to verify the accuracy of the actual control intention based on the query result, and adjust the weight based on the accuracy if the accuracy does not meet the preset verification qualification conditions.

[0136] Optionally, in one embodiment of the present application, the vehicle voice recognition device 10 further includes: a second acquisition module and an execution module.

[0137] The second acquisition module is used to acquire at least one control instruction from the actual control intention.

[0138] The execution module is used to determine the execution priority of each control instruction based on the voice source of each control instruction, the current driving state and state parameters of the vehicle, so as to control the vehicle to perform the control action based on the execution priority.

[0139] It should be noted that the above explanation of the embodiment of the vehicle voice recognition method is also applicable to the vehicle voice recognition device of this embodiment, and will not be repeated here.

[0140] The vehicle voice recognition device proposed in the embodiment of the present application can respond to voice commands, obtain video data corresponding to the voice commands, and perform multimodal data fusion based on the weights of the voice commands and video data determined by the vehicle's internal noise level. This allows the device to utilize the complementarity of voice and video data to identify the user's actual control intent, achieve more accurate voice recognition, reduce restrictions on the user's pronunciation standards, make voice recognition more humane, and control the vehicle to perform control actions corresponding to the actual control intent, thereby reducing the probability of misidentification and improving the user's interactive experience. This solves the technical problem in related technologies where the accuracy of voice recognition is limited by the sound environment and the user's pronunciation standards, resulting in a high probability of misidentification and difficulty in meeting user needs.

[0141] Figure 5 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle may include:

[0142] Memory 501 , processor 502 , and computer programs stored in the memory 501 and executable on the processor 502 .

[0143] When the processor 502 executes the program, the vehicle voice recognition method provided in the above embodiment is implemented.

[0144] Furthermore, the vehicle further comprises:

[0145] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0146] The memory 501 is used to store computer programs that can be run on the processor 502 .

[0147] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0148] If the memory 501, processor 502, and communication interface 503 are implemented independently, the communication interface 503, memory 501, and processor 502 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0149] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0150] The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0151] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned vehicle voice recognition method when executed by a processor.

[0152] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the vehicle voice recognition method provided by an embodiment of the present invention.

[0153] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0154] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0155] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0156] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0157] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0158] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0159] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0160] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A vehicle voice recognition method, characterized in that: The following steps are involved: Collecting vehicle voice commands; In response to the voice command, acquiring video data corresponding to the voice command, and determining a weight between the voice command and the video data based on an interior noise level of the vehicle; The video data and the voice command are fused based on weights of the voice command and the video data to identify the actual control intention of the user, and control the vehicle to perform a control action corresponding to the actual control intention.

2. The method according to claim 1, characterized in that The acquiring of video data corresponding to the voice command includes: collecting initial video data of the vehicle; Time-aligning the initial video data with the voice command to obtain time-series corresponding video data; Determining whether the context in the time-series corresponding video data is consistent with the voice command; If the context is consistent with the voice instruction, the time-series corresponding video data is determined to be the video data.

3. The method according to claim 1, characterized in that After determining the weights of the voice command and the video data according to the interior noise level of the vehicle, the method further includes: Acquiring internal environment data of the vehicle; Determining whether the video data meets a preset reliability condition based on the internal environment data; If the preset confidence condition is not met, the weight is adjusted based on the internal environment data.

4. The method according to claim 1, wherein After identifying the user's actual control intention, it also includes: generating a corresponding vehicle control inquiry instruction based on the actual control intention; receiving the query result of the vehicle control query command within a preset time period; The accuracy of the actual control intention is verified based on the query result, and if the accuracy does not meet a preset verification qualification condition, the weight is adjusted based on the accuracy.

5. The method according to claim 1, wherein Also includes: Acquire at least one control instruction from the actual control intention; Based on the voice source of each control instruction, the current driving state and state parameters of the vehicle, the execution priority of each control instruction is determined, so as to control the vehicle to perform the control action based on the execution priority.

6. A speech recognition device for a vehicle, characterized in that: include: A collection module, used to collect voice commands from the vehicle; a determination module configured to, in response to the voice command, obtain video data corresponding to the voice command, and determine a weight between the voice command and the video data based on an interior noise level of the vehicle; A recognition module is used to fuse the video data and the voice command based on the weights of the voice command and the video data to identify the user's actual control intention and control the vehicle to perform a control action corresponding to the actual control intention.

7. The device according to claim 6, characterized in that The determination module includes: an acquisition unit, configured to acquire initial video data of the vehicle; an alignment unit, configured to perform time alignment on the initial video data and the voice command to obtain time-series corresponding video data; a judgment unit, configured to judge whether the context in the time-series corresponding video data is consistent with the voice command; A determination unit is configured to determine, when the context is consistent with the voice instruction, that the time-series corresponding video data is the video data.

8. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the speech recognition method for a vehicle as claimed in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the speech recognition method for a vehicle as described in any one of claims 1 to 5.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed, it is used to implement the vehicle speech recognition method according to any one of claims 1 to 5.

Images