Speech recognition method and related equipment

Abstract

The invention discloses a voice recognition method and related equipment, and relates to the technical field of vehicle control, and the method comprises the steps: obtaining audio data from an audio collection device, image data from a visual collection device, and vehicle state data from a vehicle bus; performing feature extraction on the audio data and the image data to obtain audio features and visual features; based on the vehicle state data, determining a fusion weight of the audio features and the visual features; based on the fusion weight, the audio features and the visual features are fused, and fusion feature representation is generated; and performing feature analysis on the fused feature representation to obtain a target text recognition result. According to the method and the device, the audio features, the visual features and the vehicle state data are fused, and the fusion weight of the multi-modal features is adaptively adjusted based on the vehicle running state, so that the dynamic optimization of the speech recognition process in the vehicle-mounted scene can be realized, and the accuracy, the stability and the practical application adaptability of speech recognition are improved.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and more specifically, to a speech recognition method and related equipment. Background Technology

[0002] With the rapid development of intelligent vehicles and in-vehicle infotainment systems, voice interaction has gradually become an important way for drivers to obtain information, control vehicle functions, and interact with intelligent cockpit systems. Compared with traditional physical buttons or touch operations, voice recognition has the advantages of freeing the driver's hands and eyes, improving driving safety, and enhancing operational convenience, and has therefore been widely used in the fields of intelligent driving and intelligent cockpits. However, the complex environment in which vehicles operate presents many challenges to the practical application of in-vehicle voice recognition.

[0003] In related technologies, in-vehicle voice recognition methods typically rely on audio acquisition devices to obtain voice signals and then perform feature extraction and recognition processing on the acquired audio data. Although existing technologies have incorporated audio processing techniques such as noise reduction and echo cancellation, audio signals are easily affected by wind noise, road noise, and background noise under conditions such as changes in vehicle speed, open windows, air conditioning operation, or conversations among multiple people inside the vehicle. This leads to a decline in audio feature quality, making it difficult to guarantee voice recognition accuracy. Furthermore, when the vehicle's status changes, existing in-vehicle voice recognition solutions often struggle to meet the needs of different scenarios, thus affecting the stability and reliability of the voice recognition system in diverse in-vehicle environments. In other words, related technologies generally suffer from insufficient voice recognition accuracy and stability in complex in-vehicle environments, as well as inadequate utilization of multi-source information. Summary of the Invention

[0004] The summary section of this application introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. The summary section of this application is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0005] The speech recognition method and related equipment provided in this application can achieve dynamic optimization of the speech recognition process in vehicle scenarios by fusing audio features, visual features and vehicle status data, and adaptively adjusting the fusion weights of multimodal features based on the vehicle's operating status, thereby improving the accuracy, stability and practical application adaptability of speech recognition.

[0006] In a first aspect, this application provides a speech recognition method applied to a target vehicle, comprising: acquiring audio data from an audio acquisition device, image data from a visual acquisition device, and vehicle status data from a vehicle bus; extracting features from the audio data and the image data respectively to obtain audio features and visual features; determining a fusion weight between the audio features and the visual features based on the vehicle status data; fusing the audio features and the visual features based on the fusion weight to generate a fused feature representation; and performing feature parsing on the fused feature representation to obtain a target text recognition result.

[0007] In some implementations, determining the fusion weight of the audio feature and the visual feature based on the vehicle state data includes: determining whether the target vehicle is in a preset noise scene based on the vehicle state data; if the target vehicle is in the preset noise scene, determining the fusion weight as a first fusion weight configuration, wherein the weight of the visual feature in the first fusion weight configuration is greater than the weight of the audio feature; otherwise, determining the fusion weight as a second fusion weight configuration, wherein the weight of the visual feature in the second fusion weight configuration is greater than the weight of the audio feature.

[0008] In some implementations, the vehicle status data includes at least one of vehicle speed, window status parameters, and air conditioning fan speed setting; determining whether the target vehicle is in a preset noise scenario based on the vehicle status data includes: if at least one of the following is true, the target vehicle is determined to be in a preset noise scenario: the vehicle speed is greater than a preset speed threshold, the window status parameters indicate an opening gap greater than a preset gap threshold, and the air conditioning fan speed setting is greater than a preset setting threshold.

[0009] In some embodiments, the step of extracting features from the audio data and the image data to obtain audio features and visual features includes: if a preset wake-up word is detected in the audio data, then obtaining image sub-data corresponding to the time period in which the preset wake-up word is detected in the image data; based on the image sub-data, determining whether the target speaker's lips have performed a target speech activity corresponding to the preset wake-up word; if it is determined that the lips have performed the target speech activity, then extracting features from the audio data and the image data to obtain audio features and visual features.

[0010] In some implementations, determining whether the target speaker's lips have engaged in a target speech activity corresponding to the preset wake-up word based on the image sub-data includes: performing feature extraction processing on the image sub-data to obtain a lip region image sequence and gaze direction information; determining whether the target speaker's lips have a periodic movement associated with the preset wake-up word based on the lip region image sequence; if the periodic movement exists, and the gaze direction is determined to be towards a preset area based on the gaze direction information, then determining that the lips have engaged in a target speech activity corresponding to the preset wake-up word, wherein the preset area includes the vehicle center console area and the road area.

[0011] In some embodiments, the step of extracting features from the audio data and the image data to obtain audio features and visual features includes: performing frame segmentation on the audio data to obtain multiple audio frames; converting the multiple audio frames into multiple corresponding acoustic feature vectors based on a preset acoustic feature extraction algorithm; arranging the multiple acoustic feature vectors in chronological order to constitute the audio features; determining the lip region image sequence of the target speaker based on the image data; extracting features from the lip region image sequence to obtain lip geometric features, wherein the lip geometric features include lip contour, lip opening degree, and lip opening frequency; and encoding and fusing the lip contour, lip opening degree, and lip opening frequency to obtain the visual features.

[0012] In some implementations, fusing the audio features and the visual features based on the fusion weights to generate a fused feature representation includes: weighting the audio features and the visual features based on the fusion weights to obtain preliminary fused features; and performing feature enhancement processing on the preliminary fused features through a preset attention mechanism network to obtain the fused feature representation.

[0013] Secondly, this application also provides a speech recognition device applied to a target vehicle, comprising: a data acquisition unit for acquiring audio data from an audio acquisition device, image data from a visual acquisition device, and vehicle status data from a vehicle bus; a feature extraction unit for extracting features from the audio data and the image data respectively to obtain audio features and visual features; a weight determination unit for determining the fusion weight of the audio features and the visual features based on the vehicle status data; a feature fusion unit for fusing the audio features and the visual features based on the fusion weight to generate a fused feature representation; and a result determination unit for performing feature parsing on the fused feature representation to obtain a target text recognition result.

[0014] Thirdly, this application also provides an electronic device, including: a memory and a processor, wherein the processor is configured to implement the steps of the speech recognition method described in the first aspect when executing a computer program stored in the memory.

[0015] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions or a computer program, which, when executed by a processor, implement the steps of the speech recognition method described in the first aspect.

[0016] Fifthly, this application also provides a computer program product, including a computer program or computer executable instructions, wherein when the computer program or computer executable instructions are executed by a processor, the steps of the speech recognition method provided in the embodiments of this application are implemented.

[0017] In summary, this application simultaneously acquires audio and image data, and extracts audio and visual features separately to participate in the speech recognition process. Compared to recognition methods based solely on single audio information, this approach can supplement speech recognition with visual features even when audio quality is compromised, thereby improving the overall accuracy of speech recognition. By introducing vehicle status data provided by the vehicle bus and determining the fusion weights of audio and visual features based on this data, the speech recognition process can adaptively adjust according to the vehicle's operating status, avoiding the insufficient adaptability issues associated with fixed fusion strategies. Furthermore, by introducing fusion weights in the feature fusion stage, the fusion of audio and visual features is optimized. Weighted fusion of features allows multimodal features to play their respective roles more rationally during the fusion process, thereby improving the effectiveness and expressive power of the fused feature representation. Since the generated fused feature representation comprehensively considers audio, visual, and vehicle status information, other modal features can still participate in the recognition process even when the quality of one type of input data deteriorates, improving the stability and reliability of the speech recognition output. This speech recognition method is applied to the target vehicle and incorporates vehicle status data into the speech recognition process, creating a collaborative relationship between the speech recognition process and the actual vehicle operating state, enhancing the relevance and practicality of the speech recognition method in in-vehicle applications. In summary, the speech recognition method provided in this application, by fusing audio features, visual features, and vehicle status data, and adaptively adjusting the fusion weights of multimodal features based on the vehicle's operating state, can achieve dynamic optimization of the speech recognition process in in-vehicle scenarios, thereby improving the accuracy, stability, and practical application adaptability of speech recognition. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a speech recognition method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the composition structure of a speech recognition device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The terms used in the specification, claims, and drawings of this application, such as "first," "second," "third," "fourth," etc. (if any), are used to distinguish similar objects and not to describe a specific order or sequence. Therefore, it is to be understood that these terms can be used interchangeably where appropriate, allowing the described embodiments to be used in different orders, unless specifically required by the illustrations or description. Furthermore, the terms "is" and "has," and any variations thereof, are intended to cover, non-exclusively, all possible constituent elements. For example, a process, method, system, product, or apparatus comprising several steps or units is not necessarily limited to the steps or units explicitly listed, but may also include other steps or units not explicitly listed, or steps or units inherent to the process, method, product, or apparatus.

[0020] In this application, a "module" or "unit" refers to a computer program or part of a computer program that has a specific function and works in conjunction with other related parts to achieve a predetermined goal. These modules or units can be implemented by software, hardware (e.g., processing circuitry or memory), or a combination of both. One or more processors or memories can implement one or more modules or units. Furthermore, each module or unit can also be part of a larger module or unit.

[0021] The technical solutions of this application will be described in detail below with reference to the accompanying drawings of the embodiments. It should be noted that the described embodiments are only a part of this application, and not all embodiments. In the following description, the "some embodiments" mentioned are only a subset of all possible embodiments, which may be the same or different subsets, and different embodiments can be combined with each other without conflict.

[0022] Figure 1 This is a schematic flowchart of a speech recognition method provided in an embodiment of this application. For example, see [link to example]. Figure 1The speech recognition method provided in this application is applied to a target vehicle, which is the application carrier of the speech recognition method provided in this application. The target vehicle can be a motor vehicle equipped with an audio acquisition device, a visual acquisition device, and a vehicle bus system, and possessing intelligent cockpit voice interaction functionality. The speech recognition method provided in this application may include the following steps 101 to 105: Step 101: Acquire audio data from the audio acquisition device, image data from the vision acquisition device, and vehicle status data from the vehicle bus.

[0023] In some examples, the audio acquisition device is a multi-channel sound pickup device adapted for in-vehicle scenarios. It can be an in-vehicle microphone array, which can be installed in a convenient location to capture the driver's voice, such as the front of the roof, near the steering wheel, or the cabin pillars, and can comprehensively pick up voice-related signals in the vehicle. The audio data is the unprocessed acoustic signal collected by the audio acquisition device, which includes the driver's target voice command, in-vehicle environmental noise (such as wind noise, tire noise, air conditioning operation sound), and other acoustic information. The microphone unit of the audio acquisition device can convert sound wave vibrations into electrical signals, and then perform analog-to-digital conversion to obtain a digital audio stream. For example, when the driver says "open the sunroof," the microphone array collects a mixed digital audio stream containing the command voice and the simultaneously existing high-speed wind noise.

[0024] A visual acquisition device is an in-vehicle imaging device used to capture images of the face and lips of occupants inside the vehicle. It can be a Driver Monitoring System (DMS) or a dedicated cockpit camera. Its installation location can be above the dashboard, behind the steering wheel, or in the rearview mirror area to ensure clear capture of the driver's facial and lip movements. For example, a high-definition cockpit camera mounted above the dashboard can continuously capture changes in the driver's facial expressions and lip movements. The image data consists of a sequence of dynamic images of the driver's face and lips generated by the visual acquisition device. It records visual information such as the driver's facial posture, lip contours, and lip movement trajectories during voice interaction. The visual acquisition device can continuously capture images at a preset frame rate (e.g., 30 frames per second) to form a temporally coherent set of digital image frames. For example, when the driver speaks, the cockpit camera captures a continuous sequence of image frames including lip opening and closing and mouth corner movements.

[0025] The vehicle bus is an onboard data communication bus used to transmit status information of various vehicle systems. It can be a Controller Area Network (CAN) bus or a Local Interconnect Network (LIN) bus, serving as a data transmission channel between various electronic control units (ECUs) in the vehicle. For example, the CAN bus can be used to acquire operational status data transmitted by the vehicle's engine control unit and body control unit. Vehicle status data is a set of parameters reflecting the current physical operating status and equipment working status of the vehicle, transmitted via the vehicle bus. This data can include key parameters related to the vehicle's environmental noise, such as vehicle speed, window opening / closing status, and air conditioning fan speed. These physical signals collected by various vehicle sensors (such as vehicle speed sensors and window position sensors) are converted into digital signals, processed by the electronic control unit, and transmitted in real time via the vehicle bus, read by the system through the bus interface. For example, vehicle status data could include a vehicle speed of 65 km / h, the half-open window position, and the air conditioning fan speed setting (level 3).

[0026] By implementing step 101, audio data, image data, and vehicle status data are acquired simultaneously, enabling the speech recognition process to have a multi-source information input basis. This not only reflects the speaker's voice information and lip movements but also senses the current operating status of the vehicle, providing more comprehensive and realistic environmental information support for subsequent feature processing and fusion.

[0027] Step 102: Extract features from the audio data and image data respectively to obtain audio features and visual features.

[0028] In some examples, feature extraction involves using pre-defined algorithms to filter, transform, and refine raw audio and image data based on their characteristics. This process removes redundant information and noise interference, extracting structured data that characterizes the core attributes of speech and key visual information. This transforms unstructured raw data into effective features that can be processed by a computer, supporting subsequent fusion and recognition. For example, it may include using acoustic feature extraction algorithms to extract spectral features from audio data and using computer vision algorithms to extract lip dynamic features from image data. Audio features are a set of structured parameters extracted from audio data that reflect the essential attributes of speech. These can include key information such as frequency distribution, energy changes, and formant positions, serving as the core basis for characterizing speech content. Visual features are a set of key information extracted from image data that characterizes the lip movement state of the target speaker. Focusing on lip dynamic features directly related to speech pronunciation, they are an important support for improving recognition accuracy in complex environments by assisting audio features.

[0029] By implementing step 102, feature extraction is performed on the audio data and image data respectively, and the original signal is transformed into structured audio features and visual features. This can effectively reduce the redundancy and complexity of the original data, highlight the key information related to speech recognition, and facilitate subsequent efficient and stable multimodal fusion and recognition processing.

[0030] Step 103: Determine the fusion weights of audio features and visual features based on vehicle status data.

[0031] In some examples, fusion weights are numerical parameter combinations used to characterize the relative importance of audio and visual features when fusing them. These weights are used to dynamically adjust the contribution of the two types of features to the final recognition result based on the actual conditions of the in-vehicle environment, ensuring that the fused features accurately reflect the core information of the voice commands. The audio feature weights and visual feature weights can be calculated based on the environmental noise level represented by vehicle state data, through preset rules or a trained model, and the sum of the two weights is 1. For example, the combination of audio feature weight 0.8 and visual feature weight 0.2, and the combination of audio feature weight 0.4 and visual feature weight 0.6, correspond to the feature importance allocation under different noise scenarios.

[0032] Bioengineering based on vehicle status data to determine the fusion weights of audio and visual features involves analyzing the vehicle's driving status and in-vehicle environmental conditions reflected in the vehicle status data. This allows for the assessment of the degree of interference from the current environment on the audio data, and then the targeted allocation of fusion weights between the two types of features. The core logic is that vehicle status data is directly related to environmental noise intensity; the stronger the noise, the lower the reliability of the audio features, requiring a higher weight for visual features to compensate for the shortcomings of audio features. Conversely, the role of audio features is emphasized when the noise level is low. For example, if vehicle status data shows a speed of 85 km / h and all windows are open, it is considered a high-noise scene, and a fusion weight combination with a higher weight for visual features than audio features is determined. Conversely, if vehicle status data shows a speed of 25 km / h and all windows are closed, it is considered a low-noise scene, and a fusion weight combination with a higher weight for audio features than visual features is determined.

[0033] By implementing step 103, the fusion weights of audio features and visual features are determined based on vehicle status data, enabling the speech recognition process to be dynamically adjusted in combination with the vehicle's operating status. This avoids the problem of poor recognition performance in different vehicle scenarios caused by using a fixed fusion ratio, and improves the adaptability of the speech recognition method to the actual use environment.

[0034] Step 104: Based on the fusion weights, the audio features and visual features are fused to generate a fused feature representation.

[0035] In some examples, the fusion feature representation is a structured feature data that integrates and optimizes audio features and visual features according to preset fusion weights. This data comprehensively reflects the core information of the voice command and the dynamic correlation information of the lips. It can take into account the advantages of both types of features and offset the limitations of a single feature in a complex in-vehicle environment, providing more comprehensive and reliable feature support for subsequent speech recognition. Based on the fusion weights determined in step 103, audio features and visual features can be processed through weighted combination and feature enhancement algorithms to finally generate feature data with unified dimensions and complementary information. For example, it can include a fusion feature vector sequence with a dimension of T×208 (where T is the number of time frames). This sequence contains both the frequency and energy information of the audio features and integrates the lip movement information of the visual features, which can accurately represent the comprehensive features of voice commands such as "adjust the air conditioning temperature".

[0036] By implementing step 104, the fusion weights are used to weight and fuse audio features and visual features, enabling different modal features to play a more reasonable role in the fusion process, forming a fusion feature representation with complementary information. This can improve the completeness and effectiveness of feature expression and provide a more reliable input basis for speech recognition.

[0037] Step 105: Perform feature parsing on the fused feature representation to obtain the target text recognition result.

[0038] In some examples, feature parsing involves semantic extraction, temporal correlation, and decoding of the fused feature representation, transforming the structured fused features into human-understandable natural language text. This process can uncover the semantic information of speech commands contained in the fused features, achieving a precise mapping from feature data to text information. The fused feature representation can be input into a pre-defined end-to-end speech recognition model (E2E-SRM), and through the model's internal logic of temporal modeling, probability calculation, and sequence decoding, the corresponding text sequence can be output. For example, this could include parsing a fused feature representation containing the semantics of "open the sunroof" and obtaining the corresponding text through model decoding; or processing a fused feature representation representing "adjust the seat angle" to extract its core semantics and transform it into text form.

[0039] It should be noted that the preset end-to-end speech recognition model is built on a Transformer Architecture, which consists of three main modules: an encoder, a decoder, and a cross-attention layer. These modules are connected in series through fully connected layers and normalization layers to form an end-to-end semantic decoding chain. The encoder uses a 12-layer stacked encoder block structure, with each block containing a multi-head self-attention mechanism and a feed-forward neural network. The multi-head self-attention mechanism has 8 heads and a hidden layer dimension of 512, used to capture temporal dependencies and key semantic associations in the fused feature representation. The decoder uses a 6-layer decoder block, each layer also integrating a multi-head self-attention mechanism and a cross-attention layer. The cross-attention layer is responsible for associating the encoder output with the decoder's current prediction state, achieving accurate mapping between feature semantics and the text sequence. The feed-forward neural network has a hidden layer dimension of 2048, and the activation function is a Gaussian Error Linear Unit (GaOL). (GELU Units) to improve the model's ability to fit complex semantics in vehicles.Model training should follow these specific steps and parameter settings: First, construct a training dataset specifically for in-vehicle scenarios. The dataset contains 100,000 multimodal samples covering different noise scenarios (high-speed wind noise, open window ventilation, high air conditioning speed, etc.). Each sample consists of a T×208-dimensional fusion feature representation generated by fusing audio features, visual features, and vehicle status data, corresponding labeled text commands (such as "turn on the air conditioning" or "navigate to XX intersection"), and scene labels. The data is divided into training, validation, and test sets in an 8:1:1 ratio. Second, preprocess the training data, including temporal alignment verification of the fusion features, word segmentation and dictionary construction of the labeled text (the dictionary contains 5000 core in-vehicle terms and common expressions), and converting the text annotations to one-hot encoding format. Third, initialize the model using a Xavier normal distribution to initialize all weight parameters, and set the bias parameter to 0. Fourth, start iterative training, setting the batch size to 32, the initial learning rate to 1e-4, and using an adaptive moment estimation optimizer. The parameters are updated using Estimation (Adam), and the learning rate adopts a linear decay strategy. When the character error rate (CER) of the validation set does not decrease for 5 consecutive epochs, the learning rate is halved. The total number of training epochs is set to 100, and the early stopping threshold is set to 10 epochs. In the fifth step, the error between the predicted text and the labeled text is calculated using the cross-entropy loss function during training. At the same time, the sentence error rate (SER) of the validation set is monitored. When the training reaches the preset epoch or the early stopping condition, the model with the lowest SER is saved as the final deployment model. In in-vehicle applications, the model's input strictly matches the fusion feature representation output in step 104. The input data format is a T×208-dimensional temporal feature matrix (T is the number of time frames of the fusion feature, which is positively correlated with the duration of the original voice command). Each element in the matrix corresponds to the multimodal semantic information after weighted fusion and attention enhancement. The model's output is a standardized text sequence for in-vehicle scenarios, with an output dimension of N×V (N is the length of the text sequence, and V is the size of the dictionary). The output is transformed into a word probability distribution through the Softmax function, and finally, the word combination with the highest probability is selected to form the target text recognition result.To meet the real-time requirements of in-vehicle voice interaction, TensorRT is used for inference optimization during model deployment, keeping the inference latency within 200 milliseconds. Furthermore, for the three core semantic scenarios commonly encountered in vehicles—"vehicle control," "navigation services," and "multimedia operation"—a scenario adaptation layer is added to the decoder output layer. This layer standardizes and corrects the output text using pre-set semantic templates, ensuring that the recognition results match the command format of the in-vehicle system. For example, "turn up the air conditioning" is corrected to "increase the air conditioning fan speed by 1 level," ensuring that the vehicle control system can directly parse and execute the commands.

[0040] The target text recognition result is the natural language text that completely corresponds to the driver's original voice command after feature parsing is completed. It is the final output of the voice recognition process and is directly used by the vehicle control system to perform subsequent operations. The semantic content of the voice command can be restored from the fused feature representation through the decoding operation of the end-to-end voice recognition model during feature parsing, forming a standardized text expression. For example, it can be a vehicle control-related text command such as "turn on the air conditioner", "close the windows", or "navigate to XX building", or it can include in-vehicle service text commands such as "check current fuel consumption" or "play XX song".

[0041] For example, the fused feature representation generated in step 104 can first be completely input into a pre-trained end-to-end speech recognition model in time sequence. This model adopts a Transformer Model (TM) architecture, which has powerful temporal feature modeling and semantic decoding capabilities. The model performs multi-layer encoding processing on the input fused feature representation, captures long-term dependencies in the feature sequence through a self-attention mechanism (SAM), and calculates the probability distribution of different words corresponding to each time step by combining a pre-set in-vehicle voice command dictionary. Based on the probability distribution results, the model uses a beam search algorithm (BSA) to select the word combination with the highest probability, forming a text sequence that is semantically consistent with the original voice command. Finally, the text sequence is formatted and standardized to remove redundant characters and semantically ambiguous parts, outputting accurate and standardized target text recognition results, providing a direct basis for the vehicle central control system to perform corresponding operations.

[0042] By implementing step 105, feature parsing of the fused feature representation can be performed, which can comprehensively utilize the influence of audio information, visual information and vehicle status information, reduce the impact of single-modal feature fluctuations on the recognition results, obtain more stable and accurate target text recognition results, and improve the overall speech recognition performance.

[0043] In summary, this application's embodiments, by simultaneously acquiring audio and image data and extracting audio and visual features respectively to participate in the speech recognition process, compared to recognition methods based solely on single audio information, can supplement with visual features even when audio quality is compromised, thereby improving the overall accuracy of speech recognition. By introducing vehicle status data provided by the vehicle bus and determining the fusion weights of audio and visual features based on this vehicle status data, the speech recognition process can adaptively adjust according to the vehicle's operating status, avoiding the insufficient adaptability problem caused by using a fixed fusion strategy. By introducing fusion weights in the feature fusion stage, the audio and visual features are combined... Weighted fusion of visual features allows multimodal features to play their respective roles more rationally during the fusion process, thereby improving the effectiveness and expressive power of the fused feature representation. Since the generated fused feature representation comprehensively considers audio, visual, and vehicle status information, other modal features can still participate in the recognition process even when the quality of one type of input data deteriorates, improving the stability and reliability of the speech recognition output. This speech recognition method is applied to the target vehicle and incorporates vehicle status data into the speech recognition process, creating a collaborative relationship between the speech recognition process and the actual vehicle operating state, enhancing the relevance and practicality of the speech recognition method in in-vehicle application scenarios. In summary, the speech recognition method provided in this application, by fusing audio features, visual features, and vehicle status data, and adaptively adjusting the fusion weights of multimodal features based on the vehicle's operating state, can achieve dynamic optimization of the speech recognition process in in-vehicle scenarios, thereby improving the accuracy, stability, and practical application adaptability of speech recognition.

[0044] In some embodiments, the aforementioned step 103 may include: determining whether the target vehicle is in a preset noise scene based on vehicle state data; if the target vehicle is in a preset noise scene, determining the fusion weight as a first fusion weight configuration, wherein the weight of visual features in the first fusion weight configuration is greater than the weight of audio features; otherwise, determining the fusion weight as a second fusion weight configuration, wherein the weight of visual features in the second fusion weight configuration is greater than the weight of audio features.

[0045] In some examples, the preset noise scenario is a specific driving or usage scenario defined in advance based on the characteristics of the in-vehicle environment, where the intensity of environmental noise can affect the reliability of audio data. Noise in the preset noise scenario will cause a decrease in the recognizability of audio features, requiring the reliance on visual features to improve recognition accuracy. Common high noise sources in in-vehicle scenarios can be combined with experimental testing and scenario summarization to pre-set typical high noise scenarios related to driving status and vehicle equipment operation status. For example, extracting vehicle status data such as vehicle speed of 80 km / h, window opening gap of 8 cm, and air conditioning fan speed of level 5, and comparing them with preset vehicle speed threshold of 60 km / h, gap threshold of 5 cm, and gear threshold of level 3, respectively, the target vehicle is determined to be in the preset noise scenario because all three exceed the standard.

[0046] The first fusion weight configuration is a pre-set weight combination of audio and visual features for a preset noise scenario. Its characteristic is that the weight value of visual features is greater than that of audio features, so as to make up for the defect of audio features being affected by noise through the high reliability of visual features. It can query the pre-stored scene-weight mapping database based on the feature reliability test results under the preset noise scenario, and directly call it according to the scene determination result. For example, it can include a combination of visual feature weight 0.7 and audio feature weight 0.3, or a combination of visual feature weight 0.65 and audio feature weight 0.35. This kind of configuration can ensure that visual features play a dominant role in the fusion process.

[0047] The second fusion weight configuration is a pre-set weight combination of audio and visual features for non-preset noise scenarios. Its characteristic is that the weight value of visual features is greater than that of audio features, while the weight of visual features is lower than that of visual features in the first fusion weight configuration. This is to balance the effectiveness of audio features and the auxiliary role of visual features in scenarios with low environmental noise. It can be verified by multimodal fusion effect in non-preset noise scenarios by querying the pre-stored scene-weight mapping database and automatically matching and calling it when it is determined to be a non-preset noise scenario. For example, it can include a combination of visual feature weight 0.6 and audio feature weight 0.4, or a combination of visual feature weight 0.55 and audio feature weight 0.45, which retains the auxiliary role of visual features and can give full play to the semantic representation advantages of audio features in low-noise environments.

[0048] In preset noise scenarios, audio features suffer from severe interference, necessitating higher weighting for visual features. In non-preset noise scenarios, audio feature reliability is improved, allowing for a slight reduction in the weighting of visual features while maintaining their dominant position, thus maximizing feature advantages across different scenarios. A scenario determination process can be executed first, followed by calling the corresponding first or second fusion weight configuration from the preset weight configuration library based on the determination result, forming a closed-loop scenario-weight matching mechanism. For example, when the vehicle is determined to be in a preset noise scenario with a speed of 90 km / h and all windows open, the system calls the first fusion weight configuration (visual 0.7, audio 0.3); when the vehicle is determined to be in a non-preset noise scenario with a speed of 25 km / h, all windows closed, and air conditioning at level 1, the system calls the second fusion weight configuration (visual 0.6, audio 0.4).

[0049] Through the implementation of the above embodiments, it is possible to determine whether the vehicle is in a preset noise scene based on vehicle status data, and to use different fusion weight configurations under different noise scenes, so that the voice recognition process has a clear scene discrimination capability; increasing the visual feature weight in high noise scenes is beneficial to maintaining a high recognition effect even when the audio quality deteriorates; and reasonably allocating the audio and visual feature weights in non-high noise scenes helps to give full play to the recognition advantage when the audio signal is clear, thereby improving the adaptability of vehicle voice control under different operating conditions.

[0050] In some embodiments, the aforementioned vehicle status data may include at least one of vehicle speed, window status parameters, and air conditioning fan speed level; the aforementioned determination of whether the target vehicle is in a preset noise scenario based on the vehicle status data may include: if at least one of the following is true, the target vehicle is determined to be in a preset noise scenario: the vehicle speed is greater than a preset vehicle speed threshold, the window status parameter indicates that the opening gap is greater than a preset gap threshold, and the air conditioning fan speed level is greater than a preset level threshold.

[0051] In some examples, vehicle speed is the distance traveled by the target vehicle per unit time during its journey. It is a core parameter reflecting the vehicle's driving status, and its value is directly related to the intensity of driving noise such as wind noise and tire noise. The physical signals of the vehicle's movement can be collected by the vehicle speed sensor (VSS), processed by the vehicle's electronic control unit (ECU) and converted into digital signals, and then acquired in real time through the controller area network bus. For example, the vehicle speed can be a specific speed value such as 50 km / h, 75 km / h, or 90 km / h acquired in real time during the vehicle's journey.

[0052] Window status parameters are quantitative data characterizing the degree of opening and closing of vehicle windows. They are used to accurately reflect the opening status of the windows and thus determine the intensity of external wind noise entering the vehicle. Real-time position signals of the window glass can be collected by the Window Position Sensor (WPS), processed by the electronic control unit, and converted into standardized digital parameters, which are then obtained through the controller local area network bus. For example, the parameter value can include 0 when the window is fully closed, 5 when it is half open, and 10 when it is fully open (the parameter value is positively correlated with the degree of opening).

[0053] The air conditioning fan speed setting is the airflow intensity level set by the vehicle's air conditioning system (Heating, Ventilation and Air Conditioning, HVAC). The higher the setting, the greater the noise generated by the air conditioning fan, which is one of the important sources of noise in the vehicle's interior. The air conditioning control unit (ACCU) can collect the fan speed setting information set by the user, directly convert it into a digital signal, and then obtain it through the controller area network bus. For example, it can include the fan speed settings of 1, 3, 5, etc., set by the air conditioning system. Different settings correspond to different fan speeds and airflow intensities.

[0054] The preset vehicle speed threshold is a pre-set critical speed value used to determine whether the driving noise meets the standard. It is one of the reference standards for determining whether the vehicle is in a preset noise scenario. Based on a large number of vehicle noise test experiments, the impact of noise intensity at different vehicle speeds on speech recognition can be statistically analyzed to calibrate the vehicle speed critical value that can significantly affect the reliability of audio features, and store it in the system's threshold configuration library. For example, it can include specific threshold values ​​such as 60 km / h and 70 km / h set after experimental calibration. When the vehicle speed exceeds this value, the driving noise will cause significant interference to audio recognition.

[0055] The window opening gap indicated by the window status parameters is the actual opening width of the window, calculated from the window status parameters. It is a key indicator that directly reflects the degree to which external wind noise enters the vehicle. A mapping relationship between window status parameters and opening gaps can be preset. After receiving the window status parameters, a preset conversion formula or mapping table is used to convert the parameter value into the corresponding actual opening gap length. For example, when the window status parameter is 3, the calculated opening gap is 3 cm; when the parameter is 8, the calculated opening gap is 8 cm. The preset gap threshold is a pre-set critical value for determining whether wind noise interference meets the standard. It serves as the basis for judging preset noise scenarios. By simulating in-vehicle wind noise tests under different window opening gaps and combining data on changes in voice recognition accuracy, the critical gap values ​​that will significantly interfere with audio features can be calibrated and pre-stored in a threshold configuration library. For example, specific thresholds such as 5 cm and 6 cm can be included based on tests. When the window opening gap exceeds this value, external wind noise will seriously affect the reliability of the audio data.

[0056] The preset threshold is a pre-set critical value for the air conditioning noise level to determine whether the noise generated by the in-vehicle equipment will interfere with voice recognition. By testing the in-vehicle noise intensity at different air conditioning fan speed levels, the impact on audio feature extraction can be analyzed, and the critical threshold value can be calibrated and stored in the threshold configuration library. For example, it can include specific thresholds such as level 3 and level 4 determined by experiments. When the air conditioning fan speed exceeds this value, the noise generated by the air conditioning fan will cause significant interference to audio recognition.

[0057] Vehicle speed, window opening gap, and air conditioning fan speed level correspond to the three main sources of vehicle noise: driving noise, wind noise, and equipment noise, respectively. If the noise intensity of any of these sources exceeds the standard (parameters exceeding the corresponding preset thresholds), the ambient noise inside the vehicle will be sufficient to interfere with the reliability of the audio characteristics, and thus it will be determined as a preset noise scenario. For example, the preset vehicle speed threshold is 60 km / h, the preset window opening gap threshold is 5 cm, and the preset fan speed threshold is 3. If the acquired vehicle speed is 75 km / h (greater than 60 km / h), or the window opening gap is 6 cm (greater than 5 cm), or the air conditioning fan speed level is 4 (greater than 3), or any two or three of these conditions are met simultaneously, the target vehicle is determined to be in a preset noise scenario. If the vehicle speed is 50 km / h, the window opening gap is 3 cm, and the air conditioning fan speed level is 2, and none of the three conditions exceed the corresponding thresholds, then it is determined to be a non-preset noise scenario.

[0058] By implementing the above embodiments, vehicle state parameters directly related to noise, such as vehicle speed, window status parameters, and air conditioning fan speed, are incorporated into the noise scene judgment conditions. This allows noise scene recognition to be based on objective and quantifiable vehicle operation data, avoiding reliance on subjective or single indicators for judgment. It can more accurately reflect the vehicle's current real noise environment, thereby providing a reliable basis for the reasonable determination of subsequent fusion weights and further improving the stability and consistency of speech recognition in complex in-vehicle environments.

[0059] In some embodiments, the aforementioned step 102 may include: if a preset wake-up word is detected in the audio data, then obtaining image sub-data corresponding to the time period in which the preset wake-up word is detected; based on the image sub-data, determining whether the lips of the target speaker have performed target speech activity corresponding to the preset wake-up word; if it is determined that the lips have performed target speech activity, then performing feature extraction on the audio data and image data respectively to obtain audio features and visual features.

[0060] In some examples, the preset wake-up word is a specific word or phrase pre-set to activate the vehicle's voice recognition system. It is used to establish the trigger mechanism for driver-vehicle voice interaction and to clarify the target starting point of voice recognition. It can be pre-stored in the wake-up word library of the Automatic Speech Recognition (ASR) engine through the system configuration interface before the vehicle leaves the factory, and can be customized in the vehicle settings according to user needs. For example, it can include "Hello, Smart Driving", "Open Voice Assistant", "Vehicle Manager", etc. Such words need to have clear pronunciation and not be easily confused with everyday language to ensure the accuracy of wake-up.

[0061] The time interval for detecting the preset wake word is a continuous time range from the initial capture of the preset wake word's audio signal to the complete end of the wake word's audio signal. This allows for precise location of the time range corresponding to the wake-up behavior, providing a time reference for subsequent image sub-data extraction. When detecting audio data, the start and end timestamps of the preset wake word being recognized can be recorded, and the interval between the two timestamps is the time interval. For example, it can include a 0.6-second interval corresponding to a start timestamp of 1.2 seconds and an end timestamp of 1.8 seconds, or a 0.8-second interval corresponding to a start timestamp of 3.5 seconds and an end timestamp of 4.3 seconds.

[0062] Image sub-data refers to the portion of the image sequence that corresponds exactly to the time period when the preset wake word is detected from the complete image data acquired by the visual acquisition device. Its characteristics are that the time dimension is synchronized with the wake word audio signal and it only contains visual information when the wake-up behavior occurs. Based on the start and end timestamps of the detected preset wake word, continuous image frames within the corresponding time interval can be extracted from the complete image data stream to form independent image sub-data. For example, it may include 18 consecutive images within a time interval of 1.2 seconds to 1.8 seconds (the frame rate of the visual acquisition device is 30 frames / second). This image sequence completely records the lip dynamics during the pronunciation of the wake word.

[0063] The target speaker is the user who intends to control the vehicle through voice commands. In an in-vehicle scenario, this could be the driver, the initiator of the voice interaction with the vehicle, and the core service target of the voice recognition system. It could also be the speaker located and identified by combining image data collected by a visual acquisition device with facial positioning algorithms and default rules for in-vehicle scenarios (with the driver as the primary interaction target). For example, it could include a person in the driver's seat captured by a cockpit camera, or a registered driver confirmed through facial feature matching.

[0064] Target speech activity refers to the specific dynamic behavior of the lips when a target speaker pronounces a preset wake word, corresponding to the pronunciation action of that wake word. Its core feature is the consistency between the lip movements and the pronunciation rhythm and manner of the wake word, which is a key visual basis for verifying the authenticity of the wake-up behavior. It can be determined whether it matches the standard lip movement features corresponding to the preset wake word by analyzing the lip movement trajectory, opening and closing frequency, and contour changes of the target speaker in the image sub-data. For example, it can include the continuous dynamic of "closing-opening-closing-slight opening and closing" when pronouncing "Hello, Smart Drive", or the lip opening and closing rhythm and contour changes corresponding to pronouncing "Open Voice Assistant".

[0065] If the target speech activity is determined to have occurred in the lips, feature extraction is performed on both the audio and image data to obtain audio and visual features. This process involves dual verification through audio wake-up word detection and visual lip movement verification to eliminate false wake-ups initiated by non-target speakers, such as broadcasts or passenger chatter, ensuring that features are extracted only from genuine driver voice commands, thus improving the accuracy and anti-interference capability of speech recognition. The wake-up word can be detected first through audio recognition, and then lip movements can be verified based on the corresponding image sub-data. Once the dual verification is successful, the feature extraction process begins. For example, if the wake-up word "Hello, Smart Driving" is detected in the audio data, and the image sub-data confirms that the driver's lips have made a corresponding opening and closing movement, then the extraction of audio and visual features is initiated. If only the wake-up word audio is detected but there is no corresponding lip movement, feature extraction is terminated to avoid invalid calculations.

[0066] Through the implementation of the above embodiments, after detecting the preset wake-up word, a visual confirmation mechanism is introduced. By judging whether the target speaker's lips have made corresponding speech movements, it is decided whether to perform feature extraction and recognition processing. This can reduce the problem of false wake-up caused by broadcasts, passenger conversations or environmental sounds. In vehicle control scenarios, it can ensure that voice commands mainly come from the target driver, improve the reliability of voice recognition, and reduce the impact of false triggering on vehicle operation safety.

[0067] In some embodiments, the aforementioned determination of whether the target speaker's lips have performed a target speech activity corresponding to a preset wake-up word based on image sub-data may include: performing feature extraction processing on the image sub-data to obtain a lip region image sequence and gaze direction information; determining whether the target speaker's lips have a periodic movement associated with a preset wake-up word based on the lip region image sequence; if a periodic movement exists, and the gaze direction is determined to be towards a preset area based on the gaze direction information, then determining that the lips have performed a target speech activity corresponding to the preset wake-up word, wherein the preset area may include the vehicle center console area and the road area.

[0068] In some examples, feature extraction is a process of using computer vision algorithms (CVA) to filter, separate, and extract key visual information related to the determination of target speech activity for image sub-data. This process can accurately separate dynamic data of the lip region and gaze direction from image sub-data, providing a valid basis for subsequent judgment. For example, key points of the lip contour can be extracted using a convolutional neural network (CNN), and eye rotation angle and gaze vector can be extracted using eye feature analysis algorithms.

[0069] A lip region image sequence is a collection of consecutive image frames extracted from image sub-data that contains only the lips and surrounding area of ​​the target speaker. The lip region image sequence is temporally coherent and focuses on lip dynamics, fully reflecting the lip movement process during the pronunciation of the wake word. First, the facial region of the target speaker can be located using a Face Detection Model (FDM). Then, the Lip Segmentation Algorithm (LSA) can be used to crop images of the lips and surrounding area within a preset range from the facial region, and arranged in chronological order to form a sequence. For example, it can include 24 consecutive lip images cropped from image sub-data from 2.1 seconds to 2.9 seconds (with a visual acquisition device frame rate of 30 frames / second), with each image focusing only on the lips and the surrounding area of ​​the upper and lower jaws.

[0070] Gaze direction information is quantitative data characterizing the direction of a target speaker's eye gaze, including parameters such as gaze vector and gaze angle, used to determine whether the target speaker intends to actively interact with the vehicle. The coordinates of key points such as the center of the pupil and the corner of the eye can be extracted using the Eye Keypoint Detection Algorithm (EKDA), and combined with facial pose parameters to calculate the quantitative data related to gaze direction. For example, gaze direction information may include specific quantitative results such as the angle between the gaze vector and the vehicle's central control area being 25 degrees, and the gaze angle pointing towards the road area within the windshield's field of vision.

[0071] The periodic movements associated with the preset wake word are the regular opening and closing and deformation movements of the lips when the target speaker pronounces the preset wake word, which match the rhythm and manner of pronunciation of the wake word. The core feature is that the movements have a fixed period and sequence pattern, and are highly synchronized with the pronunciation logic of the preset wake word. A large amount of lip movement data when users pronounce the wake word can be collected in advance to establish a standard movement template library. The image sequence of the lip region to be detected is compared with the template library. If the matching degree reaches a preset threshold, it is determined that the periodic movement exists. For example, it can include the periodic movement of the lips "closing-rapid opening-slow closing-slight opening and closing" when pronouncing the preset wake word "Hello, Smart Drive", and its opening and closing period is completely synchronized with the pronunciation rhythm.

[0072] By comparing the lip region image sequence with the standard lip action template corresponding to the preset wake word, it can be analyzed whether the periodicity and sequence of lip actions are consistent with the template. First, the lip region image sequence can be processed by motion quantification to extract feature parameters such as lip opening and closing frequency and contour change trajectory. Then, the similarity is calculated with the feature parameters of the standard action template. If the similarity exceeds the preset matching threshold, it is determined that the periodic action exists. For example, if the extracted lip action feature parameters are compared with the standard template corresponding to "open voice assistant", and the similarity reaches more than 85%, it is determined that there is a periodic action of the lips associated with the wake word.

[0073] The preset area is a pre-defined gaze area based on the in-vehicle voice interaction scenario, representing the intended interaction by the target speaker (driver). It is used to filter valid interaction behaviors by gaze direction, eliminating non-initiated wake-up interference. It can be determined and pre-stored in the system before the vehicle leaves the factory, taking into account the driver's driving posture and in-vehicle interaction habits, and can also be fine-tuned according to vehicle model design. For example, it can include the vehicle's central control area and the road area, both of which together constitute the driver's main gaze range during normal driving or interaction. The vehicle's central control area is the core interaction area of ​​the center console and surrounding area in the vehicle's cockpit, and is the main gaze area for the driver to operate vehicle functions and initiate voice interaction. The road area is the driving road and surrounding environment area within the driver's field of vision of the windshield during vehicle movement, and is the core gaze area for the driver during normal driving.

[0074] The system employs a dual verification mechanism, combining lip movement periodicity matching with gaze direction compliance, to ensure that the target voice activity is a valid action initiated by the driver, thus eliminating interference factors such as passenger lip movements or unintentional lip movements by the driver. The system first determines the lip movement periodicity, then determines the gaze direction based on gaze direction information. Only when both determinations are "yes" is the conclusion "target voice activity has occurred" output. For example, if a lip movement matches the standard periodic movement of the preset wake-up word "Vehicle Butler," and the gaze direction is pointing towards the vehicle's central control area (gaze angle less than 30 degrees), then the corresponding target voice activity is determined to have occurred. If only the lip movement matches but the gaze is directed towards a non-road area outside the vehicle, then no target voice activity is determined to have occurred.

[0075] By implementing the above embodiments, combining the periodic movement features in the lip region image sequence with gaze direction information, the target voice activity is further confirmed, making the visual judgment more consistent with the interactive behavior in real driving scenarios. Only when a lip movement related to the wake word is detected and the gaze is directed towards the central control area or the road area is a valid voice activity confirmed. This can exclude non-target voice behaviors such as head-turning conversations and rear-seat conversations, thereby further improving the accuracy and safety of voice recognition in vehicle control scenarios.

[0076] In some embodiments, the aforementioned feature extraction of audio data and image data to obtain audio features and visual features may include: performing frame segmentation on the audio data to obtain multiple audio frames; converting the multiple audio frames into multiple corresponding acoustic feature vectors based on a preset acoustic feature extraction algorithm; arranging the multiple acoustic feature vectors in chronological order to form audio features; determining the image sequence of the lip region of the target speaker based on image data; extracting features from the lip region image sequence to obtain lip geometric features, wherein the lip geometric features may include lip contour, lip opening degree, and lip opening frequency; and encoding and fusing the lip contour, lip opening degree, and lip opening frequency to obtain visual features.

[0077] In some examples, frame segmentation is a preprocessing operation that cuts continuous audio data into multiple short-duration continuous segments according to a preset time length and overlap ratio. This transforms long-duration audio that cannot be directly processed into short, independently analyzable units, adapting to the algorithm requirements for acoustic feature extraction. A sliding window technique can be used to extract audio data with a fixed frame length, while setting a reasonable frame shift to ensure the continuity of information between frames. For example, this could include sliding segmentation of audio data with a frame length of 20 milliseconds and a frame shift of 10 milliseconds, achieving frame segmentation without information loss. Multiple audio frames are the multiple short-duration audio segments obtained after frame segmentation. Each segment contains acoustic information within a specific time interval. The core features are fixed duration and reasonable overlap between frames, completely covering all information in the original audio data. The number of frames can be directly output through frame segmentation, determined by the total duration of the original audio data, the frame length, and the frame shift. For example, this could include processing 3 seconds of audio data with a frame length of 20 milliseconds and a frame shift of 10 milliseconds to obtain 299 consecutive audio frames.

[0078] The preset acoustic feature extraction algorithms are a set of algorithms pre-stored in the system to transform audio frames into structured data representing the essential acoustic properties. They can convert the physical characteristics of audio signals into digital features that can be processed by a computer. Mature acoustic feature extraction algorithms can be selected and pre-stored based on the needs of in-vehicle voice recognition scenarios before the vehicle leaves the factory. For example, preset acoustic feature extraction algorithms may include Mel Frequency Cepstral Coefficients (MFCC) and Filter Bank Features (FBank). Multiple acoustic feature vectors are high-dimensional digital vectors corresponding to each audio frame obtained after processing each audio frame using the preset acoustic feature extraction algorithms. Each acoustic feature vector contains key acoustic information such as the frequency distribution and energy changes of that audio frame. Multiple audio frames can be input one by one into the preset acoustic feature extraction algorithms, and the corresponding digital vectors are output after processing. For example, the filter bank feature algorithm can process each 20-millisecond audio frame to obtain an 80-dimensional acoustic feature vector, and 299 audio frames can output 299 80-dimensional acoustic feature vectors.

[0079] The process of arranging multiple acoustic feature vectors in chronological order to form audio features involves maintaining the temporal correspondence between the multiple acoustic feature vectors and the original audio frames, and sequentially concatenating the feature vectors according to the order of the audio frames to form a temporally coherent feature set (audio features). It can record the timestamp of the audio frame corresponding to each acoustic feature vector in the original audio data, and arrange and combine the feature vectors in ascending order of timestamps to form a two-dimensional feature matrix. For example, it can include arranging 299 80-dimensional acoustic feature vectors in chronological order to form an audio feature matrix with a dimension of 299×80, thus completely preserving the temporal acoustic information of the original speech.

[0080] A lip region image sequence is a collection of continuous image frames selected from complete image data, containing only the lips of the target speaker and a predetermined surrounding area. It can fully reflect the lip movement trajectory during the speaker's pronunciation process. The facial region of the target speaker can be located from each frame of image data using face detection and lip segmentation algorithms. The lips and the predetermined surrounding area are then cropped and arranged in chronological order to form a sequence. The sequence can be formed by first using a Face Detection Model (FDM) to locate the target speaker's face in each frame, then using a Lip Segmentation Algorithm (LSA) to accurately segment the lips and the predetermined surrounding area (e.g., 10 pixels above and below the lips, and 15 pixels to the left and right). Finally, all lip images from all frames are arranged in chronological order. For example, for image data containing the driver, passengers, and the vehicle's interior environment, the driver's lip region image can be extracted frame by frame using the above algorithm, ultimately forming a 90-frame continuous lip region image sequence.

[0081] Lip geometric features are a set of quantitative parameters extracted from lip region image sequences, representing the shape and movement of the lips. They reflect lip pronunciation actions and provide core data support for visual features. Image feature extraction algorithms can be used to analyze lip region image sequences to extract key quantitative information such as lip contour, degree of opening and closing, and movement frequency. For example, this can include the coordinates of key points on the lip contour, the numerical value of lip opening and closing, and statistical results of lip opening and closing frequency. The lip contour is a set of key point coordinates representing the lip edge shape, accurately reflecting changes in lip shape and directly related to pronunciation actions. Lip keypoint detection algorithms (LKDA) can be used to process lip region images, identifying and extracting the pixel coordinates of key points such as the upper lip edge, lower lip edge, and corners of the mouth. For example, this can include extracting the coordinates of 68 key lip points, which together constitute the complete lip contour, showing the dynamic changes in contour coordinates corresponding to different pronunciation actions. Lip opening degree is a parameter that quantitatively represents the distance between the upper and lower lips. Its value changes synchronously with the opening and closing movements of the lips during pronunciation, reflecting the force and amplitude of the sound. It can be obtained by calculating the straight-line distance between key points on the upper and lower lips in the lip contour, thus determining the degree of lip opening degree in each frame of the image. For example, in one frame, the distance between the key points on the upper and lower lips might be 5 mm, corresponding to a slightly open lip state; in another frame, the distance might be 12 mm, corresponding to a significantly open lip state. Lip opening frequency is the number of times the lips complete an "open-close" movement per unit of time. Frequency changes are related to the rhythm of pronunciation, reflecting the speed and rhythm of the sound. It can be analyzed chronologically to count the number of complete cycles of lip opening degree from minimum to maximum and back to minimum per unit of time. For example, in a 1-second lip region image sequence, if the lips complete 5 complete "open-close" movements, the corresponding lip opening frequency is 5 times / second.

[0082] The process of extracting features from a sequence of lip region images to obtain lip geometric features involves analyzing consecutive lip region image frames frame by frame using computer vision algorithms to extract and quantify key information such as lip contour, opening degree, and opening frequency. This can be achieved by first preprocessing the lip region image sequence (e.g., grayscale conversion, noise reduction), then using a lip keypoint detection algorithm to extract the lip contour coordinates of each frame, calculating the distance between key points on the upper and lower lips to obtain the opening degree, and counting the number of opening cycles per unit time to obtain the opening frequency. These are then integrated to form the lip geometric features. For example, processing a 90-frame lip region image sequence can extract 68 contour keypoint coordinates and 1 opening degree value from each frame, and calculating an opening frequency of 3 times / second, which together constitute the lip geometric features.

[0083] The process of encoding and fusing lip contour, lip opening degree, and lip opening frequency to obtain visual features involves transforming three separate types of lip geometric features—lip contour, lip opening degree, and lip opening frequency—into structured feature data with unified dimensions and information fusion through an encoding algorithm. This integrates multi-dimensional visual information to form visual features that comprehensively represent lip articulation movements. A Feature Encoding Algorithm (FEA) can be used to standardize the three types of geometric features, converting the contour coordinate sequence, lip opening degree numerical sequence, and lip opening frequency statistical values ​​into vectors of fixed dimensions. These vectors are then integrated into a single feature vector using a fusion algorithm. For example, encoding the lip contour coordinate sequence into a 64-dimensional vector, the lip opening degree numerical sequence into a 32-dimensional vector, and the lip opening frequency into a 32-dimensional vector, after processing by the fusion algorithm, yields a 128-dimensional visual feature vector.

[0084] Through the implementation of the above embodiments, targeted feature extraction methods are used for audio data and image data respectively, transforming continuous speech signals and lip movement information into structured, temporal audio features and visual features, so that multimodal information can maintain consistency in the time dimension; by extracting lip geometric features and encoding and fusing them, the visual features can accurately reflect the changes in mouth shape during the speaking process, and can provide an effective supplement to speech recognition when the audio is interfered with, thereby improving the overall recognition effect.

[0085] In some embodiments, step 104 may include: weighting and combining audio features and visual features based on fusion weights to obtain preliminary fusion features; and performing feature enhancement processing on the preliminary fusion features through a preset attention mechanism network to obtain a fusion feature representation.

[0086] In some examples, the initial fusion feature is the intermediate feature data obtained by weighting and combining audio features and visual features according to preset fusion weights. It integrates the basic information of the two types of features and retains their core representation content, but has not yet undergone key information enhancement and redundancy interference suppression processing. The audio features and visual features can first be unified in dimensions and aligned in time sequence, and then the two types of features can be weighted and summed element by element according to the fusion weights to output intermediate feature data. For example, it can include an intermediate feature matrix with T×208 dimensions (T is the number of time frames) obtained by multiplying the unified audio feature vector and visual feature vector by their corresponding weights and then summing them element by element when the audio feature weight is 0.4 and the visual feature weight is 0.6. This matrix contains both audio frequency information and visual lip movement information. First, the audio and visual features can be adjusted to the same dimension using the Feature Dimensionality Unification Algorithm (FDUA) (e.g., unifying audio feature T×80 and visual feature T×128 to T×208). Then, a frame-by-frame, element-by-element weighted summation operation is performed on the two types of features after time alignment (formula: preliminary fused feature = audio feature × audio weight + visual feature × visual weight). For example, when the fusion weight is configured as 0.8 for audio features and 0.2 for visual features, the audio feature vector (80 dimensions) and visual feature vector (128 dimensions) of each frame are unified to 208 dimensions, multiplied by 0.8 and 0.2 respectively, and then added element-by-element to obtain the preliminary fused feature vector corresponding to each frame. All frames are arranged in chronological order to form the preliminary fused feature matrix.

[0087] The pre-trained attention mechanism network is a deep learning network that is pre-trained and stored in the system. It is used to enhance key information and suppress redundant interference of the initial fused features. It can automatically identify and focus on feature regions that are strongly related to the semantics of voice commands, weaken the influence of noise or irrelevant information, and improve the recognition and reliability of features. Before the vehicle leaves the factory, the model can be trained based on massive in-vehicle multimodal voice data to select attention mechanism networks suitable for in-vehicle scenarios and pre-store them in the computing module of the speech recognition system. For example, the pre-trained attention mechanism network can include self-attention mechanism networks and multi-head attention mechanism networks based on transformer architecture, etc., with the attention mechanism network (AMN) as the core to perform feature enhancement operations.

[0088] The process of enhancing the initial fused features through a pre-defined attention mechanism network to obtain the fused feature representation involves inputting the initial fused features into the network. Through attention weight calculation and feature interaction operations within the network, key semantic features are strengthened and invalid interference features are suppressed, ultimately outputting high-quality structured features. This approach leverages the "selective focusing" capability of the attention mechanism to improve the semantic representation accuracy of the fused features. A temporally aligned initial fused feature matrix can be input into the network. The network generates attention weights by calculating the correlation between features, and then uses these weights to weight and enhance the initial fused features, highlighting effective information related to speech commands and weakening redundant information caused by noise, ultimately outputting the final fused feature representation. For example, if the initial fused features contain audio information interfered by wind noise and invalid lip background information, after processing by the attention mechanism network, the audio frequency features and lip opening / closing features related to the semantics of "adjusting the air conditioner temperature" are strengthened, while invalid features corresponding to wind noise and background features around the lips are suppressed, ultimately obtaining a fused feature representation that focuses on the core semantics.

[0089] Through the implementation of the above embodiments, based on the weighted fusion of audio and visual features, an attention mechanism network is introduced to enhance the fused features, enabling the speech recognition model to pay more attention to the feature information that contributes more to the current recognition task; in complex vehicle environments, it can further strengthen key information, suppress noise interference, improve the discriminative ability of the fused feature representation, and enhance the accuracy and stability of speech recognition in vehicle control applications.

[0090] Furthermore, as an implementation of the foregoing method embodiments, this application also provides a speech recognition device for implementing the foregoing method embodiments. This device embodiment corresponds to the foregoing method embodiments. For ease of reading, this speech recognition device embodiment will not repeat the details of the foregoing method embodiments one by one, but it should be understood that the device in this application embodiment can correspondingly implement all the contents of the foregoing method embodiments. For example... Figure 2As shown, the speech recognition device 20 includes: a data acquisition unit 201, a feature extraction unit 202, a weight determination unit 203, a feature fusion unit 204, and a result determination unit 205. The data acquisition unit 201 acquires audio data from an audio acquisition device, image data from a visual acquisition device, and vehicle status data from a vehicle bus. The feature extraction unit 202 extracts features from the audio data and image data respectively to obtain audio features and visual features. The weight determination unit 203 determines the fusion weights of the audio features and visual features based on the vehicle status data. The feature fusion unit 204 fuses the audio features and visual features based on the fusion weights to generate a fused feature representation. The result determination unit 205 performs feature parsing on the fused feature representation to obtain the target text recognition result.

[0091] In some embodiments, the weight determination unit 203 is further configured to determine whether the target vehicle is in a preset noise scene based on vehicle state data; if the target vehicle is in a preset noise scene, the fusion weight is determined to be a first fusion weight configuration, wherein the weight of visual features in the first fusion weight configuration is greater than the weight of audio features; otherwise, the fusion weight is determined to be a second fusion weight configuration, wherein the weight of visual features in the second fusion weight configuration is greater than the weight of audio features.

[0092] In some embodiments, the vehicle status data includes at least one of vehicle speed, window status parameters, and air conditioning fan speed level; the weight determination unit 203 is further configured to determine that the target vehicle is in a preset noise scene if at least one of the following is true: the vehicle speed is greater than a preset vehicle speed threshold, the window status parameters indicate that the opening gap is greater than a preset gap threshold, and the air conditioning fan speed level is greater than a preset level threshold.

[0093] In some embodiments, the feature extraction unit 202 is further configured to, if a preset wake-up word is detected in the audio data, acquire image sub-data corresponding to the time period in which the preset wake-up word is detected in the image data; based on the image sub-data, determine whether the lips of the target speaker have performed target speech activity corresponding to the preset wake-up word; if it is determined that the lips have performed target speech activity, perform feature extraction on the audio data and image data respectively to obtain audio features and visual features.

[0094] In some embodiments, the feature extraction unit 202 is further configured to perform feature extraction processing on the image sub-data to obtain a lip region image sequence and gaze direction information; based on the lip region image sequence, determine whether there is a periodic movement of the target speaker's lips associated with a preset wake-up word; if there is a periodic movement, and the gaze direction is determined to be towards a preset area based on the gaze direction information, then determine that the lips have performed a target speech activity corresponding to the preset wake-up word, wherein the preset area includes the vehicle center console area and the road area.

[0095] In some embodiments, the feature extraction unit 202 is further configured to perform frame-segmentation processing on the audio data to obtain multiple audio frames; convert the multiple audio frames into multiple corresponding acoustic feature vectors based on a preset acoustic feature extraction algorithm; arrange the multiple acoustic feature vectors in chronological order to form audio features; determine the image sequence of the lip region of the target speaker based on image data; perform feature extraction on the lip region image sequence to obtain lip geometric features, wherein the lip geometric features include lip contour, lip opening degree and lip opening frequency; encode and fuse the lip contour, lip opening degree and lip opening frequency to obtain visual features.

[0096] In some embodiments, the feature fusion unit 204 is further configured to perform weighted combination of audio features and visual features based on fusion weights to obtain preliminary fused features; and to perform feature enhancement processing on the preliminary fused features through a preset attention mechanism network to obtain a fused feature representation.

[0097] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, will cause the processor to perform any step of the speech recognition method provided in this application.

[0098] In some embodiments, the computer-readable storage medium may be a random access memory (RAM), a read-only memory (ROM), flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); or it may be a variety of devices that include one or any combination of the above-mentioned memories.

[0099] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0100] In some embodiments, computer-executable instructions may, but do not necessarily, correspond to files in a file system, and may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0101] In some embodiments, computer-executable instructions may be deployed to execute on an electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0102] like Figure 3 As shown, this application also provides an electronic device 30, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, it implements any step of the above-described speech recognition method.

[0103] This application also provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform any step of the speech recognition method described above.

[0104] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A voice recognition method, characterized by, The voice recognition method is applied to a target vehicle and comprises the following steps: obtaining audio data from an audio acquisition device, image data from a visual acquisition device, and vehicle state data from a vehicle bus; performing feature extraction on the audio data and the image data respectively to obtain audio features and visual features; determining a fusion weight of the audio features and the visual features based on the vehicle state data; fusing the audio features and the visual features based on the fusion weight to generate a fusion feature representation; performing feature analysis on the fusion feature representation to obtain a target text recognition result.

2. The voice recognition method of claim 1, wherein, The step of determining the fusion weight of the audio features and the visual features based on the vehicle state data comprises the following steps: determining whether the target vehicle is in a preset noise scene based on the vehicle state data; if the target vehicle is in the preset noise scene, determining the fusion weight as a first fusion weight configuration, wherein the weight of the visual features is greater than the weight of the audio features in the first fusion weight configuration; otherwise, determining the fusion weight as a second fusion weight configuration, wherein the weight of the visual features is greater than the weight of the audio features in the second fusion weight configuration.

3. The voice recognition method of claim 2, wherein, The vehicle state data comprises at least one of a vehicle speed, a vehicle window state parameter, and an air conditioner air volume gear position; The step of determining whether the target vehicle is in a preset noise scene based on the vehicle state data comprises the following steps: if at least one of the following conditions is met, it is determined that the target vehicle is in a preset noise scene: the vehicle speed is greater than a preset vehicle speed threshold, the opening gap indicated by the vehicle window state parameter is greater than a preset gap threshold, and the air conditioner air volume gear position is greater than a preset gear threshold.

4. The voice recognition method of claim 1, wherein, The step of performing feature extraction on the audio data and the image data respectively to obtain audio features and visual features comprises the following steps: if a preset wake-up word is detected in the audio data, obtaining image sub-data corresponding to a time period in which the preset wake-up word is detected from the image data; determining whether the lips of a target speaker have a target speech activity corresponding to the preset wake-up word based on the image sub-data; if it is determined that the lips have the target speech activity, performing feature extraction on the audio data and the image data respectively to obtain audio features and visual features.

5. The voice recognition method of claim 4, wherein, The step of determining whether the lips of a target speaker have a target speech activity corresponding to the preset wake-up word based on the image sub-data comprises the following steps: performing feature extraction processing on the image sub-data to obtain a lip region image sequence and a gaze direction information; determining whether the lips of a target speaker have a periodic action associated with the preset wake-up word based on the lip region image sequence; if the periodic action exists and the gaze direction is determined to be towards a preset area based on the gaze direction information, it is determined that the lips have a target speech activity corresponding to the preset wake-up word, wherein the preset area comprises a vehicle center control area and a road area.

6. The speech recognition method of any one of claims 1 to 5, characterized in that, The step of performing feature extraction on the audio data and the image data respectively to obtain audio features and visual features comprises the following steps: The audio data is segmented into frames to obtain multiple audio frames; Based on a preset acoustic feature extraction algorithm, the multiple audio frames are converted into multiple corresponding acoustic feature vectors; The audio features are formed by arranging the multiple acoustic feature vectors in chronological order. Based on the image data, a sequence of images of the lip region of the target speaker is determined; Feature extraction is performed on the image sequence of the lip region to obtain lip geometric features, wherein the lip geometric features include lip contour, lip opening degree, and lip opening frequency; The visual features are obtained by encoding and fusing the lip contour, lip opening degree, and lip opening frequency.

7. The speech recognition method of any one of claims 1 to 5, characterized in that, The step of fusing the audio features and the visual features based on the fusion weights to generate a fused feature representation includes: Based on the fusion weights, the audio features and the visual features are weighted and combined to obtain preliminary fusion features; The initial fused features are enhanced by a preset attention mechanism network to obtain the fused feature representation.

8. A speech recognition apparatus characterized by comprising: Applied to the target vehicle, the voice recognition device includes: The data acquisition unit is used to acquire audio data from the audio acquisition device, image data from the vision acquisition device, and vehicle status data from the vehicle bus. The feature extraction unit is used to extract features from the audio data and the image data respectively to obtain audio features and visual features; The weight determination unit is used to determine the fusion weight of the audio features and the visual features based on the vehicle state data. The feature fusion unit is used to fuse the audio features and the visual features based on the fusion weights to generate a fused feature representation; The result determination unit is used to perform feature parsing on the fused feature representation to obtain the target text recognition result.

9. An electronic device comprising: A memory and a processor, characterized in that the processor, when executing a computer program stored in the memory, implements the steps of the speech recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer- executable instructions or a computer program, characterized in that, When the computer-executable instructions or the computer program are executed by a processor, the steps of the speech recognition method as described in any one of claims 1 to 7 are implemented.

Images