Small model-based in-vehicle multi-mode interaction identification method and system

Abstract

The invention relates to the technical field of vehicle machine intelligent interaction, and discloses a vehicle machine multi-mode interaction identification method and system based on a small model. The method comprises the following steps: collecting multi-modal data, preprocessing the multi-modal data to form a synchronous data stream, inputting a lightweight small model with a parameter quantity less than 5M, extracting features, evaluating quality, judging a low-quality modal according to a quality threshold value, and calculating a compensation gain coefficient to form a self-adaptive weight group; and performing weighted summation on the feature vectors, keeping the dimensionality unchanged, inputting the feature vectors into a small model classifier, and outputting an identification result. According to the method, the identification accuracy and the real-time response capability of the small model in the resource-constrained environment are improved.

Description

Technical Field

[0001] This application relates to the field of vehicle-machine intelligent interaction technology, and in particular to a vehicle-machine multimodal interaction recognition method and system based on a small model. Background Technology

[0002] Multimodal interaction recognition technology for in-vehicle systems achieves intelligent understanding of driver intentions by integrating multiple input modalities such as voice, vision, and gestures. Existing technologies typically employ large-scale deep learning models to extract and fuse features from each modality. Common implementations include: using deep convolutional networks such as ResNet and VGG to extract visual features, using sequence models such as LSTM and Transformer to process audio signals, achieving multimodal fusion through feature concatenation or attention mechanisms, and finally inputting the data into a classifier to complete intention recognition. These methods can achieve high recognition accuracy in cloud server environments with sufficient computing power, but they face severe resource constraints in edge computing scenarios for in-vehicle systems.

[0003] The main shortcomings of existing technologies are as follows: First, the number of parameters in large-scale models typically reaches tens to hundreds of megabytes, exceeding the storage and computing power limitations of vehicle chips, resulting in excessively long inference latency that cannot meet the requirements of real-time interaction; Second, multimodal fusion generally adopts a fixed weight strategy, fusing data from each modality according to a preset ratio regardless of the data quality. When changes in the driving environment cause degradation in the quality of a certain modality (such as strong light affecting visual recognition or noise interfering with speech recognition), the fixed weights cannot be dynamically adjusted, and the noise from low-quality modalities will severely drag down the overall recognition performance, resulting in a significant drop in accuracy; Third, fusion methods such as feature splicing will cause the feature dimensions to increase exponentially after fusion, further increasing the computational burden of small models, which contradicts the goal of lightweight design. Summary of the Invention

[0004] This application provides a vehicle-to-machine multimodal interaction recognition method and system based on a small model, which solves the technical problems in existing vehicle-to-machine multimodal interaction recognition methods, such as the decline in recognition performance of small models in modal quality fluctuation scenarios, the adjustment lag caused by passive response, and the lack of historical experience accumulation mechanism. It improves the recognition accuracy and real-time response capability of small models in resource-constrained environments.

[0005] Firstly, this application provides a vehicle-to-machine multimodal interaction recognition method based on a small model, the method comprising:

[0006] Step S1: Acquire visual image sequences, audio signals, and gaze data; preprocess the visual image sequences, audio signals, and gaze data; and align them by timestamps to form a synchronous multimodal data stream.

[0007] Step S2: Input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector;

[0008] Step S3: Determine the modes whose confidence scores are lower than the quality threshold as low-quality modes. For non-low-quality modes, calculate the compensation gain coefficient by accumulating the difference between the low-quality modes and the quality threshold. Normalize the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality modes. Directly normalize the confidence scores of the low-quality modes to obtain the weight of the low-quality modes, forming an adaptive fusion weight reassembly.

[0009] Step S4: Use the adaptive fusion weight recombination to perform weighted summation on the multimodal feature vector while keeping the feature dimension unchanged to obtain a fused feature vector. Input the fused feature vector into the small model classifier to output the interactive intent recognition result.

[0010] Secondly, this application provides a vehicle-to-machine multimodal interaction recognition system based on a small model, the vehicle-to-machine multimodal interaction recognition system based on a small model includes:

[0011] The acquisition module is used to acquire visual image sequences, audio signals, and gaze data, preprocess the visual image sequences, audio signals, and gaze data, and align them according to timestamps to form a synchronous multimodal data stream;

[0012] The input module is used to input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and to run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector.

[0013] The calculation module is used to determine that modes with confidence scores lower than the quality threshold are low-quality modes. For non-low-quality modes, it calculates a compensation gain coefficient by accumulating the difference between the low-quality mode and the quality threshold. It then normalizes the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality mode. Finally, it directly normalizes the confidence score of the low-quality mode to obtain the weight of the low-quality mode, thus forming an adaptive fusion weight reassembly.

[0014] The execution module is used to perform weighted summation on the multimodal feature vector using the adaptive fusion weight recombination while keeping the feature dimension unchanged to obtain a fused feature vector, and input the fused feature vector into a small model classifier to output the interactive intent recognition result.

[0015] The technical solution provided in this application establishes a strict validity judgment standard during the data acquisition stage through a multimodal data synchronous preprocessing mechanism. Subsequent inference processes are only triggered when a clear interaction intent is detected, avoiding resource consumption in invalid scenarios. This is significant for vehicle systems with limited computing power and power consumption. The lightweight feature extraction model with fewer than 5M parameters employs compression techniques such as depthwise separable convolution and inverse residual structures. While maintaining feature representation capabilities, this significantly reduces the model parameter size and computational complexity, enabling the entire recognition process to be completed in real-time on the vehicle's edge computing unit, meeting the dual requirements of low latency and high energy efficiency in in-vehicle interaction scenarios. The design of parallel execution of the quality assessment sub-network and feature extraction provides a reliable quality quantification index for each modal feature vector without increasing additional inference latency. This index directly reflects the usability level of the current modal data, providing an accurate decision-making basis for subsequent adaptive weight adjustment and avoiding the blind fusion problem caused by a lack of quality awareness in traditional methods. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an embodiment of the vehicle-machine multimodal interaction recognition method based on a small model in this application.

[0018] Figure 2 This is a schematic diagram illustrating the verification of the adaptive weight adjustment effect in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram illustrating the influence analysis of compensation strength parameters in the embodiments of this application. Detailed Implementation

[0020] This application provides a vehicle-to-everything (V2X) multimodal interaction recognition method and system based on a small model. The terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0021] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the vehicle-to-machine multimodal interaction recognition method based on a small model in this application includes:

[0022] Step S1: Acquire visual image sequences, audio signals, and gaze data; preprocess the visual image sequences, audio signals, and gaze data; and align them by timestamps to form a synchronous multimodal data stream.

[0023] Specifically, during the multimodal data acquisition phase, driver interaction information is synchronously acquired through an in-vehicle sensor array. Visual image sequences capture changes in the driver's facial expressions and hand gestures, audio signals record the acoustic characteristics of voice commands, and gaze data tracks the direction of eye gaze. Preprocessing operations employ differentiated strategies for different modalities: inter-frame difference detection identifies motion regions by calculating changes in pixel values ​​across consecutive frames, triggering valid frame extraction when the change exceeds a threshold; short-time energy reflects the intensity characteristics of the speech signal, and short-time zero-crossing rate reflects the frequency characteristics of the signal; their combined determination accurately separates valid speech segments from silent segments; a Kalman filter smooths jitter noise in the gaze trajectory through state prediction and observation update steps. A timestamp alignment mechanism maps modal data with different sampling rates to a unified time axis, forming a frame-level synchronized multimodal data stream.

[0024] Step S2: Input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector;

[0025] Specifically, the lightweight feature extraction mini-model employs compression techniques such as depthwise separable convolution and inverse residual structures to reduce parameter size. In the visual mini-model, the inverse residual module first expands the number of feature channels to enhance representational capability through a first convolutional layer, then extracts spatial features independently in each channel through depthwise convolution. The attention module assigns weights based on channel importance, and finally, a second convolutional layer compresses the number of channels to output compact features. The audio mini-model converts audio signals into acoustic features, then progressively reduces temporal resolution through convolutional subsampling layers, captures long-term dependencies through a multi-head self-attention mechanism, and extracts local patterns through depthwise separable convolution. The gesture mini-model uses dilated convolution to capture the temporal evolution patterns of hand movements by increasing the receptive field. The quality evaluation sub-network receives feature vectors, and the first fully connected layer maps them to the hidden space to learn quality-related patterns. The second fully connected layer compresses the hidden representation to the zero-to-one interval using a sigmoid function, outputting a confidence score that quantifies the reliability level of the feature vector.

[0026] Step S3: Determine the modes with confidence scores below the quality threshold as low-quality modes. For non-low-quality modes, calculate the compensation gain coefficient by accumulating the difference between the low-quality mode and the quality threshold. Normalize the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality mode. For low-quality modes, directly normalize the confidence score to obtain the weight of the low-quality mode, forming an adaptive fusion weight reassembly.

[0027] Specifically, a quality threshold is used as the criterion to classify modes into low-quality and non-low-quality categories. For low-quality modes, the difference between their confidence score and the quality threshold reflects the degree of quality loss. The sum of the differences of multiple low-quality modes yields the total quality loss. The compensation gain coefficient amplifies the contribution of non-low-quality modes proportionally to the total quality loss; the larger the loss, the higher the gain coefficient, thus achieving an active compensation mechanism. Normalization ensures that the sum of the weights of all modes is one: the confidence score of each non-low-quality mode is multiplied by the corresponding gain coefficient to obtain the post-gain confidence score. The post-gain confidence scores or original confidence scores of all modes are summed to obtain the total confidence score. The weight of each mode is equal to its post-gain confidence score or original confidence score divided by the total confidence score. This weight allocation strategy dynamically adjusts the influence of each mode in the fusion process, with high-quality modes receiving greater weights and low-quality mode weights being suppressed.

[0028] Step S4: Use adaptive fusion weight recombination to perform weighted summation on the multimodal feature vectors while keeping the feature dimensions unchanged to obtain the fused feature vector. Input the fused feature vector into the small model classifier to output the interactive intent recognition result.

[0029] Specifically, the weighted summation operation achieves modality fusion in the feature space. Each modality feature vector is element-wise multiplied by its corresponding weight value and then summed, resulting in a fused feature vector with the same dimension as the single-modality feature vector. The small model classifier extracts abstract semantics step-by-step through a multi-layer fully connected network: the first layer maps the fused features to the first hidden layer to learn the initial pattern; the activation function introduces non-linear transformation capabilities, and dropout randomly discards some neurons to prevent overfitting; the second layer continues mapping to the smaller-dimensional second hidden layer to extract higher-level semantics; the third layer outputs the raw scores for each interaction intent category. The softmax function converts the raw scores into a probability distribution, amplifies the differences between categories through exponential operations, and normalizes the scores to ensure the probability sums to one. The category with the highest probability corresponds to the final recognition result. The entire inference process is completed in the vehicle's edge computing unit. The lightweight design of the small model keeps the end-to-end latency within the requirements of real-time interaction.

[0030] In one specific embodiment, step S1 includes:

[0031] The system acquires visual image sequences at a preset resolution and frame rate using a camera, acquires audio signals at a preset sampling rate using a microphone, and acquires line-of-sight data using an infrared camera.

[0032] Perform inter-frame difference detection on the visual image sequence, and filter out valid interactive frames when the sum of pixel differences in consecutive frames exceeds a preset motion threshold;

[0033] The short-time energy and short-time zero-crossing rate are calculated for the audio signal. When the short-time energy is greater than the preset energy threshold and the short-time zero-crossing rate is less than the preset zero-crossing threshold, it is determined to be a valid speech segment. The valid speech segment is then subjected to spectral subtraction for noise reduction to obtain the noise-reduced frequency band.

[0034] A Kalman filter is applied to the gaze direction data to smooth the trajectory. The state transition matrix of the Kalman filter is set according to the reciprocal of the preset frame rate to obtain a smooth gaze trajectory.

[0035] The effective interaction frames, noise reduction frequency bands, and smooth line-of-sight trajectories are aligned by timestamps to form a synchronous multimodal data stream.

[0036] Specifically, inter-frame difference detection identifies moving regions by calculating the difference in grayscale values ​​of corresponding pixels between adjacent frames. The calculation involves calculating the absolute value of the difference between the current frame and the previous frame pixel by pixel, then summing the differences of all pixels to obtain the total difference. When this total difference exceeds a preset motion threshold, it indicates a significant driver interaction in the frame. The corresponding frame is then marked as a valid interaction frame and retained for subsequent processing. Frames with differences below the threshold are filtered out to avoid consuming computational resources with invalid data. Short-time energy measures signal strength by summing the squares of the sample amplitudes within a short-time frame of the audio signal. Short-time zero-crossing rate reflects frequency characteristics by counting the number of times the signal waveform crosses zero points within a short-time frame. Spoken speech segments typically have high short-time energy and low short-time zero-crossing rates, while background noise and silent segments exhibit the opposite characteristics. Spectral subtraction denoising recovers the clean speech signal by estimating the noise spectrum and subtracting the noise component from the noisy speech spectrum.

[0037] The state transition matrix of the Kalman filter describes the motion of the gaze position in adjacent time steps. The time interval parameter in the matrix is ​​set to the reciprocal of the preset frame rate to match the actual sampling period. The filtering process includes a prediction step and an update step. The prediction step estimates the current position based on the state at the previous time step and the state transition matrix. The update step corrects the prediction result based on the current observation. Through repeated iterations, random jitter and measurement noise in the gaze data can be effectively suppressed. The timestamp alignment mechanism maps asynchronous data collected by different sensors to a unified time axis. By interpolation or nearest-neighbor selection, it ensures that each time point corresponds to a set of multimodal data. The timestamp of the visual frame serves as a reference, the audio data is matched to the nearest visual frame according to the sampling time, and the gaze data is also matched to the corresponding frame according to the timestamp, forming a frame-level synchronized multimodal data stream for subsequent small model processing.

[0038] In one specific embodiment, step S2 includes:

[0039] The visual feature extraction mini-model is constructed by using depthwise separable convolution to input visual data from a synchronous multimodal data stream. The visual feature extraction mini-model contains multiple inverse residual modules. Each inverse residual module contains a first convolutional layer to expand channels, a depthwise convolutional layer to extract spatial features, an attention module to perform channel weighting, and a second convolutional layer to compress channels, outputting a visual feature vector.

[0040] The audio data in the synchronous multimodal data stream is converted into acoustic features through a filter bank and then input into a lightweight audio feature extraction mini-model. The audio feature extraction mini-model contains convolutional subsampling layers and multiple block structures. Each block structure contains a multi-head self-attention mechanism and a depthwise separable convolutional module, and outputs an audio feature vector.

[0041] The hand key point coordinate sequence of consecutive frames extracted from gaze data in synchronous multimodal data stream is input into a gesture feature extraction mini-model constructed by a temporal convolutional network. The gesture feature extraction mini-model contains multiple layers of dilated convolutions and outputs a gesture feature vector.

[0042] The visual feature vector, audio feature vector, and gesture feature vector are respectively input into the corresponding quality assessment sub-network. The quality assessment sub-network is a two-layer fully connected network structure, and the corresponding confidence score is output through the sigmoid function.

[0043] Specifically, depthwise separable convolution decomposes standard convolution into two steps: depthwise convolution and pointwise convolution. Depthwise convolution independently applies a kernel to each input channel to extract spatial features, while pointwise convolution uses a 1x1 kernel to linearly combine all channels. This decomposition significantly reduces the number of parameters and computational cost. The inverse residual module expands the number of channels in the input features to a larger dimension through the first convolutional layer to enhance feature representation. Then, the depthwise convolutional layer extracts rich spatial patterns in the expanded high-dimensional space. The attention module calculates the importance weights for each channel and performs weighted modulation on the channel features. Finally, the second convolutional layer compresses the high-dimensional features back to a smaller number of channels, outputting a compact visual feature vector. The entire module uses residual connections to directly add the input to the output, preserving the original information and promoting gradient propagation. The audio filter bank transforms the time-domain audio signal into the frequency domain to obtain acoustic features. The convolutional sub-sampling layer gradually reduces the resolution of the time dimension and extracts local features through convolution operations with strides. The multi-head self-attention mechanism divides the input features into multiple heads and computes attention weights in parallel. Each head focuses on a different semantic subspace. Weights are calculated by the similarity between the query vector and the key vector, and the value vectors are weighted and summed to capture long-term dependencies.

[0044] Dilated convolutions expand the receptive field without increasing the number of parameters by inserting holes between kernel elements. The dilation rate controls the size of the hole spacing. Stacking multiple layers of dilated convolutions allows for covering a longer time span with fewer layers. Temporal convolutional networks take the coordinates of hand keypoints in consecutive frames as the input sequence and extract temporal patterns from local to global through dilated convolutional layers, ultimately outputting a fixed-dimensional gesture feature vector representing the entire action sequence. The first fully connected layer of the quality evaluation subnetwork linearly transforms the feature vector to the hidden space and introduces non-linearity through an activation function. The second fully connected layer maps the hidden representation to a single scalar value, and the sigmoid function compresses this scalar to the zero-to-one interval as a confidence score. This score reflects the intrinsic quality level of the input feature vector. During training, the subnetwork learns the mapping relationship between feature quality and recognition performance by using a large number of samples of different quality levels and their corresponding manually labeled quality scores, enabling it to automatically evaluate the reliability of the feature vector based on its distribution characteristics.

[0045] Figure 2This diagram illustrates the verification of the adaptive weight adjustment effect in this application's embodiments. The horizontal axis represents the test scenario number, and the vertical axis represents the recognition accuracy. The dashed line represents the recognition accuracy using the fixed weight fusion method, and the solid line represents the recognition accuracy using the adaptive weight fusion method. As can be seen from the diagram, under different test scenarios, the recognition accuracy of the adaptive weight fusion method consistently remains at a high level above 0.75, while the fixed weight fusion method shows a significant decrease in accuracy in certain scenarios (such as scenarios 3, 9, and 18), dropping as low as 0.58, indicating that the fixed weight fusion method cannot cope with modal quality fluctuations. The adaptive weight fusion method of this application, by evaluating the confidence scores of each modality in real time and dynamically adjusting the fusion weights, maintains high recognition performance even in scenarios with degraded modal quality, achieving an average accuracy improvement of approximately 6 percentage points compared to the fixed weight method.

[0046] In one specific embodiment, step S3, determining that a mode with a confidence score lower than a quality threshold is a low-quality mode, includes:

[0047] The confidence score is compared with a preset quality threshold. When the confidence score is less than the preset quality threshold, the corresponding mode is marked as a low-quality mode. When the confidence score is greater than or equal to the preset quality threshold, the corresponding mode is marked as a non-low-quality mode.

[0048] Specifically, a preset quality threshold is used as the criterion for judging modality quality, usually set to 0.4. The confidence score of each modality is compared with this threshold one by one. When the confidence score of a modality is lower than 0.4, it means that the current data quality of the modality is insufficient to reliably support the recognition task. It is marked as a low-quality modality and its influence weight is reduced in subsequent fusion. On the other hand, the confidence score of a modality is equal to or higher than 0.4, indicating that its data quality is within an acceptable range. It is marked as a non-low-quality modality and given normal or enhanced weight in fusion. This binary classification labeling mechanism transforms continuous confidence scores into discrete quality levels, which facilitates the use of differentiated processing strategies in subsequent weight calculations.

[0049] In one specific embodiment, step S3, calculating the compensation gain coefficient for non-low-quality modes based on the cumulative difference between low-quality modes and the quality threshold, includes:

[0050] Iterate through all low-quality modes and calculate the difference between the confidence score of each low-quality mode and the preset quality threshold.

[0051] The total quality loss is obtained by summing the differences of all low-quality modes;

[0052] For each non-low-quality mode, the corresponding compensation gain coefficient is obtained by multiplying the preset compensation strength parameter by the total amount of missing quality and adding one.

[0053] Specifically, the calculation of the compensation gain coefficient first iterates through all modes marked as low quality, calculating the difference between the preset quality threshold of 0.4 and the confidence score of each low-quality mode. This difference quantifies the degree of quality loss in a single mode. Then, the differences of all low-quality modes are summed to obtain the total quality loss, which reflects the severity of quality loss in the overall multimodal data. Next, for each non-low-quality mode, the preset compensation intensity parameter of 0.6 is multiplied by the total quality loss and then 1 is added to obtain the compensation gain coefficient of that mode. A coefficient greater than 1 means that the contribution of non-low-quality modes needs to be amplified to compensate for the information loss caused by low-quality modes. The larger the total quality loss, the higher the gain coefficient, thereby realizing a quality-driven active compensation mechanism.

[0054] Figure 3 This is a schematic diagram illustrating the impact analysis of the compensation intensity parameter in this application embodiment. The horizontal axis represents the value of the compensation intensity parameter α, and the vertical axis represents the recognition accuracy. The line-filled bar chart represents the recognition accuracy in normal scenarios, and the grid-filled bar chart represents the recognition accuracy in quality degradation scenarios. As can be seen from the figure, in normal scenarios, the compensation intensity parameter α has a relatively small impact on the recognition accuracy, with the accuracy remaining stable between 0.82 and 0.85 for each α value. However, in quality degradation scenarios, the compensation intensity parameter α has a significant impact on performance: when α is 0, the compensation mechanism is not enabled, and the accuracy is only 0.58; as α increases to 0.6, the accuracy increases to 0.78, an improvement of 20 percentage points; when α continues to increase to 0.8, the accuracy reaches a peak of 0.80. Experimental results show that choosing α as 0.6 can achieve a significant performance improvement in quality degradation scenarios without negatively impacting normal scenarios, verifying the effectiveness of the compensation gain mechanism in this application.

[0055] In one specific embodiment, step S3 involves normalizing the product of the compensation gain coefficient and the corresponding confidence score to obtain the non-low-quality mode weights, and directly normalizing the confidence scores of the low-quality modes to obtain the low-quality mode weights, forming an adaptive fusion weight reassembly, including:

[0056] The post-gain confidence is obtained by multiplying the compensation gain coefficient of each non-low-quality mode by the corresponding confidence score.

[0057] The total confidence score is obtained by summing the post-gain confidence scores of all non-low-quality modes with the confidence scores of all low-quality modes.

[0058] The weights of non-low-quality modes are obtained by dividing the post-gain confidence of each non-low-quality mode by the total confidence.

[0059] The weight of a low-quality mode is obtained by dividing the confidence score of each low-quality mode by the total confidence score.

[0060] All non-low-quality modal weights and low-quality modal weights are combined to form an adaptive fusion weight reorganization.

[0061] Specifically, the post-gain confidence score is obtained by multiplying the compensation gain coefficient of each non-low-quality mode by its original confidence score. This operation applies the gain coefficient as an amplification factor to the confidence score, resulting in higher numerical representations for higher-quality modes, while low-quality modes retain their original confidence scores without gain amplification. The total confidence score is calculated by summing the post-gain confidence scores of all non-low-quality modes with the original confidence scores of all low-quality modes. This sum serves as the normalized denominator, ensuring that the sum of the mode weights in subsequent calculations is strictly equal to 1.

[0062] Normalized weights are calculated using division. For each non-low-quality mode, its post-gain confidence score is divided by the total confidence score to obtain its final weight value. This weight value reflects the proportion that mode should occupy in the fusion process. Due to gain amplification, the weight of non-low-quality modes is usually higher than the proportion of their original confidence scores to the total. For each low-quality mode, its original confidence score is directly divided by the total confidence score to obtain its weight value. This relatively small weight value reflects the suppression of the influence of low-quality data. All non-low-quality mode weights and low-quality mode weights are arranged and combined in modality order to form an adaptive fusion weight reassembly. The sum of all elements in this weight reassembly is 1, and the value of each element is dynamically adjusted according to the real-time quality status. High-quality modes receive larger weights, while the weights of low-quality modes are compressed, realizing a quality-aware adaptive weight allocation mechanism.

[0063] In one specific embodiment, step S4 includes:

[0064] Each weight value in the adaptive fusion weighting process is multiplied element-wise with its corresponding multimodal feature vector to obtain a weighted feature vector;

[0065] The fused feature vector is obtained by summing all weighted feature vectors element by element. The dimension of the fused feature vector is the same as that of the individual multimodal feature vectors.

[0066] The fused feature vector is input into a small model classifier, which contains a multi-layer fully connected network. The first fully connected layer maps the fused feature vector to the first hidden layer dimension and processes it through an activation function and dropout. The second fully connected layer maps it to the second hidden layer dimension and processes it through an activation function and dropout. The third fully connected layer outputs a logits vector with a preset number of categories.

[0067] The softmax function is applied to the logits vector to calculate the probability distribution of each category, and the category with the highest probability is selected as the result of interaction intent recognition.

[0068] Specifically, the weighted feature vector is generated through element-wise multiplication. The process involves multiplying the visual modality weights in the adaptive fusion weighting by each element of the visual feature vector, the audio modality weights by each element of the audio feature vector, and the gesture modality weights by each element of the gesture feature vector. This multiplication operation broadcasts the scalar weight values ​​to each dimension of the entire feature vector, achieving uniform scaling. The fused feature vector is obtained by summing the elements of all weighted feature vectors at their corresponding positions. Specifically, the first elements of the visual, audio, and gesture weighted feature vectors are added together to obtain the first element of the fused feature vector; the second elements are added together to obtain the second element, and so on, until all dimensions are accumulated. Since the modal feature vectors have the same dimension and the weighted summation does not change the dimension, the dimension of the fused feature vector remains consistent with the dimension of a single multimodal feature vector—256 dimensions.

[0069] The first fully connected layer of the small model classifier maps 256-dimensional features to a 128-dimensional first hidden layer space by multiplying the weight matrix with the fused feature vector and adding a bias term. The activation function applies a non-linear transformation to the hidden layer output element by element to enhance the network's expressive power, and dropout randomly sets some neuron outputs to zero to prevent overfitting. The second fully connected layer maps the 128-dimensional hidden representation to a 64-dimensional second hidden layer space and is similarly processed by the activation function and dropout. The third fully connected layer maps the 64-dimensional features to a preset number of categories, outputting a logits vector. Each element of this vector corresponds to the original score of an interaction intent category. The softmax function first calculates the exponential function value for each element in the logits vector, then sums all the exponential values ​​as the normalized denominator. The probability of each category is equal to its exponential value divided by the sum. The exponential operation amplifies the score differences between categories, making the probability distribution more concentrated. Finally, the index of the category with the highest probability value is selected as the interaction intent recognition result output.

[0070] The above describes the vehicle-to-machine multimodal interaction recognition method based on a small model in the embodiments of this application. The following describes the vehicle-to-machine multimodal interaction recognition system based on a small model in the embodiments of this application. One embodiment of the vehicle-to-machine multimodal interaction recognition system based on a small model in the embodiments of this application includes:

[0071] The acquisition module is used to acquire visual image sequences, audio signals, and gaze data, preprocess the visual image sequences, audio signals, and gaze data, and align them according to timestamps to form a synchronous multimodal data stream;

[0072] The input module is used to input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and to run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector.

[0073] The calculation module is used to determine that modes with confidence scores lower than the quality threshold are low-quality modes. For non-low-quality modes, it calculates a compensation gain coefficient by accumulating the difference between the low-quality mode and the quality threshold. It then normalizes the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality mode. Finally, it directly normalizes the confidence score of the low-quality mode to obtain the weight of the low-quality mode, thus forming an adaptive fusion weight reassembly.

[0074] The execution module is used to perform weighted summation on the multimodal feature vector using the adaptive fusion weight recombination while keeping the feature dimension unchanged to obtain a fused feature vector, and input the fused feature vector into a small model classifier to output the interactive intent recognition result.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A small model-based car-machine multi-modal interaction recognition method, characterized in that, The method comprises: Step S1: collecting a visual image sequence, an audio signal and a line-of-sight data, preprocessing the visual image sequence, the audio signal and the line-of-sight data, and aligning the visual image sequence, the audio signal and the line-of-sight data according to timestamps to form a synchronous multi-modal data stream; Step S2: inputting the synchronous multi-modal data stream into a plurality of lightweight feature extraction small models with a parameter quantity less than 5M to extract multi-modal feature vectors of a fixed dimension, and running a quality evaluation subnetwork in parallel to calculate confidence scores corresponding to each feature vector; Step S3: determining a modality with a confidence score lower than a quality threshold as a low-quality modality, calculating a compensation gain coefficient according to a difference between the low-quality modality and the quality threshold for a non-low-quality modality, obtaining a non-low-quality modality weight by multiplying the compensation gain coefficient by the corresponding confidence score, directly normalizing the confidence score of the low-quality modality to obtain a low-quality modality weight, and forming an adaptive fusion weight set; Step S4: using the adaptive fusion weight set to perform weighted summation on the multi-modal feature vectors to obtain a fusion feature vector without changing the feature dimension, and inputting the fusion feature vector into a small model classifier to output an interactive intent recognition result.

2. The small model-based car-machine multi-modal interaction recognition method according to claim 1, characterized in that, The step S1 comprises: collecting a visual image sequence of a preset resolution and a preset frame rate through a camera, collecting an audio signal of a preset sampling rate through a microphone, and collecting line-of-sight direction data through an infrared camera; performing inter-frame difference detection on the visual image sequence, and screening valid interactive frames when the pixel difference sum of consecutive frame images exceeds a preset motion threshold; calculating a short-time energy and a short-time zero-crossing rate for the audio signal, determining an effective speech segment when the short-time energy is greater than a preset energy threshold and the short-time zero-crossing rate is less than a preset zero-crossing rate threshold, and applying spectral subtraction to the effective speech segment to obtain a noise-reduced audio segment; applying a Kalman filter to the line-of-sight direction data for trajectory smoothing, setting a state transition matrix of the Kalman filter according to the reciprocal of the preset frame rate, and obtaining a smoothed line-of-sight trajectory; aligning the valid interactive frames, the noise-reduced audio segment and the smoothed line-of-sight trajectory according to timestamps to form the synchronous multi-modal data stream. 3.The small model-based car-machine multi-modal interaction recognition method according to claim 1, characterized in that, The step S2 comprises: inputting visual data in the synchronous multi-modal data stream into a visual feature extraction small model constructed using a depth separable convolution, the visual feature extraction small model comprising a plurality of reverse residual modules, each reverse residual module comprising a first convolution layer for expanding channels, a depth convolution layer for extracting spatial features, an attention module for channel weighting, and a second convolution layer for compressing channels, and outputting a visual feature vector; inputting audio data in the synchronous multi-modal data stream into a lightweight audio feature extraction small model after converting the audio data into acoustic features through a filter bank, the audio feature extraction small model comprising a convolution subsampling layer and a plurality of block structures, each block structure comprising a multi-head self-attention mechanism and a depth separable convolution module, and outputting an audio feature vector; Input the hand key point coordinate sequence of the continuous frame of the line-of-sight data extracted from the synchronous multi-modal data stream into a gesture feature extraction small model constructed by a time convolution network, the gesture feature extraction small model comprising multiple layers of dilated convolution, and output a gesture feature vector; Input the visual feature vector, the audio feature vector and the gesture feature vector into corresponding quality evaluation sub-networks respectively, the quality evaluation sub-networks being two-layer fully connected network structures, and output corresponding confidence scores through a sigmoid function.

4. The small model-based car-machine multi-modal interaction recognition method according to claim 1, characterized in that, The step S3 comprises: Compare the confidence score with a preset quality threshold, and mark the corresponding modality as a low-quality modality when the confidence score is less than the preset quality threshold, and mark the corresponding modality as a non-low-quality modality when the confidence score is greater than or equal to the preset quality threshold.

5. The small model-based car-machine multi-modal interaction recognition method according to claim 4, characterized in that, The step S3 comprises: Calculate the difference between the confidence score of each low-quality modality and the preset quality threshold; Sum up the differences of all low-quality modalities to obtain a total quality loss; For each non-low-quality modality, obtain a corresponding compensation gain coefficient according to the product of a preset compensation intensity parameter and the total quality loss plus one.

6. The small model-based car-machine multi-modal interaction recognition method according to claim 5, characterized in that, The step S3 comprises: Multiply each non-low-quality modality compensation gain coefficient with the corresponding confidence score to obtain a gain-after-confidence score; Sum up the gain-after-confidence scores of all non-low-quality modalities and the confidence scores of all low-quality modalities to obtain a total confidence score; Divide the gain-after-confidence score of each non-low-quality modality by the total confidence score to obtain a non-low-quality modality weight; Divide the confidence score of each low-quality modality by the total confidence score to obtain a low-quality modality weight; Combine all non-low-quality modality weights and low-quality modality weights to form the adaptive fusion weight group.

7. The small model-based car machine multi-modal interaction recognition method according to claim 1, characterized in that, The step S4 comprises: Multiply each weight value in the adaptive fusion weight group with the corresponding multi-modal feature vector element by element to obtain a weighted feature vector; Sum up all weighted feature vectors element by element to obtain the fusion feature vector, the dimension of the fusion feature vector being the same as that of a single multi-modal feature vector; Input the fusion feature vector into a small model classifier, the small model classifier comprising multiple layers of fully connected networks, a first fully connected layer mapping the fusion feature vector to a first hidden layer dimension and being subjected to an activation function and dropout processing, a second fully connected layer mapping to a second hidden layer dimension and being subjected to an activation function and dropout processing, and a third fully connected layer outputting a logits vector of a preset number of categories; A softmax function is applied to the logits vector to calculate a probability distribution of each category, and a category with the maximum probability is selected as the interaction intent recognition result.

8. A small model-based vehicle-machine multi-modal interaction recognition system, characterized in that, The small model-based vehicle-machine multi-modal interaction recognition system comprises: The acquisition module is configured to acquire a visual image sequence, an audio signal, and line-of-sight data, pre-process the visual image sequence, the audio signal, and the line-of-sight data, and align the pre-processed visual image sequence, the pre-processed audio signal, and the pre-processed line-of-sight data according to timestamps to form a synchronous multi-modal data stream. The input module is configured to input the synchronous multi-modal data stream into a plurality of lightweight feature extraction small models each having a parameter quantity less than 5M to extract a multi-modal feature vector of a fixed dimension, and run a quality evaluation subnetwork in parallel to calculate a confidence score corresponding to each feature vector. The calculation module is configured to determine a modality with a confidence score lower than a quality threshold as a low-quality modality, calculate a compensation gain coefficient according to a difference between the low-quality modality and the quality threshold for a non-low-quality modality, obtain a non-low-quality modality weight by normalizing a product of the compensation gain coefficient and the corresponding confidence score, directly normalize the confidence score of the low-quality modality to obtain a low-quality modality weight, and form an adaptive fusion weight set. The execution module is configured to perform weighted summation on the multi-modal feature vector using the adaptive fusion weight set to obtain a fusion feature vector without changing the feature dimension, input the fusion feature vector into a small model classifier, and output an interaction intent recognition result.

9. The system of claim 8, wherein, The acquisition module is configured to acquire a visual image sequence, an audio signal, and line-of-sight data, pre-process the visual image sequence, the audio signal, and the line-of-sight data, and align the pre-processed visual image sequence, the pre-processed audio signal, and the pre-processed line-of-sight data according to timestamps to form a synchronous multi-modal data stream. The visual image sequence is acquired by a camera at a preset resolution and a preset frame rate, the audio signal is acquired by a microphone at a preset sampling rate, and the line-of-sight direction data is acquired by an infrared camera. Frame difference detection is performed on the visual image sequence, and an effective interaction frame is obtained by screening when the pixel difference sum of consecutive frame images exceeds a preset motion threshold. The short-time energy and the short-time zero-crossing rate of the audio signal are calculated, and when the short-time energy is greater than a preset energy threshold and the short-time zero-crossing rate is less than a preset zero-crossing rate threshold, the audio signal is determined as an effective speech segment. A Kalman filter is applied to the line-of-sight direction data to perform trajectory smoothing, and a state transition matrix of the Kalman filter is set according to the reciprocal of the preset frame rate to obtain a smoothed line-of-sight trajectory. The effective interaction frame, the noise-reduced audio segment, and the smoothed line-of-sight trajectory are aligned according to timestamps to form the synchronous multi-modal data stream.

10. The system of claim 8, wherein, The synchronous multi-modal data stream is input into a plurality of lightweight feature extraction small models each having a parameter quantity less than 5M to extract a multi-modal feature vector of a fixed dimension, and a quality evaluation subnetwork is run in parallel to calculate a confidence score corresponding to each feature vector. The visual data in the synchronous multi-modal data stream is input into a visual feature extraction small model constructed by using a depth separable convolution, the visual feature extraction small model comprises a plurality of reverse residual modules, each reverse residual module comprises a first convolution layer for expanding channels, a depth convolution layer for extracting spatial features, an attention module for channel weighting, and a second convolution layer for compressing channels, and outputs a visual feature vector; The audio data in the synchronous multi-modal data stream is converted into acoustic features by a filter bank and then input into a lightweight audio feature extraction small model, the audio feature extraction small model comprises a convolution subsampling layer and a plurality of block structures, each block structure comprises a multi-head self-attention mechanism and a depth separable convolution module, and outputs an audio feature vector; The continuous frame hand key point coordinate sequence extracted from the line of sight data in the synchronous multi-modal data stream is input into a gesture feature extraction small model constructed by using a time sequence convolution network, the gesture feature extraction small model comprises a plurality of layers of dilated convolution, and outputs a gesture feature vector; The visual feature vector, the audio feature vector, and the gesture feature vector are respectively input into corresponding quality evaluation sub-networks, the quality evaluation sub-networks are two-layer fully connected network structures, and corresponding confidence scores are output by using a sigmoid function.

Images