Small model-based car-machine multi-modal interaction recognition method and system

Abstract

The application relates to the technical field of intelligent interaction of vehicle machines, and discloses a small-model-based multi-modal interaction recognition method and system of a vehicle machine. The method comprises the following steps: collecting multi-modal data and preprocessing the multi-modal data to form a synchronous data stream; inputting a small model with a light weight and a parameter quantity less than 5M to extract features and evaluate quality; determining a low-quality mode according to a quality threshold and calculating a compensation gain coefficient to form an adaptive weight group; weighting and summing feature vectors to keep the dimension unchanged and then inputting the small model classifier to output a recognition result. The application improves the recognition accuracy and real-time response capability of the small model in a resource-limited environment.

Description

Technical Field

[0001] This application relates to the field of vehicle-machine intelligent interaction technology, and in particular to a vehicle-machine multimodal interaction recognition method and system based on a small model. Background Technology

[0002] Multimodal interaction recognition technology for in-vehicle systems achieves intelligent understanding of driver intentions by integrating multiple input modalities such as voice, vision, and gestures. Existing technologies typically employ large-scale deep learning models to extract and fuse features from each modality. Common implementations include: using deep convolutional networks such as ResNet and VGG to extract visual features, using sequence models such as LSTM and Transformer to process audio signals, achieving multimodal fusion through feature concatenation or attention mechanisms, and finally inputting the data into a classifier to complete intention recognition. These methods can achieve high recognition accuracy in cloud server environments with sufficient computing power, but they face severe resource constraints in edge computing scenarios for in-vehicle systems.

[0003] The main shortcomings of existing technologies are as follows: First, the number of parameters in large-scale models typically reaches tens to hundreds of megabytes, exceeding the storage and computing power limitations of vehicle chips, resulting in excessively long inference latency that cannot meet the requirements of real-time interaction; Second, multimodal fusion generally adopts a fixed weight strategy, fusing data from each modality according to a preset ratio regardless of the data quality. When changes in the driving environment cause degradation in the quality of a certain modality (such as strong light affecting visual recognition or noise interfering with speech recognition), the fixed weights cannot be dynamically adjusted, and the noise from low-quality modalities will severely drag down the overall recognition performance, resulting in a significant drop in accuracy; Third, fusion methods such as feature splicing will cause the feature dimensions to increase exponentially after fusion, further increasing the computational burden of small models, which contradicts the goal of lightweight design. Summary of the Invention

[0004] This application provides a vehicle-to-machine multimodal interaction recognition method and system based on a small model, which solves the technical problems in existing vehicle-to-machine multimodal interaction recognition methods, such as the decline in recognition performance of small models in modal quality fluctuation scenarios, the adjustment lag caused by passive response, and the lack of historical experience accumulation mechanism. It improves the recognition accuracy and real-time response capability of small models in resource-constrained environments.

[0005] Firstly, this application provides a vehicle-to-machine multimodal interaction recognition method based on a small model, the method comprising:

[0006] Step S1: Acquire visual image sequences, audio signals, and gaze data; preprocess the visual image sequences, audio signals, and gaze data; and align them by timestamps to form a synchronous multimodal data stream.

[0007] Step S2: Input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector;

[0008] Step S3: Determine the modes whose confidence scores are lower than the quality threshold as low-quality modes. For non-low-quality modes, calculate the compensation gain coefficient by accumulating the difference between the low-quality modes and the quality threshold. Normalize the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality modes. Directly normalize the confidence scores of the low-quality modes to obtain the weight of the low-quality modes, forming an adaptive fusion weight reassembly.

[0009] Step S4: Use the adaptive fusion weight recombination to perform weighted summation on the multimodal feature vector while keeping the feature dimension unchanged to obtain a fused feature vector. Input the fused feature vector into the small model classifier to output the interactive intent recognition result.

[0010] Secondly, this application provides a vehicle-to-machine multimodal interaction recognition system based on a small model, the vehicle-to-machine multimodal interaction recognition system based on a small model includes:

[0011] The acquisition module is used to acquire visual image sequences, audio signals, and gaze data, preprocess the visual image sequences, audio signals, and gaze data, and align them according to timestamps to form a synchronous multimodal data stream;

[0012] The input module is used to input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and to run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector.

[0013] The calculation module is used to determine that modes with confidence scores lower than the quality threshold are low-quality modes. For non-low-quality modes, it calculates a compensation gain coefficient by accumulating the difference between the low-quality mode and the quality threshold. It then normalizes the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality mode. Finally, it directly normalizes the confidence score of the low-quality mode to obtain the weight of the low-quality mode, thus forming an adaptive fusion weight reassembly.

[0014] The execution module is used to perform weighted summation on the multimodal feature vector using the adaptive fusion weight recombination while keeping the feature dimension unchanged to obtain a fused feature vector, and input the fused feature vector into a small model classifier to output the interactive intent recognition result.

[0015] The technical solution provided in this application establishes a strict validity judgment standard during the data acquisition stage through a multimodal data synchronous preprocessing mechanism. Subsequent inference processes are only triggered when a clear interaction intent is detected, avoiding resource consumption in invalid scenarios. This is significant for vehicle systems with limited computing power and power consumption. The lightweight feature extraction model with fewer than 5M parameters employs compression techniques such as depthwise separable convolution and inverse residual structures. While maintaining feature representation capabilities, this significantly reduces the model parameter size and computational complexity, enabling the entire recognition process to be completed in real-time on the vehicle's edge computing unit, meeting the dual requirements of low latency and high energy efficiency in in-vehicle interaction scenarios. The design of parallel execution of the quality assessment sub-network and feature extraction provides a reliable quality quantification index for each modal feature vector without increasing additional inference latency. This index directly reflects the usability level of the current modal data, providing an accurate decision-making basis for subsequent adaptive weight adjustment and avoiding the blind fusion problem caused by a lack of quality awareness in traditional methods. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an embodiment of the vehicle-machine multimodal interaction recognition method based on a small model in this application.

[0018] Figure 2 This is a schematic diagram illustrating the verification of the adaptive weight adjustment effect in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram illustrating the influence analysis of compensation strength parameters in the embodiments of this application. Detailed Implementation

[0020] This application provides a vehicle-to-everything (V2X) multimodal interaction recognition method and system based on a small model. The terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0021] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the vehicle-to-machine multimodal interaction recognition method based on a small model in this application includes:

[0022] Step S1: Acquire visual image sequences, audio signals, and gaze data; preprocess the visual image sequences, audio signals, and gaze data; and align them by timestamps to form a synchronous multimodal data stream.

[0023] Specifically, during the multimodal data acquisition phase, driver interaction information is synchronously acquired through an in-vehicle sensor array. Visual image sequences capture changes in the driver's facial expressions and hand gestures, audio signals record the acoustic characteristics of voice commands, and gaze data tracks the direction of eye gaze. Preprocessing operations employ differentiated strategies for different modalities: inter-frame difference detection identifies motion regions by calculating changes in pixel values ​​across consecutive frames, triggering valid frame extraction when the change exceeds a threshold; short-time energy reflects the intensity characteristics of the speech signal, and short-time zero-crossing rate reflects the frequency characteristics of the signal; their combined determination accurately separates valid speech segments from silent segments; a Kalman filter smooths jitter noise in the gaze trajectory through state prediction and observation update steps. A timestamp alignment mechanism maps modal data with different sampling rates to a unified time axis, forming a frame-level synchronized multimodal data stream.

[0024] Step S2: Input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector;

[0025] Specifically, the lightweight feature extraction mini-model employs compression techniques such as depthwise separable convolution and inverse residual structures to reduce parameter size. In the visual mini-model, the inverse residual module first expands the number of feature channels to enhance representational capability through a first convolutional layer, then extracts spatial features independently in each channel through depthwise convolution. The attention module assigns weights based on channel importance, and finally, a second convolutional layer compresses the number of channels to output compact features. The audio mini-model converts audio signals into acoustic features, then progressively reduces temporal resolution through convolutional subsampling layers, captures long-term dependencies through a multi-head self-attention mechanism, and extracts local patterns through depthwise separable convolution. The gesture mini-model uses dilated convolution to capture the temporal evolution patterns of hand movements by increasing the receptive field. The quality evaluation sub-network receives feature vectors, and the first fully connected layer maps them to the hidden space to learn quality-related patterns. The second fully connected layer compresses the hidden representation to the zero-to-one interval using a sigmoid function, outputting a confidence score that quantifies the reliability level of the feature vector.

[0026] Step S3: Determine the modes with confidence scores below the quality threshold as low-quality modes. For non-low-quality modes, calculate the compensation gain coefficient by accumulating the difference between the low-quality mode and the quality threshold. Normalize the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality mode. For low-quality modes, directly normalize the confidence score to obtain the weight of the low-quality mode, forming an adaptive fusion weight reassembly.

[0027] Specifically, a quality threshold is used as the criterion to classify modes into low-quality and non-low-quality categories. For low-quality modes, the difference between their confidence score and the quality threshold reflects the degree of quality loss. The sum of the differences of multiple low-quality modes yields the total quality loss. The compensation gain coefficient amplifies the contribution of non-low-quality modes proportionally to the total quality loss; the larger the loss, the higher the gain coefficient, thus achieving an active compensation mechanism. Normalization ensures that the sum of the weights of all modes is one: the confidence score of each non-low-quality mode is multiplied by the corresponding gain coefficient to obtain the post-gain confidence score. The post-gain confidence scores or original confidence scores of all modes are summed to obtain the total confidence score. The weight of each mode is equal to its post-gain confidence score or original confidence score divided by the total confidence score. This weight allocation strategy dynamically adjusts the influence of each mode in the fusion process, with high-quality modes receiving greater weights and low-quality mode weights being suppressed.

[0028] Step S4: Use adaptive fusion weight recombination to perform weighted summation on the multimodal feature vectors while keeping the feature dimensions unchanged to obtain the fused feature vector. Input the fused feature vector into the small model classifier to output the interactive intent recognition result.

[0029] Specifically, the weighted summation operation achieves modality fusion in the feature space. Each modality feature vector is element-wise multiplied by its corresponding weight value and then summed, resulting in a fused feature vector with the same dimension as the single-modality feature vector. The small model classifier extracts abstract semantics step-by-step through a multi-layer fully connected network: the first layer maps the fused features to the first hidden layer to learn the initial pattern; the activation function introduces non-linear transformation capabilities, and dropout randomly discards some neurons to prevent overfitting; the second layer continues mapping to the smaller-dimensional second hidden layer to extract higher-level semantics; the third layer outputs the raw scores for each interaction intent category. The softmax function converts the raw scores into a probability distribution, amplifies the differences between categories through exponential operations, and normalizes the scores to ensure the probability sums to one. The category with the highest probability corresponds to the final recognition result. The entire inference process is completed in the vehicle's edge computing unit. The lightweight design of the small model keeps the end-to-end latency within the requirements of real-time interaction.

[0030] In one specific embodiment, step S1 includes:

[0031] The system acquires visual image sequences at a preset resolution and frame rate using a camera, acquires audio signals at a preset sampling rate using a microphone, and acquires line-of-sight data using an infrared camera.

[0032] Perform inter-frame difference detection on the visual image sequence, and filter out valid interactive frames when the sum of pixel differences in consecutive frames exceeds a preset motion threshold;

[0033] The short-time energy and short-time zero-crossing rate are calculated for the audio signal. When the short-time energy is greater than the preset energy threshold and the short-time zero-crossing rate is less than the preset zero-crossing threshold, it is determined to be a valid speech segment. The valid speech segment is then subjected to spectral subtraction for noise reduction to obtain the noise-reduced frequency band.

[0034] A Kalman filter is applied to the gaze direction data to smooth the trajectory. The state transition matrix of the Kalman filter is set according to the reciprocal of the preset frame rate to obtain a smooth gaze trajectory.

[0035] The effective interaction frames, noise reduction frequency bands, and smooth line-of-sight trajectories are aligned by timestamps to form a synchronous multimodal data stream.

[0036] Specifically, inter-frame difference detection identifies moving regions by calculating the difference in grayscale values ​​of corresponding pixels between adjacent frames. The calculation involves calculating the absolute value of the difference between the current frame and the previous frame pixel by pixel, then summing the differences of all pixels to obtain the total difference. When this total difference exceeds a preset motion threshold, it indicates a significant driver interaction in the frame. The corresponding frame is then marked as a valid interaction frame and retained for subsequent processing. Frames with differences below the threshold are filtered out to avoid consuming computational resources with invalid data. Short-time energy measures signal strength by summing the squares of the sample amplitudes within a short-time frame of the audio signal. Short-time zero-crossing rate reflects frequency characteristics by counting the number of times the signal waveform crosses zero points within a short-time frame. Spoken speech segments typically have high short-time energy and low short-time zero-crossing rates, while background noise and silent segments exhibit the opposite characteristics. Spectral subtraction denoising recovers the clean speech signal by estimating the noise spectrum and subtracting the noise component from the noisy speech spectrum.

[0037] The state transition matrix of the Kalman filter describes the motion of the gaze position in adjacent time steps. The time interval parameter in the matrix is ​​set to the reciprocal of the preset frame rate to match the actual sampling period. The filtering process includes a prediction step and an update step. The prediction step estimates the current position based on the state at the previous time step and the state transition matrix. The update step corrects the prediction result based on the current observation. Through repeated iterations, random jitter and measurement noise in the gaze data can be effectively suppressed. The timestamp alignment mechanism maps asynchronous data collected by different sensors to a unified time axis. By interpolation or nearest-neighbor selection, it ensures that each time point corresponds to a set of multimodal data. The timestamp of the visual frame serves as a reference, the audio data is matched to the nearest visual frame according to the sampling time, and the gaze data is also matched to the corresponding frame according to the timestamp, forming a frame-level synchronized multimodal data stream for subsequent small model processing.

[0038] In one specific embodiment, step S2 includes:

[0039] The visual feature extraction mini-model is constructed by using depthwise separable convolution to input visual data from a synchronous multimodal data stream. The visual feature extraction mini-model contains multiple inverse residual modules. Each inverse residual module contains a first convolutional layer to expand channels, a depthwise convolutional layer to extract spatial features, an attention module to perform channel weighting, and a second convolutional layer to compress channels, outputting a visual feature vector.

[0040] The audio data in the synchronous multimodal data stream is converted into acoustic features through a filter bank and then input into a lightweight audio feature extraction mini-model. The audio feature extraction mini-model contains convolutional subsampling layers and multiple block structures. Each block structure contains a multi-head self-attention mechanism and a depthwise separable convolutional module, and outputs an audio feature vector.

[0041] The hand key point coordinate sequence of consecutive frames extracted from gaze data in synchronous multimodal data stream is input into a gesture feature extraction mini-model constructed by a temporal convolutional network. The gesture feature extraction mini-model contains multiple layers of dilated convolutions and outputs a gesture feature vector.

[0042] The visual feature vector, audio feature vector, and gesture feature vector are respectively input into the corresponding quality assessment sub-network. The quality assessment sub-network is a two-layer fully connected network structure, and the corresponding confidence score is output through the sigmoid function.

[0043] Specifically, depthwise separable convolution decomposes standard convolution into two steps: depthwise convolution and pointwise convolution. Depthwise convolution independently applies a kernel to each input channel to extract spatial features, while pointwise convolution uses a 1x1 kernel to linearly combine all channels. This decomposition significantly reduces the number of parameters and computational cost. The inverse residual module expands the number of channels in the input features to a larger dimension through the first convolutional layer to enhance feature representation. Then, the depthwise convolutional layer extracts rich spatial patterns in the expanded high-dimensional space. The attention module calculates the importance weights for each channel and performs weighted modulation on the channel features. Finally, the second convolutional layer compresses the high-dimensional features back to a smaller number of channels, outputting a compact visual feature vector. The entire module uses residual connections to directly add the input to the output, preserving the original information and promoting gradient propagation. The audio filter bank transforms the time-domain audio signal into the frequency domain to obtain acoustic features. The convolutional sub-sampling layer gradually reduces the resolution of the time dimension and extracts local features through convolution operations with strides. The multi-head self-attention mechanism divides the input features into multiple heads and computes attention weights in parallel. Each head focuses on a different semantic subspace. Weights are calculated by the similarity between the query vector and the key vector, and the value vectors are weighted and summed to capture long-term dependencies.

[0044] Dilated convolutions expand the receptive field without increasing the number of parameters by inserting holes between kernel elements. The dilation rate controls the size of the hole spacing. Stacking multiple layers of dilated convolutions allows for covering a longer time span with fewer layers. Temporal convolutional networks take the coordinates of hand keypoints in consecutive frames as the input sequence and extract temporal patterns from local to global through dilated convolutional layers, ultimately outputting a fixed-dimensional gesture feature vector representing the entire action sequence. The first fully connected layer of the quality evaluation subnetwork linearly transforms the feature vector to the hidden space and introduces non-linearity through an activation function. The second fully connected layer maps the hidden representation to a single scalar value, and the sigmoid function compresses this scalar to the zero-to-one interval as a confidence score. This score reflects the intrinsic quality level of the input feature vector. During training, the subnetwork learns the mapping relationship between feature quality and recognition performance by using a large number of samples of different quality levels and their corresponding manually labeled quality scores, enabling it to automatically evaluate the reliability of the feature vector based on its distribution characteristics.

[0045] Figure 2This diagram illustrates the verification of the adaptive weight adjustment effect in this application's embodiments. The horizontal axis represents the test scenario number, and the vertical axis represents the recognition accuracy. The dashed line represents the recognition accuracy using the fixed weight fusion method, and the solid line represents the recognition accuracy using the adaptive weight fusion method. As can be seen from the diagram, under different test scenarios, the recognition accuracy of the adaptive weight fusion method consistently remains at a high level above 0.75, while the fixed weight fusion method shows a significant decrease in accuracy in certain scenarios (such as scenarios 3, 9, and 18), dropping as low as 0.58, indicating that the fixed weight fusion method cannot cope with modal quality fluctuations. The adaptive weight fusion method of this application, by evaluating the confidence scores of each modality in real time and dynamically adjusting the fusion weights, maintains high recognition performance even in scenarios with degraded modal quality, achieving an average accuracy improvement of approximately 6 percentage points compared to the fixed weight method.

[0046] In one specific embodiment, step S3, determining that a mode with a confidence score lower than a quality threshold is a low-quality mode, includes:

[0047] The confidence score is compared with a preset quality threshold. When the confidence score is less than the preset quality threshold, the corresponding mode is marked as a low-quality mode. When the confidence score is greater than or equal to the preset quality threshold, the corresponding mode is marked as a non-low-quality mode.

[0048] Specifically, a preset quality threshold is used as the criterion for judging modality quality, usually set to 0.4. The confidence score of each modality is compared with this threshold one by one. When the confidence score of a modality is lower than 0.4, it means that the current data quality of the modality is insufficient to reliably support the recognition task. It is marked as a low-quality modality and its influence weight is reduced in subsequent fusion. On the other hand, the confidence score of a modality is equal to or higher than 0.4, indicating that its data quality is within an acceptable range. It is marked as a non-low-quality modality and given normal or enhanced weight in fusion. This binary classification labeling mechanism transforms continuous confidence scores into discrete quality levels, which facilitates the use of differentiated processing strategies in subsequent weight calculations.

[0049] In one specific embodiment, step S3, calculating the compensation gain coefficient for non-low-quality modes based on the cumulative difference between low-quality modes and the quality threshold, includes:

[0050] Iterate through all low-quality modes and calculate the difference between the confidence score of each low-quality mode and the preset quality threshold.

[0051] The total quality loss is obtained by summing the differences of all low-quality modes;

[0052] For each non-low-quality mode, the corresponding compensation gain coefficient is obtained by multiplying the preset compensation strength parameter by the total amount of missing quality and adding one.

[0053] Specifically, the calculation of the compensation gain coefficient first iterates through all modes marked as low quality, calculating the difference between the preset quality threshold of 0.4 and the confidence score of each low-quality mode. This difference quantifies the degree of quality loss in a single mode. Then, the differences of all low-quality modes are summed to obtain the total quality loss, which reflects the severity of quality loss in the overall multimodal data. Next, for each non-low-quality mode, the preset compensation intensity parameter of 0.6 is multiplied by the total quality loss and then 1 is added to obtain the compensation gain coefficient of that mode. A coefficient greater than 1 means that the contribution of non-low-quality modes needs to be amplified to compensate for the information loss caused by low-quality modes. The larger the total quality loss, the higher the gain coefficient, thereby realizing a quality-driven active compensation mechanism.

[0054] Figure 3 This is a schematic diagram illustrating the impact analysis of the compensation intensity parameter in this application embodiment. The horizontal axis represents the value of the compensation intensity parameter α, and the vertical axis represents the recognition accuracy. The line-filled bar chart represents the recognition accuracy in normal scenarios, and the grid-filled bar chart represents the recognition accuracy in quality degradation scenarios. As can be seen from the figure, in normal scenarios, the compensation intensity parameter α has a relatively small impact on the recognition accuracy, with the accuracy remaining stable between 0.82 and 0.85 for each α value. However, in quality degradation scenarios, the compensation intensity parameter α has a significant impact on performance: when α is 0, the compensation mechanism is not enabled, and the accuracy is only 0.58; as α increases to 0.6, the accuracy increases to 0.78, an improvement of 20 percentage points; when α continues to increase to 0.8, the accuracy reaches a peak of 0.80. Experimental results show that choosing α as 0.6 can achieve a significant performance improvement in quality degradation scenarios without negatively impacting normal scenarios, verifying the effectiveness of the compensation gain mechanism in this application.

[0055] In one specific embodiment, step S3 involves normalizing the product of the compensation gain coefficient and the corresponding confidence score to obtain the non-low-quality mode weights, and directly normalizing the confidence scores of the low-quality modes to obtain the low-quality mode weights, forming an adaptive fusion weight reassembly, including:

[0056] The post-gain confidence is obtained by multiplying the compensation gain coefficient of each non-low-quality mode by the corresponding confidence score.

[0057] The total confidence score is obtained by summing the post-gain confidence scores of all non-low-quality modes with the confidence scores of all low-quality modes.

[0058] The weights of non-low-quality modes are obtained by dividing the post-gain confidence of each non-low-quality mode by the total confidence.

[0059] The weight of a low-quality mode is obtained by dividing the confidence score of each low-quality mode by the total confidence score.

[0060] All non-low-quality modal weights and low-quality modal weights are combined to form an adaptive fusion weight reorganization.

[0061] Specifically, the post-gain confidence score is obtained by multiplying the compensation gain coefficient of each non-low-quality mode by its original confidence score. This operation applies the gain coefficient as an amplification factor to the confidence score, resulting in higher numerical representations for higher-quality modes, while low-quality modes retain their original confidence scores without gain amplification. The total confidence score is calculated by summing the post-gain confidence scores of all non-low-quality modes with the original confidence scores of all low-quality modes. This sum serves as the normalized denominator, ensuring that the sum of the mode weights in subsequent calculations is strictly equal to 1.

[0062] Normalized weights are calculated using division. For each non-low-quality mode, its post-gain confidence score is divided by the total confidence score to obtain its final weight value. This weight value reflects the proportion that mode should occupy in the fusion process. Due to gain amplification, the weight of non-low-quality modes is usually higher than the proportion of their original confidence scores to the total. For each low-quality mode, its original confidence score is directly divided by the total confidence score to obtain its weight value. This relatively small weight value reflects the suppression of the influence of low-quality data. All non-low-quality mode weights and low-quality mode weights are arranged and combined in modality order to form an adaptive fusion weight reassembly. The sum of all elements in this weight reassembly is 1, and the value of each element is dynamically adjusted according to the real-time quality status. High-quality modes receive larger weights, while the weights of low-quality modes are compressed, realizing a quality-aware adaptive weight allocation mechanism.

[0063] In one specific embodiment, step S4 includes:

[0064] Each weight value in the adaptive fusion weighted recombination is multiplied element-wise with the corresponding multimodal feature vector to obtain the weighted feature vector;

[0065] The fused feature vector is obtained by summing all weighted feature vectors element by element. The dimension of the fused feature vector is the same as that of the individual multimodal feature vectors.

[0066] The fused feature vector is input into a small model classifier, which contains a multi-layer fully connected network. The first fully connected layer maps the fused feature vector to the first hidden layer dimension and processes it through an activation function and dropout. The second fully connected layer maps it to the second hidden layer dimension and processes it through an activation function and dropout. The third fully connected layer outputs a logits vector with a preset number of categories.

[0067] The softmax function is applied to the logits vector to calculate the probability distribution of each category, and the category with the highest probability is selected as the result of interaction intent recognition.

[0068] Specifically, the weighted feature vector is generated through element-wise multiplication. The process involves multiplying the visual modality weights in the adaptive fusion weighting by each element of the visual feature vector, the audio modality weights by each element of the audio feature vector, and the gesture modality weights by each element of the gesture feature vector. This multiplication operation broadcasts the scalar weight values ​​to each dimension of the entire feature vector, achieving uniform scaling. The fused feature vector is obtained by summing the elements of all weighted feature vectors at their corresponding positions. Specifically, the first elements of the visual, audio, and gesture weighted feature vectors are added together to obtain the first element of the fused feature vector; the second elements are added together to obtain the second element, and so on, until all dimensions are accumulated. Since the modal feature vectors have the same dimension and the weighted summation does not change the dimension, the dimension of the fused feature vector remains consistent with the dimension of a single multimodal feature vector—256 dimensions.

[0069] The first fully connected layer of the small model classifier maps 256-dimensional features to a 128-dimensional first hidden layer space by multiplying the weight matrix with the fused feature vector and adding a bias term. The activation function applies a non-linear transformation to the hidden layer output element by element to enhance the network's expressive power, and dropout randomly sets some neuron outputs to zero to prevent overfitting. The second fully connected layer maps the 128-dimensional hidden representation to a 64-dimensional second hidden layer space and is similarly processed by the activation function and dropout. The third fully connected layer maps the 64-dimensional features to a preset number of categories, outputting a logits vector. Each element of this vector corresponds to the original score of an interaction intent category. The softmax function first calculates the exponential function value for each element in the logits vector, then sums all the exponential values ​​as the normalized denominator. The probability of each category is equal to its exponential value divided by the sum. The exponential operation amplifies the score differences between categories, making the probability distribution more concentrated. Finally, the index of the category with the highest probability value is selected as the interaction intent recognition result output.

[0070] The above describes the vehicle-to-machine multimodal interaction recognition method based on a small model in the embodiments of this application. The following describes the vehicle-to-machine multimodal interaction recognition system based on a small model in the embodiments of this application. One embodiment of the vehicle-to-machine multimodal interaction recognition system based on a small model in the embodiments of this application includes:

[0071] The acquisition module is used to acquire visual image sequences, audio signals, and gaze data, preprocess the visual image sequences, audio signals, and gaze data, and align them according to timestamps to form a synchronous multimodal data stream;

[0072] The input module is used to input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and to run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector.

[0073] The calculation module is used to determine that modes with confidence scores lower than the quality threshold are low-quality modes. For non-low-quality modes, it calculates a compensation gain coefficient by accumulating the difference between the low-quality mode and the quality threshold. It then normalizes the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality mode. Finally, it directly normalizes the confidence score of the low-quality mode to obtain the weight of the low-quality mode, thus forming an adaptive fusion weight reassembly.

[0074] The execution module is used to perform weighted summation on the multimodal feature vector using the adaptive fusion weight recombination while keeping the feature dimension unchanged to obtain a fused feature vector, and input the fused feature vector into a small model classifier to output the interactive intent recognition result.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle-to-machine multimodal interaction recognition method based on a small model, characterized in that, The method includes: Step S1: Acquire visual image sequences, audio signals, and gaze data; preprocess the visual image sequences, audio signals, and gaze data; and align them by timestamps to form a synchronous multimodal data stream. Step S2: Input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector; Step S3: Determine the modes whose confidence scores are lower than the quality threshold as low-quality modes. For non-low-quality modes, calculate the compensation gain coefficient by accumulating the difference between the low-quality modes and the quality threshold. Normalize the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality modes. Directly normalize the confidence scores of the low-quality modes to obtain the weight of the low-quality modes, forming an adaptive fusion weight reassembly. Step S4: Use the adaptive fusion weight recombination to perform weighted summation on the multimodal feature vector while keeping the feature dimension unchanged to obtain a fused feature vector. Input the fused feature vector into the small model classifier to output the interactive intent recognition result.

2. The vehicle-to-machine multimodal interaction recognition method based on a small model according to claim 1, characterized in that, Step S1 includes: The system acquires visual image sequences at a preset resolution and frame rate using a camera, acquires audio signals at a preset sampling rate using a microphone, and acquires line-of-sight data using an infrared camera. Frame difference detection is performed on the visual image sequence, and valid interactive frames are obtained when the sum of pixel differences in consecutive frame images exceeds a preset motion threshold. The short-time energy and short-time zero-crossing rate are calculated for the audio signal. When the short-time energy is greater than a preset energy threshold and the short-time zero-crossing rate is less than a preset zero-crossing threshold, it is determined to be a valid speech segment. The valid speech segment is then subjected to spectral subtraction for noise reduction processing to obtain a noise-reduced frequency band. A Kalman filter is applied to the gaze direction data to smooth the trajectory. The state transition matrix of the Kalman filter is set according to the reciprocal of the preset frame rate to obtain a smooth gaze trajectory. The effective interaction frames, noise-reduced frequency bands, and smoothed line-of-sight trajectories are aligned by timestamps to form the synchronous multimodal data stream.

3. The vehicle-to-machine multimodal interaction recognition method based on a small model according to claim 1, characterized in that, Step S2 includes: The visual data input from the synchronous multimodal data stream is used to construct a visual feature extraction mini-model using depthwise separable convolution. The visual feature extraction mini-model contains multiple inverse residual modules. Each inverse residual module includes a first convolutional layer to expand channels, a deep convolutional layer to extract spatial features, an attention module to perform channel weighting, and a second convolutional layer to compress channels, outputting a visual feature vector. The audio data in the synchronous multimodal data stream is converted into acoustic features through a filter bank and then input into a lightweight audio feature extraction mini-model. The audio feature extraction mini-model contains a convolutional subsampling layer and multiple block structures. Each block structure contains a multi-head self-attention mechanism and a depthwise separable convolutional module, and outputs an audio feature vector. The continuous frame hand key point coordinate sequence extracted from the gaze data in the synchronous multimodal data stream is input into a gesture feature extraction mini-model constructed by a temporal convolutional network. The gesture feature extraction mini-model contains multiple layers of dilated convolutions and outputs a gesture feature vector. The visual feature vector, audio feature vector, and gesture feature vector are respectively input into the corresponding quality assessment sub-network. The quality assessment sub-network is a two-layer fully connected network structure, and the corresponding confidence score is output through the sigmoid function.

4. The vehicle-to-machine multimodal interaction recognition method based on a small model according to claim 1, characterized in that, Step S3, which determines that a mode with a confidence score below a quality threshold is a low-quality mode, includes: The confidence score is compared with a preset quality threshold. When the confidence score is less than the preset quality threshold, the corresponding mode is marked as a low-quality mode. When the confidence score is greater than or equal to the preset quality threshold, the corresponding mode is marked as a non-low-quality mode.

5. The vehicle-to-machine multimodal interaction recognition method based on a small model according to claim 4, characterized in that, In step S3, the compensation gain coefficient for non-low-quality modes is calculated by accumulating the difference between the low-quality mode and the quality threshold, including: Iterate through all low-quality modes and calculate the difference between the confidence score of each low-quality mode and the preset quality threshold. The total quality loss is obtained by summing the differences of all low-quality modes; For each non-low-quality mode, the corresponding compensation gain coefficient is obtained by multiplying the preset compensation intensity parameter by the total amount of quality loss and adding one.

6. The vehicle-to-machine multimodal interaction recognition method based on a small model according to claim 5, characterized in that, In step S3, the non-low-quality mode weights are obtained by normalizing the product of the compensation gain coefficient and the corresponding confidence score, and the low-quality mode weights are obtained by directly normalizing the confidence scores of the low-quality modes, forming an adaptive fusion weight reassembly, including: The post-gain confidence is obtained by multiplying the compensation gain coefficient of each non-low-quality mode by the corresponding confidence score. The total confidence score is obtained by summing the post-gain confidence scores of all non-low-quality modes with the confidence scores of all low-quality modes. The weights of the non-low-quality modes are obtained by dividing the post-gain confidence of each non-low-quality mode by the total confidence. The low-quality mode weight is obtained by dividing the confidence score of each low-quality mode by the total confidence score. All non-low-quality modal weights and low-quality modal weights are combined to form the adaptive fusion weight recombination.

7. The vehicle-to-machine multimodal interaction recognition method based on a small model according to claim 1, characterized in that, Step S4 includes: Each weight value in the adaptive fusion weighting is multiplied element-wise with the corresponding multimodal feature vector to obtain a weighted feature vector; The fused feature vector is obtained by summing all weighted feature vectors element by element, and the dimension of the fused feature vector is the same as the dimension of a single multimodal feature vector. The fused feature vector is input into a small model classifier, which contains a multi-layer fully connected network. The first fully connected layer maps the fused feature vector to the first hidden layer dimension and processes it through an activation function and dropout. The second fully connected layer maps the fused feature vector to the second hidden layer dimension and processes it through an activation function and dropout. The third fully connected layer outputs a logits vector with a preset number of categories. The softmax function is applied to the logits vector to calculate the probability distribution of each category, and the category with the highest probability is selected as the result of the interaction intent recognition.

8. A vehicle-to-everything (V2X) multimodal interaction recognition system based on a small model, characterized in that, For implementing the vehicle-to-machine multimodal interaction recognition method based on a small model as described in any one of claims 1-7, the vehicle-to-machine multimodal interaction recognition system based on a small model comprises: The acquisition module is used to acquire visual image sequences, audio signals, and gaze data, preprocess the visual image sequences, audio signals, and gaze data, and align them according to timestamps to form a synchronous multimodal data stream; The input module is used to input the synchronous multimodal data stream into multiple lightweight feature extraction mini-models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors, and to run the quality assessment sub-network in parallel to calculate the confidence score corresponding to each feature vector. The calculation module is used to determine that modes with confidence scores lower than the quality threshold are low-quality modes. For non-low-quality modes, it calculates a compensation gain coefficient by accumulating the difference between the low-quality mode and the quality threshold. It then normalizes the product of the compensation gain coefficient and the corresponding confidence score to obtain the weight of the non-low-quality mode. Finally, it directly normalizes the confidence score of the low-quality mode to obtain the weight of the low-quality mode, thus forming an adaptive fusion weight reassembly. The execution module is used to perform weighted summation on the multimodal feature vector using the adaptive fusion weight recombination while keeping the feature dimension unchanged to obtain a fused feature vector, and input the fused feature vector into a small model classifier to output the interactive intent recognition result.

9. The system according to claim 8, characterized in that, Acquire visual image sequences, audio signals, and gaze data; preprocess the visual image sequences, audio signals, and gaze data; align the data by timestamp to form a synchronous multimodal data stream, including: The system acquires visual image sequences at a preset resolution and frame rate using a camera, acquires audio signals at a preset sampling rate using a microphone, and acquires line-of-sight data using an infrared camera. Frame difference detection is performed on the visual image sequence, and valid interactive frames are obtained when the sum of pixel differences in consecutive frame images exceeds a preset motion threshold. The short-time energy and short-time zero-crossing rate are calculated for the audio signal. When the short-time energy is greater than a preset energy threshold and the short-time zero-crossing rate is less than a preset zero-crossing threshold, it is determined to be a valid speech segment. The valid speech segment is then subjected to spectral subtraction for noise reduction processing to obtain a noise-reduced frequency band. A Kalman filter is applied to the gaze direction data to smooth the trajectory. The state transition matrix of the Kalman filter is set according to the reciprocal of the preset frame rate to obtain a smooth gaze trajectory. The effective interaction frames, noise-reduced frequency bands, and smoothed line-of-sight trajectories are aligned by timestamps to form the synchronous multimodal data stream.

10. The system according to claim 8, characterized in that, The synchronous multimodal data stream is input into multiple lightweight feature extraction models with fewer than 5M parameters to extract fixed-dimensional multimodal feature vectors. A quality assessment sub-network is then run in parallel to calculate the confidence score corresponding to each feature vector, including: The visual data input from the synchronous multimodal data stream is used to construct a visual feature extraction mini-model using depthwise separable convolution. The visual feature extraction mini-model contains multiple inverse residual modules. Each inverse residual module includes a first convolutional layer to expand channels, a deep convolutional layer to extract spatial features, an attention module to perform channel weighting, and a second convolutional layer to compress channels, outputting a visual feature vector. The audio data in the synchronous multimodal data stream is converted into acoustic features through a filter bank and then input into a lightweight audio feature extraction mini-model. The audio feature extraction mini-model contains a convolutional subsampling layer and multiple block structures. Each block structure contains a multi-head self-attention mechanism and a depthwise separable convolutional module, and outputs an audio feature vector. The continuous frame hand key point coordinate sequence extracted from the gaze data in the synchronous multimodal data stream is input into a gesture feature extraction mini-model constructed by a temporal convolutional network. The gesture feature extraction mini-model contains multiple layers of dilated convolutions and outputs a gesture feature vector. The visual feature vector, audio feature vector, and gesture feature vector are respectively input into the corresponding quality assessment sub-network. The quality assessment sub-network is a two-layer fully connected network structure, and the corresponding confidence score is output through the sigmoid function.

Images