Vehicle active service system based on end-side multi-modal large model and dynamic inference method thereof

Abstract

The application provides a vehicle active service system based on an end-side multimodal large model and a dynamic reasoning method thereof, and belongs to the field of Internet of Vehicles, and comprises a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module, a multimodal data acquisition module

Description

Technical Field

[0001] This invention relates to the field of vehicle networking technology, and in particular to a vehicle active service system based on a multimodal large model on the edge side and its dynamic reasoning method. Background Technology

[0002] With the rapid development of automotive intelligence, in-vehicle active service systems have become an important technological direction for enhancing the driving experience. Existing in-vehicle systems primarily rely on driver-initiated functions, and still have many shortcomings in understanding driving scenarios and providing personalized services.

[0003] Traditional in-vehicle service systems typically employ large cloud-based models for data processing and decision-making, which suffers from the following technical drawbacks: First, cloud reliance leads to high network latency, with response times often exceeding 500-1000 milliseconds, making it difficult to meet the real-time requirements of in-vehicle scenarios. Second, user data needs to be uploaded to the cloud for processing, posing a risk of privacy breaches. Third, the accuracy of single-modal sensor recognition is limited, especially in low-light environments, where traditional RGB cameras cannot effectively recognize complex driving scenarios. Finally, existing systems lack dynamic reasoning mechanisms, failing to dynamically acquire the necessary parameters based on intermediate results of the reasoning process, resulting in inaccurate and inflexible reasoning decisions.

[0004] Chinese invention patent CN115175138B discloses a method and in-vehicle system for providing online services for in-vehicle applications based on user characteristics. This patent uses a first in-vehicle device to detect the characteristic data of each user currently served by multiple second in-vehicle devices. Based on the characteristic data, it determines the user's in-vehicle application usage preferences, obtains the corresponding online access token, and shares it with the second in-vehicle devices. When the target in-vehicle device receives an online access command from the user, it obtains the corresponding token from the shared access tokens and provides online services to the user based on the access command and the token. This technical solution mainly solves the problem of users enjoying personalized services without registering a personal in-vehicle account. However, its service triggering still relies on user-initiated operation rather than system-initiated perception and lacks environmental understanding capabilities. Summary of the Invention

[0005] The technical solution of this invention is implemented as follows: This invention provides a vehicle active service system based on an edge-side multimodal large model, comprising:

[0006] The multimodal data acquisition module is used to collect multimodal perception data of the vehicle; the multimodal perception data includes image data, in-vehicle audio signal data and vehicle driving status parameter data;

[0007] The multimodal large model processing unit is used to extract features from multimodal perception data to obtain features of different modalities; the multimodal large model is compressed using knowledge distillation and quantization and deployed on the vehicle-mounted edge.

[0008] The cross-modal attention fusion module is used to convert feature data from different modalities into query, key, and value triples, calculate cross-modal similarity weights through a bidirectional attention mechanism, and fuse them based on query, key, and value triples and cross-modal similarity weights to obtain fused features;

[0009] The MCP dynamic inference controller is used to make a preliminary judgment on the vehicle scenario based on fused features. When the preliminary judgment result needs to be further confirmed, it generates a supplementary data acquisition instruction to control the multimodal data acquisition module to collect specific parameter data. The supplementary data is combined with the fused features to perform multi-step inference, determine the current scenario type of the vehicle, and generate proactive service decisions by combining user profiles and historical behavior data.

[0010] The proactive service execution module is used to receive proactive service decisions, generate on-board equipment control commands, and execute proactive service operations.

[0011] Based on the above technical solutions, preferably, the image data includes RGB image data and infrared image data, and the multimodal data acquisition module further includes:

[0012] The brightness scoring unit is used to convert RGB image data into color space and calculate the brightness score value. When the brightness score value is lower than a preset threshold, the system switches from the visible light camera to the infrared camera for image acquisition.

[0013] The infrared vision enhancement unit is used to acquire infrared thermal radiation image data using an infrared camera, and to extract thermal radiation features and perform dynamic exposure compensation on the infrared thermal radiation images to obtain infrared image data.

[0014] Based on the above technical solutions, preferably, the multimodal large model processing unit adopts a teacher-student network architecture for knowledge distillation, specifically including:

[0015] In the output layer distillation, the output probability distributions of the teacher model and the student model are smoothed. The KL divergence loss function is used to calculate the difference between the smoothed probability distributions of the teacher model and the student model. The KL divergence loss is weighted and combined with the cross-entropy loss of the student model to the true label to form a joint loss function for optimization training.

[0016] In intermediate feature distillation, the intermediate feature map sizes of the teacher model and the student model are adjusted by a 1×1 convolutional adaptation layer, and the intermediate feature representations of the teacher model and the student model are aligned using the mean squared error loss function.

[0017] The weight parameters and activation values ​​of the student model are quantized using 8-bit integer quantization technology. By reducing the storage precision of the model parameters through channel-by-channel quantization, the large multimodal model is compressed to less than 500MB.

[0018] Based on the above technical solutions, preferably, the step of converting feature data of different modalities into query, key, and value triples specifically includes:

[0019] Visual feature vectors are extracted from image data using a visual transformer encoder, audio embedding feature vectors are extracted from audio signal data, and state feature vectors are generated by feature encoding of vehicle driving state parameter data.

[0020] The feature vectors of different modalities are projected and transformed into query vector, key vector and value vector respectively through a linear transformation matrix;

[0021] The feature dimensions of the query vector, key vector, and value vector are uniformly set to the same feature space dimension, forming a triplet representation with a unified dimension.

[0022] Based on the above technical solutions, preferably, the calculation method of the bidirectional attention mechanism specifically includes:

[0023] The forward attention weight matrix is ​​obtained by performing a dot product operation between the query vector and the key vector, and then normalizing it using softmax based on the scaling factor.

[0024] The reverse attention weight matrix is ​​obtained by performing a dot product operation between the transpose of the key vector and the query vector, and then normalizing it using softmax based on the scaling factor.

[0025] The forward attention output is obtained by weighted summation of the forward attention weight matrix and the value vector, and the reverse attention output is obtained by weighted summation of the reverse attention weight matrix and the query vector. The weighted average of the forward attention output and the reverse attention output is calculated, and after residual connection and layer normalization, the fused feature is output.

[0026] Based on the above technical solutions, preferably, the MCP dynamic inference controller includes a dynamic inference client and multiple multi-parameter servers, and the processing steps of the MCP dynamic inference controller specifically include:

[0027] The dynamic inference client communicates with the multi-parameter server through a multi-step conditional inference protocol. Upon startup, it automatically identifies all available multi-parameter servers and obtains descriptions of the data acquisition capabilities provided by each server.

[0028] Based on the fusion features, a preliminary classification judgment of vehicle scenarios is made through a preset scenario classification model, generating a preliminary judgment result containing scenario type identifier and confidence value;

[0029] When the confidence level of the preliminary judgment result is lower than the set threshold, the dynamic inference client sends a supplementary data collection instruction to the corresponding multi-parameter server based on the scenario type identifier.

[0030] Based on the above technical solutions, preferably, the step of generating proactive service decisions by combining user profiles and historical behavior data specifically includes:

[0031] A driving state detection algorithm is used to extract the driver's physiological and behavioral feature parameters as the first inference parameters. The driving state detection algorithm includes at least one of the following: a fatigue detection algorithm based on facial key point detection, a steering wheel grip force detection algorithm, and an attention distraction detection algorithm.

[0032] When the first inference parameter indicates an abnormal driving state, the dynamic inference client requests the multi-parameter server to collect the corresponding verification parameters as the second inference parameter. The first and second inference parameters are then concatenated and input into the multi-step inference network for scene confirmation.

[0033] User personal preference features are extracted from the user profile database, and historical behavior pattern features are extracted from the database. The confirmed scenario type is then weighted and fused with the user personal preference features and historical behavior pattern features to generate proactive service decisions. The proactive service decisions include service type, service parameters, and execution sequence.

[0034] Based on the above technical solutions, preferably, the active service execution module includes:

[0035] The decision parsing unit is used to parse the service type, service parameters and execution timing information in the proactive service decision, match the corresponding vehicle equipment control protocol according to the service type, and generate equipment control parameters.

[0036] The instruction generation unit is used to generate control instructions based on the device control parameters, wherein the control instructions include seat vibration control instructions, air conditioning temperature adjustment control instructions, navigation rest area search control instructions, and audio volume adjustment control instructions.

[0037] The execution monitoring unit is used to monitor the execution status of control commands of the vehicle-mounted equipment through the device execution interface, record the control command issuance time, execution completion time, and execution result status, and generate execution feedback information including execution success rate and response time.

[0038] Furthermore, preferably, the proactive service execution module also includes a user interaction and feedback processing mechanism:

[0039] Before performing proactive service operations, proactive service semantic prompts are generated through voice synthesis technology to explain the detected driving scenario and the service measures to be taken to the user. At the same time, a proactive service confirmation card with confirmation and cancellation options is displayed on the in-vehicle display interface.

[0040] The system receives feedback information input by the user via voice commands or touch operation. When a confirmation feedback is received, the system executes the control command of the in-vehicle device. When a cancellation feedback is received, the system terminates the current service execution process and updates the user's feedback information to the database as personalized preference data.

[0041] This invention also provides a dynamic reasoning method for vehicle proactive services based on an edge-side multimodal large model, applied to the vehicle proactive service system based on an edge-side multimodal large model as described above, comprising the following steps:

[0042] S1. Collect multimodal perception data of the vehicle; the multimodal perception data includes image data, in-vehicle audio signal data and vehicle driving status parameter data;

[0043] S2. Feature extraction is performed on the multimodal perception data to obtain features of different modalities; among them, the multimodal large model is compressed using knowledge distillation and quantization and deployed on the vehicle-mounted edge.

[0044] S3. Convert feature data from different modalities into query, key, and value triples, calculate cross-modal similarity weights through a bidirectional attention mechanism, and fuse them based on query, key, and value triples and cross-modal similarity weights to obtain fused features;

[0045] S4. Make a preliminary judgment on the vehicle scenario based on the fusion features. When the preliminary judgment result needs to be further confirmed, generate a supplementary data collection instruction to control the multimodal data collection module to collect specific parameter data, combine the supplementary data with the fusion features to perform multi-step reasoning, determine the current scenario type of the vehicle, and generate proactive service decisions by combining user profiles and historical behavior data.

[0046] S5. Receive proactive service decisions, generate on-board equipment control commands, and execute proactive service operations.

[0047] The vehicle active service system and its dynamic reasoning method based on an edge-side multimodal large model of the present invention have the following advantages over the prior art:

[0048] (1) By using the edge-side multimodal large model and the MCP dynamic inference protocol, the real-time response and intelligent decision-making capabilities of the vehicle active service system are realized. Through multimodal feature fusion and cross-modal attention mechanism, the system transforms from a passive response mode to an active service mode, which significantly improves the inference flexibility and environmental adaptability.

[0049] (2) By using knowledge distillation technology, the large-scale multimodal model is compressed into a lightweight version suitable for vehicle-mounted deployment. Through the teacher-student network architecture and intermediate feature alignment mechanism, the model storage requirements and computational complexity are greatly reduced while maintaining inference accuracy. The dependence on network connection is eliminated, and a millisecond-level response speed is achieved.

[0050] (3) The MCP protocol adopts a multi-step inference network architecture, which can determine whether a second inference parameter needs to be added based on the confidence level of the first inference parameter. This realizes the on-demand allocation of computing resources, which significantly reduces the computing load and energy consumption of the end-side devices while ensuring the accuracy of decision-making, and improves the overall operating efficiency of the system. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is an architecture diagram of the vehicle active service system based on a multimodal large model on the edge side according to the present invention;

[0053] Figure 2 This is a hardware deployment diagram of the vehicle active service system based on a multimodal large model on the edge side according to the present invention;

[0054] Figure 3 This is a flowchart of the MCP protocol for the vehicle active service system based on a multimodal large model on the edge side, as described in this invention.

[0055] Figure 4 This is a schematic diagram of the MCP protocol of the vehicle active service system based on the edge-side multimodal large model of the present invention;

[0056] Figure 5 This is a schematic diagram of the key eye points in the fatigue detection algorithm of the vehicle active service system based on the edge-side multimodal large model of the present invention. Detailed Implementation

[0057] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0058] like Figure 1 As shown, this invention provides a vehicle active service system based on an edge-side multimodal large model, comprising:

[0059] The multimodal data acquisition module is used to collect multimodal perception data of the vehicle; the multimodal perception data includes image data, in-vehicle audio signal data and vehicle driving status parameter data;

[0060] The multimodal large model processing unit is used to extract features from multimodal perception data to obtain features of different modalities; the multimodal large model is compressed using knowledge distillation and quantization and deployed on the vehicle-mounted edge.

[0061] The cross-modal attention fusion module is used to convert feature data from different modalities into query, key, and value triples, calculate cross-modal similarity weights through a bidirectional attention mechanism, and fuse them based on query, key, and value triples and cross-modal similarity weights to obtain fused features;

[0062] The MCP dynamic inference controller is used to make a preliminary judgment on the vehicle scenario based on fused features. When the preliminary judgment result needs to be further confirmed, it generates a supplementary data acquisition instruction to control the multimodal data acquisition module to collect specific parameter data. The supplementary data is combined with the fused features to perform multi-step inference, determine the current scenario type of the vehicle, and generate proactive service decisions by combining user profiles and historical behavior data.

[0063] The proactive service execution module is used to receive proactive service decisions, generate on-board equipment control commands, and execute proactive service operations.

[0064] This invention achieves real-time response and intelligent decision-making capabilities for in-vehicle active service systems through edge-side multimodal large models and the MCP dynamic inference protocol. By using multimodal feature fusion and cross-modal attention mechanisms, it transforms from a passive response mode to an active service mode, significantly improving inference flexibility and environmental adaptability, and providing a more reliable technical solution for in-vehicle intelligent interaction.

[0065] like Figure 2As shown, the vehicle active service system includes data acquisition components such as RGB cameras, infrared cameras, voice sensors, and in-vehicle SDKs. It intelligently switches visual sensors through a brightness scoring unit, extracts multimodal features using ASR speech recognition and VLM visual language models, and performs preliminary scene analysis through an active service inference engine. Multiple parameter servers, including MCPServer (multi-parameter server) 1, 2, and 3, are uniformly coordinated and managed through the MCP Client (dynamic inference client). The inference agent's multi-step inference module dynamically requests additional parameters based on the preliminary analysis results and executes multi-step conditional inference. The active service execution module includes an active service engine and an execution engine, controlling specific in-vehicle hardware to perform active service operations through in-vehicle device interfaces such as RYT SDK1, SDK2, and SDK3. The entire architecture achieves modular collaborative work of data acquisition, inference decision-making, and service execution.

[0066] In one embodiment of the present invention, the image data includes RGB image data and infrared image data, and the multimodal data acquisition module further includes:

[0067] The brightness scoring unit is used to convert RGB image data into color space and calculate the brightness score value. When the brightness score value is lower than a preset threshold, the system switches from the visible light camera to the infrared camera for image acquisition.

[0068] The infrared vision enhancement unit is used to acquire infrared thermal radiation image data using an infrared camera, and to extract thermal radiation features and perform dynamic exposure compensation on the infrared thermal radiation images to obtain infrared image data.

[0069] Understandably, the RGB image data is captured using a high-definition RGB camera installed near the rearview mirror inside the vehicle, primarily responsible for collecting visual information such as the driver's facial expressions, eye state, and head posture. The infrared thermal radiation image data is captured using an 850nm infrared array camera, which forms images by detecting the infrared radiation emitted by objects at different temperatures. The thermal radiation feature extraction algorithm analyzes the radiation intensity distribution in different regions of the infrared image to identify key feature points such as facial contours and eye areas. The dynamic exposure compensation algorithm automatically adjusts the camera's exposure parameters based on the ambient temperature and the thermal radiation intensity of the target object, ensuring the infrared image has appropriate contrast and clarity. The processed infrared image data has the same resolution and frame rate as the visible light image, guaranteeing data format consistency.

[0070] Specifically, the low-light recognition rate of this invention is compared with that of traditional cloud solutions and local fixed models. The data is shown in Table 1:

[0071] Table 1

[0072]

[0073] As shown in Table 1, this scheme significantly improves the low-light recognition rate and image detection accuracy through the brightness scoring unit and infrared vision enhancement unit.

[0074] In one embodiment of the present invention, the luminance scoring algorithm is implemented using a V-channel detection method based on the HSV color space, specifically including:

[0075] The image captured by the RGB camera is converted from the RGB color space to the HSV format. In the HSV color space, the V channel directly represents the brightness value.

[0076] V channel data is extracted, and two algorithms are used in parallel to calculate the brightness score: the average value method calculates the average value of all pixels in the V channel, and the environment is judged to be too dark when the average value is lower than the 30% threshold; the proportion method counts the proportion of pixels with V values ​​less than 20% of the total pixels, and the environment is judged to be dark when the proportion exceeds 50%.

[0077] When both algorithms simultaneously meet the low-light detection criteria, the system automatically switches from the RGB camera to the infrared camera for image acquisition.

[0078] In one embodiment of the present invention, in-vehicle audio signal data is collected through an in-vehicle microphone array, including sound information such as driver's voice, in-vehicle conversations, and ambient noise. The in-vehicle audio signal data is preprocessed using steps such as noise reduction, echo cancellation, and voice enhancement. The noise reduction algorithm employs adaptive filtering technology to effectively suppress background interference such as engine noise and road noise. The voice enhancement algorithm uses spectral analysis and time-domain processing to highlight the human voice frequency band and improve the clarity of the voice signal.

[0079] In one embodiment of the present invention, vehicle driving status parameter data is collected by vehicle status sensors, including key parameters such as vehicle speed, steering wheel angle, braking status, turn signal status, and gear information. These sensors are connected to the vehicle's electronic control unit via the vehicle's CAN bus, enabling them to acquire various operating status information of the vehicle in real time.

[0080] In one embodiment of the present invention, the multimodal large model processing unit adopts a modal encoder design, using a dedicated feature extraction network for different types of input data. For image data, a visual transformer encoder (ViTEncoder) is used to extract visual feature vectors, capable of processing RGB and infrared image data simultaneously. The ViT encoder divides the input image into fixed-size image blocks, each of which is converted into an initial embedding vector through a linear projection layer, and then hierarchical visual features are extracted through a multi-layer Transformer structure. For audio signal data, an audio feature extraction network is used to generate audio embedding feature vectors. The audio processing flow includes pre-emphasis, temporal segmentation and smoothing, fast Fourier transform, and other preprocessing steps. Spectral features and temporal features are extracted through a deep convolutional neural network. The audio feature vector encodes multi-dimensional information such as speech content, speaker features, and emotional state. Vehicle driving state parameter data is processed by a dedicated state feature encoder. This encoder adopts a multi-layer perceptron structure, uniformly encoding discrete and continuous parameters such as vehicle speed, steering wheel angle, braking state, and turn signal state into state feature vectors. The dimension of the state feature vector is consistent with that of the visual feature vector and the audio feature vector.

[0081] Specifically, the multimodal large model processing unit employs a teacher-student network architecture for knowledge distillation, which includes:

[0082] In the output layer distillation, the output probability distributions of the teacher model and the student model are smoothed. The KL divergence loss function is used to calculate the difference between the smoothed probability distributions of the teacher model and the student model. The KL divergence loss is weighted and combined with the cross-entropy loss of the student model to the true label to form a joint loss function for optimization training.

[0083] In intermediate feature distillation, the intermediate feature map sizes of the teacher model and the student model are adjusted by a 1×1 convolutional adaptation layer, and the intermediate feature representations of the teacher model and the student model are aligned using the mean squared error loss function.

[0084] The weight parameters and activation values ​​of the student model are quantized using 8-bit integer quantization technology. By reducing the storage precision of the model parameters through channel-by-channel quantization, the large multimodal model is compressed to less than 500MB.

[0085] Furthermore, the quantization process employs Post-Training Quantization (PTQ), enabling model compression without retraining. Weight quantization uses a channel-by-channel approach, calculating independent quantization parameters for each output channel of each convolutional or fully connected layer. Activation quantization employs a dynamic quantization strategy, dynamically calculating quantization parameters based on the actual distribution of the input data during inference. For each activation tensor, the quantization parameters are updated using a moving average method to adapt to varying input data distributions.

[0086] Specifically, quantization processing employs two strategies: Post-Train Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ eliminates the need for retraining and is suitable for deployment. The process includes: preparing the original FP32 model, collecting 100-1000 representative calibration data samples, applying quantization algorithms such as MinMax, Histogram, or MSE approximation, and exporting the model in INT8 / INT4 format. Weight quantization uses a channel-by-channel quantization approach, calculating independent quantization parameters for each output channel of each convolutional or fully connected layer. Activation quantization employs a dynamic quantization strategy, dynamically calculating quantization parameters based on the actual distribution of the input data and updating them using a moving average method to adapt to changes in the distribution of different input data.

[0087] This invention employs knowledge distillation technology to compress large-scale multimodal models into lightweight versions suitable for vehicle-mounted deployment. Through a teacher-student network architecture and intermediate feature alignment mechanism, it significantly reduces model storage requirements and computational complexity while maintaining inference accuracy, eliminates dependence on network connectivity, achieves millisecond-level response speed, and ensures localized processing of user data. This effectively solves the technical challenges of network latency and privacy security in cloud-based solutions.

[0088] In one embodiment of the present invention, the cross-modal attention fusion module receives visual feature vectors, audio embedding feature vectors, and state feature vectors from a multimodal large model processing unit. The cross-modal attention fusion module performs dimensional unification and numerical normalization on the input feature vectors. The dimension of all feature vectors is uniformly set to 512 dimensions and represented using 32-bit floating-point numbers. Features of different original dimensions are mapped to a unified feature space through a linear projection layer. The feature vector normalization uses the L2 normalization method.

[0089] Specifically, the conversion of feature data of different modalities into query, key, and value triples includes:

[0090] Visual feature vectors are extracted from image data using a visual transformer encoder, audio embedding feature vectors are extracted from audio signal data, and state feature vectors are generated by feature encoding of vehicle driving state parameter data.

[0091] The feature vectors of different modalities are projected and transformed into query vector Q, key vector K, and value vector V respectively through a linear transformation matrix;

[0092] The feature dimensions of the query vector Q, key vector K, and value vector V are uniformly set to the same feature space dimension, forming a triplet representation with a unified dimension.

[0093] In one embodiment of the present invention, the calculation method of the bidirectional attention mechanism specifically includes:

[0094] The forward attention weight matrix is ​​obtained by performing a dot product operation between the query vector Q and the key vector K, and then normalizing it using softmax based on a scaling factor; the calculation formula is as follows:

[0095]

[0096] in, This represents the forward attention weight matrix. This represents the softmax normalization function. Represents the query vector. Represents the key vector. Indicates transpose. The feature space dimension is 512 in this invention. Indicates the scaling factor;

[0097] The reverse attention weight matrix is ​​obtained by performing a dot product operation between the transpose of the key vector K and the query vector Q, and then normalizing it using softmax based on a scaling factor; the calculation formula is as follows:

[0098]

[0099] in, Represents the reverse attention weight matrix;

[0100] The forward attention output is obtained by weighted summation of the forward attention weight matrix and the value vector, and the reverse attention output is obtained by weighted summation of the reverse attention weight matrix and the query vector. The weighted average of the forward attention output and the reverse attention output is calculated, and after residual connection and layer normalization, the fused feature is output.

[0101] In one embodiment of the present invention, the MCP dynamic inference controller includes a dynamic inference client (MCPClient) and multiple multi-parameter servers (MCP Server), and the processing steps of the MCP dynamic inference controller specifically include:

[0102] The dynamic inference client communicates with the multi-parameter server through a multi-step conditional inference protocol. Upon startup, it automatically identifies all available multi-parameter servers and obtains descriptions of the data acquisition capabilities provided by each server.

[0103] Based on the fusion features, a preliminary classification judgment of vehicle scenarios is made through a preset scenario classification model, generating a preliminary judgment result containing scenario type identifier and confidence value;

[0104] When the confidence level of the preliminary judgment result is lower than the set threshold, the dynamic inference client sends a supplementary data collection instruction to the corresponding multi-parameter server based on the scenario type identifier.

[0105] Understandably, the dynamic inference client, as the core coordination unit, is responsible for interacting with the in-vehicle large model, controlling inference logic, and scheduling services. Multi-parameter servers are responsible for different types of data acquisition and preprocessing tasks, including user information management servers, vehicle status query servers, environmental perception servers, and multimedia control servers.

[0106] The MCP protocol implementation adopts a three-step architecture deployment: First, an MCP Client module is built as a bridge connecting the large model and in-vehicle services. This module has the ability to communicate with the in-vehicle MCP Server via the standard MCP protocol, automatically discover available MCP Servers upon startup, and retrieve and cache service description information. Second, multiple MCP Server modules are developed, including a user information management server, a vehicle status query server, an environment perception server, and a multimedia control server. Each server provides independent functional modules and registers with the MCP Client. Third, available MCP Server information is preset in the MCP Client configuration file, supporting hot reloading and dynamic discovery mechanisms. The large model obtains the service list by calling the MCP Client through Function Calling, selects the appropriate service based on the service description, and executes the specific call.

[0107] Specifically, the scene classification model employs a Transformer-based multi-classification network structure, capable of recognizing various driving and riding scenario types such as fatigued driving, distracted driving, environmental anomalies, and user needs. The dynamic inference client intelligently selects the types of supplementary parameters to be collected based on the scene type identifier and confidence distribution, and sends supplementary data collection instructions to the corresponding multi-parameter server. The data structure of the supplementary data collection instructions includes information such as request type, parameter specifications, collection accuracy, and time window.

[0108] like Figure 3As shown, after a user submits a request, the large model requests a service description from the MCP Client via the Function Call mechanism. The MCP Client then retrieves service description information from MCP Servers A and B and returns a service list. The large model selects a suitable service based on the service description (such as the query service from Server A). The MCP Client forwards the request to the corresponding MCP Server to execute the specific operation. The service execution result is returned to both the large model and the user through the same path. The entire process reflects the standardized interaction mechanism of the MCP protocol in service discovery, dynamic invocation, and result return, ensuring efficient coordination between the client-side large model and the vehicle service module.

[0109] This invention uses a step-by-step conditional reasoning mechanism to dynamically request the required sensor data based on the preliminary reasoning results, avoiding the computational redundancy of full data acquisition and processing in traditional solutions. The MCP protocol adopts a multi-step reasoning network architecture, which can determine whether to supplement the second reasoning parameter based on the confidence level of the first reasoning parameter, realizing on-demand allocation of computing resources. While ensuring the accuracy of decision-making, it significantly reduces the computing load and energy consumption of the end-side devices and improves the overall operating efficiency of the system.

[0110] In one embodiment of the present invention, the step of generating proactive service decisions by combining user profiles and historical behavior data specifically includes:

[0111] The driver's physiological and behavioral feature parameters are extracted using a driving state detection algorithm as the first inference parameter. The driving state detection algorithm includes at least one of the following: a fatigue detection algorithm based on facial key point detection, a steering wheel grip force detection algorithm, and an attention distraction detection algorithm.

[0112] When the first inference parameter indicates an abnormal driving state, the dynamic inference client requests the multi-parameter server to collect the corresponding verification parameters as the second inference parameter. The first and second inference parameters are then concatenated and input into the multi-step inference network for scene confirmation.

[0113] User personal preference features are extracted from the user profile database, and historical behavior pattern features are extracted from the database. The confirmed scenario type is then weighted and fused with the user personal preference features and historical behavior pattern features to generate proactive service decisions. The proactive service decisions include service type, service parameters, and execution sequence. Service parameters define the specific configuration of the service, such as vibration intensity, temperature setting, and search range. The execution sequence specifies the execution order and time interval of the service to ensure the coordinated operation of multiple service operations.

[0114] Understandably, the output of the MCP dynamic inference controller is an active service decision containing complete decision information. This decision is encoded in JSON format and includes information such as service identifier, parameter configuration, execution sequence, and priority, providing clear execution guidance for the active service execution module.

[0115] like Figure 4 As shown, after the inference engine initiates the initial request, the MCP controller calls the first parameter set P1 for preliminary analysis. When the preliminary results require further confirmation, the system triggers the sensor cluster to collect supplementary data and obtain the second parameter set P2. The MCP controller fuses and analyzes the multi-step parameters to generate the final service instruction output. The entire process embodies the core idea of ​​dynamic inference, that is, dynamically obtaining the required parameters based on the intermediate results of the inference process to achieve an adaptive decision-making mechanism.

[0116] In one embodiment of the present invention, the multi-step inference network refers to the core network architecture in the MCP (Multi-step Conditional Protocol) dynamic inference controller that implements step-by-step conditional inference. Specifically,

[0117] The multi-step inference network employs a Transformer-based multi-classification network structure, capable of receiving a concatenated feature vector of the first inference parameter (such as EAR value) and the second inference parameter (such as steering wheel grip force data, vehicle lateral displacement, etc.) as input. The network progressively analyzes the correlation between different parameters through a multi-layer attention mechanism, outputting an inference result that includes scene confirmation and confidence score. The core advantage of this network lies in its ability to dynamically request and integrate new parameter information based on intermediate results during the inference process, achieving an adaptive inference decision-making process.

[0118] like Figure 5 As shown, specifically, taking fatigue driving detection as an example, a facial landmark detection algorithm is used to extract the driver's eye landmarks P1-P6, and the aspect ratio of the eyes is calculated. The calculation formula is as follows:

[0119]

[0120] in, Indicates the aspect ratio of the eyes. This represents the coordinates of the first key point in the eye area. This indicates the coordinates of the second key point of the eye. This indicates the coordinates of the third key point of the eye. This indicates the coordinates of the fourth key point of the eye. This indicates the coordinates of the fifth key point of the eye. This indicates the coordinates of the sixth key point of the eye. It represents Euclidean distance.

[0121] In a normal eye-opening state, the EAR value is approximately 0.25-0.35, while in a closed-eye state, the EAR value is close to 0. When the EAR parameter is continuously below the threshold of 0.2 for more than 2 seconds, it is determined that there may be a fatigue state. At this time, the dynamic inference client requests the multi-parameter server to collect steering wheel grip force data, vehicle lateral displacement, vehicle speed change rate, etc. as the second inference parameter. The first inference parameter (EAR value) and the second inference parameter are vector-connected, and the concatenated feature vector is input into the multi-step inference network for scene confirmation. The network output includes the scene confirmation result and confidence score.

[0122] In one embodiment of the present invention, the active service execution module includes:

[0123] The decision parsing unit is used to parse the service type, service parameters and execution timing information in the proactive service decision, match the corresponding vehicle equipment control protocol according to the service type, and generate equipment control parameters.

[0124] The instruction generation unit is used to generate control instructions based on the device control parameters, wherein the control instructions include seat vibration control instructions, air conditioning temperature adjustment control instructions, navigation rest area search control instructions, and audio volume adjustment control instructions.

[0125] The execution monitoring unit is used to monitor the execution status of control commands of the vehicle-mounted equipment through the device execution interface, record the control command issuance time, execution completion time, and execution result status, and generate execution feedback information including execution success rate and response time.

[0126] Understandably, the decision parsing unit is responsible for comprehensively parsing and processing the proactive service decisions from the MCP dynamic inference controller. It receives three key information dimensions: service type, service parameters, and execution sequence. The service type is identified by predefined enumeration values, including standard service types such as SEAT_VIBRATION (seat vibration), AC_TEMPERATURE (air conditioning temperature adjustment), NAVIGATION_SEARCH (navigation rest area search), and AUDIO_VOLUME (audio volume adjustment). The service parameter parsing adopts a key-value pair mapping mechanism to convert abstract service configurations into specific device control parameters. The execution sequence parsing uses a time series analysis algorithm to convert the time sequence information in the decision into a precise execution plan.

[0127] Specifically, for seat vibration services, parameters such as vibration intensity level, vibration mode, and duration are analyzed; for air conditioning temperature adjustment services, parameters such as target temperature value, fan speed level, and air outlet mode are analyzed.

[0128] In one embodiment of the present invention, the generation process of the seat vibration control command includes three steps: device address configuration, vibration parameter setting, and execution triggering. The device address adopts the CAN bus standard address format, and the vibration parameters control the motor speed and vibration intensity through PWM signals. The air conditioning temperature adjustment control command is implemented through the HVAC control protocol and includes functions such as temperature setting, fan speed adjustment, and air vent control. The navigation rest area search control command is implemented through the vehicle navigation system API interface and includes functions such as search range setting, POI type filtering, and route planning. The audio volume adjustment control command is implemented through the audio system control interface and supports adjustment of master volume, zone volume, and sound effect mode.

[0129] In one embodiment of the present invention, the execution monitoring unit adopts an asynchronous monitoring mechanism, allocating an independent monitoring thread for each control command to avoid the impact of individual device execution anomalies on overall system performance. The monitoring thread obtains execution status information, including command reception confirmation, execution progress updates, and completion status notifications, through the device feedback interface. Execution status monitoring employs a finite state machine model, defining standard states such as PENDING, EXECUTING, COMPLETED, FAILED, and TIMEOUT. State transition triggering conditions include time events, device feedback events, and abnormal events.

[0130] In one embodiment of the present invention, the active service execution module further includes a user interaction and feedback processing mechanism:

[0131] Before performing proactive service operations, proactive service semantic prompts are generated through voice synthesis technology to explain the detected driving scenario and the service measures to be taken to the user. At the same time, a proactive service confirmation card with confirmation and cancellation options is displayed on the in-vehicle display interface.

[0132] The system receives feedback information input by the user via voice commands or touch operation. When a confirmation feedback is received, the system executes the control command of the in-vehicle device. When a cancellation feedback is received, the system terminates the current service execution process and updates the user's feedback information to the database as personalized preference data.

[0133] Understandably, user interaction is achieved through an in-vehicle voice synthesis system, employing natural language generation technology to convert detected driving scenarios and impending service measures into easily understandable voice prompts. Voice synthesis utilizes a parametric TTS model, supporting dynamic adjustment of tone, speed, and timbre to adapt to different user preferences and driving environments. Card display adheres to in-vehicle HMI design specifications, ensuring readability and operational safety during driving. The card display time employs a dynamic adjustment mechanism, automatically adjusting the display duration based on the complexity of the driving environment.

[0134] The feedback processing mechanism adopts a state machine design, which executes the corresponding processing flow according to different feedback types from users. When a confirmation feedback is received, the preset vehicle equipment control command is executed immediately; when a cancellation feedback is received, the current service execution flow is terminated and the user's rejection preference is recorded; when a later feedback is received, the service request is added to a delay queue and the user is prompted again after a preset time.

[0135] The updated personalized preference data is stored in the database to provide personalized references for subsequent proactive service decisions. The database adopts a NoSQL architecture, which supports flexible expansion and efficient querying of user profiles. The storage format of personalized data includes fields such as user identifier, scenario type, service type, preference weight, and update time to ensure data integrity and traceability.

[0136] This invention also provides a dynamic reasoning method for vehicle proactive services based on an edge-side multimodal large model, applied to the vehicle proactive service system based on an edge-side multimodal large model as described above, comprising the following steps:

[0137] S1. Collect multimodal perception data of the vehicle; the multimodal perception data includes image data, in-vehicle audio signal data and vehicle driving status parameter data;

[0138] S2. Feature extraction is performed on the multimodal perception data to obtain features of different modalities; among them, the multimodal large model is compressed using knowledge distillation and quantization and deployed on the vehicle-mounted edge.

[0139] S3. Convert feature data from different modalities into query, key, and value triples, calculate cross-modal similarity weights through a bidirectional attention mechanism, and fuse them based on query, key, and value triples and cross-modal similarity weights to obtain fused features;

[0140] S4. Make a preliminary judgment on the vehicle scenario based on the fusion features. When the preliminary judgment result needs to be further confirmed, generate a supplementary data collection instruction to control the multimodal data collection module to collect specific parameter data, combine the supplementary data with the fusion features to perform multi-step reasoning, determine the current scenario type of the vehicle, and generate proactive service decisions by combining user profiles and historical behavior data.

[0141] S5. Receive proactive service decisions, generate on-board equipment control commands, and execute proactive service operations.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A vehicle active service system based on an edge-side multimodal large model, characterized in that: include: The multimodal data acquisition module is used to collect multimodal perception data of the vehicle. The multimodal perception data includes image data, in-vehicle audio signal data, and vehicle driving status parameter data; The multimodal large model processing unit is used to extract features from multimodal sensing data to obtain features of different modalities; The multimodal large model is compressed using knowledge distillation and quantization and deployed on the vehicle-mounted edge. The cross-modal attention fusion module is used to convert feature data from different modalities into query, key, and value triples, calculate cross-modal similarity weights through a bidirectional attention mechanism, and fuse them based on query, key, and value triples and cross-modal similarity weights to obtain fused features; The MCP dynamic inference controller is used to make a preliminary judgment on the vehicle scenario based on fused features. When the preliminary judgment result needs to be further confirmed, it generates a supplementary data acquisition instruction to control the multimodal data acquisition module to collect specific parameter data. The supplementary data is combined with the fused features to perform multi-step inference, determine the current scenario type of the vehicle, and generate proactive service decisions by combining user profiles and historical behavior data. The process of generating proactive service decisions by combining user profiles and historical behavior data specifically includes: The driver's physiological and behavioral feature parameters are extracted using a driving state detection algorithm as the first inference parameter. The driving state detection algorithm includes at least one of the following: a fatigue detection algorithm based on facial key point detection, a steering wheel grip force detection algorithm, and an attention distraction detection algorithm. When the first inference parameter indicates an abnormal driving state, the dynamic inference client requests the multi-parameter server to collect the corresponding verification parameters as the second inference parameter. The first and second inference parameters are then concatenated and input into the multi-step inference network for scene confirmation. User personal preference features are extracted from the user profile database, and historical behavior pattern features are extracted from the database. The confirmed scenario type is then weighted and fused with the user personal preference features and historical behavior pattern features to generate proactive service decisions. The proactive service decisions include service type, service parameters, and execution sequence. The proactive service execution module is used to receive proactive service decisions, generate on-board equipment control commands, and execute proactive service operations.

2. The vehicle active service system based on an edge-side multimodal large model as described in claim 1, characterized in that: The image data includes RGB image data and infrared image data, and the multimodal data acquisition module also includes: The brightness scoring unit is used to convert RGB image data into color space and calculate the brightness score value. When the brightness score value is lower than a preset threshold, the system switches from the visible light camera to the infrared camera for image acquisition. The infrared vision enhancement unit is used to acquire infrared thermal radiation image data using an infrared camera, and to extract thermal radiation features and perform dynamic exposure compensation on the infrared thermal radiation images to obtain infrared image data.

3. The vehicle active service system based on an edge-side multimodal large model as described in claim 1, characterized in that: The multimodal large model processing unit employs a teacher-student network architecture for knowledge distillation, specifically including: In the output layer distillation, the output probability distributions of the teacher model and the student model are smoothed. The KL divergence loss function is used to calculate the difference between the smoothed probability distributions of the teacher model and the student model. The KL divergence loss is weighted and combined with the cross-entropy loss of the student model to the true label to form a joint loss function for optimization training. In intermediate feature distillation, the intermediate feature map sizes of the teacher model and the student model are adjusted by a 1×1 convolutional adaptation layer, and the intermediate feature representations of the teacher model and the student model are aligned using the mean squared error loss function. The weight parameters and activation values ​​of the student model are quantized using 8-bit integer quantization technology. By reducing the storage precision of the model parameters through channel-by-channel quantization, the large multimodal model is compressed to less than 500MB.

4. The vehicle active service system based on an edge-side multimodal large model as described in claim 1, characterized in that: The process of converting feature data of different modalities into query, key, and value triples specifically includes: Visual feature vectors are extracted from image data using a visual transformer encoder, audio embedding feature vectors are extracted from audio signal data, and state feature vectors are generated by feature encoding of vehicle driving state parameter data. The feature vectors of different modalities are projected and transformed into query vector, key vector and value vector respectively through a linear transformation matrix; The feature dimensions of the query vector, key vector, and value vector are uniformly set to the same feature space dimension, forming a triplet representation with a unified dimension.

5. A vehicle active service system based on an edge-side multimodal large model as described in claim 4, characterized in that: The calculation method for the bidirectional attention mechanism specifically includes: The forward attention weight matrix is ​​obtained by performing a dot product operation between the query vector and the key vector, and then normalizing it using softmax based on the scaling factor. The reverse attention weight matrix is ​​obtained by performing a dot product operation between the transpose of the key vector and the query vector, and then normalizing it using softmax based on the scaling factor. The forward attention output is obtained by weighted summation of the forward attention weight matrix and the value vector, and the reverse attention output is obtained by weighted summation of the reverse attention weight matrix and the query vector. The weighted average of the forward attention output and the reverse attention output is calculated, and after residual connection and layer normalization, the fused feature is output.

6. The vehicle active service system based on an edge-side multimodal large model as described in claim 1, characterized in that: The MCP dynamic inference controller includes a dynamic inference client and multiple multi-parameter servers. The specific processing steps of the MCP dynamic inference controller include: The dynamic inference client communicates with the multi-parameter server through a multi-step conditional inference protocol. Upon startup, it automatically identifies all available multi-parameter servers and obtains descriptions of the data acquisition capabilities provided by each server. Based on the fusion features, a preliminary classification judgment of vehicle scenarios is made through a preset scenario classification model, generating a preliminary judgment result containing scenario type identifier and confidence value; When the confidence level of the preliminary judgment result is lower than the set threshold, the dynamic inference client sends a supplementary data collection instruction to the corresponding multi-parameter server based on the scenario type identifier.

7. The vehicle active service system based on an edge-side multimodal large model as described in claim 1, characterized in that: The active service execution module includes: The decision parsing unit is used to parse the service type, service parameters and execution timing information in the proactive service decision, match the corresponding vehicle equipment control protocol according to the service type, and generate equipment control parameters. The instruction generation unit is used to generate control instructions based on the device control parameters, wherein the control instructions include seat vibration control instructions, air conditioning temperature adjustment control instructions, navigation rest area search control instructions, and audio volume adjustment control instructions. The execution monitoring unit is used to monitor the execution status of control commands of the vehicle-mounted equipment through the device execution interface, record the control command issuance time, execution completion time, and execution result status, and generate execution feedback information including execution success rate and response time.

8. A vehicle active service system based on an edge-side multimodal large model as described in claim 7, characterized in that: The proactive service execution module also includes a user interaction and feedback processing mechanism: Before performing proactive service operations, proactive service semantic prompts are generated through voice synthesis technology to explain the detected driving scenario and the service measures to be taken to the user. At the same time, a proactive service confirmation card with confirmation and cancellation options is displayed on the in-vehicle display interface. The system receives feedback information input by the user via voice commands or touch operation. When a confirmation feedback is received, the system executes the control command of the in-vehicle device. When a cancellation feedback is received, the system terminates the current service execution process and updates the user's feedback information to the database as personalized preference data.

9. A dynamic reasoning method for vehicle active services based on an edge-side multimodal large model, characterized in that: The vehicle active service system based on a multimodal large model on the edge side, as described in any one of claims 1-8, includes the following steps: S1. Collect multimodal perception data of the vehicle; the multimodal perception data includes image data, in-vehicle audio signal data and vehicle driving status parameter data; S2. Feature extraction is performed on the multimodal perception data to obtain features of different modalities; among them, the multimodal large model is compressed using knowledge distillation and quantization and deployed on the vehicle-mounted edge. S3. Convert feature data from different modalities into query, key, and value triples, calculate cross-modal similarity weights through a bidirectional attention mechanism, and fuse them based on query, key, and value triples and cross-modal similarity weights to obtain fused features; S4. Make a preliminary judgment on the vehicle scenario based on the fusion features. When the preliminary judgment result needs to be further confirmed, generate a supplementary data collection instruction to control the multimodal data collection module to collect specific parameter data, combine the supplementary data with the fusion features to perform multi-step reasoning, determine the current scenario type of the vehicle, and generate proactive service decisions by combining user profiles and historical behavior data. S5. Receive proactive service decisions, generate on-board equipment control commands, and execute proactive service operations.

Images