End side dynamic optimization method and device of vehicle-mounted multi-mode large model and electronic equipment

Abstract

The invention provides an end side dynamic optimization method and device for a vehicle-mounted multi-mode large model and electronic equipment, and the method comprises the steps: carrying out the lightweight processing of an original multi-mode large model, and obtaining a lightweight model suitable for vehicle-mounted end side deployment; processing the in-vehicle multi-modal input data using a lightweight model to perform an inference task; based on the task context of the reasoning task and the operation state of the vehicle-mounted end side equipment, the reasoning calculation resources of the lightweight model are dynamically configured; performing calculations through the lightweight model based on the dynamic configuration, generating intermediate calculation results; detecting and multiplexing an intermediate calculation result; and on the basis of local feedback and vehicle-mounted environment information for executing the reasoning task, carrying out adaptive updating on the lightweight model. In the mode, through dynamic quantization and heterogeneous computing power scheduling, the power consumption and delay are remarkably reduced while the accuracy of the key task is ensured; and in combination with intermediate result multiplexing and edge adaptive updating, efficient, real-time and robust reasoning of the end side model is realized.

Description

Technical Field

[0001] This invention relates to the field of dynamic optimization technology, and in particular to an end-side dynamic optimization method, apparatus and electronic equipment for a vehicle-mounted multimodal large model. Background Technology

[0002] Currently, multimodal large models bring powerful interactive capabilities to in-vehicle intelligent cockpits, but actual deployment faces a dilemma: uploading data to the cloud for processing poses risks of privacy leaks, high latency, and network dependence; while deploying the complete model on the vehicle side is limited by chip computing power and power consumption, making it difficult to achieve real-time, high-precision inference.

[0003] Current static model compression methods cannot adapt to dynamic tasks and changing environments, making it difficult to balance performance, efficiency, and user experience. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an edge-side dynamic optimization method, device and electronic device for vehicle-mounted multimodal large models. Through dynamic quantization and heterogeneous computing power scheduling, power consumption and latency are significantly reduced while ensuring the accuracy of critical tasks. By combining intermediate result reuse and edge adaptive update, efficient, real-time and robust inference of edge-side models is achieved.

[0005] In a first aspect, embodiments of the present invention provide an edge-side dynamic optimization method for a large multimodal vehicle model. The method includes: lightweighting the original large multimodal model to obtain a lightweight model suitable for deployment on the vehicle edge; using the lightweight model to process vehicle multimodal input data to perform inference tasks; dynamically configuring the inference computing resources of the lightweight model based on the task context of the inference task and the operating status of the vehicle edge device during the execution of the inference task; performing calculations through the lightweight model based on the dynamic configuration to generate intermediate calculation results; detecting and reusing the intermediate calculation results in continuous inference tasks; and adaptively updating the lightweight model based on local feedback from the execution of the inference task and vehicle environment information.

[0006] In a preferred embodiment of the present invention, the above-described lightweighting process of the original multimodal large model to obtain a lightweight model suitable for vehicle-mounted deployment includes: decomposing the original multimodal large model into a speech coding module, a visual coding module, and a text decoding module; performing knowledge distillation on the speech coding module and the visual coding module to map the intermediate layer features of the speech coding module and the visual coding module to a shared feature space; and replacing one or more of the speech coding module, the visual coding module, and the text decoding module in the original multimodal large model with a lightweight network to generate a lightweight model.

[0007] In a preferred embodiment of the present invention, the above-mentioned dynamic configuration of inference computing resources for the lightweight model includes: dynamically selecting and applying the corresponding quantization accuracy level for the lightweight model based on the security level of the task context and the temperature or load information in the running state; and dynamically scheduling different computing subgraphs of the lightweight model to NPU, GPU or CPU for execution based on a pre-established hardware performance profile.

[0008] In a preferred embodiment of the present invention, the aforementioned security level based on the task context and the temperature or load information in the operating state are used to dynamically select and apply the corresponding quantization accuracy level for the lightweight model, including: if the inference task is safety-critical and the device temperature is lower than a preset temperature threshold, then a first quantization accuracy level is applied; if the inference task is non-critical and the system load is higher than a preset first load threshold, then a second quantization accuracy level is applied; the first quantization accuracy level is higher than the second quantization accuracy level.

[0009] In a preferred embodiment of the present invention, the above-mentioned dynamic scheduling of different computational subgraphs of the lightweight model to NPU, GPU or CPU for execution based on a pre-established hardware performance profile includes: real-time monitoring of the processor load of NPU, GPU and CPU; when the processor load of NPU is higher than a pre-set second load threshold, scheduling the subsequent computationally intensive operators or subgraphs in the lightweight model to be processed to GPU for execution.

[0010] In a preferred embodiment of the present invention, the detection and reuse of intermediate calculation results in the continuous inference task includes: calculating the feature similarity between the current input data and the historical input data in a specified network layer of the lightweight model; if the feature similarity exceeds a preset similarity threshold, the intermediate calculation results of the specified network layer and subsequent layers cached by the historical input data are reused.

[0011] In a preferred embodiment of the present invention, the lightweight model is adaptively updated based on local feedback from the inference task and in-vehicle environment information, including: monitoring the task execution error rate of the inference task and changes in in-vehicle environment information; freezing the parameters of the speech encoding module and the visual encoding module in the lightweight model when the task execution error rate continuously exceeds a preset error rate threshold or the in-vehicle environment information changes by more than a preset amount; and updating the weight parameters of the top N layers of the neural network of the text decoding module in the lightweight model; wherein N is a positive integer greater than or equal to 1.

[0012] Secondly, embodiments of the present invention also provide an end-side dynamic optimization device for a vehicle-mounted multimodal large model, comprising: a lightweight processing module for lightweighting the original multimodal large model to obtain a lightweight model suitable for vehicle-mounted end-side deployment; a vehicle-mounted multimodal input data processing module for processing vehicle-mounted multimodal input data using the lightweight model to perform inference tasks; a dynamic configuration module for dynamically configuring the inference computing resources of the lightweight model based on the task context of the inference task and the operating status of the vehicle-mounted end-side device during the execution of the inference task; an intermediate calculation result generation module for generating intermediate calculation results by performing calculations through the lightweight model based on the dynamic configuration; a detection and reuse module for detecting and reusing the intermediate calculation results in continuous inference tasks; and an adaptive update module for adaptively updating the lightweight model based on local feedback from the execution of the inference task and vehicle environment information.

[0013] Thirdly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the on-side dynamic optimization method for the vehicle-mounted multimodal large model described in the first aspect.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to implement the on-side dynamic optimization method for the vehicle multimodal large model described in the first aspect.

[0015] The embodiments of the present invention bring the following beneficial effects: This invention provides a method, apparatus, and electronic device for edge-side dynamic optimization of a large multimodal vehicle model. By lightweighting the original large multimodal model, a lightweight model suitable for vehicle-side deployment is obtained. The lightweight model is used to process vehicle-side multimodal input data to perform inference tasks. During the inference task execution, the inference computing resources of the lightweight model are dynamically configured based on the task context of the inference task and the operating status of the vehicle-side device. Based on the dynamic configuration, calculations are performed through the lightweight model to generate intermediate calculation results. In continuous inference tasks, the intermediate calculation results are detected and reused. Based on local feedback from the inference task execution and vehicle environment information, the lightweight model is adaptively updated. This approach, through dynamic quantization and heterogeneous computing power scheduling, significantly reduces power consumption and latency while ensuring the accuracy of critical tasks. Combined with intermediate result reuse and edge adaptive updates, it achieves efficient, real-time, and robust inference of the edge-side model.

[0016] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0017] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart of an end-side dynamic optimization method for a vehicle-mounted multimodal large model provided in an embodiment of the present invention; Figure 2 A flowchart of another end-side dynamic optimization method for a vehicle-mounted multimodal large model provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an end-side dynamic optimization device for a vehicle-mounted multimodal large model provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In recent years, multimodal large models have received widespread attention in the field of intelligent cockpits. These models achieve human-computer interaction capabilities, ranging from semantic understanding to contextual reasoning, by integrating multi-source data such as speech, images, text, and sensors. For example, representative open-source models include GPT-4V, LLaVA, and Qwen-VL, which are typically composed of a visual encoder (ViT, VisionTransformer) and a large language model (LLM, LargeLanguageModel).

[0022] In relevant vehicle systems, OEMs and suppliers typically employ the following main approaches to deploy multimodal models: Option A: Full cloud deployment. Raw video, image, and audio data are uploaded to the cloud, where a cloud-based model performs feature extraction and inference, and then returns the results to the vehicle.

[0023] Existing problems: 1. Privacy and security risks: Images captured by cameras inside and outside the vehicle contain sensitive information such as people's faces, postures, license plates, and geographical locations. Uploading these images directly to the cloud may violate privacy requirements.

[0024] 2. High bandwidth consumption and instability: Uploading multiple high-definition video streams consumes a large amount of 5G bandwidth, and even slight fluctuations in communication quality can affect the timeliness of results.

[0025] 3. High response latency: From data upload and cloud computing to result delivery, the link is long and the latency is high, which cannot meet the interaction requirements of less than 2 seconds in the cockpit.

[0026] Option B: Complete vehicle-side deployment. The entire multimodal model (ViT+LLM) is compressed and run on the in-vehicle computing platform.

[0027] Existing problems: 1. Significant bottleneck in computing power: Existing automotive-grade chips such as SA8295P / Orin are limited by power consumption and NPU computing power, making it difficult to run models with tens of billions of parameters in real time under 30fps image input and multi-tasking scenarios.

[0028] 2. Static compression results in significant accuracy loss: Unified quantization models cannot meet the accuracy requirements of different tasks (visual recognition, semantic understanding, etc.).

[0029] 3. High power consumption and heat dissipation pressure: Long-term high-load inference will cause power consumption to soar and the whole machine to overheat.

[0030] 4. Difficult to update and maintain: Model updates require a full OTA update, which is time-consuming and costly.

[0031] Therefore, it is difficult to balance real-time performance, energy consumption, and model accuracy in related technologies.

[0032] Based on this, the present invention provides an edge-side dynamic optimization method, apparatus, and electronic device for a vehicle-mounted multimodal large model. This method involves lightweighting the original multimodal large model to obtain a lightweight model suitable for vehicle-mounted deployment. The lightweight model is then used to process vehicle-mounted multimodal input data to perform inference tasks. During the inference task execution, the inference computing resources of the lightweight model are dynamically configured based on the task context of the inference task and the operating status of the vehicle-mounted device. Based on this dynamic configuration, calculations are performed using the lightweight model to generate intermediate calculation results. In continuous inference tasks, these intermediate calculation results are detected and reused. The lightweight model is adaptively updated based on local feedback from the inference task execution and vehicle environment information. This approach, through dynamic quantization and heterogeneous computing power scheduling, significantly reduces power consumption and latency while ensuring the accuracy of critical tasks. Combined with intermediate result reuse and edge adaptive updates, it achieves efficient, real-time, and robust inference for the edge-side model.

[0033] To facilitate understanding of this embodiment, a detailed description of the end-side dynamic optimization method for a vehicle-mounted multimodal large model disclosed in this embodiment of the invention will be provided first.

[0034] Example 1 This invention provides an end-side dynamic optimization method for a large-scale multimodal vehicle model. Figure 1 This is a flowchart illustrating an end-side dynamic optimization method for a large-scale multimodal vehicle model, provided as an embodiment of the present invention. Figure 1 As shown, the end-side dynamic optimization method for this vehicle-mounted multimodal large model may include the following steps: Step S101: Lightweight processing is performed on the original multimodal large model to obtain a lightweight model suitable for vehicle-mounted deployment.

[0035] Among them, original multimodal large models refer to pre-trained artificial intelligence models with a huge number of parameters (usually in the billions or even hundreds of billions) that can simultaneously process and understand multiple modalities of data (such as text, images, and speech). Examples include open-source models such as LLaVA, Qwen-VL, or GPT-4V.

[0036] Lightweight models refer to models obtained by compressing and pruning the original large models, which significantly reduce the number of parameters and computational load, but strive to retain core performance, aiming to adapt to the limited computing power and storage resources of in-vehicle computing platforms.

[0037] Specifically, the original multimodal large model is lightweighted to obtain a lightweight model suitable for vehicle-mounted deployment. This can include: decomposing the original multimodal large model into a speech coding module, a visual coding module, and a text decoding module; performing knowledge distillation on the speech coding module and the visual coding module, mapping the intermediate layer features of the speech coding module and the visual coding module to a shared feature space; and replacing one or more of the speech coding module, the visual coding module, and the text decoding module in the original multimodal large model with a lightweight network to generate a lightweight model.

[0038] The original multimodal large model decomposition decouples the complex multimodal model into functionally independent sub-networks. This is a deployment-oriented structured analysis that lays the foundation for subsequent differential processing (such as distilling only the visual and speech components).

[0039] Knowledge distillation is a model compression technique that learns by having a lightweight model mimic the behavior or internal feature representations of a larger model. Specifically, it refers to cross-modal feature distillation, which aims to semantically align the outputs of encoders from different modalities to a common space.

[0040] Regarding distillation, multi-task distillation, self-supervised learning, and other methods can be used to generate a shared feature space.

[0041] The shared feature space is an abstract conceptual space where semantic information from images (output of the visual encoder) and speech (output of the speech encoder) is mapped to vectors with the same dimension of meaning. The dimension and representation of the shared feature space are not limited to a single vector; they can be multidimensional tensors or other embedding forms. For example, the image feature vector and the speech feature vector for "dog" are similar in this space. This provides a unified foundation for subsequent multimodal fusion and understanding, and is key to reducing model complexity.

[0042] Lightweight network replacement involves replacing bulky and cumbersome modules in the original model with a more sophisticated network architecture that has fewer parameters. For example, replacing the original Vision Transformer (ViT) with MobileViT or EfficientNet, and replacing the original Whisper model with Conformer-Tiny. Besides direct replacement, lightweighting techniques include, but are not limited to: network pruning (removing unimportant neuron connections), low-rank decomposition (breaking down large matrices into smaller matrix products), and automated neural architecture search (NAS) to find optimal lightweight quantum structures. Replacing "one or more" reflects flexibility; for example, a powerful text decoder might be retained while only the visual and speech encoders are lightweighted.

[0043] Taking the LLaVA model as an example: Breakdown: Visual encoder (ViT-L / 14), text decoder (Vicuna-7B), speech component needs to be integrated separately (e.g., Whisper-small).

[0044] Distillation: Training is performed using a large amount of image-text-speech pairing data. The objective function not only requires the lightweight visual / speech encoders to complete their own tasks (such as image classification and speech recognition), but also requires that the features of their intermediate layers, after projection, be as close as possible in the shared feature space.

[0045] Replacement: Replace ViT-L / 14 with MobileViT-XS (4 layers), and Whisper-small with Conformer-Tiny (3 layers). The text decoding module is retained but the underlying layer is frozen. The resulting lightweight model has a significantly reduced number of parameters, meeting the requirements for edge deployment.

[0046] In particular, by generating lightweight models, not only is the number of model parameters significantly reduced (more than 40% less than the original model), but the underlying network structure can also be reused for different tasks, saving storage and computing resources.

[0047] Step S102: Use a lightweight model to process the in-vehicle multimodal input data to perform inference tasks.

[0048] Among them, vehicle multimodal input data refers to various sensor data generated in real time during vehicle operation, typically including: driver / occupant images and video streams captured by in-vehicle cameras, voice commands captured by microphone arrays, road environment images captured by external vehicle cameras, and vehicle status signals provided by the vehicle's CAN bus, etc.

[0049] The reasoning task refers to the process of using the lightweight model to process input data to arrive at a specific conclusion or response. Examples include visual question answering (What is the road sign ahead?), driver state recognition (fatigue, distraction), voice assistant interaction, and scene semantic understanding.

[0050] Step S103: During the execution of the inference task, the inference computing resources of the lightweight model are dynamically configured based on the task context of the inference task and the operating status of the vehicle-mounted terminal device.

[0051] Task context refers to the set of attribute information related to the current inference task, used to guide resource allocation strategies. Its core components include: 1) Task type (e.g., visual recognition, semantic understanding); 2) Functional safety level (classified according to ISO 26262 standards, such as ASIL-B safety-critical tasks – driver monitoring, or QM-level entertainment information tasks – music recommendation); 3) Real-time requirements (latency budget, such as collision warning requiring <100ms, navigation queries tolerating 1-2s).

[0052] The operating status of the vehicle-mounted terminal equipment refers to the real-time physical and logical parameters of the vehicle-mounted computing platform (such as Qualcomm SA8295P and NVIDIA Orin chips) that carries the model inference. This mainly includes: 1) processor load (NPU / GPU / CPU utilization); 2) temperature (chip junction temperature); 3) power consumption; and 4) memory usage.

[0053] Among them, inference computing resources refer to the total amount of software and hardware resources that can be dynamically allocated when executing model inference, mainly including: 1) computing precision resources (i.e. quantization bit width, which determines the precision of numerical representation and computing speed); 2) hardware execution unit resources (computing power of NPU, GPU, and CPU); 3) memory and cache resources.

[0054] Step S104: Calculations are performed using a lightweight model based on dynamic configuration to generate intermediate calculation results.

[0055] Intermediate calculation results refer to the activation values ​​or feature maps generated by the intermediate layers (not the final output layer) of the network during the forward propagation process. These data contain abstract representations of the input data.

[0056] Step S105: In the continuous reasoning task, the intermediate calculation results are detected and reused.

[0057] Specifically, in continuous inference tasks, the detection and reuse of intermediate calculation results can include: calculating the feature similarity between the current input data and historical input data in a specified network layer of the lightweight model; if the feature similarity exceeds a pre-set similarity threshold, the intermediate calculation results of the specified network layer and subsequent layers cached from historical input data are reused.

[0058] Among these are continuous reasoning tasks, such as video streaming or multi-turn dialogues.

[0059] Feature similarity calculation typically uses cosine similarity or Euclidean distance to measure the similarity between two feature vectors. The calculation occurs in a specified network layer, usually an intermediate layer that captures the core semantics of the input data but is not too deep (e.g., the middle and later layers of a visual encoder).

[0060] In addition to direct comparison, feature similarity calculation methods can also employ techniques such as approximate matching based on hash functions and Locality Sensitive Hashing (LSH).

[0061] Among these features, reuse is defined as follows: if the similarity between the current input frame and the previous frame exceeds a threshold (e.g., 0.9), it is determined that the scene has not undergone any fundamental change. The system will directly read all intermediate activation values ​​stored in the cache from the specified layer to the output layer, and use these values ​​as the basis for subsequent calculations (such as fusion and decoding), completely skipping the repeated calculations of all layers before the specified layer.

[0062] Among them, the cache: The cache is usually managed by a fixed-size first-in-first-out (FIFO) or least recently used (LRU) queue, and needs to work with the memory management unit of the vehicle system. It can retain a maximum of 5 sets of active data and occupy no more than 20MB, ensuring that it can be safely reclaimed when memory is tight, effectively reducing redundant calculations and improving the smoothness of multi-turn voice-visual interaction.

[0063] In addition to FIFO, cache management strategies can also employ common cache replacement algorithms such as LRU (Least Recently Used) and LFU (Least Frequently Used).

[0064] For example, in a highway assisted driving scenario, the camera captures images at 30fps. Background changes are minimal between consecutive frames, with only slight shifts in vehicle position. The system calculates a feature similarity of 0.98 for consecutive frames at a certain intermediate layer, thus reusing 90% of the calculation results from the previous frame. This allows for target detection and tracking of the current frame with minimal computational power, significantly improving efficiency.

[0065] Step S106: Based on the local feedback from the inference task and the vehicle environment information, the lightweight model is adaptively updated.

[0066] Local feedback refers to the performance metrics that can be recorded locally after the inference task is executed on the vehicle side. These mainly include: task accuracy / error rate, single inference latency, task success rate, and system-recorded exception logs.

[0067] Among them, vehicle environment information refers to the external physical environment parameters of the vehicle, which can be obtained through vehicle body sensors, such as ambient light intensity (day / night / tunnel), background noise level (noisy / quiet), weather conditions (sunny / rainy / foggy), GPS geographical location information, etc.

[0068] Specifically, the lightweight model is adaptively updated based on local feedback from the inference task and in-vehicle environment information. This may include: monitoring the task execution error rate of the inference task and changes in in-vehicle environment information; freezing the parameters of the speech encoding module and the visual encoding module in the lightweight model when the task execution error rate continuously exceeds a preset error rate threshold or the in-vehicle environment information changes by more than a preset amount; and updating the weight parameters of the top N layers of the neural network in the text decoding module of the lightweight model, where N is a positive integer greater than or equal to 1.

[0069] In this process, if the weights exceed 15% three times consecutively in the same inference task, or if there is a sudden change in the environment (such as entering a tunnel from daytime), fine-tuning is triggered. After fine-tuning, the weight parameters are encrypted and uploaded to the cloud; the cloud aggregates data from multiple vehicles to generate a new distillation model, which is then distributed via OTA, forming a Cloud-Edge linked update system.

[0070] The system continuously tracks two types of metrics: 1) performance metrics, such as the error rate of a specific task ("identifying stop signs"); and 2) environmental covariates, such as a sudden change in lighting conditions from daytime to entering a tunnel. When the error rate consistently exceeds one of these metrics, or when there is a sudden change in the environment, it is considered that the model's performance has deteriorated under the current conditions, triggering an update.

[0071] Freezing the speech / visual encoders is for functional safety and stability reasons. Encoders are the model's "perceptual organs," and accidental modification could lead to a collapse of fundamental perceptual capabilities, causing serious safety risks. Freezing them ensures the consistency of the perceptual foundation.

[0072] This involves updating the top layer of the text decoder: the text decoder is responsible for "thinking and expression," and its top-level network is more related to specific task logic and language style. Updating this part is equivalent to adjusting the model's "decision-making and expression methods" to adapt to new scenarios or correct errors while maintaining the same "worldview" (perceptual ability). "N layers" is a configurable parameter, for example, updating the last 1-2 layers.

[0073] The update process involves the following steps: After triggering, the system utilizes recently collected local data with implicit or explicit feedback (such as user-corrected samples or system-marked failure cases) to perform small-scale, low-learning-rate fine-tuning training (e.g., SGD). To prevent catastrophic forgetting, algorithms such as elastic weight consolidation can be employed.

[0074] In addition to mini-batch SGD, online fine-tuning algorithms can also employ other optimizers such as Adam and RMSprop.

[0075] For example, a car that has been driving in the dry environment of the north for a long time may have a poor recognition performance in rainy weather. One day, when driving in a rainy area in the south, raindrops and fog cause the visual recognition error rate to increase. The system triggers an update, using the newly collected data from the rainy day that showed poor recognition performance to fine-tune the top-level parameters of the decoder, enabling the model to learn to make more robust judgments in rainy conditions without forgetting how to recognize objects in sunny weather.

[0076] This mechanism can also be used to achieve limited user personalization. For example, the system can learn a driver's specific voice command habits (such as using "navigate home" instead of "please plan the route to my residence") and update the relevant parameters of the decoder without changing the basic voice recognition and map search capabilities.

[0077] The edge-side dynamic optimization method for a large multimodal vehicle model provided in this invention can obtain a lightweight model suitable for vehicle-side deployment by lightweighting the original large multimodal model. The lightweight model is then used to process vehicle-side multimodal input data to perform inference tasks. During the inference task execution, the inference computing resources of the lightweight model are dynamically configured based on the task context of the inference task and the operating status of the vehicle-side device. Based on this dynamic configuration, calculations are performed through the lightweight model to generate intermediate calculation results. In continuous inference tasks, these intermediate calculation results are detected and reused. Based on local feedback from the inference task execution and vehicle environment information, the lightweight model is adaptively updated. This method, through dynamic quantization and heterogeneous computing power scheduling, significantly reduces power consumption and latency while ensuring the accuracy of critical tasks. Combined with intermediate result reuse and edge adaptive updates, it achieves efficient, real-time, and robust inference of the edge-side model.

[0078] Example 2 This invention also provides another end-side dynamic optimization method for vehicle-mounted multimodal large models; this method is implemented based on the method in the above embodiments; this method focuses on describing the specific implementation of dynamically configuring the inference computing resources of lightweight models.

[0079] Figure 2 A flowchart of another end-side dynamic optimization method for a vehicle-mounted multimodal large model provided in an embodiment of the present invention is shown below. Figure 2 As shown, dynamically configuring the inference computing resources for the lightweight model can include the following steps: Step S201: Based on the security level of the task context and the temperature or load information in the running state, dynamically select and apply the corresponding quantization accuracy level for the lightweight model.

[0080] The quantization precision level refers to the bit width used to represent the model weights and activation values. Examples include FP32 (full precision), INT8 (8-bit integer), and INT4 (4-bit integer). A lower bit width results in faster computation, less memory usage, and lower power consumption, but may compromise precision.

[0081] Specifically, based on the safety level of the task context and the temperature or load information in the operating state, the corresponding quantization accuracy level is dynamically selected and applied to the lightweight model. This may include: if the inference task is safety-critical and the equipment temperature is lower than a preset temperature threshold, then the first quantization accuracy level is applied; if the inference task is non-critical and the system load is higher than a preset first load threshold, then the second quantization accuracy level is applied; the first quantization accuracy level is higher than the second quantization accuracy level.

[0082] For example, when the chip temperature is normal, driver attention monitoring (a safety-critical task) uses W8A8 quantization to prioritize recognition accuracy; when the system load is high, passenger song request (a non-critical task) uses W4A4 quantization to prioritize system smoothness and reduce power consumption.

[0083] Furthermore, different quantization bit widths can be used for layers with different sensitivities in the model, with high precision for sensitive layers and low precision for insensitive layers.

[0084] Furthermore, the system can learn to select the optimal quantization level to achieve the best overall "accuracy-latency-power consumption" benefits under different combinations of (task context, device state), rather than using fixed rules.

[0085] Furthermore, when the vehicle battery is low, lower precision quantization can be used more aggressively to save energy.

[0086] Step S202: Based on the pre-established hardware performance profile, dynamically schedule different computational subgraphs of the lightweight model to NPU, GPU or CPU for execution.

[0087] The hardware performance profile is a database or configuration file pre-built through extensive benchmarking. It records the performance data of different computational subgraphs of the lightweight model (such as a convolutional block or an attention layer) running on the target automotive chip (NPU / GPU / CPU), including execution latency, power consumption, memory usage, and temperature impact factor. This serves as the basis for intelligent scheduling decisions.

[0088] Specifically, based on a pre-established hardware performance profile, different computational subgraphs of the lightweight model are dynamically scheduled to be executed by the NPU, GPU, or CPU. This may include: real-time monitoring of the processor load of the NPU, GPU, and CPU; when the processor load of the NPU is higher than a pre-set second load threshold, subsequent computationally intensive operators or subgraphs in the lightweight model to be processed are scheduled to be executed by the GPU.

[0089] Among them, computationally intensive operators or subgraphs refer to operations such as convolution and matrix multiplication.

[0090] For example, in a multimodal task, the visual convolutional part is highly efficient on the NPU for hardware image display, so it is scheduled to the NPU. However, when the NPU load exceeds the 85% threshold, the scheduler redirects the visual part of subsequent new tasks to the GPU to prevent the NPU from becoming overloaded and a bottleneck. The text generation part (rich in logical judgments) is scheduled to the CPU.

[0091] Furthermore, scheduling requires balancing the computational acceleration gains with the latency and power consumption overhead incurred by moving data between the CPU, GPU, and NPU.

[0092] Furthermore, for tasks with the highest security level, a hardware preemption mechanism can be set up to ensure their computing resources.

[0093] Furthermore, instead of aiming for the shortest possible latency, the scheduling objective is "task completion power consumption" or "computational throughput per unit power consumption".

[0094] By feeding back real-time scheduling results to the power consumption monitoring module to form a closed-loop optimization, a dynamic balance between power consumption, latency, and accuracy is achieved.

[0095] As an embodiment of this application: a dynamic quantitative reasoning process for a speech and vision fusion task.

[0096] Scene: The driver asks, "What does the red road sign ahead say?"

[0097] process: The camera captures image frames, and the voice module transcribes the speech into text. The system's recognition task is "visual question answering," with the NPU temperature at 65 degrees Celsius and a load of 78%. The dynamic quantization module is selected as q = w4a16; Visual encoding is performed on the NPU, while text parsing is performed on the CPU. If the similarity of consecutive frames detected by the activation cache is equal to 0.93, which is greater than 0.9, then the cache is activated after reusing the 4th layer. The fusion layer generates the answer "speed limit 60 km / h" in the shared feature space Z; The voice broadcast result takes 1.1 seconds from end to end, and power consumption is reduced by 22%.

[0098] As another embodiment of this application: heterogeneous computing power optimization process in multi-task concurrent scenarios.

[0099] Scenario: Navigation, music playback, and driver status detection are running simultaneously.

[0100] process: Real-time load: GPU 75%, NPU idle 45%, CPU 60%; Scheduling decision: Driver detection (visual type, ASIL B) is assigned to the NPU, q = w8a16; Navigation semantic parsing is performed on the GPU; The music command "next song" is processed by the CPU; If the similarity of the driver's face is greater than 0.95 for 5 consecutive frames after activation, the first two convolutional layers are skipped. With three tasks running in parallel, overall power consumption is reduced by 18%, and the maximum latency is less than or equal to 1.2 seconds.

[0101] Example 3 Corresponding to the above method embodiments, this invention provides an end-side dynamic optimization device for a vehicle-mounted multimodal large model. Figure 3 This is a schematic diagram of the structure of an end-side dynamic optimization device for a vehicle-mounted multimodal large model provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the end-side dynamic optimization device for the vehicle-mounted multimodal large model may include: The lightweight processing module 301 is used to perform lightweight processing on the original multimodal large model to obtain a lightweight model suitable for vehicle-mounted deployment.

[0102] The vehicle-mounted multimodal input data processing module 302 is used to process vehicle-mounted multimodal input data using a lightweight model to perform inference tasks.

[0103] The dynamic configuration module 303 is used to dynamically configure the inference computing resources of the lightweight model based on the task context of the inference task and the operating status of the vehicle-mounted terminal device during the execution of the inference task.

[0104] The intermediate calculation result generation module 304 is used to perform calculations based on dynamic configuration through a lightweight model and generate intermediate calculation results.

[0105] The detection and reuse module 305 is used to detect and reuse intermediate calculation results in continuous inference tasks.

[0106] The adaptive update module 306 is used to adaptively update the lightweight model based on local feedback from the inference task and information about the vehicle environment.

[0107] The on-device dynamic optimization device for a vehicle-mounted multimodal large model provided in this invention can obtain a lightweight model suitable for on-device deployment by lightweighting the original multimodal large model. The lightweight model is then used to process on-device multimodal input data to perform inference tasks. During the inference task execution, the inference computing resources of the lightweight model are dynamically configured based on the task context of the inference task and the operating status of the on-device device. Based on this dynamic configuration, calculations are performed through the lightweight model to generate intermediate calculation results. In continuous inference tasks, these intermediate calculation results are detected and reused. Based on local feedback from the inference task execution and on-device environment information, the lightweight model is adaptively updated. This approach, through dynamic quantization and heterogeneous computing power scheduling, significantly reduces power consumption and latency while ensuring the accuracy of critical tasks. Combined with intermediate result reuse and edge adaptive updates, it achieves efficient, real-time, and robust inference of the on-device model.

[0108] In some embodiments, the lightweight processing module is further configured to decompose the original multimodal large model into a speech coding module, a visual coding module, and a text decoding module; perform knowledge distillation on the speech coding module and the visual coding module, mapping the intermediate layer features of the speech coding module and the visual coding module to a shared feature space; and replace one or more of the speech coding module, the visual coding module, and the text decoding module in the original multimodal large model with a lightweight network to generate a lightweight model.

[0109] In some embodiments, the dynamic configuration module is also used to dynamically select and apply the corresponding quantization accuracy level for the lightweight model based on the security level of the task context and the temperature or load information in the running state; and to dynamically schedule different computational subgraphs of the lightweight model to NPU, GPU or CPU for execution based on a pre-established hardware performance profile.

[0110] In some embodiments, the dynamic configuration module is further configured to apply a first quantization accuracy level if the inference task is safety-critical and the device temperature is lower than a preset temperature threshold; and apply a second quantization accuracy level if the inference task is non-critical and the system load is higher than a preset first load threshold; wherein the first quantization accuracy level is higher than the second quantization accuracy level.

[0111] In some embodiments, the dynamic configuration module is also used to monitor the processor load of the NPU, GPU and CPU in real time; when the processor load of the NPU is higher than a preset second load threshold, the computationally intensive operators or subgraphs in the subsequent lightweight model to be processed are scheduled to be executed by the GPU.

[0112] In some embodiments, the detection and reuse module is further configured to calculate the feature similarity between the current input data and the historical input data in a specified network layer of the lightweight model; if the feature similarity exceeds a preset similarity threshold, it is reused as an intermediate calculation result of the historical input data cached in the specified network layer and subsequent layers.

[0113] In some embodiments, the adaptive update module is further configured to monitor the task execution error rate of the inference task and changes in the vehicle environment information; when the task execution error rate continues to exceed a preset error rate threshold or the vehicle environment information changes by more than a preset amount, the parameters of the speech encoding module and the visual encoding module in the lightweight model are frozen; the weight parameters of the top N layers of the neural network of the text decoding module in the lightweight model are updated; where N is a positive integer greater than or equal to 1.

[0114] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0115] Example 4 This invention also provides an electronic device for running the aforementioned end-side dynamic optimization method for a large-scale vehicle-mounted multimodal model; see [link to related documentation]. Figure 4 The diagram shows the structure of an electronic device, which includes a memory 400 and a processor 401. The memory 400 is used to store one or more computer instructions, which are executed by the processor 401 to implement the above-mentioned end-side dynamic optimization method for vehicle-mounted multimodal large models.

[0116] Furthermore, Figure 4 The electronic device shown also includes a bus 402 and a communication interface 403. The processor 401, the communication interface 403 and the memory 400 are connected via the bus 402.

[0117] The memory 400 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 403 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 402 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0118] Processor 401 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 401 or by instructions in software form. Processor 401 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 400, and processor 401 reads information from memory 400 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0119] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the aforementioned on-side dynamic optimization method for a multimodal large vehicle model. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0120] The computer program product for the end-side dynamic optimization method for a large multimodal vehicle model provided in this embodiment of the invention includes a computer-readable storage medium storing non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0121] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0122] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0125] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for end-side dynamic optimization of a vehicle-mounted multimodal large model, characterized in that, The method includes: The original multimodal large model is lightweighted to obtain a lightweight model suitable for vehicle-mounted deployment. The lightweight model is used to process in-vehicle multimodal input data to perform inference tasks; During the execution of the inference task, the inference computing resources of the lightweight model are dynamically configured based on the task context of the inference task and the operating status of the vehicle-mounted device. Based on the dynamic configuration, calculations are performed through the lightweight model to generate intermediate calculation results; In continuous reasoning tasks, the intermediate calculation results are detected and reused; The lightweight model is adaptively updated based on local feedback from the inference task and information from the in-vehicle environment.

2. The method according to claim 1, characterized in that, The process of lightweighting the original multimodal large model to obtain a lightweight model suitable for vehicle-mounted deployment includes: The original multimodal large model is decomposed into a speech coding module, a visual coding module, and a text decoding module; Knowledge distillation is performed on the speech coding module and the visual coding module to map the intermediate layer features of the speech coding module and the visual coding module to a shared feature space; The lightweight model is generated by replacing one or more of the speech encoding module, visual encoding module, and text decoding module in the original multimodal large model with a lightweight network.

3. The method according to claim 1, characterized in that, The dynamic configuration of inference computing resources for the lightweight model includes: Based on the security level of the task context and the temperature or load information in the running state, the corresponding quantization accuracy level is dynamically selected and applied to the lightweight model. Based on a pre-established hardware performance profile, different computational subgraphs of the lightweight model are dynamically scheduled to be executed by NPU, GPU, or CPU.

4. The method according to claim 3, characterized in that, The dynamic selection and application of the corresponding quantization accuracy level for the lightweight model based on the security level of the task context and the temperature or load information in the operating state includes: If the inference task is safety-critical and the equipment temperature is below a preset temperature threshold, then the first quantization accuracy level is applied. If the inference task is non-critical and the system load is higher than the preset first load threshold, then the second quantization accuracy level is applied. The first quantization precision level is higher than the second quantization precision level.

5. The method according to claim 3, characterized in that, The dynamic scheduling of different computational subgraphs of the lightweight model to NPU, GPU, or CPU for execution based on a pre-established hardware performance profile includes: Real-time monitoring of processor load for NPU, GPU, and CPU; When the processor load of the NPU exceeds a pre-set second load threshold, the computationally intensive operators or subgraphs in the lightweight model that are to be processed are scheduled to be executed on the GPU.

6. The method according to claim 1, characterized in that, In the continuous reasoning task, the detection and reuse of the intermediate calculation results includes: Calculate the feature similarity between the current input data and the historical input data in the specified network layer of the lightweight model; If the feature similarity exceeds a preset similarity threshold, it is reused as the intermediate calculation result of the specified network layer and subsequent layers, which is cached as historical input data.

7. The method according to claim 1, characterized in that, The adaptive update of the lightweight model based on local feedback from the inference task and in-vehicle environment information includes: Monitor the task execution error rate of the inference task and the changes in the in-vehicle environment information; When the task execution error rate continues to exceed the preset error rate threshold or the vehicle environment information changes by more than a preset amount, the parameters of the speech coding module and the vision coding module in the lightweight model are frozen. Update the weight parameters of the top N layers of the neural network in the text decoding module of the lightweight model; where N is a positive integer greater than or equal to 1.

8. An end-side dynamic optimization device for a vehicle-mounted multimodal large model, characterized in that, The device includes: The lightweight processing module is used to perform lightweight processing on the original multimodal large model to obtain a lightweight model suitable for vehicle-mounted deployment. The vehicle-mounted multimodal input data processing module is used to process vehicle-mounted multimodal input data using the lightweight model to perform inference tasks; The dynamic configuration module is used to dynamically configure the inference computing resources of the lightweight model based on the task context of the inference task and the operating status of the vehicle-mounted terminal device during the execution of the inference task. The intermediate calculation result generation module is used to perform calculations based on the dynamic configuration through the lightweight model and generate intermediate calculation results. The detection and reuse module is used to detect and reuse the intermediate calculation results in continuous inference tasks; An adaptive update module is used to adaptively update the lightweight model based on local feedback from the inference task and information about the vehicle environment.

9. An electronic device, characterized in that, The system includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the end-side dynamic optimization method for a vehicle-mounted multimodal large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the end-side dynamic optimization method for a vehicle-mounted multimodal large model as described in any one of claims 1 to 7.

Images