Model deployment method and apparatus, electronic device, medium, and product

Abstract

The application discloses a model deployment method and device, electronic equipment, medium and product, relates to the technical field of intelligent driving, divides a driving intelligence model into a plurality of sub-modules that can be separately deployed, deploys the plurality of sub-modules on a plurality of target computing nodes through a multistage pipeline, reduces the resource pressure of a single target computing node, and thereby improves the model inference efficiency.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to model deployment methods, devices, electronic devices, media and products. Background Technology

[0002] With the rapid development of intelligent connected vehicle technology, driver assistance technology is becoming increasingly mature. It can reduce driver intervention to a certain extent and assist drivers (e.g., automatic obstacle avoidance, intelligent lane changing, intelligent speed adjustment), thereby improving safety. Driver assistance is achieved through decision-making by intelligent driving models deployed in the vehicle's infotainment system.

[0003] In related technologies, intelligent driving models are deployed on multiple computing nodes of the vehicle's infotainment system, and the resources of these nodes are used to execute the intelligent driving model. However, the scale of the vehicle's infotainment system limits the resources of the computing nodes, leading to low inference efficiency of the intelligent driving model. Summary of the Invention

[0004] This application provides model deployment methods, apparatus, electronic devices, media, and products to at least address the problem of low model inference efficiency in related technologies.

[0005] This application provides a model deployment method, including:

[0006] Receive model deployment requests, which include multiple target computing nodes for the intelligent driving model and the vehicle system;

[0007] Based on the model deployment request, the intelligent driving model is functionally decoupled and divided into multiple sub-modules and their execution order, with each sub-module corresponding to multiple functions.

[0008] According to the load balancing strategy, multiple sub-modules are deployed on multiple target computing nodes, and corresponding computing and storage resources are allocated to each sub-module.

[0009] Based on the execution order, the asynchronous non-blocking communication function of the message passing interface is called to establish data transmission connections between multiple target computing nodes, thereby forming a multi-level pipeline of multiple sub-modules to deploy the intelligent driving model on multiple target computing nodes.

[0010] This application also provides a model deployment apparatus, including:

[0011] The receiving module is used to receive model deployment requests, which include multiple target computing nodes of the intelligent driving model and the vehicle system.

[0012] The module division is used to decouple the intelligent driving model according to the model deployment request, resulting in multiple sub-modules and the execution order of the multiple sub-modules. The multiple sub-modules correspond to multiple functions.

[0013] The deployment module is used to deploy multiple sub-modules on multiple target computing nodes according to the load balancing strategy, and to allocate corresponding computing and storage resources to each sub-module.

[0014] A module is established to call the asynchronous non-blocking communication function of the message passing interface according to the execution order, and establish data transmission connections between multiple target computing nodes. This allows multiple sub-modules to form a multi-level pipeline to deploy the intelligent driving model on multiple target computing nodes.

[0015] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model deployment methods.

[0016] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described model deployment methods.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model deployment methods.

[0018] This application divides the intelligent driving model into multiple independently deployable sub-modules. By deploying these sub-modules on multiple target computing nodes through a multi-level pipeline, the resource pressure on individual target computing nodes is reduced, thereby improving the model's inference efficiency. Attached Figure Description

[0019] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of a model deployment method provided in an embodiment of this application.

[0021] Figure 2 A schematic flowchart illustrating a model deployment method provided in an embodiment of this application;

[0022] Figure 3 A flowchart illustrating another model deployment method provided in an embodiment of this application;

[0023] Figure 4 A schematic diagram illustrating functional decoupling provided for embodiments of this application;

[0024] Figure 5A schematic diagram of distributed deployment provided for an embodiment of this application;

[0025] Figure 6 A schematic diagram of model reasoning provided in the embodiments of this application;

[0026] Figure 7 This is a schematic diagram of the structure of a model deployment device provided in an embodiment of this application;

[0027] Figure 8 A schematic diagram of another model deployment device provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0030] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0031] With the rapid development of intelligent connected vehicle technology, intelligent driving can be achieved through in-vehicle intelligent systems. In-vehicle intelligent systems include, but are not limited to, at least one of the following devices: information collection equipment, vehicle-mounted systems, or execution devices.

[0032] The information acquisition devices (including but not limited to at least one of the following: cameras, LiDAR, or millimeter-wave radar, etc.) can collect environmental information such as road information and generate sensor data, which is then sent to the vehicle's infotainment system. The intelligent driving model deployed on the vehicle's infotainment system makes decisions based on the sensor data and pre-set map data, generates control commands, and sends them to the execution devices. The execution devices execute the control commands to achieve intelligent driving.

[0033] Specifically, intelligent driving models are trained on a large number of samples, possessing accurate semantic understanding and decision-making capabilities, thereby improving the understanding and generalization performance of in-vehicle intelligent systems in complex scenarios. Intelligent driving models typically have a massive number of model parameters (reaching billions to tens of billions), resulting in extremely high inference complexity.

[0034] However, the scale of the vehicle's infotainment system dictates that the computing nodes have limited resources. In related technologies, deploying intelligent driving models with massive parameters on computing nodes can lead to low model inference efficiency due to the resource bottleneck of these nodes.

[0035] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] This section describes the specific application scenarios upon which the model deployment method depends. (Reference) Figure 1 , Figure 1 This diagram illustrates an application scenario for the model deployment method. The vehicle's infotainment system receives sensor data, uses the sensor data as input to perform model inference through the vehicle's intelligent driving model, obtains control commands, and instructs the vehicle to perform corresponding operations.

[0037] Figure 2 This is a flowchart illustrating the model deployment method provided in an embodiment of this application, as shown below. Figure 2 As shown, embodiments of this application provide a model deployment method, which is described in detail below:

[0038] S201. Receive model deployment request. The model deployment request includes multiple target computing nodes of the intelligent driving model and the vehicle system.

[0039] For example, a model deployment request is used to trigger the deployment of an intelligent driving model on the vehicle's infotainment system.

[0040] For example, multiple target computing nodes refer to multiple computing nodes of the vehicle's infotainment system specified in the model deployment request. These multiple computing nodes provide resources for inference on the intelligent driving model. The multiple computing nodes implement distributed computing.

[0041] With the example of a scenario, when the vehicle system deploys the intelligent driving model for the first time or updates the intelligent driving model, the deployment of the intelligent driving model is triggered by the model deployment request.

[0042] S202. Based on the model deployment request, the intelligent driving model is functionally decoupled and divided into multiple sub-modules and their execution order. Each sub-module corresponds to a different function.

[0043] For example, functional decoupling means splitting the intelligent driving model according to multiple different functions in the model.

[0044] For example, multiple submodules correspond one-to-one with multiple functions. The execution order of the multiple submodules is a preset order, and each submodule must wait for the upstream submodule to complete before it can execute.

[0045] Optionally, multiple functions may include, but are not limited to, at least one of the following: a visual encoder, a projection module, a language processor, or an action decoder. The visual encoder processes and understands the raw visual signals to determine environmental features such as roads. The projection module maps the features to a semantic space so that subsequent sub-modules can accurately understand the features. The language processor performs contextual reasoning based on the features to generate decision information. The action decoder converts the decision information into executable control commands so that the vehicle's equipment can function correctly.

[0046] With the help of scenario examples, the decision-making of the intelligent driving model is completed by multiple functions executed in a preset order. The division of functions into decoupled parts can ensure that each function is divided into an independent sub-module, so that each sub-module can accurately complete its corresponding function.

[0047] Based on the above implementation methods, the complexity of the intelligent driving model is reduced by the functional decoupling approach, allowing each sub-module to run independently, thus laying the foundation for subsequent distributed deployment.

[0048] S203. According to the load balancing strategy, deploy multiple sub-modules on multiple target computing nodes and allocate corresponding computing and storage resources to each sub-module.

[0049] For example, the load balancing strategy determines the deployment method, that is, which submodule is specifically deployed on which target computing node.

[0050] For example, the balancing strategy ensures the maximum utilization of multiple target computing nodes, avoiding a situation where some target computing nodes are overloaded while others are excessively idle. By maximizing the utilization of multiple target computing nodes, the resource utilization of the target computing nodes is improved, thereby enhancing model inference efficiency.

[0051] S204. According to the execution order, call the asynchronous non-blocking communication function of the message passing interface to establish a data transmission connection between multiple target computing nodes, thereby forming a multi-level pipeline of multiple sub-modules to deploy the intelligent driving model on multiple target computing nodes.

[0052] For example, data transmission connections are established to enable data to be transmitted normally between multiple target computing nodes, and the order of the data transmission connections is consistent with the execution order.

[0053] With the scenario example, in a data transmission connection established according to the execution order, each target computing node can only receive data sent by the designated target computing node, and can only send data to the designated target computing node. The connection order of the target computing nodes is consistent with the execution order of the submodules deployed on the target computing nodes.

[0054] For example, the messaging interface is a parallel computing communication library. This includes asynchronous non-blocking communication functions. These functions indicate that any target computing node can return to execute its next task without waiting for all data to be sent to the next target computing node; data transmission is completed in the background.

[0055] For example, by establishing a data transmission connection, the raw data can pass through multiple target computing nodes in the order of execution, perform different processing on each node, and finally obtain control instructions, thereby realizing a multi-stage pipeline.

[0056] Based on the above implementation methods, a large intelligent driving model can be deployed reasonably and efficiently on multiple distributed target computing nodes. The collaboration of multiple target computing nodes can realize model inference.

[0057] The model deployment method provided in this application receives a model deployment request, which includes multiple target computing nodes for the intelligent driving model and the vehicle's infotainment system. Based on the deployment request, the intelligent driving model is functionally decoupled and divided into multiple sub-modules and their execution order, with each sub-module corresponding to multiple functions. According to a load balancing strategy, the sub-modules are deployed on multiple target computing nodes, and corresponding computing and storage resources are allocated to each sub-module. Based on the execution order, an asynchronous non-blocking communication function of the message passing interface is invoked to establish data transmission connections between the multiple target computing nodes, thereby forming a multi-level pipeline to deploy the intelligent driving model across these nodes. This solution divides the intelligent driving model into multiple independently deployable sub-modules and deploys them on multiple target computing nodes through a multi-level pipeline, reducing the resource pressure on individual target computing nodes and thus improving model inference efficiency.

[0058] Based on any of the above embodiments, the following, in conjunction with Figure 3 The detailed process of model deployment is explained.

[0059] Figure 3 This is a flowchart illustrating another model deployment method provided in an embodiment of this application. Figure 3 As shown, the method includes:

[0060] S301. Receive model deployment request, which includes multiple target computing nodes for the intelligent driving model and the vehicle system.

[0061] It should be noted that the execution process of S301 is the same as that of S201, and will not be repeated here.

[0062] S302. Determine the functional logic of the intelligent driving model.

[0063] For example, the functional logic refers to the working principle and data flow within the intelligent driving model. After determining the functional logic, functional decoupling can be performed.

[0064] Optionally, by analyzing the computational graph or architecture design of the intelligent driving model, it can be determined how the input data gradually generates control commands, thereby obtaining the functional logic of the intelligent driving model.

[0065] S303. Based on the functional logic, determine the multiple sub-functions of the intelligent driving model and the logical order of the multiple sub-functions.

[0066] For example, after determining the overall workflow of the intelligent driving model, naturally existing and relatively independent sub-parts are identified, resulting in multiple sub-functions. Based on the overall workflow of the intelligent driving model, the logical order of these multiple sub-functions is determined.

[0067] As illustrated by the scenario example, the visual encoder is used to receive input data, and the motion decoder is used to generate the final control commands. Therefore, the motion decoder should be placed after the visual encoder.

[0068] S304. Based on multiple sub-functions and logical order, the intelligent driving model is logically divided to obtain multiple sub-modules corresponding to multiple sub-functions and the execution order corresponding to the logical order.

[0069] Below, in conjunction with Figure 4 Explain the functional decoupling.

[0070] Figure 4 This is a schematic diagram illustrating the functional decoupling provided in the embodiments of this application. For example... Figure 4 As shown, the visual encoder and projection module process the data acquired by the sensor to obtain a feature tensor. The language processor and motion decoder process the feature tensor to obtain control commands so that the execution device can execute the control commands. The sub-modules corresponding to the visual encoder and projection module can be deployed on one target computing node. The sub-modules corresponding to the language processor and motion decoder can be deployed on another target computing node. The target nodes can be directly scheduled through a containerized scheduler.

[0071] For example, the division positions in the intelligent driving model are determined based on multiple sub-functions and logical order. The model is then divided according to these division positions to obtain multiple sub-functions. The logical order is then determined as the execution order.

[0072] Optionally, based on multiple sub-functions and logical order, multiple boundary positions between multiple sub-functions can be determined from the computational graph of the intelligent driving model or the architecture design, and multiple sub-modules can be obtained by dividing based on the boundary positions.

[0073] Based on the above implementation method, the system is divided according to multiple sub-functions and logical order, ensuring that the resulting sub-modules are functionally cohesive and complete. This avoids functional errors caused by the division and improves the reliability of model deployment.

[0074] S305. Determine multiple hardware processing capabilities corresponding to multiple target computing nodes. The multiple hardware processing capabilities include at least one of the following: computing power value or storage power value.

[0075] For example, hardware processing power characterizes how many resources a target computing node has for model inference. Deploying intelligent driving models based on hardware processing power can maximize resource utilization.

[0076] For example, the computing power value is used to quantify the computing performance of the target computing node.

[0077] Optionally, the computing power value can be the theoretical computing power of the target computing node.

[0078] For example, the storage capacity value is used to quantify the storage capacity of the target computing node.

[0079] Optionally, the storage capacity value can be the video memory capacity or the RAM capacity, etc.

[0080] S306. Determine multiple hardware requirement information corresponding to multiple sub-modules. The multiple hardware requirement information includes at least one of the following: calculated requirement value or storage occupancy value.

[0081] For example, hardware requirement information represents the amount of hardware resources required for a submodule to operate.

[0082] For example, the computational requirement value is used to quantify the amount of computation required for a submodule to run. For instance, it could be the required computing power.

[0083] For example, the storage occupancy value is used to quantify the storage space required for the submodule to run.

[0084] Optionally, the storage footprint may include the space occupied by the model parameters of the submodule and the space occupied when the submodule runs to its peak.

[0085] S307. Based on the balancing strategy, the processing capabilities of multiple hardware components, and the information on multiple hardware requirements, deploy multiple sub-modules on multiple target computing nodes.

[0086] For example, hardware processing power limits the upper limit of target memory nodes. Under this constraint, a balanced distribution strategy is used for reasonable deployment. This ensures that each target computing node is sufficient to handle the deployed sub-modules and that the resource utilization of each target computing node is similar.

[0087] One feasible implementation involves deploying multiple sub-modules across multiple target computing nodes using the following methods: determining the mapping relationship between model types and thresholds; determining the current model type of the intelligent driving model; determining the first and second thresholds corresponding to the intelligent driving model based on the current model type and the mapping relationship; deploying the multiple sub-modules across multiple target computing nodes based on a computational load balancing strategy, multiple hardware processing capabilities, and multiple hardware requirement information, such that the difference in the number of operations between any two target computing nodes is less than or equal to the first threshold, and the computational requirement value of each target computing node does not exceed its computational capability value; or, deploying the multiple sub-modules across multiple target computing nodes based on a storage occupancy balancing strategy, multiple hardware processing capabilities, and multiple hardware requirement information, such that the storage occupancy ratio of any two target computing nodes is less than or equal to the second threshold, and the storage occupancy value of each target computing node does not exceed its storage capability value.

[0088] For example, a large amount of data is summarized in advance to determine the mapping relationship, and the threshold that is suitable for the model type can be determined based on the mapping relationship.

[0089] Optionally, the current model type is the model architecture of the intelligent driving model itself. Factors affecting the model architecture include, but are not limited to, at least one of the following: the application scenario of the intelligent driving model (e.g., expressway or urban road), the data type of the input of the intelligent driving model (e.g., LiDAR data or visual data), and the technical level of the intelligent driving model (e.g., assisted driving, conditional autonomous driving).

[0090] For example, a computational load balancing strategy is used to control whether multiple target computing nodes are computationally overloaded and to control the difference in computational load among multiple target computing nodes, avoiding low model inference efficiency caused by excessive computational tasks on some target computing nodes. A first threshold can be used to control the difference in the number of operations between the target computing node with the highest computational load and the target computing node with the lowest computational load, thereby ensuring that the number of operations on multiple target computing nodes is similar.

[0091] For example, a storage balancing strategy is used to control whether multiple target compute nodes are overloaded and to control the difference in storage load among multiple target compute nodes, avoiding low model inference efficiency caused by excessive storage tasks on some target compute nodes. A second threshold can be used to control the difference in storage occupancy between the target compute node with the highest storage load and the target compute node with the lowest storage load. This ensures that the storage resource utilization of multiple target compute nodes is similar.

[0092] In this feasible implementation, the deployment of multiple sub-modules and multiple target computing nodes is adaptively adjusted through a balancing strategy, thereby maximizing the utilization of the resources of the target computing nodes and improving the efficiency of model inference.

[0093] A feasible implementation method can also be achieved by the following approach to achieve balanced deployment: according to the layer balancing strategy, multiple sub-modules are deployed on multiple target computing nodes so that the difference in the number of sub-modules deployed on any two target computing nodes is less than or equal to a third threshold.

[0094] For example, the number of submodules deployed on each target computing node is determined by counting the submodules deployed on each target computing node, and then balanced based on the number and a third threshold to make the number of submodules deployed on each target computing node similar.

[0095] With scenario examples, layer balancing ensures that submodules are not heavily skewed onto certain target computing nodes. Layer balancing only requires simple counting, and the deployment process does not require complex calculations.

[0096] In this feasible implementation, deployment efficiency can be improved by calculating simple layer balance.

[0097] One feasible implementation method for model deployment may also include: encapsulating the sub-modules of each target computing node as container instances.

[0098] For example, a separate container instance can be created for one or more sub-modules deployed on each target compute node.

[0099] For example, container instances can be managed through a containerized scheduler (e.g., k3s) to dynamically manage the tasks executed by the container instances.

[0100] In this feasible implementation, containerization improves scalability, allowing for the rapid addition or replacement of modules, thereby improving the efficiency of model inference.

[0101] S308. According to the execution order, call the asynchronous non-blocking communication function of the message passing interface to establish a data transmission connection between multiple target computing nodes, thereby forming a multi-level pipeline of multiple sub-modules to deploy the intelligent driving model on multiple target computing nodes.

[0102] Below, in conjunction with Figure 5 This section explains distributed deployment.

[0103] Figure 5 This is a schematic diagram illustrating a distributed deployment provided in an embodiment of this application. For example... Figure 5As shown, the intelligent driving model W is divided into multiple sub-modules. , , ...Identify multiple target computing nodes , ...Deploy multiple submodules on multiple target compute nodes. Adjust the connection order of the target compute nodes according to the execution order of the submodules. Manage communication between the multiple target compute nodes through communication functions.

[0104] One feasible implementation method for model deployment may further include: establishing data transmission connections between multiple target computing nodes according to a pipelined processing mechanism, wherein each target computing node processes data from different frames simultaneously; and setting up an intermediate result buffer, which is used to store compressed files of feature tensors to be transmitted, wherein the feature tensors are the intermediate computation results of the target nodes.

[0105] For example, a high-speed, low-latency data transmission channel is established between multiple target computing nodes, allowing each target computing node to process data from different frames simultaneously to achieve pipelined parallel processing.

[0106] For example, the feature tensor is an intermediate computation result generated by a submodule of the upstream target computing node and needs to be transmitted to the downstream target computing node for further computation.

[0107] For example, the intermediate result cache is a temporary storage area between target computing nodes, used to receive feature tensors generated by upstream target computing nodes, call compression algorithms to compress the feature tensors, and temporarily store compressed files.

[0108] For example, by storing compressed files in an intermediate results buffer, the size of the data can be effectively reduced.

[0109] With the aid of scenario examples, each target computing node is only responsible for the computation of its own submodule, asynchronously passing the results to the next target computing node, thus forming a pipelined continuous inference. In terms of scheduling strategy, if any target computing node is still idle after completing the processing of the current frame, it can immediately begin the inference task for the next frame. Simultaneously, each target computing node transmits intermediate results to the next target computing node via asynchronous communication, allowing multiple frames to be processed in parallel without blocking each other.

[0110] In this feasible implementation, the data transmission connection established through the pipelined processing mechanism can improve the throughput of data transmission, thereby improving the efficiency of model inference.

[0111] One feasible implementation method for model deployment may also include: decoupling computation and communication operations using double buffering technology, establishing data transmission connections between multiple target computing nodes, so that the target computing nodes can transmit the processing results of the previous frame while processing the current frame data.

[0112] For example, for each target computing node that needs to send data outward, two buffers are configured, and the computing and communication processes use different buffers to achieve parallel operation.

[0113] With a scenario example, within a processing cycle, the computing unit performs calculations on the data of the current frame and writes the result to the first buffer. Simultaneously, the communication unit reads the calculated data of the previous frame from the second buffer and sends it to the next target computing node via the network interface. When the target computing unit completes the calculation of the current frame (filling the first buffer) and the communication unit completes the transmission of the previous frame's data (emptying the second buffer), the computing unit and the communication unit exchange their target buffers. In the next processing cycle, the computing unit uses the recently released second buffer to process the data of the new frame, while the communication unit reads data from the recently filled first buffer and sends it. The use of two buffers decouples the computation and communication operations, enabling parallel processing.

[0114] In this feasible implementation, double buffering technology can eliminate serial latency in computation and communication, thereby improving the efficiency of model inference.

[0115] One feasible implementation method for the model inference method may include: receiving a model inference request, the model inference request including a task to be executed; determining multiple current loads corresponding to multiple target computing nodes based on the model inference request; and performing multi-level pipelined model inference on the task to be executed based on the multiple current loads to obtain the inference result.

[0116] For example, the task to be performed is the input data for model inference, such as sensor data, user instructions (e.g., "navigate to the nearest charging station"), etc.

[0117] For example, multiple current loads represent real-time resource usage status. Inference is then performed based on the real-time resource usage status according to a pre-defined multi-stage pipeline.

[0118] In this feasible implementation, model inference is performed based on multiple current loads, which can maximize the utilization of the real-time resources of the target computing node, thereby improving the efficiency of model inference.

[0119] One feasible implementation method is to perform model inference by: determining a load threshold; if multiple current loads are all less than the load threshold, then performing multi-level pipeline model inference on the task to be executed to obtain the inference result.

[0120] For example, the load threshold is determined based on the amount of security reserved resources. When the load of each target computing node is less than the load threshold, it means that each target computing node can efficiently execute model inference. At this time, multi-level model inference can give full play to the performance of each target computing node.

[0121] Below, in conjunction with Figure 6 Explain the model reasoning.

[0122] Figure 6 This is a schematic diagram of model reasoning provided in an embodiment of this application. For example... Figure 6 As shown, in response to a model inference request, multiple real-time current loads are determined. A judgment is made based on these current loads and a load threshold. If the current load is sufficient, model inference is performed. If the current load is insufficient, resource scheduling is performed first, followed by model inference.

[0123] In this feasible implementation, the load can be quantified by using a load threshold, thereby improving the accuracy of model inference.

[0124] One feasible implementation includes a model inference request that also includes a task priority; the model deployment method further includes: if at least one current load among multiple current loads is greater than or equal to a load threshold, then a priority threshold is determined; if the task priority is greater than or equal to the priority threshold, then an abnormal computing node is determined from multiple target computing nodes; and the number of sub-modules of the abnormal computing node is adjusted through the containerized scheduler of the abnormal computing node so that the adjusted load of the abnormal computing node is less than the load threshold.

[0125] For example, task priority is used to identify the urgency and importance of tasks to be performed.

[0126] Using scenario examples, tasks directly related to safety, such as emergency avoidance, should be given higher priority. Tasks such as casual conversation or navigation route planning should be given lower priority. Higher-priority tasks should be executed first.

[0127] For example, if at least one current load is greater than or equal to the load threshold, it means that the current resources are insufficient to handle multiple tasks, and a decision needs to be made to prioritize ensuring that important tasks can be executed efficiently.

[0128] For example, if a task priority is greater than or equal to a priority threshold, it indicates that the task to be executed is an important task and needs to be executed efficiently. The number of sub-modules on abnormal computing nodes is dynamically scheduled through a containerized scheduler until the resources of the abnormal computing node are sufficient to support the sub-modules deployed on the abnormal computing node.

[0129] In this feasible implementation, dynamic scheduling ensures sufficient resources for model inference, thereby improving the efficiency of model inference.

[0130] One feasible implementation method for model deployment may also include: in each processing stage of multi-level pipelined model inference, after the current frame data is processed, a prefetch operation is triggered to load the next frame of data to be processed from the storage location into the video memory of the target computing node of the current stage.

[0131] For example, each submodule runs as a resident container service on its respective target compute node. When a task arrives at a submodule, it is placed in the corresponding task queue. The containerized scheduler allocates computing resources to the tasks in the queue and schedules their execution according to preset resource quotas and priority policies. After processing its current task, each submodule sends its intermediate results to the task queue of the downstream submodule through an asynchronous data transmission channel, until the last submodule outputs its inference result.

[0132] For example, the prefetch operation is used to preload the data to be processed in the next frame from its storage location into the target computing node where the current stage is located. The prefetch operation is executed by a separate thread and does not block the computing task.

[0133] Optionally, a checkpoint mechanism can be set up at each stage of the pipeline to save the processing status periodically. When a node failure is detected, the task of the failed node can be automatically migrated to the backup node. A data retransmission mechanism can be used to ensure communication reliability, and an acknowledgment retransmission strategy can be implemented for important intermediate result data.

[0134] Optionally, the power consumption status and temperature information of each target computing node can be monitored in real time, and the computing frequency and voltage of each node can be dynamically adjusted according to the power consumption budget. A task migration strategy can be adopted to migrate high-load tasks from high-power nodes to low-power nodes.

[0135] In this feasible implementation, the prefetching operation can reduce the waiting time for the target computing node to load data after completing the current computing task, thereby improving the efficiency of model inference.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0137] Figure 7 This is a schematic diagram of the structure of the model deployment device provided in an embodiment of this application. Figure 7 As shown, embodiments of this application also provide a model deployment apparatus 70, which may include: a receiving module 71, a partitioning module 72, a deployment module 73, and a creation module 74.

[0138] The receiving module 71 is used to receive model deployment requests, which include multiple target computing nodes of the intelligent driving model and the vehicle system.

[0139] Module 72 is used to divide the intelligent driving model into functional decoupling parts according to the model deployment request, resulting in multiple sub-modules and the execution order of the sub-modules. The multiple sub-modules correspond to multiple functions.

[0140] Deployment module 73 is used to deploy multiple sub-modules on multiple target computing nodes according to the load balancing strategy, and to allocate corresponding computing and storage resources to each sub-module.

[0141] Module 74 is established to call the asynchronous non-blocking communication function of the message passing interface according to the execution order, establish data transmission connections between multiple target computing nodes, thereby forming a multi-level pipeline of multiple sub-modules to deploy the intelligent driving model on multiple target computing nodes.

[0142] Optionally, the receiving module 71 can perform... Figure 2 S201 in the embodiment.

[0143] Optionally, module 72 can be executed. Figure 2 S202 in the embodiment.

[0144] Optionally, deployment module 73 can execute Figure 2 S203 in the embodiment.

[0145] Optionally, module 74 can be executed. Figure 2 S204 in the embodiment.

[0146] It should be noted that the model deployment device shown in the embodiments of this application can execute the technical solutions shown in the above method embodiments, and its implementation principle and beneficial effects are similar, so they will not be described again here.

[0147] In one possible implementation, the partitioning module 72 is specifically used for:

[0148] Determine the functional logic of the intelligent driving model;

[0149] Based on the functional logic, determine the multiple sub-functions of the intelligent driving model and the logical order of the multiple sub-functions;

[0150] Based on multiple sub-functions and logical order, the intelligent driving model is logically divided to obtain multiple sub-modules corresponding to multiple sub-functions and the execution order corresponding to the logical order.

[0151] In one possible implementation, deployment module 73 is specifically used for:

[0152] Determine multiple hardware processing capabilities corresponding to multiple target computing nodes, wherein the multiple hardware processing capabilities include at least one of the following: computing power value or storage capacity value;

[0153] Determine multiple hardware requirements for multiple sub-modules, including at least one of the following: calculated requirement value or storage usage value;

[0154] Based on the balancing strategy, the processing capabilities of multiple hardware components, and the hardware requirements, multiple sub-modules are deployed on multiple target computing nodes.

[0155] In one possible implementation, deployment module 73 is specifically used for:

[0156] Determine the mapping relationship between model type and threshold;

[0157] Determine the current model type of the intelligent driving model;

[0158] Based on the current model type and mapping relationship, determine the first and second thresholds corresponding to the intelligent driving model;

[0159] Based on a computational load balancing strategy, the processing capabilities of multiple hardware components, and information on multiple hardware requirements, multiple sub-modules are deployed on multiple target computing nodes, ensuring that the difference in the number of operations performed on any two target computing nodes is less than or equal to a first threshold, and that the computational requirement of each target computing node does not exceed its computational capacity; or,

[0160] Based on the storage occupancy balancing strategy, the processing capabilities of multiple hardware devices, and the hardware requirements, multiple sub-modules are deployed on multiple target computing nodes so that the storage occupancy ratio of any two target computing nodes is less than or equal to the second threshold, and the storage occupancy value of each target computing node does not exceed the storage capacity value.

[0161] In one possible implementation, the deployment module 73 is specifically used to: deploy multiple sub-modules on multiple target computing nodes according to a layer balancing strategy, so that the difference in the number of sub-modules deployed on any two target computing nodes is less than or equal to a third threshold.

[0162] Figure 8 This is a schematic diagram of another model deployment device provided in an embodiment of this application. Figure 7 Based on the illustrated embodiments, as Figure 8As shown, the model deployment device 70 also includes: an encapsulation module 75, a connection module 76, a decoupling module 77, an execution module 78, a scheduling module 79, and a prefetching module 710.

[0163] Encapsulation module 75 is used to encapsulate the sub-modules of each target computing node into container instances.

[0164] Connection module 76, used for:

[0165] Based on the pipelined processing mechanism, data transmission connections are established between multiple target computing nodes, where each target computing node processes data from different frames simultaneously.

[0166] Set up an intermediate result buffer, which is used to store compressed files of feature tensors to be transmitted. The feature tensors are the intermediate calculation results of the target node.

[0167] The decoupling module 77 is used to: decouple computation and communication operations according to double buffering technology, and establish data transmission connections between multiple target computing nodes so that the target computing nodes can transmit the processing results of the previous frame while processing the current frame data.

[0168] Execution module 78 is used for:

[0169] Receive a model inference request, which includes tasks to be executed.

[0170] Based on the model inference request, determine the current load corresponding to multiple target computing nodes;

[0171] Based on multiple current loads, a multi-level pipeline model inference is performed on the task to be executed to obtain the inference result.

[0172] In one possible implementation, execution module 78 is specifically used for:

[0173] Determine the load threshold;

[0174] If multiple current loads are all less than the load threshold, then a multi-level pipeline model inference is performed on the task to be executed to obtain the inference result.

[0175] Scheduling module 79 is used for:

[0176] If at least one of the multiple current loads is greater than or equal to the load threshold, then the priority threshold is determined.

[0177] If the task priority is greater than or equal to the priority threshold, then the abnormal computing node is determined from multiple target computing nodes;

[0178] By using a containerized scheduler for the abnormal computing node, the number of sub-modules of the abnormal computing node is adjusted so that the adjusted load of the abnormal computing node is less than the load threshold.

[0179] The prefetch module 710 is used to: trigger a prefetch operation to load the next frame of data to be processed from the storage location into the video memory of the target computing node of the current stage after the current frame data is processed in each processing stage of the multi-stage pipeline model inference.

[0180] For a description of the features in the embodiment corresponding to the model deployment device, please refer to the relevant description in the embodiment corresponding to the model deployment method, which will not be repeated here.

[0181] Figure 9 A schematic diagram of the structure of the electronic device provided in this application. Figure 9 As shown, the electronic device 90 provided in this embodiment includes at least one processor 901 and a memory 902. Optionally, the electronic device 90 further includes a communication component 903. The processor 901, memory 902, and communication component 903 are connected via a bus.

[0182] In a specific implementation, at least one processor 901 executes computer execution instructions stored in memory 902, causing at least one processor 901 to execute the above-described model deployment method embodiment.

[0183] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0184] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0185] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0186] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0187] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model deployment method embodiments at runtime.

[0188] In one exemplary embodiment, the aforementioned non-volatile computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0189] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model deployment method embodiments.

[0190] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model deployment method embodiments.

[0191] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0192] The foregoing has provided a detailed description of a model deployment method, apparatus, electronic device, medium, and product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model deployment method, characterized in that, include: Receive a model deployment request, which includes multiple target computing nodes for the intelligent driving model and the vehicle system; Based on the model deployment request, the intelligent driving model is functionally decoupled and divided into multiple sub-modules and the execution order of the multiple sub-modules, with the multiple sub-modules corresponding to multiple functions; Determine multiple hardware processing capabilities corresponding to multiple target computing nodes, wherein the multiple hardware processing capabilities include at least one of the following: computing power value or storage capacity value; Determine multiple hardware requirement information corresponding to the multiple sub-modules, wherein the multiple hardware requirement information includes at least one of the following: calculated requirement value or storage occupancy value; Determine the mapping relationship between model type and threshold; Determine the current model type of the intelligent driving model; Based on the current model type and the mapping relationship, determine the first threshold and the second threshold corresponding to the intelligent driving model; Based on the computational load balancing strategy, the processing capabilities of the multiple hardware components, and the hardware requirement information, the multiple sub-modules are deployed on the multiple target computing nodes so that the difference in the number of operations between any two target computing nodes is less than or equal to a first threshold, and the computational requirement value of each target computing node does not exceed the computational capability value. or, Based on the storage occupancy balancing strategy, the processing capabilities of the multiple hardware components, and the hardware requirements information, the multiple sub-modules are deployed on the multiple target computing nodes so that the storage occupancy ratio of any two target computing nodes is less than or equal to the second threshold, and the storage occupancy value of each target computing node does not exceed the storage capacity value. Allocate corresponding computing and storage resources to each submodule; According to the execution order, the asynchronous non-blocking communication function of the message passing interface is called to establish a data transmission connection between the multiple target computing nodes, thereby forming a multi-level pipeline of the multiple sub-modules to deploy the intelligent driving model on the multiple target computing nodes; The method further includes: Receive a model inference request, the model inference request including a task to be executed; Based on the model inference request, determine multiple current loads corresponding to multiple target computing nodes; Determine the load threshold; if all the current loads are less than the load threshold, perform multi-level pipeline model inference on the task to be executed to obtain the inference result; The model inference request also includes task priority; the method further includes: If at least one of the multiple current loads is greater than or equal to the load threshold, then a priority threshold is determined; If the task priority is greater than or equal to the priority threshold, then an abnormal computing node is determined from the plurality of target computing nodes; The number of sub-modules of the abnormal computing node is adjusted by the containerized scheduler of the abnormal computing node so that the adjusted load of the abnormal computing node is less than the load threshold.

2. The model deployment method according to claim 1, characterized in that, The intelligent driving model is functionally decoupled and divided into multiple sub-modules and their execution order, including: Determine the functional logic of the intelligent driving model; Based on the functional logic, determine multiple sub-functions of the intelligent driving model and the logical order of the multiple sub-functions; Based on the multiple sub-functions and the logical order, the intelligent driving model is logically divided to obtain multiple sub-modules corresponding to the multiple sub-functions and the execution order corresponding to the logical order.

3. The model deployment method according to claim 1, characterized in that, According to the load balancing strategy, the multiple sub-modules are deployed on the multiple target computing nodes, including: According to the layer balancing strategy, the multiple sub-modules are deployed on the multiple target computing nodes so that the difference in the number of sub-modules deployed on any two target computing nodes is less than or equal to the third threshold.

4. The model deployment method according to claim 1, characterized in that, The method further includes: Each target computing node's sub-modules are encapsulated as container instances.

5. The model deployment method according to claim 1, characterized in that, The method further includes: According to the pipelined processing mechanism, a data transmission connection is established between the multiple target computing nodes, wherein each target computing node processes data from different frames simultaneously. An intermediate result buffer is set up to store compressed files of feature tensors to be transmitted, wherein the feature tensors are intermediate calculation results of the target node.

6. The model deployment method according to claim 1, characterized in that, The method further includes: By decoupling computation and communication operations using double buffering technology, a data transmission connection is established between the multiple target computing nodes, so that the target computing nodes can transmit the processing results of the previous frame while processing the current frame data.

7. The model deployment method according to claim 1, characterized in that, The method further includes: In each processing stage of the multi-stage pipelined model inference, after the current frame data is processed, a prefetch operation is triggered to load the next frame of data to be processed from the storage location into the GPU memory of the target computing node of the current stage.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the model deployment method as described in any one of claims 1 to 7.

Images