Cross-node computing power scheduling method and device, medium and vehicle

Abstract

The embodiment of the invention relates to a cross-node computing power scheduling method and device, a medium and a vehicle, and the method comprises the steps: receiving a first target computing power service of a model use message, and detecting whether a first target computing power node at which the first target computing power service is deployed meets a preset computing power condition or not; if the first target computing power node does not meet the preset computing power condition, the main computing power service screens out a second target computing power node from the plurality of computing power nodes and screens out a second target computing power service from computing power services deployed on the second target computing power node; and the second target computing power service loads the target model and executes an inference task corresponding to the model use message by using the target model. According to the embodiment of the invention, the computing power resources of the whole vehicle can be scheduled in a unified manner to achieve the sharing of the computing power of the whole vehicle, so that the high computing power node can undertake the reasoning tasks of other computing power nodes in the idle period, and the overall utilization rate of the computing power resources of the whole vehicle is improved.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle technology, and in particular to a cross-node computing power scheduling method, apparatus, medium, and vehicle. Background Technology

[0002] In current vehicles, each functional module (such as autonomous driving and smart cockpit) is typically handled by an independent Electronic Control Unit (ECU). Each ECU's Operating System (OS) has its own models pre-deployed. When the vehicle is running, the OS on each ECU calls the models deployed on its own OS, and each OS manages its own models without interfering with each other.

[0003] However, this can lead to some ECUs having excessive computing power while others have insufficient computing power, resulting in an uneven utilization of computing resources across the vehicle. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, embodiments of this disclosure provide a cross-node computing power scheduling method, apparatus, medium, and vehicle.

[0005] A first aspect of this disclosure provides a cross-node computing power scheduling method applied to a vehicle-side model deployment framework. The vehicle-side model deployment framework includes multiple computing power nodes. Each computing power node has at least one operating system deployed on it, and each operating system has a computing power service deployed on it. At least one model is deployed on each operating system. Among all the computing power services deployed on the multiple computing power nodes, one computing power service is a master computing power service, and the remaining computing power services are slave computing power services. The method includes: The receiving model uses the first target computing power service to detect whether the first target computing power node it is deployed on meets the preset computing power conditions; If the first target computing power node does not meet the preset computing power conditions, the main computing power service selects a second target computing power node from the plurality of computing power nodes and selects a second target computing power service from the computing power services deployed on the second target computing power node. The model usage message is received by the third target computing power service from the application and forwarded to the first target computing power service because the target model is not deployed on the third target computing power node on which the third target computing power service is deployed. The second target computing power node meets the preset computing power conditions, and the second target computing power service and the target model are deployed on the same operating system. The second target computing power service loads the target model and uses the target model to execute the inference task corresponding to the model usage message.

[0006] A second aspect of this disclosure provides a cross-node computing power scheduling device applied to a vehicle-side model deployment framework. The vehicle-side model deployment framework includes multiple computing power nodes. Each computing power node has at least one operating system deployed on it, and each operating system has a computing power service deployed on it. At least one model is deployed on each operating system. Among all the computing power services deployed on the multiple computing power nodes, one computing power service is a master computing power service, and the remaining computing power services are slave computing power services. The device includes: The first detection module is used to detect whether the first target computing power service that receives the model usage message meets the preset computing power conditions on the first target computing power node it is deployed on. The first filtering module is configured to, if the first target computing power node does not meet the preset computing power conditions, the main computing power service filters out a second target computing power node from the plurality of computing power nodes and filters out a second target computing power service from the computing power services deployed on the second target computing power node, wherein the model usage message is received by the third target computing power service from the application and forwarded to the first target computing power service because the target model is not deployed on the third target computing power node on which the third target computing power service is deployed, the second target computing power node meets the preset computing power conditions, and the second target computing power service and the target model are deployed on the same operating system; The first inference module is used by the second target computing power service to load the target model and use the target model to execute the inference task corresponding to the model usage message.

[0007] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the method of the first aspect described above.

[0008] A fourth aspect of this disclosure provides a vehicle that includes the cross-node computing power scheduling device and / or computer-readable storage medium described in the second aspect.

[0009] The technical solution provided in this disclosure has the following advantages compared with the prior art: In this embodiment of the disclosure, the first target computing power service that receives the model usage message checks whether the first target computing power node on which it is deployed meets the preset computing power conditions. If the first target computing power node does not meet the preset computing power conditions, the main computing power service selects a second target computing power node from multiple computing power nodes and selects a second target computing power service from the computing power services deployed on the second target computing power node. The model usage message is received by the third target computing power service from the application and forwarded to the first target computing power service because the target model is not deployed on the third target computing power node on which the third target computing power service is deployed. The target model refers to the model indicated by the model usage message. The second target computing power node meets the preset computing power conditions, and the second target computing power service and the target model are deployed on the same operating system. The second target computing power service loads the target model and uses the target model to execute the inference task corresponding to the model usage message. As can be seen, by adopting the above technical solution, if the target model indicated by the model usage message is not deployed on the third target computing power node, the third target computing power service (the computing power service that receives the model usage message from the application) forwards the model usage message to the first target computing power service (deployed on the first target computing power node with the target model); if the first target computing power node does not meet the preset computing power conditions, the second target computing power service (deployed on the second target computing power node that meets the preset computing power conditions) completes the target model loading and inference task execution. Therefore, this embodiment of the present disclosure allows the second target computing power node, which has the target model deployed and meets the preset computing power conditions, to respond to the model usage message. This not only improves the success rate of responding to the model usage message but also enables unified scheduling of the vehicle's computing power resources to achieve vehicle-wide computing power sharing. This allows high-computing-power nodes to take over the inference tasks of other computing power nodes during idle periods, thereby improving the overall utilization rate of the vehicle's computing power resources. Furthermore, the third target computing node prioritizes finding the first target computing node that may be able to respond to the model usage message. Only when the first target computing node has insufficient computing power will the model usage message be upgraded to the main computing node. The main computing node then coordinates with the second target computing node that has the target model and sufficient computing power. In this way, the main computing node can be avoided from being disturbed frequently, allowing it to focus more on other complex tasks. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a cross-node computing power scheduling method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of a vehicle-side model deployment framework provided in an embodiment of this disclosure; Figure 3 This is a logical schematic diagram illustrating the working process of an NN adapter provided in an embodiment of this disclosure; Figure 4 This is a flowchart illustrating a cross-node computing power scheduling example provided in this disclosure embodiment; Figure 5 This is a schematic diagram of the structure of a cross-node computing power scheduling device provided in an embodiment of this disclosure. Detailed Implementation

[0013] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0014] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0015] In current vehicles, each functional module (such as autonomous driving and smart cockpit) is typically handled by an independent Electronic Control Unit (ECU). Each ECU's Operating System (OS) pre-deploys the models required by its own OS. When the vehicle is running, the OS on each ECU calls the models deployed on its own OS according to a scheduling strategy. Each OS manages its own models independently, without interference. However, this leads to some ECUs having excess computing power while others lack sufficient computing power, resulting in uneven utilization of the vehicle's computing resources. For example, some ECUs are under low load most of the time, with a large amount of their computing resources idle; when some ECUs need to perform complex inference tasks, their local computing power is insufficient to support the operation of high-precision, low-latency models, leading to performance degradation or even functional limitations. Furthermore, this model deployment method is relatively rigid, resulting in low model utilization. Additionally, it cannot dynamically expand the vehicle's computing resources to more flexibly meet the needs of different application scenarios. In view of this, embodiments of this disclosure provide a cross-node computing power scheduling method, apparatus, medium, and vehicle. The local node computing power scheduling method will be described in detail below.

[0016] Figure 1This is a flowchart of a cross-node computing power scheduling method provided in this disclosure embodiment, which can be executed by a vehicle-side model deployment framework. Figure 1 As shown, the method provided in this embodiment includes the following steps: S110. The first target computing power service that receives the message detects whether the first target computing power node on which it is deployed meets the preset computing power conditions.

[0017] S120. If the first target computing power node does not meet the preset computing power conditions, the main computing power service selects the second target computing power node from multiple computing power nodes and selects the second target computing power service from the computing power services deployed on the second target computing power node.

[0018] The model usage message is received by the third target computing power service from the application and forwarded to the first target computing power service because the target model is not deployed on the third target computing power node where the third target computing power service is deployed. The second target computing power node meets the preset computing power conditions, and the second target computing power service and the target model are deployed on the same operating system.

[0019] S130. The second target computing power service loads the target model and uses the target model to execute the inference task corresponding to the model usage message.

[0020] To more clearly illustrate the cross-node computing power scheduling method provided in this disclosure, the vehicle-side model deployment framework will first be described in detail below. For example, Figure 2 This is a schematic diagram of the structure of a vehicle-side model deployment framework provided in an embodiment of this disclosure, such as... Figure 2 As shown, the vehicle-side model deployment framework includes: multiple computing power nodes, each computing power node having at least one operating system deployed on it, each operating system having a computing power service deployed on it, and the operating system having a model deployed on it. Among all the computing power services deployed on the multiple computing power nodes, one computing power service is the master computing power service, and the remaining computing power services are slave computing power services.

[0021] Specifically, a computing node is an ECU deployed on a vehicle within a vehicle-side model deployment framework, capable of performing inference tasks.

[0022] In some examples, all computing nodes are embedded computing nodes, where embedded computing nodes refer to computing nodes integrated into the vehicle.

[0023] In other examples, the multiple computing nodes in the vehicle-side model deployment framework include embedded computing nodes and pluggable computing nodes. Pluggable computing nodes refer to external computing nodes that can be hot-swapped via a computing interface (i.e., a physical interface), while the main computing service is deployed in the embedded computing nodes.

[0024] Specifically, pluggable computing nodes may include AIBOX, but are not limited to these.

[0025] Specifically, when a pluggable computing node is inserted into the computing interface, the pluggable computing node can dynamically access the vehicle-side model deployment framework through its own deployed computing services. For example, among all the computing services deployed on all computing nodes in the vehicle-side model deployment framework, one computing service is the master computing service (configured in the embedded computing node), and the remaining computing services are slave computing services. The process of dynamically accessing the vehicle-side model deployment framework for pluggable computing power nodes is as follows: When the main computing power service receives a node join broadcast message from a slave computing power service on a pluggable computing power node, it parses the node information of the pluggable computing power node from the broadcast message and performs authentication based on this information. The node join broadcast message is sent after the pluggable computing power node is inserted into the computing power interface. This message includes the functional domain to which the pluggable computing power node belongs, its role, the type of operating system deployed on it, its name, its publicly available services, its network address, and verification information. After successful authentication, the main computing power service sends a join confirmation message to the slave computing power services deployed on the pluggable computing power node, prompting them to send a node registration message. This node registration message includes the command type, session ID, and the role of the requesting node. The registration process includes the following information: message body length (in bytes), checksum of the entire message, node role, functional domain, operating system type, publicly available services, network address of the node, network address of the parent node, and a model list. The model list includes the model ID and model type of each deployed model. The master computing service sends a first confirmation message to the slave computing services deployed on the pluggable computing node and adds the pluggable computing node to the computing node list. Upon receiving node capability messages from slave computing services deployed on the pluggable computing node, the master service sends a second confirmation message to the slave computing services deployed on the pluggable computing node, thus completing the registration process. The node capability messages include command type, session ID, requesting node role, message body length (in bytes), checksum of the entire message, model index, model name, model type, model ID, priority, lifecycle management method, popularity metric, context window size, input data volume, output data volume, and model functionality. However, this is not the only information provided.

[0026] Understandably, in related technologies, the computing nodes on a vehicle are fixed and cannot be dynamically added, thus preventing the supplementation of the vehicle's overall computing power. This disclosure, through a customized plug-and-play protocol, incorporates pluggable computing nodes into the vehicle's on-board model deployment framework via a network (such as TCP or IP), enabling dynamic expansion of the vehicle's computing power and more flexibly responding to the needs of different application scenarios regarding the access of computing nodes.

[0027] Specifically, computing power service is a service that runs on computing power nodes, is exposed through standardized interfaces, and provides computing power to applications.

[0028] Optional, such as Figure 2 As shown, the computing power service includes: a session management module, which manages the entire lifecycle of the session to which the model uses messages, as well as the storage and querying of session data; The model management module is used to manage the entire lifecycle of models and recover from anomalies. The memory management module is used to manage the allocation of memory resources during the execution of inference tasks by the model. Optionally, it is also used to manage the sharing mechanism of the model during the execution of inference tasks. The scheduling and management module is used to manage and control the computing resources of the computing power nodes on which it is deployed. In the case that its computing power service is the main computing power service, it is also used to schedule the use of computing power resources of the computing power nodes on which the secondary computing power service is deployed. The protocol management module is used to communicate with other computing power services based on a preset network, and also to communicate with applications. NN adapters are used to load and adapt models to the hardware computing platform of computing nodes.

[0029] Specifically, the session management module is responsible for the full lifecycle management of sessions initiated by applications using model messages (such as the full lifecycle management of sessions in scenarios with concurrent access from multiple applications), supporting session creation, retrieval, binding, and destruction. Message isolation and context association are achieved through a unique session ID. It also provides a session state persistence mechanism, supporting the storage and retrieval of historical session data to meet functional safety and diagnostic traceability requirements.

[0030] Specifically, the model management module implements full lifecycle management of models, covering model loading and inference management, and also including model deployment, uninstallation, and version management. It supports unified access to multiple model formats such as ONNX, TensorRT, and TFLite, and adapts to hardware computing platforms through the NN adapter. It features model anomaly detection and fault-tolerant recovery mechanisms, supporting automatic reloading or switching to a backup model in scenarios such as inference failure, hardware anomalies, or memory overflows, ensuring service continuity and system robustness.

[0031] Specifically, the memory management module oversees the allocation and sharing of memory resources during the model inference process. This sharing mechanism enables zero-copy data transfer across systems and nodes.

[0032] Specifically, the scheduling management module is responsible for controlling the use of computing resources. Each computing service's scheduling management module can decide when to execute inference tasks based on factors such as the computing resource usage of the computing nodes on which the service is deployed, inference task priorities, and / or model priorities. It supports various scheduling strategies, including immediate execution, delayed execution, pause / resume, and degraded execution. Sub-computing services can forward model usage messages to the master computing service to initiate computing resource scheduling requests. The master computing service can then make scheduling decisions based on global computing resources, achieving optimal utilization and load balancing of computing resources.

[0033] Specifically, the protocol management module defines and implements a customized communication protocol stack. It enables communication between various computing power services based on a pre-defined network (such as a TCP / IP network), and also enables communication between computing power services and applications. The protocol system covers node registration protocols, model usage message protocols, and / or computing power scheduling protocols. For example, the protocol system includes: a node registration protocol (automatically registering computing power when a computing power node starts up); a model usage message protocol (the application initiates model usage messages); and a computing power scheduling protocol (negotiating resource scheduling between master and slave computing power services).

[0034] Specifically, the computing node integrates a hardware computing platform. The NN adapter, through hardware abstraction and interface standardization, shields the underlying differences of the hardware computing platform (such as NPU, GPU, and / or DSP) and provides a unified and portable neural network inference interface for upper-layer applications, thereby loading and adapting the model to the hardware computing platform of the computing node.

[0035] For example, Figure 3 This is a logical diagram illustrating the working process of an NN adapter provided in an embodiment of this disclosure. Figure 3As shown, the NN adapter may include QNN Adaptor, TensorRT Adaptor, and / or Other Vendor Adaptor (i.e., any other vendor adapter known to those skilled in the art). The parts of the neural network framework (NN Framework) corresponding to the QNN Adaptor include: PyTorch, TF-lite, ONNX, QCOM Neural Processing SDK, and QCOM AI Engine Direct API (including NEON, HMX, HVX, and OpenCL). PyTorch and TF-lite are model training and lightweight frameworks, ONNX is a unified intermediate representation format for models, QCOM Neural Processing SDK is a model deployment toolchain, and Qualcomm AI Engine Direct API is a hardware-level acceleration interface. The hardware (HW) corresponding to the QNN Adaptor includes CPU, HTP, cDSP, and GPU. Accordingly, the model deployment process is as follows: the model is trained in the cloud using PyTorch or TF-lite and exported to ONNX; the QCOM Neural Processing SDK converts and optimizes the model; and finally, the model is deployed on the computing nodes. The model usage process is as follows: The QNN Adaptor loads and initializes the model, and calls the QCOM AI Engine Direct API to map different operators of the model to the corresponding hardware computing platforms for inference. The components of the neural network framework corresponding to the TensorRT Adaptor include: PyTorch, TensorFlow, NVIDIA TensorRT SDK, and NVIDIA CUDA. PyTorch and TensorFlow are the model training frameworks, the NVIDIA TensorRT SDK is the model deployment toolchain, and NVIDIA CUDA is the parallel computing interface. The hardware corresponding to the TensorRT Adaptor includes GPUs and DLAs. Accordingly, the model deployment process is as follows: The model is trained in the cloud using PyTorch or TensorFlow and exported as ONNX or TorchScript. The NVIDIA TensorRT SDK transforms and optimizes the model, and finally, the model is deployed on the computing nodes. The model usage process is as follows: The QNN Adaptor loads and initializes the model, and calls NVIDIA CUDA to map different operators of the model to the corresponding hardware computing platforms for inference.The parts of the neural network framework corresponding to OtherVendor Adaptor include: Vendor NPU SDK and Vendor GPU SDK, which are toolchains corresponding to OtherVendor Adaptor and correspond to any other matching hardware computing power platform known to those skilled in the art (i.e., other NN HW).

[0036] It is understood that the computing power service provided in this disclosure abstracts and encapsulates the underlying hardware computing power platform, shields the differences between different hardware architectures, provides a unified and standardized interface for calling computing power resources to upper-layer applications, decouples application logic from the hardware computing power platform, and ensures that applications do not need to be aware of the type of underlying computing power resources when calling the model, thus providing technical support for realizing cross-system calling of the model.

[0037] Specifically, in some examples, at least one model is deployed on each operating system; in other examples, at least one model is deployed on some operating systems while no model is deployed on others. This disclosure does not limit this.

[0038] In some examples, the model usage messages received by the main computing power service include: a first type of model usage message, which refers to model usage messages sent by applications deployed on the computing power nodes on which the computing power service is located; and a second type of model usage message, which refers to model usage messages sent by applications not on computing power nodes. The model usage messages received from the computing power service include the first type of model usage messages. That is, the second type of model usage requests initiated by applications not on computing power nodes are sent directly to the main computing power service.

[0039] Specifically, non-computing nodes refer to ECUs that do not have the ability to perform inference tasks. They are usually mainly responsible for signal acquisition, simple control, communication relay, or executing fixed logic.

[0040] Specifically, for the main computing power service, it can receive model usage messages (i.e., the first type of model usage messages) sent by applications deployed on the same computing power node as it, and it can also receive model usage messages (i.e., the second type of model usage messages) sent by applications on non-computing power nodes; for the slave computing power service, it can receive model usage messages sent by applications deployed on the same computing power node as it.

[0041] It is understandable that the main computing power service can schedule the use of computing power resources on the computing power nodes where the secondary computing power service is deployed. Therefore, by setting the main computing power service to receive the second type of model usage messages, it is possible to quickly determine the computing power nodes that respond to the second type of model usage messages, which helps to improve the response speed to model usage messages.

[0042] In other examples, each computing power service can receive model usage messages from applications deployed on the same computing power node as it, and can also receive model usage messages from applications on non-computing power nodes.

[0043] In this embodiment of the disclosure, when the application sends a model usage message, the third target computing power service receiving the model usage message checks whether a target model is deployed on the third target computing power node it is deployed on; if no target model is deployed on the third target computing power node, the third computing power service forwards the model usage message to the first target computing power service; the first target computing power service checks whether the first target computing power node it is deployed on meets the preset computing power conditions; if the first target computing power node does not meet the preset computing power conditions, the main computing power service selects a second target computing power node from multiple computing power nodes and selects a second target computing power service from the computing power services deployed on the second target computing power node; the second target computing power service loads the target model and uses the target model to execute the inference task corresponding to the model usage message.

[0044] Specifically, the third-target computing power service refers to the computing power service that receives model usage messages from the application. This third-target computing power service receives model usage messages through the protocol management module.

[0045] Specifically, the third target computing node refers to the computing node on which the third target computing service is deployed.

[0046] Specifically, the first target computing power service refers to the computing power service that receives model usage messages from the third target computing power service. The first target computing power service receives these messages through the protocol management module.

[0047] Specifically, the first target computing node refers to the computing node on which the first target computing service is deployed.

[0048] Specifically, the second target computing power service refers to the computing power service that receives model usage messages from the first target computing power service. The second target computing power service receives these messages through the protocol management module.

[0049] Specifically, the second target computing node refers to the computing node on which the second target computing service is deployed.

[0050] Specifically, there are various ways to set up the model to use messages. The following are typical examples, but they do not constitute a limitation of this disclosure.

[0051] In some examples, the model uses a message that includes a third message header and a third message body; the third message header includes the command type, session ID, role of the requesting node, byte length of the message body data, and checksum of the entire message; the third message body includes the model input data that is input to the target model.

[0052] Specifically, the command type indicates the intent of this communication. For example, the command type can be represented by the command name and / or command ID.

[0053] For example, the definitions of various types of commands are shown in Table 1 below, but are not limited thereto. According to Table 1, the command type in the model usage message and the new model usage message can be represented by AI_MODEL_EXTERNA_CMD_REQUEST_REMOTE and / or 0x0007.

[0054] Table 1

[0055] Specifically, the session ID is used to identify a complete interaction process. It is empty for the first request and is returned after the request node is generated. Subsequent requests carry this ID to indicate that they belong to the same interaction process.

[0056] Specifically, the role of a request node represents its function within the vehicle-side model deployment framework. A request node refers to the computing power node that receives messages. For example, the role of a request node can be represented by a role name and / or a role ID.

[0057] For example, the various roles of the request node are defined as shown in Table 2 below, but are not limited thereto.

[0058] Table 2

[0059] Specifically, the byte length of the message body data refers to the byte length of the message body portion.

[0060] Specifically, the checksum of the entire message is used to verify whether data corruption has occurred during transmission. For example, the checksum may include a CRC checksum, but is not limited to this.

[0061] Understandably, by setting the model to use messages including a third message header and a third message body, message parsing becomes easier. The third message header is read first to determine the purpose, which computing node should process it, and the amount of data, before processing the message body. Furthermore, by setting the roles of requesting nodes, access control, resource isolation, and priority scheduling can be implemented based on their roles. Setting session IDs distinguishes different interaction processes, avoids data confusion, and ensures each session is processed independently. Setting the byte length of the message body data helps to clearly define boundaries and prevent packet fragmentation. Finally, setting checksums ensures reliable data transmission and prevents accidental operations.

[0062] Specifically, there are several ways to set the preset computing power conditions. The following is an explanation of a typical example.

[0063] In some embodiments, the preset computing power condition includes the load level corresponding to the used computing power resources of the first target computing power node being greater than a preset load level threshold. Correspondingly, the first target computing power service receiving the model's usage message checks whether the first target computing power node it is deployed on meets the preset computing power condition, including: The first target computing power service obtains the used computing power resources of the first target computing power node and determines the load level corresponding to the used computing power resources of the first target computing power node; If the load level corresponding to the used computing resources of the first target computing power node is greater than the preset load level threshold, the first target computing power service determines that the first target computing power node does not meet the preset computing power conditions.

[0064] Specifically, the computing power service can obtain the used computing power resources of the computing power nodes it is deployed on. Optionally, the computing power service can also obtain the total computing power resources, etc., without limitation.

[0065] Among them, the computing resources used refer to the amount of computing resources currently occupied, which is usually characterized by at least one of the following: how many trillion floating-point operations are actually performed per second, how many trillion operations are actually performed per second, CPU utilization, memory consumption, GPU computing cycle, etc.

[0066] Total computing power refers to the maximum computing capacity that computing power resources can provide per unit of time. It is usually measured by specific indicators, typically represented by at least one of the following: trillion floating-point operations per second (TFLOPS), trillion operations per second (TOPS), etc.

[0067] Specifically, there is a positive correlation between the computing resources used and the load level. There are several ways to determine the "load level corresponding to the computing resources used".

[0068] For example, a first mapping relationship between used computing resources and load levels can be predefined. In this way, based on the used computing resources of a computing node, the load level corresponding to the used computing resources of the computing node can be determined by querying the first mapping relationship.

[0069] For example, the resource utilization rate is obtained by dividing the used computing resources by the total computing resources. A second mapping relationship between resource utilization rate and load level is predefined. Thus, based on the resource utilization rate, the load level corresponding to the used computing resources of a computing node can be determined by querying this second mapping relationship. The second mapping relationship is shown in Table 3 below: Table 3

[0070] In some examples, if the load level corresponding to the used computing resources of the first target computing power node is less than or equal to the preset load level threshold, the first target computing power service determines that the first target computing power node meets the preset computing power conditions; if the load level corresponding to the used computing resources of the first target computing power node is greater than the preset load level threshold, the first target computing power service determines that the first target computing power node does not meet the preset computing power conditions.

[0071] In other examples, if the load level corresponding to the used computing resources of the first target computing node is less than or equal to a preset load level threshold, the first target computing service determines the computing resources required by the model and calculates the sum of the computing resources required by the model and the used computing resources of the first target computing node to obtain new used computing resources. Here, the computing resources required by the model refer to the computing power required to load the target model and perform inference tasks. The first target computing service determines the load level corresponding to the new used computing resources of the first target computing node. If the load level corresponding to the new used computing resources of the first target computing node is greater than the preset load level threshold, the first target computing service determines that the first target computing node does not meet the preset computing power condition. If the load level corresponding to the new used computing resources of the first target computing node is less than or equal to the preset load level threshold, the first target computing service determines that the first target computing node meets the preset computing power condition. If the load level corresponding to the used computing resources of the first target computing node is greater than the preset load level threshold, the first target computing service determines that the first target computing node does not meet the preset computing power condition.

[0072] Specifically, the required computing resources refer to the amount of computational resources needed for the target model to complete the inference task at a specified precision (e.g., FP32, FP16, INT8) and target performance (e.g., latency ≤100ms). Those skilled in the art can calculate the required computing resources in any known way, and this disclosure does not limit this calculation.

[0073] It should be noted that the specific method for determining the "load level corresponding to the new used computing resources" is similar to that for the "load level corresponding to the used computing resources", and will not be repeated here.

[0074] Understandably, by setting a "load level corresponding to the used computing power resources" to determine whether the computing power of a computing node meets the preset conditions (i.e., whether it is sufficient), the continuous used computing power resources can be discretized into predefined load levels, reducing the complexity of determining whether "the computing power of a computing node is sufficient." Furthermore, when determining whether "the computing power of a computing node is sufficient," evaluating it in conjunction with the computing power requirements of the target model ensures sufficient computing power resources and avoids inference task failures or excessive latency due to insufficient computing power.

[0075] In other embodiments, the preset computing power condition includes the used computing power resources of the first target computing power node being less than or equal to a preset computing power threshold. Accordingly, the first target computing power service receiving the model usage message detects whether the first target computing power node it is deployed on meets the preset computing power condition, including: If the used computing power resources of the first target computing power node are greater than the preset computing power threshold, the first target computing power service determines that the first target computing power node does not meet the preset computing power conditions; if the used computing power resources of the first target computing power node are less than or equal to the preset computing power threshold, the first target computing power service determines that the first target computing power node meets the preset computing power conditions.

[0076] Specifically, when the primary computing power service is the first target computing power service, it selects a second target computing power node from the computing power nodes deployed on the vehicle-side model deployment framework, and then selects a second target computing power service from the computing power services deployed on the second target computing power node. The primary target computing power service then forwards the model usage message to the second target computing power service. For example, it updates the role of the requesting node in the model usage message from the role of the first target computing power node to the role of the second target computing power node to update the model usage message, and then sends the updated model usage message to the second target computing power service. Similarly, when the third target computing power service forwards the model usage message to the first target computing power service, it also needs to update the role of the requesting node in the model usage message from the role of the third target computing power node to the role of the first target computing power node.

[0077] Specifically, when the first target computing power service is a slave computing power service, the first target computing power service first forwards the model usage message to the master computing power service. For example, the first target computing power service updates the role of the request node in the model usage message from the role of the first target computing power node to the role of the computing power node on which the master computing power service is deployed, thereby updating the model usage message, and then sends the updated model usage message to the master computing power service. The master computing power service then selects the second target computing power node from the computing power nodes deployed on the vehicle-side model deployment framework, and selects the second target computing power service from the computing power services deployed on the second target computing power node. If the second target computing power service is a slave computing power service, the master computing power service also needs to forward the model usage message to the second target computing power service. For example, the master computing power service updates the role of the request node in the model usage message from the role of the computing power node on which the master computing power service is deployed to the role of the second target computing power node, thereby updating the model usage message again, and then sends the updated model usage message to the second target computing power service. Similarly, when the third target computing power service forwards the model usage message to the first target computing power service, it also needs to update the role of the requesting node in the model usage message from the role of the third target computing power node to the role of the first target computing power node.

[0078] Specifically, there are various methods for selecting the first target computing power node. The following are typical examples, but they do not constitute a limitation of this disclosure.

[0079] In some embodiments, the specific steps of the third target computing power service in selecting the first target computing power node from multiple computing power nodes include: selecting the first target computing power node from multiple computing power nodes based on the model deployment information of each operating system deployed on each computing power node.

[0080] Optionally, the method further includes: each slave computing service reporting a model deployment message to the master computing service, wherein the model deployment message includes model deployment information; and the master computing service sending the aggregated model deployment information of each operating system deployed on each computing node to each slave computing service.

[0081] Specifically, for each slave computing service, it can encapsulate the model deployment information of its deployed operating system into a model deployment message through the protocol management module, and then report the model deployment message to the master computing service. The reporting method is either periodic reporting or reporting when the models deployed on the computing nodes change (e.g., adding, deleting, or disabling models). In this way, the master computing service can obtain the model deployment information of each operating system deployed on each computing node, and then distribute the aggregated model deployment information of each operating system deployed on each computing node to each slave computing service. Thus, the third target computing service can select the first target computing node based on the model deployment information of each operating system deployed on each computing node.

[0082] Specifically, there are various methods for selecting the second target computing power node. The following are typical examples, but they do not constitute a limitation of this disclosure.

[0083] In some embodiments, the main computing power service selects a second target computing power node from multiple computing power nodes, including: selecting a second target computing power node from multiple computing power nodes based on the computing power resource information of each computing power node and the model deployment information of each operating system deployed on each computing power node.

[0084] Optionally, the method further includes: each slave computing service periodically sending computing resource messages and model deployment messages to the master computing service, wherein the computing resource message includes computing resource information, which includes the used computing resources and total computing resources of the computing node, and the model deployment message includes model deployment information; Alternatively, the computing power service can send a computing power resource message to the main computing power service when the change in the used computing power resources of the computing power nodes on which it is deployed exceeds a preset change threshold, and send a model deployment message to the main computing power service when the model on the computing power nodes on which it is deployed changes.

[0085] Specifically, for each slave computing service, it can encapsulate the computing resource information of the computing nodes it deploys on into a computing resource message through the protocol management module, and then report the computing resource message to the master computing service. The reporting method is either scheduled reporting or reporting when the change in the used computing resources of the computing nodes it deploys on exceeds a preset change threshold. Furthermore, for each slave computing service, it can encapsulate the model deployment information of the operating system it deploys on into a model deployment message through the protocol management module, and then report the model deployment message to the master computing service. The reporting method is either scheduled reporting or reporting when the models deployed on the computing nodes change (e.g., adding, deleting, or disabling models). In this way, the master computing service can obtain the computing resource information of each computing node and the model deployment information of each operating system deployed on each computing node, and then select the second target computing node. For example, based on the model deployment information of each operating system deployed on each computing node, computing nodes with the target model deployed are selected from the computing nodes deployed in the vehicle-side model deployment framework as the first-stage computing nodes; based on the computing resource information of each computing node, computing nodes that meet the preset computing conditions are selected from the initial-stage computing nodes as the second-stage computing nodes; if the number of second-stage computing nodes is one, the second-stage computing node is determined as the second target computing node; if the number of second-stage computing nodes is multiple, one can be randomly selected from the second-stage computing nodes or the one with the largest remaining computing resources can be selected as the second target computing node, but it is not limited to this.

[0086] Specifically, there are various ways to set up computing resource messages and model deployment messages. The following are typical examples, but they do not constitute a limitation of this disclosure.

[0087] For example, a computing resource message includes a first message header and a first message body. The first message header includes the command type, the role of the requesting node, the byte length of the message body data, and the checksum of the entire message. The first message body includes the computing resources used and the total computing resources.

[0088] For example, the computing resource message is as follows: First message header: uses JSON format { "cmd": AI_MODEL_EXTERNA_CMD_RESOURCE_REPORT, / / Command type "role":MODEL_NODE_ROLE_SLAVER, / / The role requested for the node "length": 82, / / Byte length of the message body data "checksum": FBBA / / Checksum of the entire message } First message body: { "nntype": NN_HW_NVIDIA, / / Hardware manufacturer of the computing node "CPU": "loading=A1;total=B1", / / Used and total CPU computing resources "GPU": "loading=A2;total=B2", This represents the GPU's used computing resources and total computing resources. "NPU": "loading=A3;total=B3", This represents the used and total computing resources of the NPU. "DLA": "loading=A4;total=B4", DLA's used computing resources and total computing resources } For example, a model deployment message includes a second message header and a second message body. The second message header includes the command type, the role of the requesting node, the byte length of the message body data, and the checksum of the entire message. The second message body includes model deployment information, which may include a list of models supported by the computing power node. The model list includes model type, model ID and / or model name, etc. For example, the model ID is shown in Table 4 below, and the model type is shown in Table 5 below, but is not limited to these.

[0089] Table 4

[0090] Table 5

[0091] For example, the model deployment message is as follows: Second message header: uses JSON format { "cmd": AI_MODEL_EXTERNA_CMD_RESOURCE_REPORT, / / Command type "role":MODEL_NODE_ROLE_SLAVER, / / The role requested for the node "length":492, / / Length of message body data in bytes "checksum": 3FFE / / The checksum of the entire message } Second message body: { "modelvobs": [ / / Model list] { "id": MODEL_LLM, / / The model ID of the first model "type": "MODEL_TYPE_SINGLE" / / The model type of the first model }, { "id": MODEL_ASR, / / The model ID of the second model "type": "MODEL_TYPE_SINGLE" / / The model type of the second model }, ], } Understandably, considering the computing resource information of each computing node when selecting the second target computing node allows for the filtering out of nodes that are fully loaded or have insufficient computing resources, while selecting those with surplus or idle resources as the second target computing nodes, thereby improving the utilization rate of the vehicle's computing resources. Furthermore, considering the model deployment information of each operating system on each computing node during the selection process ensures that the second target computing node meets the operational requirements of the target model, avoiding inference task failures due to model deployment mismatches.

[0092] It's also understandable that by including message headers and message bodies in computing resource messages and model deployment messages, message parsing becomes easier. Reading the message header first reveals the purpose, which computing node it should be delivered to, and the amount of data, before processing the message body. Furthermore, setting the byte length of the message body helps to clearly define boundaries and prevent packet fragmentation. Setting checksums ensures reliable data transmission and prevents accidental operations. Standardized result feedback is provided through status codes.

[0093] Specifically, the second target computing power service loads the target model and inputs the model input data carried in the model usage message into the target model, so that the target model can perform inference tasks based on the model input data to obtain inference result data.

[0094] In some embodiments, the second target computing power service can generate inference feedback messages based on the inference result data.

[0095] For example, the inference feedback message includes a fourth message header and a fourth message body. The fourth message header includes the command type, session ID, byte length of the message body data, checksum of the entire message, and status code. The command type is the model inference result feedback. The fourth message body includes the inference result data. The status code indicates the processing result status of this request. It is usually a predefined value representing a specific semantic, such as "success", "failure", "timeout", "insufficient permissions", etc., but is not limited to these.

[0096] In one example, the second target computing power service can directly send inference feedback messages to the application via the protocol management module, enabling the application to receive the inference feedback messages and parse the inference result data. In another example, the second target computing power service can send inference feedback messages to the first target computing power service via the protocol management module. The first target computing power service then forwards the inference feedback messages to the third target computing power service, which in turn forwards the inference feedback messages to the application, enabling the application to receive the inference feedback messages and parse the inference result data.

[0097] In other embodiments, the second target computing power service can store inference result data in a shared data space so that the application can read the inference result data from the shared data space.

[0098] In this embodiment, a second target computing node with a deployed target model and meeting preset computing power conditions can respond to model usage messages. This not only improves the success rate of model usage message responses but also enables unified scheduling of the vehicle's computing resources to achieve vehicle-wide computing power sharing. This allows high-computing-power nodes to take over inference tasks from other computing-power nodes during idle periods, thereby improving the overall utilization rate of the vehicle's computing resources. Furthermore, the third target computing node prioritizes finding a first target computing node that might be able to respond to model usage messages. Only when the first target computing node lacks sufficient computing power does it escalate the model usage message to the primary computing node. The primary computing node then coordinates with the second target computing node, which possesses the target model and sufficient computing power. This avoids frequent interruptions to the primary computing node, allowing it to focus more on other complex tasks.

[0099] In another embodiment of this disclosure, the method further includes: if the first target computing power node meets the preset computing power conditions, the first target computing power service loads the target model and uses the target model to execute the inference task corresponding to the model usage message, wherein the first target computing power service and the target model are deployed on the same operating system.

[0100] Optionally, the method further includes: if the first target computing power node meets the preset computing power conditions and the target model and the first target computing power service are deployed on different operating systems, the first target computing power service forwards the model usage message to the fourth target computing power service; the fourth target computing power service loads the target model and uses the target model to execute the inference task corresponding to the model usage message, wherein the fourth target computing power service is deployed on the first target computing power node and the target model and the fourth target computing power service are deployed on the same operating system.

[0101] The cross-node computing power scheduling method provided in this disclosure embodiment will be described in detail below with reference to a specific example. Figure 4 This is a flowchart illustrating a cross-node computing power scheduling example provided in this disclosure. Figure 4 As shown, when the APP sends a model usage message, the third target computing power service (the computing power service that directly receives the model usage message) checks whether the target model (i.e., the model indicated by the model usage message) is deployed on the third target computing power node (the computing power node on which the third target computing power service is deployed). If not deployed, the third target computing power service forwards the model usage message to the first target computing power service (deployed on the first target computing power node that has the target model). The first target computing power service checks whether the first target computing power node meets the preset computing power conditions. Specifically, when the resource utilization rate of a computing power node is at level 3 or level 4, the computing power resources are scarce and the preset computing power conditions are not met. When the resource utilization rate of a computing power node is at level 1 or level 2, it is necessary to check whether the "resource utilization rate after adding the computing power resources required by the model" will reach level 3 or level 4. If it reaches level 3 or level 4, the preset computing power conditions are not met; if it does not reach level 3 or level 4, the preset computing power conditions are met. If the first target computing power node meets the preset computing power conditions, the first target computing power service uses the target model to perform the inference task. If the first target computing power node does not meet the preset computing power conditions, the first target computing power service forwards the model usage message to the main computing power service. The main computing power service then selects a second target computing power node from the computing power services deployed on the vehicle-side model deployment framework, and further selects a second target computing power service from the computing power services deployed on the second target computing power node. The main computing power service then forwards the model usage message to the second target computing power service, enabling the second target computing power service to perform inference tasks using the target model.

[0102] Figure 5 This is a schematic diagram of a cross-node computing power scheduling device provided in an embodiment of this disclosure. This cross-node computing power scheduling device can be understood as the aforementioned vehicle-side model deployment framework or a functional module within the aforementioned vehicle-side model deployment framework. For example... Figure 5 As shown, the cross-node computing power scheduling device includes: The first detection module 510 is used to detect whether the first target computing power service that receives the model usage message meets the preset computing power conditions on the first target computing power node it is deployed on. The first filtering module 520 is configured to, if the first target computing power node does not meet the preset computing power conditions, the main computing power service filters out a second target computing power node from the plurality of computing power nodes and filters out a second target computing power service from the computing power services deployed on the second target computing power node, wherein the second target computing power node meets the preset computing power conditions, the second target computing power service and the target model are deployed on the same operating system, and the target model refers to the model indicated by the model using a message. The first inference module 530 is used by the second target computing power service to load the target model and use the target model to execute the inference task corresponding to the model usage message.

[0103] Optionally, the first detection module 510 includes a first determination submodule, used by the first target computing power service to obtain the used computing power of the first target computing power node and determine the load level corresponding to the used computing power of the first target computing power node; The second determining submodule is used to determine that the first target computing power node does not meet the preset computing power condition if the load level corresponding to the used computing power of the first target computing power node is greater than the preset load level threshold.

[0104] Optionally, the first detection module 510 further includes a third determination submodule, configured to determine the computing power required by the model if the load level corresponding to the used computing power of the first target computing power node is less than or equal to the preset load level threshold, and calculate the sum of the computing power required by the model and the used computing power of the first target computing power node to obtain a new used computing power, wherein the computing power required by the model refers to the computing power required to load the target model and the target model to perform the inference task; The first target computing power service determines the load level corresponding to the newly used computing power of the first target computing power node; If the load level corresponding to the newly used computing power of the first target computing power node is greater than the preset load level threshold, the first target computing power service determines that the first target computing power node does not meet the preset computing power condition; If the load level corresponding to the newly used computing power of the first target computing power node is less than or equal to the preset load level threshold, the first target computing power service determines that the first target computing power node meets the preset computing power condition.

[0105] Optionally, the first filtering module 520 includes a first filtering submodule, used by the main computing power service to filter out a second target computing power node from the plurality of computing power nodes. Specifically, the first filtering submodule is used to filter out the second target computing power node from the plurality of computing power nodes based on the computing power resource information of each computing power node and the model deployment information of each operating system deployed on each computing power node.

[0106] Optionally, the device further includes: a reporting module, used to periodically send computing resource messages and model deployment messages from each of the slave computing services to the master computing service, wherein the computing resource message includes computing resource information, the computing resource information includes the used computing resources and total computing resources of the computing node, and the model deployment message includes model deployment information; Alternatively, the slave computing service may send the computing resource message to the master computing service when the change in the used computing resources of the computing node on which it is deployed exceeds a preset change threshold, and send the model deployment message to the master computing service when the model on the computing node on which it is deployed changes.

[0107] Optionally, the computing power resource message includes a first message header and a first message body. The first message header includes the command type, the role of the requesting node, the byte length of the message body data, and the verification value of the entire message. The first message body includes the used computing power resources and the total computing power resources. The model deployment message includes a second message header and a second message body. The first message header includes the command type, the role of the requesting node, the byte length of the message body data, and the checksum of the entire message. The first message body includes the model deployment information.

[0108] Optionally, the device further includes: a second inference module, configured to, if the first target computing power node meets the preset computing power conditions, load the target model and use the target model to execute the inference task corresponding to the model usage message, wherein the first target computing power service and the target model are deployed on the same operating system.

[0109] The apparatus provided in this embodiment can execute the methods of any of the above embodiments, and its execution method and beneficial effects are similar, so they will not be described again here.

[0110] This disclosure also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0111] This disclosure also provides a vehicle including the apparatus described in any of the above embodiments and / or the computer-readable storage medium described in any of the above embodiments, which will not be described again here.

[0112] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-node computing power scheduling method, characterized in that, An application is provided to a vehicle-side model deployment framework, the framework comprising multiple computing power nodes, each computing power node having at least one operating system deployed on it, each operating system having a computing power service deployed on it, and at least one model deployed on each operating system. Of all the computing power services deployed on the multiple computing power nodes, one is designated as a master computing power service, and the remaining services are designated as slave computing power services. The method includes: The receiving model uses the first target computing power service to detect whether the first target computing power node it is deployed on meets the preset computing power conditions; If the first target computing power node does not meet the preset computing power conditions, the main computing power service selects a second target computing power node from the plurality of computing power nodes and selects a second target computing power service from the computing power services deployed on the second target computing power node. The model usage message is received by the third target computing power service from the application and forwarded to the first target computing power service because the target model is not deployed on the third target computing power node on which the third target computing power service is deployed. The target model refers to the model indicated by the model usage message. The second target computing power node meets the preset computing power conditions, and the second target computing power service and the target model are deployed on the same operating system. The second target computing power service loads the target model and uses the target model to execute the inference task corresponding to the model usage message.

2. The method according to claim 1, characterized in that, The first target computing power service that receives the model usage message detects whether the first target computing power node on which it is deployed meets the preset computing power conditions, including: The first target computing power service obtains the used computing power resources of the first target computing power node and determines the load level corresponding to the used computing power resources of the first target computing power node; If the load level corresponding to the used computing resources of the first target computing power node is greater than the preset load level threshold, the first target computing power service determines that the first target computing power node does not meet the preset computing power conditions.

3. The method according to claim 2, characterized in that, Also includes: If the load level corresponding to the used computing power resources of the first target computing power node is less than or equal to the preset load level threshold, the first target computing power service determines the computing power resources required by the model, and calculates the sum of the computing power resources required by the model and the used computing power resources of the first target computing power node to obtain new used computing power resources. The computing power resources required by the model refer to the computing power required to load the target model and the target model to perform the inference task. The first target computing power service determines the load level corresponding to the newly used computing power resources of the first target computing power node; If the load level corresponding to the newly used computing resources of the first target computing power node is greater than the preset load level threshold, the first target computing power service determines that the first target computing power node does not meet the preset computing power condition; If the load level corresponding to the newly used computing resources of the first target computing power node is less than or equal to the preset load level threshold, the first target computing power service determines that the first target computing power node meets the preset computing power condition.

4. The method according to claim 1, characterized in that, The main computing power service selects a second target computing power node from the plurality of computing power nodes, including: Based on the computing power resource information of each computing power node and the model deployment information of each operating system deployed on each computing power node, the second target computing power node is selected from the plurality of computing power nodes.

5. The method according to claim 1, characterized in that, Also includes: Each of the sub-computing services periodically sends computing resource messages and model deployment messages to the main computing service. The computing resource message includes computing resource information, which includes the used computing resources and total computing resources of the computing node. The model deployment message includes model deployment information. Alternatively, the slave computing service may send the computing resource message to the master computing service when the change in the used computing resources of the computing node on which it is deployed exceeds a preset change threshold, and send the model deployment message to the master computing service when the model on the computing node on which it is deployed changes.

6. The method according to claim 5, characterized in that, The computing power resource message includes a first message header and a first message body. The first message header includes the command type, the role of the requesting node, the byte length of the message body data, and the verification value of the entire message. The first message body includes the used computing power resources and the total computing power resources. The model deployment message includes a second message header and a second message body. The first message header includes the command type, the role of the requesting node, the byte length of the message body data, and the checksum of the entire message. The first message body includes the model deployment information.

7. The method according to claim 1, characterized in that, Also includes: If the first target computing power node meets the preset computing power conditions, the first target computing power service loads the target model and uses the target model to execute the inference task corresponding to the model usage message, wherein the first target computing power service and the target model are deployed on the same operating system.

8. A cross-node computing power scheduling device, characterized in that, An application is provided for a vehicle-side model deployment framework, the framework comprising multiple computing power nodes, each computing power node having at least one operating system deployed on it, each operating system having a computing power service deployed on it, and at least one model deployed on each operating system. Of all the computing power services deployed on the multiple computing power nodes, one is a master computing power service, and the remaining computing power services are slave computing power services. The device includes: The first detection module is used to detect whether the first target computing power service that receives the model usage message meets the preset computing power conditions on the first target computing power node it is deployed on. The first filtering module is configured to, if the first target computing power node does not meet the preset computing power conditions, the main computing power service filters out a second target computing power node from the plurality of computing power nodes and filters out a second target computing power service from the computing power services deployed on the second target computing power node, wherein the model usage message is received by the third target computing power service from the application and forwarded to the first target computing power service because the target model is not deployed on the third target computing power node on which the third target computing power service is deployed, the second target computing power node meets the preset computing power conditions, and the second target computing power service and the target model are deployed on the same operating system; The first inference module is used by the second target computing power service to load the target model and use the target model to execute the inference task corresponding to the model usage message.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

10. A vehicle, characterized in that, include: The cross-node computing power scheduling device of claim 8 or the computer-readable storage medium of claim 9.

Images