Distributed reasoning method and device based on hybrid expert architecture, equipment and medium

Abstract

The application relates to the field of artificial intelligence and provides a distributed reasoning method, device and equipment based on a hybrid expert architecture and a medium, wherein the distributed reasoning method based on the hybrid expert architecture applied to an edge computing gateway comprises the following steps: if a reasoning task is received, determining a plurality of target intelligent vehicles participating in the reasoning task; acquiring an expert reasoning model and deploying the expert reasoning model into each target intelligent vehicle; wherein the expert reasoning model comprises a plurality of expert reasoning modules, and at least one expert reasoning module is deployed in each target intelligent vehicle; sending the reasoning task to each target intelligent vehicle, so that each target intelligent vehicle executes the reasoning task based on the expert reasoning module to obtain a reasoning result; and aggregating the reasoning results of the target intelligent vehicles to obtain a target reasoning result. Through the technical scheme provided by the application, the computing power resources of idle intelligent vehicles are fully utilized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a distributed reasoning method, apparatus, device, and medium based on a hybrid expert architecture. Background Technology

[0002] With the development of artificial intelligence technology, the number of parameters in large-scale language models and other AI models is increasing, making it difficult for a single device to efficiently complete inference tasks. Expert hybrid systems, as an effective solution, significantly reduce computational resource consumption and improve inference efficiency by splitting large models into multiple expert models, with the router deciding which expert models to activate for inference.

[0003] In traditional hybrid expert architectures, expert models are typically deployed across multiple servers within the same data center, relying on high-speed network interconnection. However, in edge computing scenarios, fully utilizing distributed, heterogeneous computing resources to achieve efficient inference systems remains a significant challenge.

[0004] In the wave of vehicle electrification and intelligentization, intelligent electric vehicles are generally equipped with high-performance artificial intelligence computing units for environmental perception, planning, and decision-making in autonomous driving, with computing power reaching hundreds of tera-flops (T). While these vehicles are parked and charging, their computing resources are idle, resulting in a waste of computing power. Summary of the Invention

[0005] This invention provides a distributed reasoning method, apparatus, device, and medium based on a hybrid expert architecture to address the limitations of computing resources in existing intelligent vehicles and to fully utilize the computing resources of idle intelligent vehicles.

[0006] This invention provides a distributed inference method based on a hybrid expert architecture, applied to an edge computing gateway. The method includes:

[0007] If a reasoning task is received, identify the multiple target intelligent vehicles participating in the reasoning task;

[0008] An expert reasoning model is obtained and deployed in each of the target intelligent vehicles; wherein the expert reasoning model includes multiple expert reasoning modules, and at least one of the expert reasoning modules is deployed in each of the target intelligent vehicles;

[0009] The reasoning task is sent to each of the target intelligent vehicles, so that each of the target intelligent vehicles executes the reasoning task based on the expert reasoning module and obtains the reasoning result.

[0010] The reasoning results of each target intelligent vehicle are aggregated to obtain the target reasoning result.

[0011] According to a distributed reasoning method based on a hybrid expert architecture provided by the present invention, the step of deploying the expert reasoning model to each of the target intelligent vehicles includes:

[0012] Obtain the first computing power information of each of the target intelligent vehicles;

[0013] The computing power score of each target intelligent vehicle is calculated based on the first computing power information;

[0014] The expert reasoning module corresponding to each of the target intelligent vehicles is determined based on the computing power score;

[0015] The expert reasoning module corresponding to each of the target intelligent vehicles is deployed in the computing power sandbox of each of the target intelligent vehicles.

[0016] According to a distributed reasoning method based on a hybrid expert architecture provided by the present invention, determining multiple target intelligent vehicles participating in the reasoning task includes:

[0017] Obtain secondary computing power information and estimated charging time for multiple currently connected smart vehicles;

[0018] The reasoning score of each intelligent vehicle is calculated based on the second computing power information and the estimated charging time.

[0019] The multiple target intelligent vehicles participating in the reasoning task are determined based on the reasoning score.

[0020] According to a distributed reasoning method based on a hybrid expert architecture provided by the present invention, after sending the reasoning task to each of the target intelligent vehicles, the method further includes:

[0021] Identify the migrating intelligent vehicles for each of the target intelligent vehicles;

[0022] If the target intelligent vehicle is detected to be offline, the expert reasoning module and reasoning data in the offline target intelligent vehicle are migrated to the migrated intelligent vehicle, so that the migrated intelligent vehicle can perform the reasoning task based on the expert reasoning module and obtain the reasoning result.

[0023] This invention also provides a distributed inference method based on a hybrid expert architecture, applied to a computing power management center, the method comprising:

[0024] If a reasoning task is received from a task requester, obtain the location information of the task requester.

[0025] The edge computing gateway corresponding to the inference task is determined based on the location information;

[0026] The inference task is sent to the edge computing gateway corresponding to the inference task, so that the edge computing gateway executes the inference task.

[0027] According to a distributed inference method based on a hybrid expert architecture provided by the present invention, the step of determining the edge computing gateway corresponding to the inference task based on the location information includes:

[0028] Candidate edge computing gateways are determined based on the location information;

[0029] Obtain the computing power resources of the candidate edge computing power gateways, and calculate the computing power resource score of the candidate edge computing power gateways based on the computing power resources;

[0030] The edge computing gateway corresponding to the inference task is determined based on the computing resource score.

[0031] According to a distributed inference method based on a hybrid expert architecture provided by the present invention, after sending the inference task to the edge computing gateway corresponding to the inference task so that the edge computing gateway executes the inference task, the method further includes:

[0032] Record the computing power contribution of each target intelligent vehicle in performing the inference task;

[0033] The incentive information for each target intelligent vehicle is calculated based on the computing power contribution.

[0034] This invention also provides a distributed reasoning method based on a hybrid expert architecture, applied to intelligent vehicles, the method comprising:

[0035] If an expert inference module and inference task are received from the edge computing power gateway, the expert inference module is deployed in the computing power sandbox.

[0036] The reasoning task is executed based on the expert reasoning module to obtain the reasoning result, and the reasoning result is sent to the edge computing gateway.

[0037] This invention also provides a distributed inference device based on a hybrid expert architecture, applied to an edge computing gateway, comprising:

[0038] The first determining module is configured to, upon receiving a reasoning task, determine multiple target intelligent vehicles participating in the reasoning task;

[0039] The first deployment module is configured to acquire an expert reasoning model and deploy the expert reasoning model to each of the target intelligent vehicles; wherein the expert reasoning model includes multiple expert reasoning modules, and at least one of the expert reasoning modules is deployed in each of the target intelligent vehicles;

[0040] The first sending module is configured to send the inference task to each of the target intelligent vehicles, so that each of the target intelligent vehicles executes the inference task based on the expert inference module and obtains the inference result.

[0041] The aggregation module is configured to aggregate the reasoning results of each of the target intelligent vehicles to obtain the target reasoning result.

[0042] This invention also provides a distributed inference device based on a hybrid expert architecture, applied in a computing power management center, comprising:

[0043] The acquisition module is configured to acquire the location information of the task requester if it receives a reasoning task published by the task requester.

[0044] The second determining module is configured to determine the edge computing gateway corresponding to the inference task based on the location information;

[0045] The second sending module is configured to send the inference task to the edge computing gateway corresponding to the inference task, so that the edge computing gateway executes the inference task.

[0046] The present invention also provides a distributed reasoning device based on a hybrid expert architecture, applicable to intelligent vehicles, comprising:

[0047] The second deployment module is configured to deploy the expert inference module in the computing power sandbox if it receives the expert inference module and inference task sent by the edge computing power gateway.

[0048] The execution module is configured to execute the reasoning task based on the expert reasoning module, obtain the reasoning result, and send the reasoning result to the edge computing gateway.

[0049] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the distributed reasoning method based on a hybrid expert architecture as described above.

[0050] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed reasoning method based on a hybrid expert architecture as described above.

[0051] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the distributed reasoning method based on a hybrid expert architecture as described above.

[0052] This invention provides a distributed inference method, apparatus, device, and medium based on a hybrid expert architecture. Upon receiving an inference task, it identifies multiple target intelligent vehicles participating in the inference task, acquires multiple expert inference modules from the expert inference model, deploys them to each target intelligent vehicle, sends the inference task to each target intelligent vehicle, and each target intelligent vehicle executes the inference task based on the expert inference modules to obtain the inference result. The inference results from all target intelligent vehicles are then aggregated to obtain the target inference result. The edge computing gateway breaks down inference computation into three stages: "routing → expert parallelism → aggregation," proposing a hybrid expert distributed inference architecture suitable for edge environments. This fully utilizes the computing resources of idle intelligent vehicles to achieve efficient inference for large-scale AI models. By distributing the expert inference model through the hybrid expert architecture, each intelligent vehicle only needs to store a portion of the parameters, significantly reducing the computational and storage burden on a single device. Compared to centralized inference, edge distributed inference reduces data transmission distance and bandwidth requirements, improves response speed, and reduces overall energy consumption. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the distributed inference method based on a hybrid expert architecture for edge computing gateways provided by the present invention.

[0055] Figure 2 This is a schematic diagram of a scenario for the distributed reasoning method based on a hybrid expert architecture provided by the present invention.

[0056] Figure 3 This is a flowchart illustrating the distributed inference method based on a hybrid expert architecture for use in computing power management centers, provided by this invention.

[0057] Figure 4 This is a flowchart illustrating the distributed reasoning method based on a hybrid expert architecture for intelligent vehicles provided by the present invention.

[0058] Figure 5 This is a schematic diagram of the structure of the distributed inference device based on a hybrid expert architecture for use in edge computing gateways provided by the present invention.

[0059] Figure 6 This is a schematic diagram of the structure of the distributed inference device based on a hybrid expert architecture, which is applied to a computing power management center, provided by the present invention.

[0060] Figure 7 This is a schematic diagram of the structure of a distributed reasoning device based on a hybrid expert architecture for use in intelligent vehicles, provided by the present invention.

[0061] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0063] Figure 1 This is a flowchart illustrating a distributed reasoning method based on a hybrid expert architecture, according to an exemplary embodiment. Figure 1 As shown in an exemplary embodiment, the distributed reasoning method based on a hybrid expert architecture is applied to an edge computing gateway. The method includes steps 110 to 140, which are described in detail below.

[0064] Step 110: If a reasoning task is received, determine the multiple target intelligent vehicles participating in the reasoning task.

[0065] In this embodiment of the invention, the edge computing gateway is the central coordinator of the distributed inference system, which stores all model parameters of the expert inference model. For example... Figure 2 As shown, the intelligent vehicle connects to the edge computing gateway via charging piles 1-4. If it receives a reasoning task from a reasoning service requester (task requester), it identifies multiple target intelligent vehicles participating in the reasoning task, such as... Figure 2 The intelligent vehicles 1-4 in the model require 2 public experts (A, B) and 2 domain experts (X, Y) to complete the reasoning task. The full set of model parameters of the expert reasoning model is divided into 4 expert reasoning modules (model weights W1-4).

[0066] Step 120: Obtain the expert reasoning model and deploy the expert reasoning model to each of the target intelligent vehicles; wherein the expert reasoning model includes multiple expert reasoning modules, and at least one of the expert reasoning modules is deployed in each of the target intelligent vehicles.

[0067] In this embodiment of the invention, an expert reasoning model is obtained and divided into multiple expert reasoning modules. The multiple expert reasoning modules are then deployed to the computing power sandboxes in each target intelligent vehicle.

[0068] Step 130: The reasoning task is sent to each of the target intelligent vehicles, so that each of the target intelligent vehicles executes the reasoning task based on the expert reasoning module and obtains the reasoning result.

[0069] In this embodiment of the invention, inference tasks are sent to each target intelligent vehicle. Each target intelligent vehicle executes the inference task based on the deployed expert inference module and obtains the inference result. The target intelligent vehicles execute the inference task in a pipeline manner; that is, after obtaining the inference result, the first target intelligent vehicle feeds it back to the edge computing gateway. The edge computing gateway then sends the inference result to the second target intelligent vehicle, enabling the second target intelligent vehicle to execute the inference task based on the inference result and obtain the corresponding inference result. This process continues until the last target intelligent vehicle has completed its inference task.

[0070] Step 140: Aggregate the reasoning results of each target intelligent vehicle to obtain the target reasoning result.

[0071] In this embodiment of the invention, the reasoning results of each target intelligent vehicle are aggregated, and the final target reasoning result is obtained through a fusion algorithm.

[0072] In this embodiment of the invention, the edge computing gateway breaks down a single inference process into three stages: "routing → expert parallelism → aggregation". It broadcasts scheduling frames via PLC (Power Line Communication) to specify the input offset, output buffer address, and time slice for each target intelligent vehicle.

[0073] This invention proposes an expert-integrated distributed inference architecture suitable for edge environments, which fully utilizes the computing resources of idle intelligent vehicles to achieve efficient inference for large-scale AI models. By distributing model parameters through the expert-integrated architecture, each intelligent vehicle only needs to store a portion of the parameters, significantly reducing the computational and storage burden on a single device. Compared to centralized inference, edge distributed inference reduces data transmission distance and bandwidth requirements, improves response speed, and reduces overall energy consumption.

[0074] In an exemplary embodiment of the present invention, deploying the expert reasoning model to each of the target intelligent vehicles includes:

[0075] Obtain the first computing power information of each of the target intelligent vehicles;

[0076] The computing power score of each target intelligent vehicle is calculated based on the first computing power information;

[0077] The expert reasoning module corresponding to each of the target intelligent vehicles is determined based on the computing power score;

[0078] The expert reasoning module corresponding to each of the target intelligent vehicles is deployed in the computing power sandbox of each of the target intelligent vehicles.

[0079] In this embodiment of the invention, the edge computing power gateway stores all the model parameters of the expert inference model and maintains a real-time "vehicle-computing power" table. The "vehicle-computing power" table records the first computing power information of the smart vehicles currently registered in the edge computing power gateway in real time, such as the peak FLOPS (Floating Point Operations Per Second) of the smart vehicle's NPU (Neural Network Processing Unit) / GPU (Graphics Processing Unit), remaining storage, communication latency, and estimated remaining charging time. The computing power score of each target smart vehicle is calculated based on the first computing power information.

[0080] The edge computing gateway quantifies the initial computing power information into a computing power score S. compute The specific calculation formula is as follows:

[0081] S compute = w1·log10(FLOPS peak ) + w2·(MEM free / MEM total ) + w3·(1 / RTT avg ) +w4·(1 / T charge )

[0082] Among them, FLOPS peak This represents the peak floating-point computing power (in Tera-FLOPS) of the target intelligent vehicle's NPU / GPU; MEM free / MEM total RTT represents the ratio of the target intelligent vehicle's remaining video memory to its total video memory. avg T represents the average round-trip latency (in milliseconds) of the N most recent communications between the target intelligent vehicle and the edge computing gateway, such as the average round-trip latency of the last 10 communications; charge This indicates the estimated remaining charging time (in hours) when the target intelligent vehicle's current State of Charge (SOC) is ≥80%; w1~w4 represent preset weights, for example, w1=0.4, w2=0.3, w3=0.2, w4=0.1, satisfying Σw i = 1, i = 1-4.

[0083] The model parameters of the expert inference model can be divided into multiple expert inference modules. Specifically, the expert inference model can be divided according to the number of target intelligent vehicles. For example, if there are 8 target intelligent vehicles, a 70-byte expert inference model can be divided into 8 expert inference modules, with each module containing 8.75 bytes of parameters. Alternatively, it can be divided according to a fixed parameter granularity. For example, a 70-byte expert inference model can be sliced ​​into 70 pieces at a granularity of 128MB. The expert inference modules corresponding to each target intelligent vehicle are then allocated based on its computing power score. For instance, a heuristic allocation algorithm (such as greedy or K-means clustering) can be used to allocate the expert inference modules in descending order of computing power score. For example, multiple adjacent expert inference modules can be deployed in target intelligent vehicles with high computing power to reduce communication, while only one expert inference module can be deployed in target intelligent vehicles with low computing power.

[0084] Taking a language model with a 24-layer Transformer architecture as an example, each 4 layers can be considered as an expert inference module, forming 6 independent expert inference modules. Each expert inference module contains specific weight parameters for the self-attention sublayer and the feedforward sublayer. During deployment, the edge computing gateway distributes these expert inference modules to target intelligent vehicles that meet the storage and computing power requirements, ensuring a reasonable workload for each vehicle, and reserving some expert processing modules for redundancy backup.

[0085] In this embodiment of the invention, the target intelligent vehicle is re-evaluated at preset time intervals, and the expert inference modules deployed in the target intelligent vehicle are dynamically adjusted based on the re-evaluated computing power score. If the computing power score of a target intelligent vehicle is less than a computing power threshold, migration is triggered.

[0086] In an exemplary embodiment of the present invention, determining the plurality of target intelligent vehicles participating in the reasoning task includes:

[0087] Obtain secondary computing power information and estimated charging time for multiple currently connected smart vehicles;

[0088] The reasoning score of each intelligent vehicle is calculated based on the second computing power information and the estimated charging time.

[0089] The multiple target intelligent vehicles participating in the reasoning task are determined based on the reasoning score.

[0090] In this embodiment of the invention, the edge computing node has the function of an expert inference router, that is, it determines which expert inference modules should handle the inference task. For example, based on parameters such as load balancing coefficient, latency penalty, and exploration rate, it controls which inference task is routed to a specific expert inference module on a specific intelligent vehicle.

[0091] On the edge computing gateway, the inference task (token) outputs the activation probability of each expert through the router, thereby determining the target intelligent vehicle. That is, the edge computing gateway combines the real-time load of the connected intelligent vehicle, the PLC link communication latency, the peak FLOPS of the intelligent vehicle's NPU / GPU, the remaining storage and other second computing power information, and performs weighted summation to calculate the inference score of each intelligent vehicle. Based on the inference score, the top-K intelligent vehicles are retained as target intelligent vehicles. At the same time, the total latency of the target intelligent vehicles should be less than the latency threshold.

[0092] The edge computing gateway quantifies the second computing power information into an inference score S. infer Used to filter target intelligent vehicles participating in reasoning tasks:

[0093] S infer = w5·log10(FLOPS peak ) + w6·(MEM free / MEM total ) + w7·(1 / RTT avg ) +w8·(T remain / T max )

[0094] Among them, T remain Indicates the estimated remaining charging time (in hours); T max This represents the maximum allowed time (in hours) to complete this reasoning task; w5~w8 represent preset weights, such as w5=0.35, w6=0.25, w7=0.2, w8=0.2, satisfying Σw i =1, i=5~8.

[0095] In each decision cycle, the edge computing gateway calculates an inference score S for all currently online intelligent vehicles. infer After sorting the inference scores in descending order, select the Top-K intelligent vehicles as the target intelligent vehicles to ensure the total latency Σ(1 / RTT). avg )< Delay threshold L max .

[0096] In an exemplary embodiment of the present invention, after sending the inference task to each of the target intelligent vehicles, the method further includes:

[0097] Identify the migrating intelligent vehicles for each of the target intelligent vehicles;

[0098] If the target intelligent vehicle is detected to be offline, the expert reasoning module and reasoning data in the offline target intelligent vehicle are migrated to the migrated intelligent vehicle, so that the migrated intelligent vehicle can perform the reasoning task based on the expert reasoning module and obtain the reasoning result.

[0099] In this embodiment of the invention, to cope with dynamic changes in vehicles, a redundant backup and dynamic migration mechanism for the expert inference module can be implemented. Specifically, each target intelligent vehicle has at least one migrating intelligent vehicle, and the computing power of the migrating intelligent vehicle is similar to that of the corresponding target intelligent vehicle. The expert inference module is redundantly stored in the migrating intelligent vehicle. Before the target intelligent vehicle leaves the network, it actively pushes model parameters to the migrating intelligent vehicle. If the target intelligent vehicle suddenly loses connection, the migrating intelligent vehicle performs the inference task.

[0100] In this embodiment of the invention, a dynamic expert allocation mechanism is implemented through redundancy backup and dynamic migration mechanisms, which can adapt to vehicle access and departure scenarios and ensure system stability.

[0101] In an exemplary embodiment of the present invention, during the execution of an inference task, inference orchestration can be performed, such as splitting a 512-token inference task into 4 micro-batches, with each batch containing 8 expert inference modules running in parallel; the data is sent to the target intelligent vehicle via PLC communication time slots of 0-3ms, the target intelligent vehicle performs inference calculations from 3-6ms, and the inference results are returned from 6-7ms; the edge computing gateway initiates All-Reduce aggregation at 7ms and outputs the target inference results at 7.5ms. The inference task is executed in a pipelined manner, with communication and computation overlapping simultaneously; for example, within the aforementioned inference execution time, the second micro-batch can be transmitted starting at 3s.

[0102] Figure 3 This is a flowchart illustrating a distributed reasoning method based on a hybrid expert architecture, according to an exemplary embodiment. Figure 3 As shown in an exemplary embodiment, this distributed reasoning method based on a hybrid expert architecture is applied to a computing power management center. The method includes steps 310 to 330, which are described in detail below.

[0103] Step 310: If a reasoning task is received from a task requester, obtain the location information of the task requester.

[0104] In this embodiment of the invention, the computing power management center is responsible for managing multiple edge computing power gateways, realizing global computing power resource scheduling, providing inference computing power interfaces to inference service providers, and selecting appropriate edge computing power gateways based on business needs and vehicle distribution.

[0105] After receiving the reasoning task issued by the task requester, determine the location information of the task requester.

[0106] Step 320: Determine the edge computing gateway corresponding to the inference task based on the location information.

[0107] In this embodiment of the invention, an edge computing gateway suitable for inference tasks is determined based on location information.

[0108] Step 330: Send the inference task to the edge computing gateway corresponding to the inference task, so that the edge computing gateway executes the inference task.

[0109] In this embodiment of the invention, the inference task is sent to the edge computing gateway corresponding to the inference task, and the edge computing gateway executes the inference task. The specific scheme for the edge computing gateway to execute the inference task has been described and will not be repeated here.

[0110] In an exemplary embodiment of the present invention, determining the edge computing gateway corresponding to the inference task based on the location information includes:

[0111] Candidate edge computing gateways are determined based on the location information;

[0112] Obtain the computing power resources of the candidate edge computing power gateways, and calculate the computing power resource score of the candidate edge computing power gateways based on the computing power resources;

[0113] The edge computing gateway corresponding to the inference task is determined based on the computing resource score.

[0114] In this embodiment of the invention, after receiving an inference request, the computing power management center first filters candidate edge computing power gateways (e.g., those less than 1 km away) based on location information and user geofencing. Then, it multiplies the number of available smart vehicles in each candidate edge computing power gateway by the average FLOPS / average RTT (Round-Trip Time) of each smart vehicle to obtain a computing power resource score. The candidate edge computing power gateway with the highest computing power resource score is selected to execute the inference task. If at least two candidate edge computing power gateways have the same highest computing power resource score, the edge computing power gateway with a high historical inference success rate and / or low electricity price is prioritized and returned to the inference service requester.

[0115] In an exemplary embodiment of the present invention, after sending the inference task to the edge computing gateway corresponding to the inference task so that the edge computing gateway executes the inference task, the method further includes:

[0116] Record the computing power contribution of each target intelligent vehicle in performing the inference task;

[0117] The incentive information for each target intelligent vehicle is calculated based on the computing power contribution.

[0118] In this embodiment of the invention, the contribution of each target intelligent vehicle to reasoning is recorded, and then the incentive information of the target intelligent vehicle is calculated. The incentive mechanism is implemented through computing power points or charging discounts.

[0119] This invention provides a contribution-based incentive mechanism that creates a win-win situation for computing power providers, computing power users, and system operators.

[0120] Figure 4 This is a flowchart illustrating a distributed reasoning method based on a hybrid expert architecture, according to an exemplary embodiment. Figure 4 As shown in an exemplary embodiment, this distributed reasoning method based on a hybrid expert architecture is applied to an intelligent vehicle. The method includes steps 410 to 420, which are described in detail below.

[0121] Step 410: If the expert inference module and inference task are received from the edge computing power gateway, deploy the expert inference module in the computing power sandbox.

[0122] In this embodiment of the invention, the intelligent vehicle is equipped with a high-performance NPU / GPU computing unit and provides a computing sandbox with high isolation and high security. The charging pile is connected to the intelligent vehicle via a charging cable, and a high-speed data transmission channel is provided through a PLC, enabling high-speed network connections between intelligent vehicles and with edge computing gateways. This invention achieves high-speed interconnection between inference nodes through power line communication technology, eliminating the need for additional network infrastructure.

[0123] Once a smart vehicle connects to a charging station, it establishes a connection with the edge computing gateway via a PLC, registers available computing resources, and initializes the computing sandbox environment. When a smart vehicle is identified as the target vehicle for performing an inference task, it receives the expert inference module and inference task sent by the edge computing gateway and deploys the expert inference module in the computing sandbox. Each smart vehicle stores only a portion of the model's parameters, acting as one or more expert nodes in a hybrid expert architecture.

[0124] Step 420: Execute the reasoning task based on the expert reasoning module to obtain the reasoning result, and send the reasoning result to the edge computing gateway.

[0125] In this embodiment of the invention, the reasoning task is executed based on the deployed expert reasoning module to obtain the reasoning result, and the reasoning result is sent to the edge computing power gateway, where the edge computing power node aggregates the results to obtain the final target reasoning result.

[0126] In an exemplary embodiment of the present invention, the inference service provider, upon receiving a user-requested inference task, determines whether to use edge inference to provide the service. Specifically, the inference service provider maintains an edge / cloud cost model. When the number of available smart vehicles at the edge computing gateway is greater than or equal to a vehicle threshold, and the estimated RTT is less than a latency threshold, the edge inference system is selected for execution. If there are insufficient available smart vehicles or the RTT is too high, the process reverts to the cloud. Simultaneously, the inference service provider supports users specifying either a "low latency priority" or a "low cost priority" mode to execute the inference task.

[0127] In this embodiment of the invention, the distributed inference method based on a hybrid expert architecture includes five stages: initialization and computing power registration stage, expert inference model allocation stage, inference service request and routing stage, expert selection and inference execution stage, and contribution recording and resource release stage. The relevant descriptions of each stage are as follows:

[0128] During the initialization and computing power registration phase, the intelligent vehicles of domain experts and public experts connect to charging piles via charging interfaces to obtain power. The charging piles establish a high-speed communication channel through PLC technology. The intelligent vehicles report information such as NPU / GPU computing power type, content storage, and computing sandbox status, and register computing power resources with the edge computing power gateway via the PLC. At the same time, the intelligent vehicles of domain experts also report their specialized computing power characteristics. The edge computing power gateway reports its available distributed computing power resources to the computing power management center, and the computing power management center confirms the resource registration with the edge computing power gateway.

[0129] During the expert inference model allocation phase, the computing power management center distributes the completed expert inference model parameters to the edge computing power gateway, including the expert partitioning strategy, routing algorithm parameters, and inter-expert communication protocol. The edge computing power gateway evaluates the computing power characteristics and stability of the intelligent vehicle, i.e., it performs a tiered evaluation based on the expected charging time and computing power information to determine the intelligent vehicle's inference score. Based on the expert partitioning strategy, the edge computing power gateway distributes the expert model parameters to the intelligent vehicle and initializes the intelligent vehicle's computing power sandbox environment. Finally, the edge computing power gateway reports the completion of the expert inference module deployment to the computing power management center.

[0130] During the inference service request and routing phase, the inference service requester initiates an inference request to the inference service provider, uploading the inference task type and input data. The inference service provider requests edge inference resources from the computing power management center, while simultaneously conveying task requirements and service quality requirements. The computing power management center assesses the global computing power distribution based on network latency, computing power load, etc., determines the edge computing power nodes to execute the inference task, and returns edge computing power gateway credentials to the inference service provider. The inference service provider establishes a connection with the edge computing power gateway based on the edge computing power gateway credentials and forwards the inference request.

[0131] During the expert selection and inference execution phases, the edge computing gateway executes an expert routing algorithm, which analyzes input features to determine which expert inference models to activate and their collaboration methods. When an inference task requires processing by public experts, input data (inference task) is distributed to these experts, and the inference task runs within a computing sandbox. When an inference task requires processing by domain experts, input data (inference task) is distributed to these experts, who then perform optimized calculations tailored to their specific domain. The edge computing gateway integrates the results from each expert (inference result) using a fusion algorithm, and the final inference result (target inference result) is sequentially transmitted between the edge computing gateway, the computing power management center, the inference service provider, and the inference service requester.

[0132] During the contribution recording and resource release phase, the computing power contribution of each intelligent vehicle is determined based on the computational load, computation time, and result quality assessment of the inference tasks performed by each intelligent vehicle. The computing power management center then updates this contribution based on incentive mechanisms such as generating charging coupons or computing power points rewards. The edge computing power gateway monitors the vehicle's connectivity status and triggers expert parameter migration and reallocation when the vehicle leaves.

[0133] In this embodiment of the invention, common inference tasks include text-to-text, text-to-video, and video-to-text inference tasks. The following describes the specific distributed inference process:

[0134] The scenario is set as follows: An underground parking garage in a commercial complex has 80 charging stations. Between 10:00 PM and 6:00 AM, approximately 60 smart vehicles are charging simultaneously. These smart vehicles are equipped with NPUs and computing sandboxes, capable of performing inference tasks. The computing power management center receives an inference task from the mall's customer service system: to provide customers with a multi-turn dialogue and question-and-answer service that includes "store recommendations + real-time coupon generation."

[0135] Decompose the reasoning task:

[0136] This inference task employs a 7 billion parameter Transformer MoE dialogue model, pre-divided into 4 common experts (A: intent recognition; B: store knowledge; C: discount strategy; D: language generation) and 2 domain experts (X: women's clothing; Y: digital products). A complete dialogue requires going through 4 stages: "intent → knowledge → strategy → generation," with 2 experts activating in parallel at each stage.

[0137] Reasoning process:

[0138] During the t0 (0 ms) phase, the computing power management center selects the nearest edge computing power gateway GW-01 with the highest computing power resource score based on the GPS location of the customer's mobile APP.

[0139] In phase t1 (1 ms), the edge computing gateway GW-01 selects 6 intelligent vehicles (V1~V6) from 60 intelligent vehicles as target intelligent vehicles by calculating the inference score formula. infer The values ​​are 9.2, 8.9, 8.7, 8.5, 8.3, and 8.0, respectively.

[0140] During phase t2 (2 ms), the edge computing gateway GW-01 distributes 4 common expert modules (each with 1.75 B parameters) + 2 domain expert modules (each with 1.75 B parameters) according to computing power scores:

[0141] V1 deploys A+B; V2 deploys C+D; V3 deploys X; V4 deploys Y; V5 and V6 serve as redundant backups.

[0142] In the t3 (3 ms) phase, the customer inputs: "I want to buy a birthday gift for xx, with a budget of xxx yuan."

[0143] In the t4 (4 ms) phase, GW-01 broadcasts the token sequence to V1 (intent recognition expert A).

[0144] In the t5 (7 ms) phase, V1 returns the intent vector I, and the edge computing gateway GW-01 sends I along with the original token to V3 (X, the women's clothing expert).

[0145] In the t6 (10 ms) phase, V3 returns the store candidate list L1, and the edge computing gateway GW-01 then sends I+L1 to V2 (discount strategy expert C).

[0146] In stage t7 (13 ms), V2 generates coupon vector C1, and the edge computing gateway GW-01 sends I+L1+C1 to V2's language generation expert D.

[0147] In the t8 (16 ms) phase, the V2 language generation expert D outputs the final reply: "Recommend a new scarf from ×× women's clothing store for her. Original price xxx yuan, now yyy yuan after coupon, limited time offer for 2 hours."

[0148] During the t9 (17 ms) phase, the edge computing gateway GW-01 performs weighted aggregation on the intermediate vectors returned by the six experts and sends the results back to the customer's mobile APP.

[0149] During phase t10 (18 ms), the customer continues the conversation, and the system repeats t4-t9, forming multiple rounds of interaction.

[0150] After the inference task is completed, the edge computing gateway GW-01 calculates the contribution based on the actual number of tokens calculated for each smart vehicle multiplied by a weighting coefficient:

[0151] Contribution i = Token i × (S infer_i / ΣS infer );

[0152] The contribution points will be converted into charging coupons, which will be distributed to the car owner's APP by the computing power management center the next day.

[0153] The following describes the distributed inference device based on a hybrid expert architecture provided by the present invention. The distributed inference device based on a hybrid expert architecture described below can be referred to in correspondence with the distributed inference method based on a hybrid expert architecture described above. It should be noted that the device provided in the following embodiments belongs to the same concept as the method provided in the above embodiments, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0154] In one exemplary embodiment of the present invention, please refer to Figure 5 , Figure 5 This is an exemplary embodiment illustrating a distributed inference device based on a hybrid expert architecture, applied to an edge computing gateway, comprising the following modules.

[0155] The first determining module 510 is configured to determine multiple target intelligent vehicles participating in the reasoning task if a reasoning task is received.

[0156] The first deployment module 520 is configured to acquire an expert reasoning model and deploy the expert reasoning model to each of the target intelligent vehicles; wherein the expert reasoning model includes multiple expert reasoning modules, and at least one of the expert reasoning modules is deployed in each of the target intelligent vehicles;

[0157] The first sending module 530 is configured to send the reasoning task to each of the target intelligent vehicles, so that each of the target intelligent vehicles executes the reasoning task based on the expert reasoning module and obtains the reasoning result.

[0158] The aggregation module 540 is configured to aggregate the reasoning results of each of the target intelligent vehicles to obtain the target reasoning result.

[0159] In one exemplary embodiment of the present invention, the first deployment module 520 includes:

[0160] The first acquisition submodule is configured to acquire the first computing power information of each of the target intelligent vehicles;

[0161] The first calculation submodule is configured to calculate the computing power score of each of the target intelligent vehicles based on the first computing power information;

[0162] The first determining submodule is configured to determine the expert reasoning module corresponding to each of the target intelligent vehicles based on the computing power score;

[0163] The deployment submodule is configured to deploy the expert inference module corresponding to each of the target intelligent vehicles in the computing power sandbox of each of the target intelligent vehicles.

[0164] In an exemplary embodiment of the present invention, the first determining module 510 includes:

[0165] The second acquisition submodule is configured to acquire the second computing power information and estimated charging time of multiple currently connected smart vehicles;

[0166] The second calculation submodule is configured to calculate the inference score of each of the intelligent vehicles based on the second computing power information and the estimated charging time;

[0167] The second determining submodule is configured to determine multiple target intelligent vehicles participating in the reasoning task based on the reasoning score.

[0168] In an exemplary embodiment of the present invention, the distributed reasoning device based on a hybrid expert architecture further includes:

[0169] The third determining module is configured to determine the migrating intelligent vehicle for each of the target intelligent vehicles;

[0170] The migration module is configured to, if the target intelligent vehicle is detected to be offline, migrate the expert inference module and inference data from the offline target intelligent vehicle to the migrating intelligent vehicle, so that the migrating intelligent vehicle can perform the inference task based on the expert inference module and obtain the inference result.

[0171] In one exemplary embodiment of the present invention, please refer to Figure 6 , Figure 6 This is an exemplary embodiment illustrating a distributed inference device based on a hybrid expert architecture, applied in a computing power management center, comprising:

[0172] The acquisition module 610 is configured to acquire the location information of the task requester if it receives a reasoning task published by the task requester.

[0173] The second determining module 620 is configured to determine the edge computing power gateway corresponding to the inference task based on the location information;

[0174] The second sending module 630 is configured to send the inference task to the edge computing gateway corresponding to the inference task, so that the edge computing gateway executes the inference task.

[0175] In an exemplary embodiment of the present invention, the second determining module 620 includes:

[0176] The third determining submodule is configured to determine candidate edge computing power gateways based on the location information;

[0177] The third calculation submodule is configured to obtain the computing power resources of the candidate edge computing power gateway and calculate the computing power resource score of the candidate edge computing power gateway based on the computing power resources.

[0178] The fourth determination submodule is configured to determine the edge computing gateway corresponding to the inference task based on the computing resource score.

[0179] In an exemplary embodiment of the present invention, the distributed reasoning device based on a hybrid expert architecture further includes:

[0180] The recording module is configured to record the computing power contribution of each target intelligent vehicle in performing the inference task;

[0181] The calculation module is configured to calculate the incentive information for each of the target intelligent vehicles based on the computing power contribution.

[0182] In one exemplary embodiment of the present invention, please refer to Figure 7 , Figure 7 This is an exemplary embodiment illustrating a distributed reasoning device based on a hybrid expert architecture, applied to an intelligent vehicle, comprising:

[0183] The second deployment module 710 is configured to deploy the expert inference module in the computing power sandbox if it receives the expert inference module and inference task sent by the edge computing power gateway.

[0184] The execution module 720 is configured to execute the reasoning task based on the expert reasoning module, obtain the reasoning result, and send the reasoning result to the edge computing gateway.

[0185] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a distributed inference method based on a hybrid expert architecture.

[0186] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the distributed reasoning method based on a hybrid expert architecture provided by the above methods.

[0188] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the distributed reasoning method based on a hybrid expert architecture provided by the above methods.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for distributed reasoning based on hybrid expert architecture, characterized in that, The method applied to an edge computing gateway comprises: If an inference task is received, a plurality of target intelligent vehicles participating in the inference task are determined; An expert inference model is obtained, and the expert inference model is deployed in each target intelligent vehicle; wherein the expert inference model comprises a plurality of expert inference modules, and at least one expert inference module is deployed in each target intelligent vehicle; The inference task is sent to each target intelligent vehicle, so that each target intelligent vehicle executes the inference task based on the expert inference module to obtain an inference result; The inference results of each target intelligent vehicle are aggregated to obtain a target inference result; The expert inference model is deployed in each target intelligent vehicle, comprising: First computing power information of each target intelligent vehicle is obtained; wherein the first computing power information comprises peak floating point operation capability of NPU / GPU, remaining video memory, communication delay, and predicted remaining charging duration; According to the first computing power information, the computing power score of each target intelligent vehicle is calculated; wherein the computing power score S compute The calculation formula is as follows: S compute = w1·log10(FLOPS peak ) + w2·(MEM free / MEM total ) + w3·(1 / RTT avg ) +w4·(1 / T charge ); where FLOPS peak MEM represents the peak floating-point computing power of the target intelligent vehicle's NPU / GPU. free / MEM total RTT represents the ratio of the remaining video memory to the total video memory of the target intelligent vehicle. avg T represents the average round-trip latency of the N most recent communications between the target intelligent vehicle and the edge computing gateway. charge This represents the estimated remaining charging time when the target intelligent vehicle's current SOC is ≥ 80%; w1~w4 represent preset weights, satisfying Σw i = 1, i = 1-4; The expert inference module corresponding to each target intelligent vehicle is determined based on the computing power score; The expert inference module corresponding to each target intelligent vehicle is deployed in the computing power sandbox of each target intelligent vehicle; The plurality of target intelligent vehicles participating in the inference task are determined, comprising: Second computing power information and predicted charging duration of a plurality of intelligent vehicles currently connected are obtained; wherein the second computing power information comprises peak floating point operation capability of NPU / GPU, remaining video memory, communication delay, and predicted remaining charging duration; Calculate an inference score of each of the intelligent vehicles based on the second computing power information and the predicted charging duration; wherein the inference score S of each of the intelligent vehicles is calculated according to the following formula: infer S = (1 - (1 - P) / 1) * (1 - (1 - Q) / 1) S infer = w5·log10(FLOPS peak ) + w6·(MEM free / MEM total ) + w7·(1 / RTT avg ) + w8·(T remain / T max ); where T remain Indicates the estimated remaining charging time, T max This represents the maximum allowed time to complete this reasoning task, and w5~w8 represent preset weights, satisfying Σw i = 1, i = 5~8; The plurality of target intelligent vehicles participating in the inference task are determined according to the inference score.

2. The method of claim 1, wherein the hybrid expert architecture based distributed reasoning method is characterized by, After the inference task is sent to each target intelligent vehicle, the method further comprises: A migration intelligent vehicle of each target intelligent vehicle is determined; If it is detected that the target intelligent vehicle is offline, the expert inference module and inference data in the offline target intelligent vehicle are migrated to the migration intelligent vehicle, so that the migration intelligent vehicle executes the inference task based on the expert inference module to obtain an inference result.

3. A method for distributed reasoning based on hybrid expert architecture, characterized in that, The method applied to a computing power management center comprises: If an inference task published by a task requester is received, position information of the task requester is obtained; An edge computing gateway corresponding to the inference task is determined based on the position information; The inference task is sent to the edge computing gateway corresponding to the inference task, so that the edge computing gateway executes the inference task; wherein the edge computing gateway corresponding to the inference task executes the method of any one of claims 1 to 2; The edge computing gateway corresponding to the inference task is determined based on the position information, comprising: Candidate edge computing gateways are determined based on the position information; Computing power resources of the candidate edge computing gateways are obtained, and a computing power resource score of each candidate edge computing gateway is calculated based on the computing power resources; wherein the computing power resource score is obtained by multiplying the number of available intelligent vehicles in each candidate edge computing gateway by the average FLOPS / average round trip delay of each intelligent vehicle; The edge computing gateway corresponding to the inference task is determined based on the computing power resource score.

4. The method of claim 3, wherein the hybrid expert architecture based distributed reasoning method is characterized by, The method further comprises, after sending the inference task to the edge computing power gateway corresponding to the inference task for the edge computing power gateway to execute the inference task: Recording the computing power contribution of each target intelligent vehicle in executing the inference task; Calculating the incentive information of each target intelligent vehicle according to the computing power contribution.

5. A method for distributed reasoning based on a hybrid expert architecture, characterized in that, The method applied to an intelligent vehicle comprises: If the expert inference module and the inference task sent by the edge computing power gateway of any one of claims 1-2 are received, deploying the expert inference module in a computing power sandbox; Based on the expert inference module executing the inference task, obtaining an inference result, and sending the inference result to the edge computing power gateway.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the distributed inference method based on the hybrid expert architecture of any one of claims 1-5.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the distributed inference method based on the hybrid expert architecture of any one of claims 1-5.

Images