A distributed collaborative reasoning method and system for multimodal models

Abstract

The present invention provides a distributed collaborative reasoning method for multimodal models. The overall execution steps are divided into two stages: offline preparation and online reasoning. The offline preparation stage includes identifying the divisible parts according to the data dependency relationship in the multimodal model, and at the same time performing corresponding segmentation on the trained weights. The feature extraction part of the multimodal model is deployed on each end-side device, and the complete model is deployed on the central device that receives all data. The multimodal model is run on each device in the reasoning system to form an offline segmentation configuration; the online reasoning stage includes completing the total reasoning delay modeling, segmentation scheme distribution and model reasoning. Then, the central device distributes the optimal segmentation scheme of the model to each device, and each device adjusts the actual operation part of the model. Each device performs distributed reasoning calculation based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device to complete the distributed multimodal reasoning task. The present invention can interpretably output the optimal model distributed deployment plan under the current bandwidth, effectively reducing the reasoning delay.

Description

Technical Field

[0001] The present invention relates to a distributed collaborative reasoning technology for multimodal models, and belongs to the field of collaborative reasoning optimization in embedded artificial intelligence. Background Art

[0002] The rapid development of the Internet of Things (IoT) has generated massive amounts of multimodal data. With the increasing deployment of sensor devices and the number of communication network connections, various types of devices and sensors are able to connect and exchange data. This data includes not only traditional text and digital information, but also multimodal data such as sound, images, and video. This surge in multimodal data provides a richer source of information for AI applications, while also bringing new challenges and opportunities.

[0003] Compared to traditional single-modal data analysis, multimodal data analysis can rationally utilize the complementary information of different modal data. In recent years, thanks to the rapid development of artificial intelligence (AI), multimodal AI has become a hot research area. Traditional AI models are primarily based on single-modal data, while multimodal AI models integrate and process multimodal data from different sensors and devices to further enhance their application in various fields.

[0004] As the application of multimodal AI continues to expand, researchers and users are increasingly concerned about privacy and security issues. To protect user privacy and data security, an increasing number of AI models are being deployed locally on embedded devices. User data can be processed and analyzed locally, eliminating the risks of transmitting sensitive data to the cloud. Furthermore, deploying AI models locally reduces cloud computing latency, improves response speed, and better adapts to the needs of real-time applications.

[0005] The extremely high computational demands of multimodal models and limited local computing resources restrict the deployment of AI applications. First, embedded devices are often limited by bandwidth and storage capacity, making them unable to support large-scale models and data. Second, the limited computing resources of embedded devices make it difficult to meet the operational demands of complex models. These limitations make the efficient deployment of multimodal AI models on embedded devices a challenging task. A common approach is to coordinate local and edge devices for joint inference. Compared to model segmentation for single-modal models, the distributed deployment of multimodal models also requires consideration of differences in transmission bandwidth, data volume, and computing power resources across different devices. Therefore, the core challenge of distributed multimodal model deployment is how to comprehensively consider various resource constraints and find the optimal segmentation solution for distributed multimodal model deployment. Summary of the Invention

[0006] The technical problem to be solved by the present invention is how to efficiently deploy multimodal artificial intelligence models on embedded devices with limited bandwidth and computing power.

[0007] The present invention solves the above technical problems through the following technical means: a distributed collaborative reasoning method for multimodal models, the overall execution steps are divided into two stages: offline preparation and online reasoning. The offline preparation stage includes:

[0008] Step 1-1, complete the training of the multimodal model as a whole;

[0009] Step 1-2: Identify the divisible parts based on the data dependency within the multimodal model and divide the trained weights accordingly.

[0010] Steps 1-3 deploy the divisible parts of the multimodal model to the corresponding devices according to data dependencies. Specifically, the feature extraction part of the multimodal model is deployed to each end-side device, and the complete model is deployed to the central device that receives all data.

[0011] In steps 1-4, the multimodal model is run on each device in the inference system. The computational requirements of each multimodal model block are recorded, namely the runtime of each component and the size of the output data. The output data size is the size of the data transmitted after the model is split at the current layer location. By summarizing the runtime and data size of each component on each device, an offline segmentation configuration is formed.

[0012] The online reasoning phase includes:

[0013] Step 2-1: Preheat the model and test the communication connection of each device;

[0014] Step 2-2: Prepare the data required for modeling. First, the central device reads the offline segmentation configuration parameters pre-set during the offline preparation phase. Simultaneously, the central device reads the data transmission rate of each end-side device. Combined with the data transmission size specified in the segmentation configuration, the central device calculates the transmission time for each end-side device at different segmentation points under the current bandwidth. The transmission time is then added to the execution time of each layer before the current segmentation point to obtain the local execution time for each end-side device when applying the segmentation policy.

[0015] Step 2-3: For a multimodal model with N modal branches, calculate the time added to the total inference latency by solving the calculation of each branch;

[0016] Steps 2-4: Complete the total inference latency modeling;

[0017] Step 2-5: Split scheme distribution and model inference. The central device monitors the data transmission rate during communication with different edge devices for prediction, and combines the model splitting configuration obtained in the offline preparation stage to solve the optimization problem of the total inference delay model, and obtains the optimal splitting scheme.

[0018] Then the central device distributes the optimal model splitting scheme to each device. Each device adjusts the actual running part of the model, and each device reconstructs the local required model structure. Then, each device performs distributed inference calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device and completes the remaining model calculations to output the complete inference result, thus completing the distributed multi-modal inference task.

[0019] Further, in Step 2-2, for a multi-modal model with N modal branches, the splitting scheme is denoted as P(N) = {p1, p2, …, p N}. According to the order in which data from different edge devices arrives at the central device, the sum of the execution time and transmission time on the edge device is denoted as {E(p1), E(p2), …, E(p N ), and the execution time on the central device is denoted as {C(p1), C(p2), …, C(p N ). The timing constraint can be expressed as "when i < j, E(p i ) < E(p j )". E j , C j are used to simplify the representation of the sum of the execution time and transmission time on the edge device and the execution time on the central device respectively.

[0020] Further, in Step 2-3, for the j-th branch, the total increased time is:

[0021]

[0022] The total inference delay of the first m branches is denoted as

[0023] In Step 2-4, under the timing constraint E1 < E2 < … < E N , the optimization problem of the inference delay of splitting the distributed multi-modal model is modeled as follows:

[0024]

[0025] where T rest represents the execution time of the remaining part of the model. After completing the calculations of all modal feature extraction branches, the central device collects all modal data and executes the remaining part of the model to complete the inference task.

[0026] Furthermore, in steps 2-5, the detailed steps for solving are as follows:

[0027] 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest of the model is not affected by the model segmentation operation and network communication bandwidth, it is considered that T rest It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into:

[0028]

[0029] 2) Next, the split points of each branch are determined in order of arrival time. The split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), for the total time added to the inference by the j-th branch (j = 2, ..., M), the optimization problem of formula (3) is transformed into the problem of solving the optimal split point of M branches in sequence. The optimal split point problem of the j-th branch is as follows:

[0030]

[0031] Among them, P * (N) = {p * (1),p * (2),...,p * (M)};

[0032] 3) Solve equation (4) to obtain the optimal segmentation point;

[0033] 4) Solve equation (4) from the Mth branch to the first branch in chronological order, repeat the previous step, and obtain M optimal segmentation points {p * (1),p * (2),...,p * (M)}, that is, the overall model segmentation solution P of the model is obtained * (N) = {p * (1),p * (2),...,p * (M)}.

[0034] Furthermore, in step 3), equation (4) is solved in two cases; first, when E j -S j-1 <0, T j =C j , read C directly from the split configuration j The value of C j The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the j-th branch; second, on the contrary, T j =E j +C j-S j-1 , where S j-1 is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, E is directly read from the split configuration. j and C j The value of E j +C j The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

[0035] The present invention also provides a distributed collaborative reasoning system for multimodal models, which is divided into an offline preparation module and an online reasoning module. The offline preparation module includes:

[0036] The training unit is used to complete the training of the multimodal model as a whole;

[0037] The segmentation unit is used to identify the divisible parts based on the data dependency relationship in the multimodal model and to segment the trained weights accordingly;

[0038] The deployment unit is used to deploy the divisible parts of the multimodal model to the corresponding devices according to the data dependency relationship. In other words, the feature extraction part of the multimodal model is deployed to each end-side device, and the complete model is deployed to the central device that receives all data.

[0039] The run unit is used to run the multimodal model on each device in the inference system and record the computational requirements of each multimodal model block, namely the runtime of each component and the size of the output data. The output data size is the size of the data transmitted after the model is split at the location of the current layer. By sorting the runtime and transmission data size of each component on each device, an offline segmentation configuration is formed.

[0040] The online reasoning stage modules include:

[0041] Preheating unit, used to preheat the model and test the communication connection of each device;

[0042] The data preparation unit is used to prepare the data required for modeling. First, the central device reads the offline segmentation configuration parameters pre-set during the offline preparation phase. Simultaneously, the central device reads the data transmission rate of each end-side device. Combined with the transmission data size in the segmentation configuration, the central device calculates the transmission time for each end-side device at different segmentation points under the current bandwidth. The transmission time is then added to the execution time of each layer before the current segmentation point to obtain the local execution time for each end-side device when applying the segmentation policy.

[0043] A solving unit, which is used to solve the time added to the total inference latency by the calculation of each branch for a multi-modal model with N modal branches;

[0044] A modeling unit, which is used to complete the modeling of the total inference latency;

[0045] A splitting unit, which is used to split the scheme distribution and model inference, monitor the data transmission rate of the central device during monitoring and prediction and communication with different end-side devices, and combine the model splitting configuration obtained in the offline preparation stage to solve the optimization problem of the total inference latency model and obtain the optimal splitting scheme;

[0046] A distribution module, which is used for the central device to distribute the optimal model splitting scheme to each device. Each device adjusts the actual running part of the model, reconstructs the model structure required locally by each device, and then each device performs distributed inference calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device and completes the remaining model calculations, outputs the complete inference result, and completes the distributed multi-modal inference task.

[0047] Further, in the data preparation unit, for a multi-modal model with N modal branches, the splitting scheme is denoted as P(N) = {p1, p2,..., p N}, and according to the order in which the data of different end-side devices is transmitted to the central device, the sum of the execution time and transmission time on the end-side device is denoted as {E(p1), E(p2),..., E(p N )}, and the execution time on the central device is denoted as {C(p1), C(p2),..., C(p N )}. The timing constraint can be expressed as "when i < j, E(p i ) < E(p j )", and E j , C j are used to simplify the representation of the sum of the execution time and transmission time on the end-side device and the execution time on the central device respectively.

[0048] Further, in the solving unit, for the j-th branch, the total increased time is:

[0049]

[0050] where the total inference latency of the first m branches is denoted as

[0051] In the modeling unit, under the timing constraint E1 < E2 <... < E N , the optimization problem of the inference latency of the distributed multi-modal model splitting is modeled as follows:

[0052]

[0053] where Trest Represents the execution time of the remaining part of the model. After completing the calculation of all modal feature extraction branches, the central device collects all modal data and executes the remaining part of the model to complete the inference task.

[0054] Furthermore, in the segmentation unit, the detailed steps of solving are as follows:

[0055] 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest of the model is not affected by the model segmentation operation and network communication bandwidth, it is considered that T rest It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into:

[0056]

[0057] 2) Next, the split points of each branch are determined in order of arrival time. The split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), for the total time added to the inference by the j-th branch (j = 2, ..., M), the optimization problem of formula (3) is transformed into the problem of solving the optimal split point of M branches in sequence. The optimal split point problem of the j-th branch is as follows:

[0058]

[0059] Among them, P * (N) = {p * (1),p * (2),...,p * (M)};

[0060] 3) Solve equation (4) to obtain the most suitable segmentation point;

[0061] 4) By solving equation (4) in chronological order from the Mth branch to the first branch, repeating the previous step, we can obtain M optimal segmentation points {p * (1),p * (2),...,p * (M)}, that is, the overall model segmentation solution P of the model is obtained * (N) = {p * (1),p * (2),...,p * (M)}.

[0062] Furthermore, in step 3), equation (4) is solved in two cases; first, when E j -S j-1 <0, T j =C j , read C directly from the split configuration jThe value of C j The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the j-th branch; second, on the contrary, T j =E j +C j -S j-1 , where S j-1 is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, E is directly read from the split configuration. j and C j The value of E j +C j The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

[0063] The advantages of the present invention are:

[0064] 1. This paper proposes, for the first time, a universal distributed collaborative reasoning technology for deep multimodal neural networks. By theoretically modeling distributed reasoning in deep multimodal neural networks, it reveals the key factors that limit the performance of distributed deployment of multimodal models. In practical applications, even when the communication bandwidth of different devices is heterogeneous and dynamically changing, this paper can interpretably output the optimal model distributed deployment solution for the current bandwidth, effectively reducing inference latency.

[0065] 2. The present invention decouples the modules and layers in the deep multimodal neural network and constructs a collaborative reasoning framework, which can effectively reduce the blocking problem caused by data transmission differences when deployed on different devices, alleviate the problem of resource consumption, and improve the overall resource utilization of the reasoning platform.

[0066] 3. The present invention has been experimentally verified on different distributed embedded platforms. Compared with traditional model segmentation methods, the present invention can adaptively adjust the deployment strategy in real time, and is superior to traditional methods in terms of computational complexity, accuracy, and latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a diagram of the reasoning effect of a distributed collaborative reasoning method for a multimodal model according to an embodiment of the present invention;

[0068] Figure 2 Schematic diagram of a reasoning framework of a distributed collaborative reasoning method for a multimodal model according to an embodiment of the present invention;

[0069] Figure 3 This is a distributed reasoning flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0071] To fully utilize the computing power of each device, a distributed approach is required to deploy multimodal models. Existing partitioning schemes are designed for single-modal models, treating the model on the central device as a whole. This ignores the blocking time caused by transmission differences between multiple data sources, resulting in low resource utilization.

[0072] Therefore, if Figure 1 As shown, this paper proposes a distributed collaborative reasoning method for multimodal models. First, the multimodal model is divided into blocks, using blocking time to execute the parts that do not depend on transmitted data. Second, the multimodal model structure executed by each device is adjusted to reduce blocking time and overall latency. During the inference process, each feature extraction branch is divided into two parts: branch 1.A and branch 1.B, which are executed separately on the terminal and the central device.

[0073] like Figure 2 As shown, the overall execution steps of a distributed collaborative reasoning method for a multimodal model of the present invention are divided into two stages: offline preparation and online reasoning.

[0074] The specific steps in the offline preparation phase are as follows:

[0075] Step 1-1, complete the training of the multimodal model as a whole;

[0076] Step 1-2: Identify the divisible parts based on the data dependency within the multimodal model and divide the trained weights accordingly.

[0077] Steps 1-3, such as Figure 2 , the present invention deploys the divisible parts of the multimodal model to the corresponding devices according to the data dependency relationship, that is, deploys the feature extraction part of the multimodal model to each end-side device, and deploys the complete model to the central device that receives all data;

[0078] Steps 1-4: Run the multimodal model on each device in the inference system, and record the computing requirements of each multimodal model block, that is, the running time of each component and the size of the output data; where the size of the output data is the size of the data transmitted after splitting the model at the position of the current layer; by collating the running time of each component and the size of the transmitted data on each device, an offline splitting configuration is formed for serving real-time model splitting in the inference phase.

[0079] Combined with Figure 3 As shown, the specific operation steps in the online inference phase are as follows:

[0080] Step 2-1: Warm up the model, and each device tests the communication connection status.

[0081] Step 2-2: Prepare the data required for modeling.

[0082] First, the central device reads the offline splitting configuration parameters preset in the offline preparation phase. At the same time, the central device reads the data transmission rate of each edge device, combines the size of the transmitted data in the splitting configuration, calculates the transmission time of different splitting points of each edge device under the current bandwidth condition, and adds the transmission time to the execution time of each layer before the current splitting point to obtain the local execution time of each edge device when applying the splitting strategy.

[0083] For a multimodal model with N modal branches, the splitting scheme is denoted as P(N) = {p1, p2, …, p N}, according to the order (timing constraint) of the transmitted data of different edge devices arriving at the central device, the sum of the execution time and the transmission time on the edge device is denoted as {E(p1), E(p2), …, E(p N )}, and the execution time on the central device is denoted as {C(p1), C(p2), …, C(p N )}, where the timing constraint can be expressed as "when i < j, E(p i ) < E(p j )", and below, E j , C j are used to simplify and represent the sum of the execution time and the transmission time on the edge device and the execution time on the central device respectively.

[0084] Step 2-3: Solve the time added to the total inference latency by the computation of each branch. For example, for the jth branch, the total added time is:

[0085]

[0086] where the total inference latency of the first m branches is denoted as

[0087] Steps 2-4 complete the inference total delay modeling. <E2<…<E N Under this circumstance, the inference delay optimization problem of distributed multimodal model segmentation is modeled as follows:

[0088]

[0089] Where T rest Represents the execution time of the remaining model. After completing the computation of all modal feature extraction branches, the central device collects all modal data and executes the remaining model to complete the inference task. By solving the above optimization problem, the ideal model segmentation solution corresponding to the minimum inference latency is obtained.

[0090] Steps 2-5: Segmentation plan distribution and model inference. The central device monitors and predicts the data transmission rate when communicating with different end-side devices. Combined with the model segmentation configuration obtained during the offline preparation phase, it solves the above optimization problem and obtains the optimal segmentation plan.

[0091] The detailed steps of the solution are as follows:

[0092] 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest of the model is not affected by the model segmentation operation and network communication bandwidth, it can be considered that T rest It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into:

[0093]

[0094] 2) Next, the split points of each branch are determined in order of arrival time. It is worth noting that the split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), for the total time added to the inference by the j-th branch (j = 2, ..., M), the optimization problem of formula (3) can be transformed into the problem of solving the optimal split point of M branches in sequence. The optimal split point problem of the j-th branch is as follows:

[0095]

[0096] Among them, P * (N) = {p * (1),p * (2),...,p * (M)}.

[0097] 3) The present invention solves equation (4) in two cases; first, when E j -S j-1 <0, T j =C j , C can be read directly from the split configurationj The value of C j The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the j-th branch; second, on the contrary, T j =E j +C j -S j-1 , where S j-1 is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, E can be directly read from the split configuration. j and C j The value of E j +C j The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

[0098] 4) Solve equation (4) from the Mth branch to the first branch in chronological order, repeat the previous step, and obtain M optimal segmentation points {p * (1),p * (2),...,p * (M)}, that is, the overall model segmentation solution P of the model is obtained * (N) = {p * (1),p * (2),...,p * (M)}.

[0099] The central device then distributes the optimal model segmentation plan to each device, which adjusts the actual running part of the model. Each device rebuilds the model structure required locally, and then each device performs distributed inference calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device, completes the remaining model calculations, outputs the complete inference results, and completes the distributed multimodal inference task.

[0100] This technology divides complex multimodal AI models into multiple sub-models, dynamically loading and executing them based on the computing and storage resources of the embedded device. In an offline environment, it analyzes the computing power resources of each device in the inference platform one by one, and conducts a fine-grained analysis of the computing power requirements and expected data transmission size of each model component. During the inference phase, it monitors and predicts the data transmission rate of each device, models distributed inference, and solves the inference latency optimization problem while meeting timing conditions to achieve optimal inference performance. By optimizing the model structure and algorithm, the overall latency of the model inference is reduced, achieving the goal of efficiently deploying multimodal AI models on embedded devices with limited bandwidth and computing power.

[0101] The present invention theoretically models distributed multimodal reasoning and constructs an adaptive collaborative reasoning framework, which can realize collaborative reasoning of multiple embedded devices. It can adaptively adjust the transmission and computing settings on each device according to the real-time communication bandwidth, achieve the theoretical lowest reasoning latency, and promote the process of multimodal models towards embedded deployment.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A distributed collaborative reasoning method for multimodal models, characterized by: The overall execution steps are divided into two stages: offline preparation and online inference. The offline preparation stage includes: Step 1-1, complete the training of the multimodal model as a whole; Step 1-2: Identify the divisible parts based on the data dependency within the multimodal model and divide the trained weights accordingly. Steps 1-3 deploy the divisible parts of the multimodal model to the corresponding devices according to data dependencies. Specifically, the feature extraction part of the multimodal model is deployed to each end-side device, and the complete model is deployed to the central device that receives all data. In steps 1-4, the multimodal model is run on each device in the inference system. The computational requirements of each multimodal model block are recorded, namely the runtime of each component and the size of the output data. The output data size is the size of the data transmitted after the model is split at the current layer location. By summarizing the runtime and data size of each component on each device, an offline segmentation configuration is formed. The online reasoning phase includes: Step 2-1: Preheat the model and test the communication connection of each device; Step 2-2: Prepare the data required for modeling. First, the central device reads the offline segmentation configuration parameters pre-set during the offline preparation phase. Simultaneously, the central device reads the data transmission rate of each end-side device. Combined with the data transmission size specified in the segmentation configuration, the central device calculates the transmission time for each end-side device at different segmentation points under the current bandwidth. The transmission time is then added to the execution time of each layer before the current segmentation point to obtain the local execution time for each end-side device when applying the segmentation policy. Step 2-3, for For a multimodal model with multiple modal branches, calculate the time added to the total inference delay by solving the calculation of each branch; Steps 2-4: Complete the total inference latency modeling; In steps 2-5, the central device monitors and predicts the data transmission rate when communicating with different end-side devices. It then combines the model segmentation configuration obtained during the offline preparation phase to solve the optimization problem of the total inference latency model and obtain the optimal segmentation solution. The central device then distributes the optimal model segmentation plan to each device, which adjusts the actual running part of the model. Each device rebuilds the model structure required locally, and then each device performs distributed inference calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device, completes the remaining model calculations, outputs the complete inference results, and completes the distributed multimodal inference task.

2. The distributed collaborative reasoning method for multimodal models according to claim 1, characterized in that: In step 2-2, The multimodal model with modal branches is divided into According to the order in which different end-side devices transmit data to the central device, the sum of the execution time and transmission time on the end-side device is recorded as , the execution time on the central device is recorded as , where the timing constraint can be expressed as "when hour, ”, To simplify the sum of the execution time and transmission time on the end device and the execution time on the central device.

3. The distributed collaborative reasoning method for multimodal models according to claim 1, characterized in that: In step 2-3, for The total time added for each branch is: （1） Among them The total delay of reasoning for branches is recorded as ; In steps 2-4, in the timing constraints Under this circumstance, the inference delay optimization problem of distributed multimodal model segmentation is modeled as follows: （2） in Represents the execution time of the remaining part of the model. After completing the calculation of all modal feature extraction branches, the central device collects all modal data and executes the remaining part of the model to complete the inference task.

4. The distributed collaborative reasoning method for multimodal models according to claim 3, characterized in that: In steps 2-5, the detailed solution steps are as follows: 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest of the model is not affected by the model splitting operation and network communication bandwidth, it is considered that It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into: （3） 2) Next, the split points of each branch are determined according to the arrival time. The split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), the total time added to the reasoning for the j-th branch is , transform the optimization problem of formula (3) into the problem of solving the optimal splitting point of M branches in sequence. The optimal splitting point problem of the j-th branch is as follows: （4） in ; 3) Solve equation (4) to obtain the most suitable segmentation point; 4) Solve equation (4) from the Mth branch to the first branch in chronological order, repeat the previous step, and obtain M optimal split points. , that is, the overall model segmentation solution of the model is obtained .

5. The distributed collaborative reasoning method for multimodal models according to claim 4, characterized in that: In step 3), equation (4) is solved in two cases; first, when hour, , read directly from the split configuration The value of The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the j-th branch; second, on the contrary, ,in is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, it is directly read from the split configuration. and The value of The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

6. A distributed collaborative reasoning system for multimodal models, characterized by: It is divided into an offline preparation module and an online reasoning module. The offline preparation module includes: The training unit is used to complete the training of the multimodal model as a whole; The segmentation unit is used to identify the divisible parts based on the data dependency relationship in the multimodal model and to segment the trained weights accordingly; The deployment unit is used to deploy the divisible parts of the multimodal model to the corresponding devices according to the data dependency relationship. In other words, the feature extraction part of the multimodal model is deployed to each end-side device, and the complete model is deployed to the central device that receives all data. The run unit is used to run the multimodal model on each device in the inference system and record the computational requirements of each multimodal model block, namely the runtime of each component and the size of the output data. The output data size is the size of the data transmitted after the model is split at the location of the current layer. By sorting the runtime and transmission data size of each component on each device, an offline segmentation configuration is formed. The online reasoning stage modules include: Preheating unit, used to preheat the model and test the communication connection of each device; The data preparation unit is used to prepare the data required for modeling. First, the central device reads the offline segmentation configuration parameters pre-set during the offline preparation phase. Simultaneously, the central device reads the data transmission rate of each end-side device. Combined with the transmission data size in the segmentation configuration, the central device calculates the transmission time for each end-side device at different segmentation points under the current bandwidth. The transmission time is then added to the execution time of each layer before the current segmentation point to obtain the local execution time for each end-side device when applying the segmentation policy. Solving unit, used for For a multimodal model with multiple modal branches, calculate the time added to the total inference delay by solving the calculation of each branch; Modeling unit, used to complete the total inference delay modeling; The segmentation unit is responsible for segmentation solution distribution and model inference. The central device monitors and predicts the data transmission rate when communicating with different end-side devices. It combines the model segmentation configuration obtained during the offline preparation phase to solve the optimization problem of the total inference delay model and obtain the optimal segmentation solution. The distribution module is used by the central device to distribute the optimal model segmentation solution to each device. Each device adjusts the actual operation part of the model, and each device rebuilds the model structure required locally. Then, each device performs distributed inference calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device, completes the remaining model calculations, outputs the complete inference results, and completes the distributed multimodal inference task.

7. The distributed collaborative reasoning system for multimodal models according to claim 6, characterized in that: In the data preparation unit, The multimodal model with modal branches is divided into According to the order in which different end-side devices transmit data to the central device, the sum of the execution time and transmission time on the end-side device is recorded as , the execution time on the central device is recorded as , where the timing constraint can be expressed as "when hour, ”, To simplify the sum of the execution time and transmission time on the end device and the execution time on the central device.

8. The distributed collaborative reasoning system for multimodal models according to claim 6, characterized in that: In the solution unit, for The total time added for each branch is: （1） Among them The total delay of reasoning for branches is recorded as ; In the modeling unit, in the timing constraint Under this circumstance, the inference delay optimization problem of distributed multimodal model segmentation is modeled as follows: （2） in Represents the execution time of the remaining part of the model. After completing the calculation of all modal feature extraction branches, the central device collects all modal data and executes the remaining part of the model to complete the inference task.

9. The distributed collaborative reasoning system for multimodal models according to claim 8, characterized in that: In the split unit, the detailed steps of solving are as follows: 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest of the model is not affected by the model splitting operation and network communication bandwidth, it is considered that It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into: （3） 2) Next, the split points of each branch are determined according to the arrival time. The split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), the total time added to the reasoning for the j-th branch is , transform the optimization problem of formula (3) into the problem of solving the optimal splitting point of M branches in sequence. The optimal splitting point problem of the j-th branch is as follows: （4） in ; 3) Solve equation (4) to obtain the optimal segmentation point; 4) Solve equation (4) from the Mth branch to the first branch in chronological order, repeat the previous step, and obtain M optimal split points. , that is, the overall model segmentation solution of the model is obtained .

10. The distributed collaborative reasoning system for multimodal models according to claim 9, characterized in that: In step 3), equation (4) is solved in two cases; first, when hour, , read directly from the split configuration The value of The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the j-th branch; second, on the contrary, ,in is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, it is directly read from the split configuration. and The value of The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

Images