Distributed cooperative reasoning method and system for multi-modal model

Abstract

According to the distributed cooperative reasoning method oriented to the multi-modal model, the overall execution steps are divided into an offline preparation stage and an online reasoning stage, the offline preparation stage part comprises the steps that a divisible part is recognized according to the data dependency relationship in the multi-modal model, meanwhile, the weight obtained through training is subjected to corresponding segmentation, and the divided part is divided into the divided part according to the data dependency relationship in the multi-modal model; the feature extraction part of the multi-modal model is deployed to each end side device, the complete model is deployed to a center device receiving all data, and the multi-modal model is operated on each device in the inference system to form offline segmentation configuration; the online reasoning stage part comprises the steps of completing reasoning total time delay modeling, distributing a segmentation scheme and model reasoning, then distributing the optimal segmentation scheme of the model to each device by the central device, adjusting the actual operation part of the model by each device, and executing distributed reasoning calculation by each device based on a new model structure, so as to obtain the optimal segmentation scheme of the model. And finally, the central device merges the intermediate features transmitted by the devices to complete a distributed multi-modal reasoning task. According to the method, the optimal model distributed deployment scheme under the current bandwidth condition can be explained and output, and the reasoning time delay is effectively reduced.

Description

Technical Field

[0001] The present invention relates to a distributed collaborative reasoning technology for multimodal models, and belongs to the field of collaborative reasoning optimization in embedded artificial intelligence. Background Art

[0002] The rapid development of the Internet of Things has created a massive amount of multimodal data. With the increase in the scale of sensor device deployment and the number of communication network connections, various types of devices and sensors can connect to each other and exchange data. This data comes not only from traditional text and digital information, but also includes data in multiple modalities such as sound, images and videos. The emergence of multimodal data provides a richer source of information for the application of artificial intelligence, but also brings new challenges and opportunities.

[0003] Compared with traditional single-modal data analysis methods, multimodal data analysis can reasonably utilize the complementary information of different modal data. In recent years, benefiting from the rapid development of artificial intelligence technology, multimodal artificial intelligence has become a hot research field. Traditional artificial intelligence models are mainly based on single-modal data, while multimodal artificial intelligence models integrate and process multiple modal data from different sensors and devices in order to further improve their application effects in various fields.

[0004] As the application scope of multimodal artificial intelligence continues to expand, researchers and users have begun to pay attention to privacy and security issues. In order to protect user privacy and data security, more and more artificial intelligence models are being deployed locally on embedded devices. User data can be processed and analyzed locally, avoiding the risk of transmitting sensitive data to the cloud. At the same time, locally deployed artificial intelligence models can also reduce the latency of cloud computing, improve response speed, and better adapt to the needs of real-time applications.

[0005] The extremely high computational workload of multimodal models and the limited local computing resources restrict the deployment of artificial intelligence applications. First, embedded devices are usually limited by bandwidth and storage capacity and cannot carry large-scale models and data. Second, the computing resources of embedded devices are limited and it is difficult to meet the computing requirements of complex models. These limitations make it a challenging task to efficiently deploy multimodal artificial intelligence models on embedded devices. A common approach is to coordinate local devices and edge devices for joint reasoning. Compared with the model segmentation of single-modal models, when distributing multimodal models, it is also necessary to consider the differences in bandwidth transmission of different devices, the differences in the amount of data transmitted, and the differences in computing resources of different devices. Therefore, how to comprehensively consider various resource constraints and find the optimal performance segmentation scheme for distributed deployment of multimodal models is the core challenge of distributed deployment of multimodal models. Summary of the invention

[0006] The technical problem to be solved by the present invention is how to efficiently deploy a multimodal artificial intelligence model on an embedded device with limited bandwidth and computing power.

[0007] The present invention solves the above technical problems through the following technical means: a distributed collaborative reasoning method for multimodal models, the overall execution steps are divided into two stages: offline preparation and online reasoning, and the offline preparation stage part includes:

[0008] Step 1-1, complete the training of the multimodal model as a whole;

[0009] Step 1-2: Identify the divisible parts according to the data dependency relationship in the multimodal model, and divide the trained weights accordingly;

[0010] Step 1-3: deploy the divisible parts of the multimodal model to the corresponding devices according to the data dependency relationship, that is, deploy the feature extraction part of the multimodal model to each end-side device, and deploy the complete model to the central device that receives all data;

[0011] Steps 1-4: Run the multimodal model on each device in the inference system, record the computational requirements of each multimodal model block, that is, the running time of each component and the size of the output data, where the size of the output data is the size of the data transmitted after the model is split at the location of the current layer, and form an offline segmentation configuration by sorting out the running time of each component on each device and the size of the transmitted data;

[0012] The online reasoning phase includes:

[0013] Step 2-1, preheat the model and test the communication connection of each device;

[0014] Step 2-2, prepare the data required for modeling. First, the central device reads the offline segmentation configuration parameters pre-set in the offline preparation stage. At the same time, the central device reads the data transmission rate of each end-side device, and calculates the transmission time of different segmentation points of each end-side device under the current bandwidth in combination with the transmission data size in the segmentation configuration. The transmission time is added to the execution time of each layer before the current segmentation point to obtain the local execution time of each end-side device when applying the segmentation strategy.

[0015] Step 2-3, for a multimodal model with N modal branches, calculate the time added to the total inference delay by solving the calculation of each branch;

[0016] Step 2-4, complete the total inference delay modeling;

[0017] Step 2-5, Split Scheme Distribution and Model Inference. The central device monitors the data transmission rate during communication with different edge devices for prediction, and combines the model splitting configuration obtained in the offline preparation stage to solve the optimization problem of the total inference delay model, obtaining the optimal splitting scheme.

[0018] Then the central device distributes the optimal model splitting scheme to each device. Each device adjusts the actual running part of the model, rebuilds the model structure required locally by each device, and then each device performs distributed inference calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device and completes the remaining model calculations, outputting the complete inference result to complete the distributed multi-modal inference task.

[0019] Further, in Step 2-2, for a multi-modal model with N modal branches, the splitting scheme is denoted as P(N) = {p1, p2, …, p N}. According to the order in which the data transmitted by different edge devices arrives at the central device, the sum of the execution time and transmission time on the edge device is denoted as {E(p1), E(p2), …, E(p N ), and the execution time on the central device is denoted as {C(p1), C(p2), …, C(p N ). Among them, the timing constraint can be expressed as "when i < j, E(p i ) < E(p j )", and E j , C j are used to simplify the representation of the sum of the execution time and transmission time on the edge device and the execution time on the central device respectively.

[0020] Further, in Step 2-3, for the j-th branch, the total increased time is:

[0021]

[0022] Among them, the total inference delay of the first m branches is denoted as

[0023] In Step 2-4, under the timing constraint E1 < E2 < … < E N , the optimization problem of the inference delay of the distributed multi-modal model splitting is modeled as follows:

[0024]

[0025] Among them, T rest represents the execution time of the remaining part of the model. After completing the calculations of all modal feature extraction branches, the central device collects all modal data and executes the remaining part of the model to complete the inference task.

[0026] Furthermore, in steps 2-5, the detailed steps of solving are as follows:

[0027] 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest is not affected by the model segmentation operation and network communication bandwidth, it is considered that T rest It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into:

[0028]

[0029] 2) Next, the split points of each branch are determined in order of arrival time. The split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), for the total time added to the reasoning by the j-th branch (j = 2, ..., M), the optimization problem of formula (3) is transformed into the problem of solving the optimal split point of M branches in sequence. The optimal split point problem of the j-th branch is as follows:

[0030]

[0031] Where P * (N) = {p * (1),p * (2),...,p * (M)};

[0032] 3) Solve equation (4) to obtain the optimal segmentation point;

[0033] 4) By solving equation (4) in chronological order from the Mth branch to the first branch, repeating the previous step, we can obtain M optimal segmentation points {p * (1),p * (2),...,p * (M)}, that is, the overall model segmentation solution P of the model is obtained * (N) = {p * (1),p * (2),...,p * (M)}.

[0034] Furthermore, in step 3), equation (4) is solved in two cases; first, when E j -S j-1 <0, T j =C j , read C directly from the split configuration j The value of C j The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the jth branch; second, on the contrary, T j =E j +C j-S j-1 , where S j-1 is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, E is directly read from the split configuration. j and C j The value of E j +C j The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

[0035] The present invention also provides a distributed collaborative reasoning system for a multimodal model, which is divided into an offline preparation module and an online reasoning module. The offline preparation module includes:

[0036] A training unit, used to complete the training of the multimodal model as a whole;

[0037] The segmentation unit is used to identify the divisible parts according to the data dependency relationship in the multimodal model, and to segment the trained weights accordingly;

[0038] A deployment unit is used to deploy the divisible parts of the multimodal model to the corresponding devices according to the data dependency relationship, that is, to deploy the feature extraction part of the multimodal model to each end-side device and to deploy the complete model to the central device that receives all data;

[0039] The running unit is used to run the multimodal model on each device in the inference system, record the computing requirements of each multimodal model block, that is, the running time of each component and the size of the output data, where the size of the output data is the size of the data transmitted after the model is split at the location of the current layer, and form an offline segmentation configuration by sorting out the running time of each component on each device and the size of the transmitted data;

[0040] The online reasoning phase modules include:

[0041] Preheating unit, used to preheat the model and test the communication connection of each device;

[0042] The data preparation unit is used to prepare the data required for modeling. First, the central device reads the offline segmentation configuration parameters pre-set in the offline preparation phase. At the same time, the central device reads the data transmission rate of each end-side device, and calculates the transmission time of different segmentation points of each end-side device under the current bandwidth in combination with the transmission data size in the segmentation configuration. The transmission time is added to the execution time of each layer before the current segmentation point to obtain the local execution time of each end-side device when applying the segmentation strategy.

[0043] A solving unit, which is used to solve the time increased by the calculation of each branch for a multi-modal model with N modal branches for the total inference latency.

[0044] A modeling unit, which is used to complete the modeling of the total inference latency.

[0045] A splitting unit, which is used to split the scheme distribution and model inference, monitor the data transmission rate when the central device communicates with different edge devices for prediction, and combine the model splitting configuration obtained in the offline preparation stage to solve the optimization problem of the total inference latency model and obtain the optimal splitting scheme.

[0046] A distribution module, which is used for the central device to distribute the optimal model splitting scheme to each device. Each device adjusts the actual running part of the model, rebuilds the local required model structure, and then each device performs distributed inference calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device and completes the remaining model calculations to output the complete inference result and complete the distributed multi-modal inference task.

[0047] Further, in the data preparation unit, for a multi-modal model with N modal branches, the splitting scheme is denoted as P(N) = {p1, p2, …, p N}, and according to the order in which the data of different edge devices is transmitted to the central device, the sum of the execution time and transmission time on the edge device is denoted as {E(p1), E(p2), …, E(p N )}, and the execution time on the central device is denoted as {C(p1), C(p2), …, C(p N )}, where the timing constraint can be expressed as "when i < j, E(p i ) < E(p j )", and E j , C j are used to simplify the representation of the sum of the execution time and transmission time on the edge device and the execution time on the central device respectively.

[0048] Further, in the solving unit, for the j-th branch, the total increased time is:

[0049]

[0050] where the total inference latency of the first m branches is denoted as

[0051] In the modeling unit, under the timing constraint E1 < E2 < … < E N , the optimization problem of the inference latency of the distributed multi-modal model splitting is modeled as follows:

[0052]

[0053] where Trest Represents the execution time of the rest of the model. After completing the calculation of all modal feature extraction branches, the central device collects all modal data and executes the rest of the model to complete the inference task.

[0054] Furthermore, in the segmentation unit, the detailed steps of solving are as follows:

[0055] 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest is not affected by the model segmentation operation and network communication bandwidth, it is considered that T rest It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into:

[0056]

[0057] 2) Next, the split points of each branch are determined in order of arrival time. The split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), for the total time added to the reasoning by the j-th branch (j = 2, ..., M), the optimization problem of formula (3) is transformed into the problem of solving the optimal split point of M branches in sequence. The optimal split point problem of the j-th branch is as follows:

[0058]

[0059] Where P * (N) = {p * (1),p * (2),...,p * (M)};

[0060] 3) Solve equation (4) to obtain the most suitable segmentation point;

[0061] 4) By solving equation (4) in chronological order from the Mth branch to the first branch, repeating the previous step, we obtain M optimal segmentation points {p * (1),p * (2),...,p * (M)}, that is, the overall model segmentation solution P of the model is obtained * (N) = {p * (1),p * (2),...,p * (M)}.

[0062] Furthermore, in step 3), equation (4) is solved in two cases; first, when E j -S j-1 <0, T j =C j , read C directly from the split configuration jThe value of C j The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the jth branch; second, on the contrary, T j =E j +C j -S j-1 , where S j-1 is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, E is directly read from the split configuration. j and C j The value of E j +C j The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

[0063] The advantages of the present invention are:

[0064] 1. This invention proposes for the first time a general distributed collaborative reasoning technology for deep multimodal neural networks. By theoretically modeling the distributed reasoning of deep multimodal neural networks, the key factors that limit the performance of distributed deployment of multimodal models are revealed. In practical applications, even if the communication bandwidth of different devices is heterogeneous and changes dynamically, this invention can explainably output the optimal model distributed deployment solution under the current bandwidth, effectively reducing the reasoning delay.

[0065] 2. The present invention decouples the modules and layers in the deep multimodal neural network and constructs a collaborative reasoning framework, which can effectively reduce the blocking problem caused by the data transmission differences when deployed on different devices, alleviate the problem of resource consumption, and improve the overall resource utilization of the reasoning platform.

[0066] 3. The present invention has been experimentally verified on different distributed embedded platforms. Compared with the traditional model segmentation method, the present invention can adaptively adjust the deployment strategy in real time, and is superior to the traditional method in terms of computational complexity, accuracy, and latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a reasoning effect diagram of a distributed collaborative reasoning method for a multimodal model according to an embodiment of the present invention;

[0068] Figure 2 It is a schematic diagram of a reasoning framework of a distributed collaborative reasoning method for a multimodal model according to an embodiment of the present invention;

[0069] Figure 3 It is a distributed reasoning flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] In order to fully utilize the computing power of each device, it is necessary to deploy multimodal models in a distributed manner. Existing partitioning schemes are designed for single-modal models, treating the model on the central device side as a whole, but ignoring the blocking time caused by transmission differences between multiple data sources, resulting in low resource utilization.

[0072] Therefore, if Figure 1 As shown, the present invention proposes a distributed collaborative reasoning method for multimodal models. First, the multimodal model is divided into blocks, and the blocking time is used to execute the part that does not depend on the transmitted data. Second, the multimodal model structure executed by each device is adjusted to reduce the blocking time and reduce the overall delay. During the reasoning process, each feature extraction branch is divided into two parts: branch 1.A and branch 1.B, which are executed on the terminal and the central device respectively.

[0073] like Figure 2 As shown, the overall execution steps of a distributed collaborative reasoning method for a multimodal model of the present invention are divided into two stages: offline preparation and online reasoning.

[0074] The specific steps of the offline preparation phase are as follows:

[0075] Step 1-1, complete the training of the multimodal model as a whole;

[0076] Step 1-2: Identify the parts that can be divided according to the data dependency relationship in the multimodal model, and divide the trained weights accordingly;

[0077] Steps 1-3, such as Figure 2 , the present invention deploys the divisible parts of the multimodal model to the divided model block structure files and weight files to the corresponding devices according to the data dependency relationship, that is, deploys the feature extraction part of the multimodal model to each end-side device, and deploys the complete model to the central device that receives all data;

[0078] Steps 1 - 4: Run the multimodal model on each device in the inference system, and record the computing requirements of each multimodal model block, i.e., the running time of each component and the size of the output data; where the size of the output data is the size of the data transmitted after splitting the model at the position of the current layer; by organizing the running time and the size of the transmitted data of each component on each device, an offline splitting configuration is formed for serving real - time model splitting in the inference stage.

[0079] Combined with Figure 3 as shown, the specific operation steps in the online inference stage are as follows:

[0080] Step 2 - 1: Warm up the model, and each device tests the communication connection status.

[0081] Step 2 - 2: Prepare the data required for modeling.

[0082] First, the central device reads the offline splitting configuration parameters preset in the offline preparation stage. At the same time, the central device reads the data transmission rate of each edge device, combines the size of the transmitted data in the splitting configuration, calculates the transmission time of different splitting points of each edge device under the current bandwidth condition, and adds the transmission time to the execution time of each layer before the current splitting point to obtain the local execution time of each edge device when applying the splitting strategy.

[0083] For a multimodal model with N modal branches, the splitting scheme is denoted as P(N) = {p1, p2, …, p N}. According to the order (timing constraint) in which the transmitted data of different edge devices arrives at the central device, the sum of the execution time and the transmission time on the edge device is denoted as {E(p1), E(p2), …, E(p N ), and the execution time on the central device is denoted as {C(p1), C(p2), …, C(p N ), where the timing constraint can be expressed as "when i < j, E(p i ) < E(p j )". Below, E j , C j are used to simplify and represent the sum of the execution time and the transmission time on the edge device and the execution time on the central device respectively.

[0084] Step 2 - 3: Solve the time added to the total inference latency by the computation of each branch. For example, for the j - th branch, the total added time is:

[0085]

[0086] where the total inference latency of the first m branches is denoted as

[0087] Step 2-4, complete the total delay modeling of inference. <E2<…<E N Under this circumstance, the inference delay optimization problem of distributed multimodal model segmentation is modeled as follows:

[0088]

[0089] Where T rest Represents the execution time of the rest of the model. After completing the calculation of all modal feature extraction branches, the central device collects all modal data and executes the rest of the model to complete the inference task. By solving the above optimization problem, the ideal model segmentation solution corresponding to the minimum inference delay is obtained.

[0090] Step 2-5, segmentation scheme distribution and model reasoning. The central device monitors and predicts the data transmission rate when communicating with different end-side devices, combines the model segmentation configuration obtained in the offline preparation phase, solves the above optimization problem, and obtains the optimal segmentation scheme.

[0091] The detailed steps of the solution are as follows:

[0092] 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest of the model is not affected by the model segmentation operation and network communication bandwidth, it can be considered that T rest It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into:

[0093]

[0094] 2) Next, the split points of each branch are determined in order of arrival time. It is worth noting that the split points of different branches of the multimodal model do not interfere with each other. Combined with equation (1), for the total time added to the reasoning by the j-th branch (j = 2, ..., M), the optimization problem of equation (3) can be transformed into the problem of solving the optimal split point of M branches in sequence. The optimal split point problem of the j-th branch is as follows:

[0095]

[0096] Where P * (N) = {p * (1),p * (2),...,p * (M)}.

[0097] 3) The present invention solves equation (4) in two cases; first, when E j -S j-1 <0, T j =C j , C can be read directly from the split configurationj The value of C j The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the jth branch; second, on the contrary, T j =E j +C j -S j-1 , where S j-1 is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, E can be directly read from the split configuration. j and C j The value of E j +C j The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

[0098] 4) By solving equation (4) in chronological order from the Mth branch to the first branch, repeating the previous step, we can obtain M optimal segmentation points {p * (1),p * (2),...,p * (M)}, that is, the overall model segmentation solution P of the model is obtained * (N) = {p * (1),p * (2),...,p * (M)}.

[0099] The central device then distributes the optimal model segmentation plan to each device, which adjusts the actual running part of the model. Each device rebuilds the model structure required locally, and then each device performs distributed reasoning calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device, completes the remaining model calculations, outputs the complete reasoning results, and completes the distributed multimodal reasoning task.

[0100] This technology cuts complex multimodal AI models into multiple sub-models, and dynamically loads and executes these sub-models based on the computing and storage resources of embedded devices. In an offline environment, the computing power resources of each device in the inference platform are analyzed one by one, and the computing power requirements and expected transmission data size of each component of the model are analyzed in a fine-grained manner; in the inference stage, the data transmission rate of each device is monitored and predicted, distributed inference is modeled, and the inference latency optimization problem is solved under the premise of meeting the timing conditions to achieve the optimization of inference performance. By optimizing the model structure and algorithm, the overall latency of the model inference is reduced, and the goal of efficiently deploying multimodal AI models on embedded devices with limited bandwidth and computing power is achieved.

[0101] The present invention theoretically models distributed multimodal reasoning and constructs an adaptive collaborative reasoning framework, which can realize collaborative reasoning of multiple embedded devices, and can adaptively adjust the transmission and computing settings on each device according to the real-time communication bandwidth to achieve the theoretical minimum reasoning delay, thus promoting the process of multimodal models towards embedded deployment.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed collaborative reasoning method for multimodal models, Features: The overall execution steps are divided into two stages: offline preparation and online reasoning. The offline preparation stage includes: Step 1-1, complete the training of the multimodal model as a whole; Step 1-2: Identify the divisible parts according to the data dependency relationship in the multimodal model, and divide the trained weights accordingly; Step 1-3: deploy the divisible parts of the multimodal model to the corresponding devices according to the data dependency relationship, that is, deploy the feature extraction part of the multimodal model to each end-side device, and deploy the complete model to the central device that receives all data; Steps 1-4: Run the multimodal model on each device in the inference system, record the computational requirements of each multimodal model block, that is, the running time of each component and the size of the output data, where the size of the output data is the size of the data transmitted after the model is split at the location of the current layer, and form an offline segmentation configuration by sorting out the running time of each component on each device and the size of the transmitted data; The online reasoning phase includes: Step 2-1, preheat the model and test the communication connection of each device; Step 2-2, prepare the data required for modeling. First, the central device reads the offline segmentation configuration parameters pre-set in the offline preparation stage. At the same time, the central device reads the data transmission rate of each end-side device, and calculates the transmission time of different segmentation points of each end-side device under the current bandwidth in combination with the transmission data size in the segmentation configuration. The transmission time is added to the execution time of each layer before the current segmentation point to obtain the local execution time of each end-side device when applying the segmentation strategy. Step 2-3, for a multimodal model with N modal branches, calculate the time added to the total inference delay by solving the calculation of each branch; Step 2-4, complete the total inference delay modeling; Step 2-5: Segmentation scheme distribution and model reasoning: The central device monitors and predicts the data transmission rate when communicating with different end-side devices, and solves the optimization problem of the total delay model of reasoning in combination with the model segmentation configuration obtained in the offline preparation phase to obtain the optimal segmentation scheme; The central device then distributes the optimal model segmentation plan to each device, which adjusts the actual running part of the model. Each device rebuilds the model structure required locally, and then each device performs distributed reasoning calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device, completes the remaining model calculations, outputs the complete reasoning results, and completes the distributed multimodal reasoning task.

2. A distributed collaborative reasoning method for a multimodal model as claimed in claim 1, Features: In step 2-2, for a multi-modal model with N modal branches, the segmentation scheme is denoted as P(N) = {p 1 , p 2 , …, p N}. According to the order in which data is transmitted from different edge devices to the central device, the sum of the execution time and transmission time on the edge device is denoted as {E(p 1 ), E(p 2 ), …, E(p N ), and the execution time on the central device is denoted as {C(p 1 ), C(p 2 ), …, C(p N ). The timing constraint can be expressed as "when i < j, E(p i ) < E(p j )". E j and C j are used to simplify the representation of the sum of the execution time and transmission time on the edge device and the execution time on the central device, respectively.

3. A distributed collaborative reasoning method for a multimodal model as claimed in claim 1, Features: In step 2-3, for the jth branch, the total additional time is: The total reasoning delay of the first m branches is recorded as In step 2-4, in the timing constraint E 1 <E 2 <…<E N Under this circumstance, the inference delay optimization problem of distributed multimodal model segmentation is modeled as follows: where Tr est Represents the execution time of the rest of the model. After completing the calculation of all modal feature extraction branches, the central device collects all modal data and executes the rest of the model to complete the inference task.

4. A distributed collaborative reasoning method for a multimodal model as claimed in claim 3, Features: In steps 2-5, the detailed steps of solving are as follows: 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest of the model is not affected by the model segmentation operation and network communication bandwidth, it is believed that Tre st It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into: 2) Next, the split points of each branch are determined in order of arrival time. The split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), for the total time added to the reasoning by the j-th branch (j=2, ..., M), the optimization problem of formula (3) is transformed into the problem of solving the optimal split point of M branches in sequence. The optimal split point problem of the j-th branch is as follows: Where P * (N) = {p * (1), p * (2), ..., p * (M)}; 3) Solve equation (4) to obtain the most suitable segmentation point; 4) By solving equation (4) in chronological order from the Mth branch to the first branch, repeating the previous step, we obtain M optimal segmentation points {p * (1), p * (2), ..., p * (M)}, that is, the overall model segmentation solution P of the model is obtained * (N) = {p * (1), p * (2), ..., p * (M)}.

5. A distributed collaborative reasoning method for a multimodal model as claimed in claim 4, Features: In step 3), equation (4) is solved in two cases; first, when E j -S j-1 When <0, T j =C j , read C directly from the split configuration j The value of C j The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the jth branch; second, on the contrary, T j =E j +C j -S j-1 , where S j-1 is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, E is directly read from the split configuration. j and C j The value of E j +C j The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

6. A distributed collaborative reasoning system for multimodal models, Features: It is divided into an offline preparation module and an online reasoning module. The offline preparation module includes: A training unit, used to complete the training of the multimodal model as a whole; The segmentation unit is used to identify the divisible parts according to the data dependency relationship in the multimodal model, and to segment the trained weights accordingly; A deployment unit is used to deploy the divisible parts of the multimodal model to the corresponding devices according to the data dependency relationship, that is, to deploy the feature extraction part of the multimodal model to each end-side device and to deploy the complete model to the central device that receives all data; The running unit is used to run the multimodal model on each device in the inference system, record the computing requirements of each multimodal model block, that is, the running time of each component and the size of the output data, where the size of the output data is the size of the data transmitted after the model is split at the location of the current layer, and form an offline segmentation configuration by sorting out the running time of each component on each device and the size of the transmitted data; The online reasoning phase modules include: Preheating unit, used to preheat the model and test the communication connection of each device; The data preparation unit is used to prepare the data required for modeling. First, the central device reads the offline segmentation configuration parameters pre-set in the offline preparation phase. At the same time, the central device reads the data transmission rate of each end-side device, and calculates the transmission time of different segmentation points of each end-side device under the current bandwidth in combination with the transmission data size in the segmentation configuration. The transmission time is added to the execution time of each layer before the current segmentation point to obtain the local execution time of each end-side device when applying the segmentation strategy. A solving unit, used to solve the time added to the total inference delay by the calculation of each branch for a multimodal model with N modal branches; A modeling unit, used to complete the total delay modeling of reasoning; The segmentation unit is used for segmentation scheme distribution and model reasoning. The central device monitors and predicts the data transmission rate when communicating with different end-side devices. Combined with the model segmentation configuration obtained in the offline preparation phase, it solves the optimization problem of the total delay model of reasoning and obtains the optimal segmentation scheme. The distribution module is used for the central device to distribute the optimal model segmentation solution to each device. Each device adjusts the actual running part of the model. Each device rebuilds the model structure required locally. Then each device performs distributed reasoning calculations based on the new model structure. Finally, the central device merges the intermediate features transmitted by each device, completes the remaining model calculations, outputs the complete reasoning results, and completes the distributed multimodal reasoning task.

7. A distributed collaborative reasoning system for multimodal models as claimed in claim 1, Features: In the data preparation unit, for a multi-modal model with N modal branches, the segmentation scheme is denoted as P(N) = {p 1 , p 2 , …, p N}. According to the order in which data is transmitted from different edge devices to the central device, the sum of the execution time and the transmission time on the edge device is denoted as {E(p 1 ), E(p 2 ), …, E(p N ), and the execution time on the central device is denoted as {C(p 1 ), C(p 2 ), …, C(p N ). Among them, the timing constraint can be expressed as "when i < j, E(p i ) < E(p j )". E j and C j are used to simplify the representation of the sum of the execution time and the transmission time on the edge device and the execution time on the central device respectively.

8. A distributed collaborative reasoning system for multimodal models as claimed in claim 1, Features: In the solution unit, for the jth branch, the total time added is: The total reasoning delay of the first m branches is recorded as In the modeling unit, in the timing constraint E 1 <E 2 <…<E N Under this circumstance, the inference delay optimization problem of distributed multimodal model segmentation is modeled as follows: where Tr e *t represents the execution time of the rest of the model. After completing the calculation of all modal feature extraction branches, the central device collects all modal data and executes the rest of the model to complete the inference task.

9. A distributed collaborative reasoning system for multimodal models as claimed in claim 8, Features: In the split unit, the detailed steps of solving are as follows: 1) Since the rest of the model is always deployed on the central device, and the computing power of the central device is stable, and the execution of the rest of the model is not affected by the model segmentation operation and network communication bandwidth, it is believed that Tre st It does not change during the reasoning process, thus transforming the optimization problem of formula (2) into: 2) Next, the split points of each branch are determined in order of arrival time. The split points of different branches of the multimodal model do not interfere with each other. Combined with formula (1), for the total time added to the reasoning by the j-th branch (j=2, ..., M), the optimization problem of formula (3) is transformed into the problem of solving the optimal split point of M branches in sequence. The optimal split point problem of the j-th branch is as follows: Where P * (N) = {p * (1), p * (2), ..., p * (M)}; 3) Solve equation (4) to obtain the most suitable segmentation point; 4) By solving equation (4) in chronological order from the Mth branch to the first branch, repeating the previous step, we obtain M optimal segmentation points {p * (1), p * (2), ..., p * (M)}, that is, the overall model segmentation solution P of the model is obtained * (N) = {p * (1), p * (2), ..., p * (M)}.

10. A distributed collaborative reasoning system for multimodal models as claimed in claim 9, Features: In step 3), equation (4) is solved in two cases; first, when E j -S j-1 When <0, T j =C j , read C directly from the split configuration j The value of C j The segmentation layer index corresponding to the minimum value is one of the optimal segmentation points of the jth branch; second, on the contrary, T j =E j +C j -S j-1 , where S j-1 is the total inference delay of the first j-1 branches, which is independent of the split position of the j-th branch. Therefore, when determining the split point of the j-th branch, E is directly read from the split configuration. j and C j The value of E j +C j The segmentation layer index corresponding to the minimum value of is one of the optimal segmentation points of the j-th branch.

Images