End-cloud collaborative reasoning method, device, equipment and program product based on model unloading and intelligent quantification

Abstract

The invention provides an end-cloud collaborative reasoning method, device, equipment and program product based on model unloading and intelligent quantification, which can be applied to the field of end-side model reasoning tasks in resource-limited and network dynamic scenes. The method comprises the steps of dividing a deep neural network model into an end side sub-model and a cloud side sub-model according to calculation complexity and an interlayer dependency relationship of each network layer in the deep neural network model; processing the reasoning sub-task based on an end side sub-model to obtain an intermediate reasoning sub-result; matching the semantic center of the intermediate reasoning sub-result with the data feature of the reasoning task to obtain a data separation degree, and adjusting the precision of the intermediate reasoning sub-result based on the data separation degree to obtain an intermediate target reasoning sub-result; processing the intermediate target reasoning sub-result based on a cloud side sub-model to obtain a reasoning sub-result corresponding to the reasoning sub-task; and based on the plurality of reasoning sub-results, determining a reasoning result of the reasoning task.

Description

Technical Field

[0001] The present disclosure relates to the field of end-side model reasoning tasks in resource-constrained and network dynamic scenarios, and more specifically to an end-cloud collaborative reasoning method, device, equipment and program product based on model offloading and intelligent quantization. Background Art

[0002] Under the end-cloud collaborative reasoning computing paradigm, complex reasoning tasks involve the coordinated scheduling of multiple resources, including computing resources of end devices, network bandwidth resources, and computing resources of cloud servers. The effective scheduling of these resources is the key to achieving efficient reasoning. The pipeline parallel computing model shows broad application potential because it allows overlapping execution of computing and transmission phases in end-cloud collaborative reasoning tasks, which helps to improve the throughput of reasoning tasks. However, the model faces the challenges of time imbalance in the execution phase and uneven distribution of computing resources, and often has a "bubble phenomenon", that is, the idle waiting time of the participating devices is too long, resulting in increased latency and reduced efficiency of reasoning tasks. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a method, device, equipment and program product for end-cloud collaborative reasoning based on model offloading and intelligent quantization.

[0004] According to the first aspect of the present disclosure, a method for end-cloud collaborative reasoning based on model offloading and intelligent quantization is provided, including: dividing the deep neural network model into an end-side sub-model and a cloud-side sub-model according to the computational complexity and inter-layer dependency of each network layer in the deep neural network model; processing the i-1th reasoning subtask based on the end-side sub-model, obtaining the i-1th intermediate reasoning sub-result, and starting the i-th reasoning subtask, the end-side device where the end-side sub-model is located has i intermediate reasoning sub-results cached, and the reasoning task includes multiple reasoning subtasks, i>1; processing the i-1th intermediate reasoning sub-result The semantic center of the i-1th data separation degree is matched with the data features of the reasoning task to obtain the i-1th data separation degree, the semantic center represents the matrix features of the i-1th intermediate reasoning sub-result, and the data features represent the semantic attributes of the input data related to the reasoning task; the accuracy of the i-1th intermediate reasoning sub-result is adjusted based on the i-1th data separation degree to obtain the i-1th intermediate target reasoning sub-result; the i-1th intermediate target reasoning sub-result is processed based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task; and the reasoning result of the reasoning task is determined based on multiple reasoning sub-results.

[0005] The second aspect of the present disclosure provides an end-cloud collaborative reasoning device based on model offloading and intelligent quantization, including: a division module, which is used to divide the deep neural network model into an end-side sub-model and a cloud-side sub-model according to the computational complexity and inter-layer dependency of each network layer in the deep neural network model; a processing module, which is used to process the i-1th reasoning sub-task based on the end-side sub-model, obtain the i-1th intermediate reasoning sub-result, and start the i-th reasoning sub-task, the end-side device where the end-side sub-model is located has i intermediate reasoning sub-results cached, and the reasoning task includes multiple reasoning sub-tasks, i>1; a matching module, which is used to match the semantic center of the i-1th intermediate reasoning sub-result Matching with the data features of the reasoning task to obtain the i-1th data separation degree, the semantic center represents the matrix features of the i-1th intermediate reasoning sub-result, and the data features represent the semantic attributes of the input data related to the reasoning task; an adjustment module is used to adjust the accuracy of the i-1th intermediate reasoning sub-result based on the i-1th data separation degree to obtain the i-1th intermediate target reasoning sub-result; an intermediate target processing module is used to process the i-1th intermediate target reasoning sub-result based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task; and a reasoning result determination module is used to determine the reasoning result of the reasoning task based on multiple reasoning sub-results.

[0006] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0007] A fourth aspect of the present disclosure also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0008] According to an embodiment of the present disclosure, the deep neural network model is divided into a terminal-side sub-model and a cloud-side sub-model according to the computational complexity of each network layer and the inter-layer dependency in the deep neural network model. The segmentation of the model realizes the refined management and scheduling of the model. The i-1th reasoning sub-task is processed based on the terminal-side sub-model to obtain the i-1th intermediate reasoning sub-result, and the i-th reasoning sub-task is started. The reasoning task includes multiple reasoning sub-tasks; the semantic center of the i-1th intermediate reasoning sub-result is matched with the data feature of the reasoning task to obtain the i-1th data separation degree; the i-1th data separation degree is adaptively adjusted based on the i-1th data separation degree. The accuracy of the i-1th intermediate reasoning sub-result is improved to obtain the i-1th intermediate target reasoning sub-result, so as to reduce the impact of pipeline bubbles and network fluctuations on the system reasoning task throughput and latency, which can further reduce the transmission delay while ensuring the accuracy of reasoning; the i-1th intermediate target reasoning sub-result is processed based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task; based on multiple reasoning sub-results, the reasoning result of the reasoning task is determined. Such a segmented execution strategy not only optimizes the data transmission and processing process, but also can preliminarily process sensitive data locally, thereby enhancing data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0010] Figure 1 An application scenario diagram of the end-cloud collaborative reasoning method for model offloading and intelligent quantization according to an embodiment of the present disclosure is shown;

[0011] Figure 2 A flowchart of a method for end-cloud collaborative reasoning of model offloading and intelligent quantization according to an embodiment of the present disclosure is shown;

[0012] Figure 3 a shows a schematic diagram of a directed acyclic graph according to an embodiment of the present disclosure;

[0013] Figure 3 b shows a schematic diagram of a chain structure diagram with a first-level virtual node according to an embodiment of the present disclosure;

[0014] Figure 3 c shows a schematic diagram of the first-level subgraph in the chain structure diagram according to an embodiment of the present disclosure;

[0015] Figure 3 d shows a schematic diagram of a second-level subgraph according to an embodiment of the present disclosure;

[0016] Figure 3 e shows a schematic diagram of adaptively adjusting the network layer to reduce bubbles according to an embodiment of the present disclosure;

[0017] Figure 4 A reasoning flow chart of a terminal-cloud collaborative reasoning method based on model offloading and intelligent quantization according to an embodiment of the present disclosure is shown;

[0018] Figure 5 A reasoning flow chart of three-stage coordination with pipeline scheduling according to an embodiment of the present disclosure is shown;

[0019] Figure 6 a is a schematic diagram showing the inference latency of a ResNet101 network using a low-performance device NVIDIA according to an embodiment of the present disclosure;

[0020] Figure 6 b is a schematic diagram showing the inference latency of a VGG16 network using a low-performance device NVIDIA according to an embodiment of the present disclosure;

[0021] Figure 6 c is a schematic diagram showing the inference delay of the ResNet101 network using the low-performance device TX2 according to an embodiment of the present disclosure;

[0022] Figure 6 d is a schematic diagram showing the inference latency of a VGG16 network using a low-performance device TX2 according to an embodiment of the present disclosure;

[0023] Figure 7 a is a schematic diagram showing the inference throughput of the ResNet101 network using a low-performance device NVIDIA under different bandwidth conditions according to an embodiment of the present disclosure;

[0024] Figure 7 b is a schematic diagram showing the inference throughput of the VGG16 network using a low-performance device NVIDIA under different bandwidth conditions according to an embodiment of the present disclosure;

[0025] Figure 7 c is a schematic diagram showing the inference throughput of the ResNet101 network under different bandwidth conditions using the low-performance device TX2 according to an embodiment of the present disclosure;

[0026] Figure 7 d is a schematic diagram showing the inference latency of a VGG16 network using a low-performance device TX2 according to an embodiment of the present disclosure;

[0027] Figure 8 a shows the change in inference throughput of different technical solutions under a situation of dynamic network change according to an embodiment of the present disclosure.

[0028] Figure 8 b shows the change in inference throughput of different technical solutions under another situation of dynamic network change according to an embodiment of the present disclosure.

[0029] Fig. 9 A structural block diagram of a device-cloud collaborative reasoning apparatus for model offloading and intelligent quantization according to an embodiment of the present disclosure is shown; and

[0030] Fig.10 A block diagram of an electronic device suitable for implementing an end-cloud collaborative reasoning method for model offloading and intelligent quantization according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0032] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0033] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0034] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0035] In the existing computing paradigm, traditional computing tasks usually involve uploading data to a remote cloud server for processing, and then returning the results to the end device after processing. This model faces many challenges, including high computing latency, large network bandwidth consumption, and potential data privacy leakage risks. These problems are particularly prominent today, especially as the amount of data increases dramatically and privacy protection requirements are strengthened.

[0036] Therefore, the end-cloud collaborative reasoning computing paradigm was proposed, aiming to solve the limitations of the traditional cloud computing model by optimizing resource allocation and utilization efficiency. Under this paradigm, the deep learning model is divided into two parts: the end-side sub-model and the cloud-side sub-model. The end-side sub-model is deployed on the end device close to the data source, which can respond quickly and effectively reduce the response time and network bandwidth requirements for preliminary data processing. The cloud-side sub-model is deployed on a cloud server with powerful computing power, responsible for handling more complex computing tasks, optimizing computing load and improving overall processing capabilities. In addition, end-cloud collaborative reasoning also involves the collaborative work of the end side and the cloud side, so that the deep learning model can be processed in a distributed manner. This processing method allows the terminal device to first execute the end-side sub-model, process and generate intermediate results, and then send these intermediate results to the cloud server. After receiving the intermediate results, the cloud server continues to execute the remaining part of the cloud-side sub-model and returns the final calculation results to the end device.

[0037] In order to improve the throughput of collaborative reasoning tasks and reduce reasoning latency, existing end-cloud collaborative reasoning methods mainly use compressed transmitted intermediate data and dynamic scheduling. Despite this, the end-cloud collaborative reasoning computing paradigm still faces some problems in practical applications. For example, due to the complexity of deep learning models and inter-layer dependencies, current methods cannot effectively reduce the idle waiting time of devices, which leads to increased reasoning task latency and reduced reasoning efficiency. There is also a lack of overall optimization strategies. In traditional collaborative reasoning methods, optimization techniques such as model segmentation, transmission compression, and dynamic scheduling are usually performed independently. There is a lack of a unified overall optimization strategy, making it difficult to achieve approximate zero-bubble optimization of reasoning tasks.

[0038] In view of this, the present disclosure provides a method, device, electronic device and medium for end-cloud collaborative reasoning of model offloading and intelligent quantization, the method comprising: dividing the deep neural network model into an end-side sub-model and a cloud-side sub-model according to the computational complexity of each network layer and the inter-layer dependency relationship in the deep neural network model; processing the i-1th reasoning sub-task based on the end-side sub-model, obtaining the i-1th intermediate reasoning sub-result, and starting the i-th reasoning sub-task, the end-side device where the end-side sub-model is located has i intermediate reasoning sub-results cached, the reasoning task includes multiple reasoning sub-tasks, i>1; processing the i-1th intermediate reasoning sub-task based on the end-side sub-model, obtaining the i-1th intermediate reasoning sub-result, and starting the i-th reasoning sub-task. The semantic center of the reasoning sub-result is matched with the data features of the reasoning task to obtain the i-1th data separation degree, the semantic center represents the matrix features of the i-1th intermediate reasoning sub-result, and the data features represent the semantic attributes of the input data related to the reasoning task; the accuracy of the i-1th intermediate reasoning sub-result is adjusted based on the i-1th data separation degree to obtain the i-1th intermediate target reasoning sub-result; the i-1th intermediate target reasoning sub-result is processed based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task; and based on multiple reasoning sub-results, the reasoning result of the reasoning task is determined.

[0039] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] Figure 1 An application scenario diagram of the end-cloud collaborative reasoning method for model offloading and intelligent quantization according to an embodiment of the present disclosure is shown.

[0041] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0042] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0043] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0044] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0045] It should be noted that the end-cloud collaborative reasoning method for model offloading and intelligent quantification provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the end-cloud collaborative reasoning device for model offloading and intelligent quantification provided in the embodiment of the present disclosure can generally be set in the server 105. The end-cloud collaborative reasoning method for model offloading and intelligent quantification provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the end-cloud collaborative reasoning device for model offloading and intelligent quantification provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0046] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0047] The following will be based on Figure 1 The scene described by Figure 2 , 3 a, 3b, 3c, 3d, 3e, 4, 5, 6a, 6b, 6c, 6d, 7a, 7b, 7c, 7d, 8a and 8b describe in detail the end-cloud collaborative reasoning method of model offloading and intelligent quantization of the disclosed embodiment.

[0048] Figure 2 A flowchart of the end-cloud collaborative reasoning method for model offloading and intelligent quantization according to an embodiment of the present disclosure is shown.

[0049] like Figure 2 As shown, the end-cloud collaborative reasoning method of model offloading and intelligent quantization in this embodiment includes operations S210 to S260.

[0050] In operation S210, the deep neural network model is divided into a device-side sub-model and a cloud-side sub-model according to the computational complexity of each network layer and the inter-layer dependency in the deep neural network model.

[0051] In operation S220 , the i-1th reasoning subtask is processed based on the terminal-side sub-model, an i-1th intermediate reasoning subresult is obtained, and the i-th reasoning subtask is started.

[0052] In operation S230 , the semantic center of the i-1th intermediate reasoning sub-result is matched with the data feature of the reasoning task to obtain the i-1th data separation degree.

[0053] In operation S240 , the accuracy of the i-1th intermediate inference sub-result is adjusted based on the i-1th data separation degree to obtain the i-1th intermediate target inference sub-result.

[0054] In operation S250, the i-1th intermediate target reasoning sub-result is processed based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task.

[0055] In operation S260 , a reasoning result of the reasoning task is determined based on the plurality of reasoning sub-results.

[0056] According to an embodiment of the present disclosure, a deep neural network model may include multiple network layers. The deep neural network model represents a type of machine learning model composed of multiple layers of neural networks, such as VGG16, ResNet101, etc. VGG16 is a widely used classic convolutional neural network, and ResNet101 is a deep residual network, but it is not limited to this. The embodiment of the present disclosure does not limit the type of specific model. The computational complexity can represent the amount of computation required for the model when performing forward propagation. The inter-layer dependency can be expressed as the connection and interaction between different network layers. The inter-layer dependency may include data flow, feature transfer, etc., but it is not limited to this. The embodiment of the present disclosure does not specifically limit the specific dependency of the inter-layer dependency. The deep neural network model can be divided into an end-side sub-model and a cloud-side sub-model according to the computational complexity and inter-layer dependency of each network layer in the deep neural network model to achieve computational load balancing between the end-side device and the cloud-side device.

[0057] According to an embodiment of the present disclosure, the end-side device where the end-side sub-model is located caches i intermediate reasoning sub-results, and the reasoning task includes multiple reasoning sub-tasks, i>1. The i-1th reasoning sub-task can be processed based on the end-side sub-model to obtain the i-1th intermediate reasoning sub-result, and the i-th reasoning sub-task can be started. The reasoning tasks of the end-side sub-model and the cloud-side sub-model do not need to wait for the entire task to be processed before executing the next task like the traditional reasoning process. Instead, after the end-side sub-model has processed the reasoning sub-task and is in an idle state, the end-side sub-model can be used immediately to process the next reasoning sub-task, thereby improving the processing efficiency of the reasoning task and reducing the processing delay of the reasoning task.

[0058] According to an embodiment of the present disclosure, the semantic center represents the matrix features of the i-1th intermediate reasoning sub-result, and the data features represent the semantic attributes of the input data related to the reasoning task. The semantic center of the i-1th intermediate reasoning sub-result can be matched with the data features of the reasoning task to obtain the i-1th data separation degree.

[0059] According to an embodiment of the present disclosure, the lower the data separation, the closer the reasoning subtask is to the semantic center, and the easier it is for the reasoning process to distinguish the task results. The precision of the i-1th intermediate reasoning subresult can be adjusted based on the i-1th data separation to obtain the i-1th intermediate target reasoning subresult. Where appropriate, the precision of the i-1th intermediate reasoning subresult can be reduced to improve data transmission efficiency.

[0060] According to an embodiment of the present disclosure, the i-1th intermediate target reasoning sub-result may be processed based on the cloud-side sub-model to obtain the i-1th reasoning sub-result. The reasoning result of the reasoning task may be determined based on multiple reasoning sub-results.

[0061] According to an embodiment of the present disclosure, the deep neural network model is divided into a terminal-side sub-model and a cloud-side sub-model according to the computational complexity of each network layer and the inter-layer dependency in the deep neural network model. The segmentation of the model realizes the refined management and scheduling of the model. The i-1th reasoning sub-task is processed based on the terminal-side sub-model to obtain the i-1th intermediate reasoning sub-result, and the i-th reasoning sub-task is started. The reasoning task includes multiple reasoning sub-tasks; the semantic center of the i-1th intermediate reasoning sub-result is matched with the data feature of the reasoning task to obtain the i-1th data separation degree; the i-1th data separation degree is adaptively adjusted based on the i-1th data separation degree. The accuracy of the i-1th intermediate reasoning sub-result is improved to obtain the i-1th intermediate target reasoning sub-result, so as to reduce the impact of pipeline bubbles and network fluctuations on the system reasoning task throughput and latency, which can further reduce the transmission delay while ensuring the accuracy of reasoning; the i-1th intermediate target reasoning sub-result is processed based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task; based on multiple reasoning sub-results, the reasoning result of the reasoning task is determined. Such a segmented execution strategy not only optimizes the data transmission and processing process, but also can preliminarily process sensitive data locally, thereby enhancing data security.

[0062] According to an embodiment of the present disclosure, the end-cloud collaborative reasoning method based on model offloading and intelligent quantization also includes: at the i-1th transmission end time of the i-1th transmission task, starting to execute the i-th transmission task, wherein the i-1th transmission task is used to transmit the i-1th intermediate target reasoning sub-result from the end-side device to the cloud-side device according to the target transmission accuracy, and the i-th transmission task is used to transmit the i-th intermediate target reasoning sub-result from the end-side device to the cloud-side device according to the target transmission accuracy, and the target transmission accuracy represents the transmission rate of the intermediate target reasoning sub-result from the end-side device to the cloud-measuring device.

[0063] According to an embodiment of the present disclosure, the i-th transmission task may be started at the i-1th transmission end time of the i-1th transmission task to reduce the idle waiting time of participating devices and improve efficiency.

[0064] According to an embodiment of the present disclosure, the target transmission accuracy can characterize the transmission rate of the intermediate target reasoning sub-result from the end-side device to the cloud-measuring device. The target transmission accuracy can also characterize the transmission bandwidth of the intermediate target reasoning sub-result from the end-side device to the cloud-measuring device. The i-1th transmission task is used to transmit the i-1th intermediate target reasoning sub-result from the end-side device to the cloud-side device according to the target transmission accuracy. The i-th transmission task is used to transmit the i-th intermediate target reasoning sub-result from the end-side device to the cloud-side device according to the target transmission accuracy.

[0065] According to the embodiments of the present disclosure, transmission needs to be performed based on a specific transmission accuracy, which can balance the computing load of the terminal-side device and the cloud-side server and ensure transmission efficiency.

[0066] According to an embodiment of the present disclosure, the target transmission accuracy is determined based on the following operations: determining a directed acyclic graph representing an end-side sub-model and a cloud-side sub-model, wherein the nodes of the directed acyclic graph represent the network layer of the end-side sub-model, or represent the network layer of the cloud-side sub-model, and the edge relationship between the nodes represents the data flow between the network layers; determining a first-level sub-graph with a first-level parallel data stream transmission mode from the directed acyclic graph, the first-level sub-graph including multiple first-level nodes with a first-level parallel data stream transmission mode; using the first-level sub-graph as a first-level virtual node, and updating the directed acyclic graph to a chain structure graph including the first-level virtual node and other nodes; processing the data transmission accuracy of the chain structure graph based on the bisection method to obtain the target transmission accuracy, and the data transmission accuracy represents the data stream transmission rate between at least two network layers.

[0067] Figure 3 a shows a schematic diagram of a directed acyclic graph according to an embodiment of the present disclosure.

[0068] Figure 3 b shows a schematic diagram of a chain structure diagram with a first-level virtual node according to an embodiment of the present disclosure.

[0069] Figure 3 c shows a schematic diagram of the first-level subgraph in the chain structure diagram according to an embodiment of the present disclosure.

[0070] Figure 3 d shows a schematic diagram of a second-level subgraph according to an embodiment of the present disclosure.

[0071] Figure 3 e shows a schematic diagram of adaptively adjusting network layers to reduce bubbles according to an embodiment of the present disclosure.

[0072] like Figure 3As shown in a, a directed acyclic graph representing the end-side sub-model and the cloud-side sub-model is determined. The nodes of the directed acyclic graph can represent the network layer of the end-side sub-model, or can represent the network layer of the cloud-side sub-model. For example, node 1, node 2, node 3, node 5, and node 6 can represent the network layer of the end-side sub-model, and node 4, node 7, node 8, and node 9 can represent the network layer of the cloud-side sub-model. The edge relationship between the nodes can represent the data flow between the network layers. For example, the directed edge between node 1 and node 2 can be represented as transferring the data output result in network layer 1 to network layer 2.

[0073] A first-level subgraph with a first-level parallel data stream transmission mode can be determined from a directed acyclic graph. The parallel data stream transmission mode can be represented as a same starting node having multiple initial degrees, and a same destination node having multiple in-degrees, and a ring topology is formed between the starting node and the destination node to realize the parallel data transmission mode between the starting node and the destination node. Figure 3 As shown in Figure c, the starting node 2 has two initial degrees, and the destination node 8 has two in-degrees. All nodes on the parallel data transmission path including node 2 and node 8 can be determined as the first-level subgraph. The first-level subgraph can include multiple first-level nodes with the first-level parallel data flow transmission mode. The first-level subgraph is used as the first-level virtual node B1, and the directed acyclic graph is updated to a chain structure graph including the first-level virtual node B1 and other nodes (i.e., node 1 and node 9), as shown in Figure 2. Figure 3 As shown in b.

[0074] According to an embodiment of the present disclosure, as shown in 310 in the figure, the conventional method is to execute the data processing processes of network layer 1, network layer 2 and network layer 3 represented by node 1, node 2 and node 3 respectively after the execution is completed. After the data processing result is transmitted to the network layer 4 represented by node 4 at a transmission rate, the network layer 4 represented by node 4 starts to execute. After the data processing result of network layer 5 is transmitted to network layer 7 represented by node 7 at a transmission rate, network layer 7 represented by node 7 starts to execute. After the data processing result of network layer 6 is transmitted to network layer 8 represented by node 8 at a transmission rate of 100,000, the network layer 8 represented by node 8 starts to execute. In this process, the device idle waiting time will be very long and the processing result will be executed intermittently, resulting in a waste of resources.

[0075] In the present disclosure, as shown in 320 in the figure, It represents the time it takes to input data into the network layer 4 represented by the cloud-side node 4 and output the result. It represents the time it takes to input data into network layer 5 and network layer 6 represented by cloud-side nodes 5 and 6 and output the results. Based on the determined target transmission accuracy, it can be seen that and After the target transmission accuracy is determined, the computing load can be balanced and the overall processing capacity can be improved. After the end-side sub-model outputs the intermediate results, based on the determined target transmission accuracy, after a certain period of time, the network layers 4, 7, 8, and 9 corresponding to nodes 4, 7, 8, and 9 can start to execute.

[0076] According to the embodiments of the present disclosure, the method provided by the present disclosure can reduce the idle time of the device. Based on the determined target transmission accuracy, the network layer 4 represented by the node 4 in 321 can be adaptively adjusted to the position of the network layer 4 represented by the node 4 in 322. By adjusting the data stream transmission rate between at least two network layers, after all the end-side output results are obtained, the results can be uniformly input into the network layer on the cloud side for processing, thereby reducing the idle waiting time of the device.

[0077] The data transmission accuracy of the chain structure diagram can be processed based on the dichotomy method to obtain the target transmission accuracy. The data transmission accuracy represents the data flow transmission rate between at least two network layers.

[0078] According to an embodiment of the present disclosure, data transmission accuracy can characterize the data flow transmission rate between at least two network layers. The binary method can represent a search algorithm for finding a data transmission accuracy that meets a preset condition in an ordered data transmission accuracy. The data transmission accuracy of the chain structure diagram can be processed based on the binary method to obtain a target transmission accuracy.

[0079] According to the embodiments of the present disclosure, by determining the target transmission accuracy, not only can the reasoning speed be improved, but also the resource waste in the execution process can be reduced, and the time difference between the calculation and transmission stages can be effectively reduced. The introduction of virtual nodes can effectively simplify the reasoning structure of the model, simplifying its originally complex directed acyclic graph into a chain structure, so as to accurately control the parallel execution between model levels and ensure the minimum pipeline bubbles.

[0080] According to an embodiment of the present disclosure, a directed acyclic graph includes multiple level 2 nodes having a level 2 parallel data stream transmission mode, a topological structure representing the parallel data transmission mode between the multiple level 2 nodes is determined as a level 2 subgraph, and the level 1 subgraph includes a level 2 virtual node representing the level 2 subgraph.

[0081] According to an embodiment of the present disclosure, a topological structure representing a parallel data transmission mode between a plurality of level 2 nodes is determined as a level 2 subgraph, such as Figure 3d, with the same starting node 5 and the same in-degree node 8, all nodes on the parallel path including node 5 and node 8 can be determined as the second-level subgraph. , The directed acyclic graph can include multiple second-level nodes with a second-level parallel data stream transmission mode. The first-level subgraph can include a second-level virtual node B2 representing the second-level subgraph.

[0082] According to an embodiment of the present disclosure, the data transmission accuracy of the chain structure graph is processed based on the dichotomy method to obtain the target transmission accuracy, including: processing the second-level data transmission accuracy of the second-level subgraph based on the dichotomy method to obtain the second-level target transmission accuracy, the second-level data transmission accuracy characterizing the inter-layer data flow transmission efficiency of parallel data flow transmission from the same network layer in the second-level subgraph to different next-level network layers; determining the first-level parallel transmission node having the same starting node and in-degree node as the second-level virtual node from the first-level subgraph; determining the first-level data transmission accuracy of the first-level parallel transmission node based on the second-level target transmission accuracy; processing the first-level data transmission accuracy based on the dichotomy method to obtain the target transmission accuracy.

[0083] According to the embodiments of the present disclosure, the chain structure diagram, virtual node B1 and virtual node B2 can be processed separately based on the dichotomy method to obtain the corresponding target transmission accuracy. The second-level data transmission accuracy can characterize the inter-layer data flow transmission efficiency of parallel data flow transmission from the same network layer in the second-level subgraph to different next-level network layers.

[0084] According to the embodiments of the present disclosure, the second-level data transmission accuracy of the second-level subgraph can be processed based on the binary method to obtain the second-level target transmission accuracy. Then, from the first-level subgraph, the first-level parallel transmission node with the same starting node and in-degree node as the second-level virtual node can be determined. Based on the second-level target transmission accuracy, the first-level data transmission accuracy of the first-level parallel transmission node can be determined; based on the binary method processing of the first-level data transmission accuracy, the target transmission accuracy can be obtained.

[0085] According to an embodiment of the present disclosure, a divide-and-conquer strategy is used on the chain structure diagram to test the quantization accuracy of different bits of intermediate data in each layer to obtain the optimal quantization accuracy under the accuracy constraint. Finally, a model partitioning scheme and corresponding transmission accuracy are determined that can balance the computing load of the terminal-side device and the cloud-side server while ensuring efficient transmission.

[0086] According to an embodiment of the present disclosure, the second-level data transmission accuracy of the second-level subgraph is processed based on the binary method to obtain the second-level target transmission accuracy, including: transmitting data to the second-level subgraph with a first preset transmission accuracy and a second preset transmission accuracy, respectively, wherein the first preset transmission accuracy is greater than the second preset transmission accuracy, the first preset transmission accuracy corresponds to the first transmission loss rate, and the second preset transmission accuracy corresponds to the second transmission loss rate; using the binary method to process the first preset transmission accuracy and the second preset transmission accuracy to obtain a first intermediate transmission accuracy, the first intermediate transmission accuracy corresponds to the first intermediate transmission loss rate; from the first intermediate transmission loss rate , the first transmission loss rate and the second transmission loss rate, determine the first target transmission loss rate and the second target transmission loss rate that meet the preset conditions; use the dichotomy method to process the first target accuracy corresponding to the first target transmission loss rate and the second target accuracy corresponding to the second target transmission loss rate to obtain a second intermediate transmission accuracy, and the second intermediate transmission accuracy corresponds to the second intermediate transmission loss rate; determine the second-level target transmission loss rate that meets the preset loss rate threshold from the first target transmission loss rate, the second target transmission loss rate and the second intermediate transmission loss rate; determine the transmission accuracy corresponding to the second-level target transmission loss rate as the second target transmission accuracy.

[0087] According to an embodiment of the present disclosure, for example, the first preset transmission accuracy is 20 megabits per second (Mbps), and the second preset transmission accuracy is 4 Mbps. Transmission at 20 Mbps corresponds to a first transmission loss rate of 55%, and transmission at 4 Mbps corresponds to a second transmission loss rate of 15%. Data can be transmitted to the second-level subgraph at 20 Mbps and 4 Mbps, respectively. By using binary search to process 20 Mbps and 4 Mbps, a first intermediate transmission accuracy of 12 Mbps can be obtained. Transmission at 12 Mbps corresponds to a first intermediate transmission loss rate of 35%. From the first intermediate transmission loss rate of 35%, the first transmission loss rate of 55%, and the second transmission loss rate of 15%, the first target transmission loss rate of 15% and the second target transmission loss rate of 35% that meet the preset conditions can be determined. The binary search method is used to process the first target transmission loss rate of 15%. The first target accuracy of 4 megabits per second and the second target accuracy of 12 megabits per second corresponding to the second target transmission loss rate of 35% can obtain a second intermediate transmission accuracy of 8 megabits per second, and transmission at 8 megabits per second corresponds to a second intermediate transmission loss rate of 20%; the second-level target transmission loss rate of 15% that meets the preset loss rate threshold is determined from the first target transmission loss rate of 15%, the second target transmission loss rate of 35% and the second intermediate transmission loss rate of 20%; the transmission accuracy of 4 megabits per second corresponding to the second-level target transmission loss rate of 15% can be determined as the second target transmission accuracy.

[0088] According to the embodiments of the present disclosure, the dichotomy method can be used to quickly decide the partitioning strategy of the model with a time complexity of O(log(n)). In each round of iteration, the system not only performs fine-grained partitioning of virtual nodes, but also searches and optimizes the data flow inside the virtual nodes, especially the part with parallel data flow transmission mode calculation. Through the abstraction and aggregation of virtual nodes, it can be transformed into a global optimization goal, and the system processes it through the divide-and-conquer method, realizing the effective partitioning and optimization of the overall data flow.

[0089] According to the embodiments of the present disclosure, in this process, the system continuously improves the quantization accuracy of each layer through iterative optimization of the layer-by-layer quantization relationship. Each round of iteration will re-evaluate the trade-off between the model partitioning strategy and the quantization accuracy to ensure that the final selected quantization scheme can not only meet the accuracy requirements, but also greatly improve the transmission efficiency. This iterative divide-and-conquer algorithm achieves a balanced distribution of computing loads through multiple rounds of optimization, and finally obtains the optimal transmission strategy by gradually adjusting the transmission accuracy. As the iteration deepens, the data flow between virtual nodes gradually tends to the optimal state. The model partitioning strategy finally generated by the system can efficiently collaborate between the end-side device and the cloud server to achieve the optimal configuration of computing resources and transmission bandwidth.

[0090] Ultimately, the system uses this iterative divide-and-conquer algorithm to determine an overall solution that maximizes transmission and computing efficiency while maintaining model reasoning accuracy. This solution performs particularly well in scenarios where end-side device resources are limited, meeting the needs of efficient reasoning.

[0091] According to an embodiment of the present disclosure, based on multiple reasoning sub-results, the reasoning result of the reasoning task is determined, including: when the accuracy of the i-th intermediate target reasoning sub-result is greater than a preset reliability threshold, the i+1-th reasoning sub-task is subjected to an early exit operation; based on the first i reasoning sub-results, the reasoning result of the reasoning task is determined.

[0092] According to the embodiments of the present disclosure, in the traditional framework, the calculation stage of each reasoning subtask needs to execute a complete deep neural network model. An early exit mechanism for reasoning subtasks can be introduced. When the accuracy of the i-th intermediate target reasoning subtask is greater than a preset reliable threshold, the calculation stage is terminated in advance, and the i+1-th reasoning subtask is taken out of operation early. The reasoning result of the reasoning task can be determined based on the first i reasoning subtask results.

[0093] According to the embodiments of the present disclosure, by introducing an early exit mechanism for reasoning, the calculation phase can be ended early, which not only reduces the waste of computing resources but also further improves the efficiency of the reasoning task.

[0094] Figure 4A reasoning flow chart of an end-cloud collaborative reasoning method based on model offloading and intelligent quantization according to an embodiment of the present disclosure is shown.

[0095] like Figure 4 As shown, an inference task S410 can be input, an inference task S420 can be processed based on the end-side sub-model, and an intermediate result can be output. When the accuracy of the intermediate result is greater than the preset reliable threshold, the next inference task is subjected to an early exit operation S430. Based on the previous inference result, the final inference result 460 is determined. When the accuracy of the intermediate result is less than the preset reliable threshold, the intermediate result can be adaptively quantized and adjusted S440. The specific adaptive quantization adjustment operation S440 may include matching the semantic center of the intermediate result with the data feature of the inference task to obtain the data separation degree. Adjust the accuracy of the intermediate result based on the data separation degree to obtain the intermediate target inference result; process the intermediate target inference result S450 based on the cloud-side sub-model to obtain the corresponding inference sub-result, and finally obtain the inference result 460 based on multiple inference sub-results.

[0096] Figure 5 A reasoning flow chart of three-stage coordination with pipeline scheduling according to an embodiment of the present disclosure is shown.

[0097] like Figure 5 As shown in Figure 1, Scheme 1 can represent the idea of ​​introducing pipeline parallel execution. While the calculation phase of a task is in progress, the transmission phase of the next task can be started. This parallel processing significantly reduces the idle waiting time and improves the throughput of the reasoning task. However, due to the inconsistency between the calculation and transmission phases, bubbles may still occur, that is, the transmission phase is completed earlier than the calculation phase, increasing the latency of the reasoning task.

[0098] According to the embodiments of the present disclosure, Solution 2 adopts an end-cloud collaborative reasoning framework based on zero-bubble pipeline optimization. That is, through the offline model offloading optimization module 410, the model is segmented and the transmission accuracy is adjusted, thereby realizing refined management and scheduling of the model. When adjusting the transmission accuracy, a recursive divide-and-conquer algorithm is used to abstract the parallel layer into a virtual node, and the directed acyclic graph is converted into a simple chain structure graph, which accurately controls the parallel execution of the network layer, effectively reducing the time difference between the calculation and transmission stages, thereby greatly reducing the bubble phenomenon of the pipeline.

[0099] According to the embodiment of the present disclosure, Solution 3, based on Solution 2, analyzes the data characteristics of the reasoning task and adaptively adjusts the accuracy of the intermediate reasoning sub-results to reduce the transmission delay. The accuracy requirements of different data are adjusted according to the characteristics of data distribution and data separation.

[0100] According to the embodiments of the present disclosure, Scheme 4 introduces an early exit mechanism for inference based on Scheme 3 to further optimize the efficiency of the inference task. The early exit mechanism for inference allows the i+1th inference task to be exited early when the accuracy of the i-th intermediate target inference sub-result is greater than a preset reliable threshold; the inference result of the inference task is determined based on the first i inference sub-results. This not only reduces the waste of computing resources, but also further improves the efficiency of the inference task.

[0101] Figure 6 a shows a schematic diagram of the inference latency of the ResNet101 network using a low-performance device NVIDIA according to an embodiment of the present disclosure.

[0102] Figure 6 b shows a schematic diagram of the inference latency of the VGG16 network using a low-performance device NVIDIA according to an embodiment of the present disclosure.

[0103] Figure 6 c shows a schematic diagram of the inference delay of the ResNet101 network using the low-performance device TX2 according to an embodiment of the present disclosure.

[0104] Figure 6 d shows a schematic diagram of the inference latency of the VGG16 network using the low-performance device TX2 according to an embodiment of the present disclosure.

[0105] like Figure 6 As shown in a-6b, a large number of simulation experiments are conducted to compare the technical solution of the present disclosure with some existing technical solutions to illustrate the technical effect. The compared technical solutions include Neurosurgeon, DADS, SPINN, and JPS. Among them, Neurosurgeon (NS) is a basic model partitioning method; DADS introduces pipeline reasoning to optimize performance under different loads; SPINN uses dynamic partitioning strategies and early exit mechanisms to reduce reasoning delays; and JPS realizes near-optimal pipeline scheduling in the calculation and transmission stages of terminal devices.

[0106] In the experiment, the ImageNet-100 image dataset and the UCF-101 video dataset can be used. ImageNet-100 contains 100 different object categories and is used to evaluate the natural scene processing capabilities in mobile computing. UCF-101 contains 101 different action categories, and a rate of 20 frames per second can be selected to construct continuous reasoning tasks to simulate real-world dynamic scenes. In order to comprehensively evaluate the technical solutions disclosed in the present invention, complex models including VGG16 and ResNet101 can be used. VGG16 is a widely used classic convolutional neural network, while ResNet101 is a deep residual network, both of which have a wide range of application bases in image and video processing tasks. In addition, each experiment will be repeated 200 times to ensure the accuracy and reliability of the results.

[0107] First, the inference latency of various technical solutions can be tested under different network conditions from 2Mbps to 100Mbps. The experimental results show that the technical solution of the present disclosure can significantly reduce the inference latency under various bandwidth conditions. Compared with the existing solutions such as Neurosurgeon, DADS, SPINN and JPS, the inference latency is reduced by 22.48% to 73.59%. Especially in the scenario of using low-performance devices such as NVIDIA, TX2 and complex DNN models such as ResNet101, the technical solution of the present disclosure shows a more obvious delay reduction advantage and effectively optimizes the complex environment of collaborative reasoning.

[0108] Figure 7 a shows a schematic diagram of the inference throughput of the ResNet101 network using a low-performance device NVIDIA under different bandwidth conditions according to an embodiment of the present disclosure.

[0109] Figure 7 b is a schematic diagram showing the inference throughput of the VGG16 network using a low-performance device NVIDIA under different bandwidth conditions according to an embodiment of the present disclosure.

[0110] Figure 7 c shows a schematic diagram of the inference throughput of the ResNet101 network under different bandwidth conditions using the low-performance device TX2 according to an embodiment of the present disclosure.

[0111] Figure 7 d shows a schematic diagram of the inference latency of the VGG16 network using the low-performance device TX2 according to an embodiment of the present disclosure.

[0112] like Figure 7As shown in a-7d, under low bandwidth conditions, transmission resources often become the bottleneck of the reasoning process. The present disclosure significantly improves the throughput, which is 6.2 times higher than Neurosurgeon and 1.6 times higher than JPS. Under high bandwidth conditions, computing resources become the limiting factor. At this time, the throughput of the technical solution of the present disclosure is 4.5 times higher than that of Neurosurgeon and 1.4 times higher than that of JPS. These data show that the technical solution of the present disclosure not only significantly optimizes the reasoning latency, but also effectively improves the reasoning throughput, demonstrating the ability to optimize resource utilization and reduce system bottlenecks under different network conditions.

[0113] Figure 8 a shows a schematic diagram of the change in inference throughput of different technical solutions under a situation of dynamic network change according to an embodiment of the present disclosure.

[0114] Figure 8 b is a schematic diagram showing the change in inference throughput of different technical solutions under another situation of dynamic network changes according to an embodiment of the present disclosure.

[0115] like Figure 8 As shown in a, the initial bandwidth is 20 Mbps, and then changes to 10 Mbps and 5 Mbps. Figure 8 As shown in Figure b, the initial bandwidth is 100Mbps, and then changes to 50Mbps and 20Mbps. The experimental results show that this method can quickly adapt to changes in network bandwidth and maintain a high throughput rate. Figure 8 In a, when the bandwidth drops from 20Mbps to 10Mbps, the throughput of the present disclosure is 1.3 times higher than that of the leading JPS; when the bandwidth drops from 10Mbps to 5Mbps, this advantage is expanded to 1.4 times. Figure 8 In Figure 2b, the present disclosure can maintain high throughput, and even if the bandwidth drops from 100Mbps to 50Mbps, the throughput is still 1.2 times higher than that of JPS; these results show that the present disclosure can quickly adapt to dynamic network changes, maintain a high reasoning throughput, and has better robustness and stability. In summary, the present disclosure can significantly improve the efficiency of end-cloud collaborative reasoning tasks, reduce reasoning latency, improve reasoning throughput, and ensure the accuracy and robustness of the reasoning system.

[0116] Based on the above-mentioned end-cloud collaborative reasoning method of model offloading and intelligent quantification, the present disclosure also provides an end-cloud collaborative reasoning device of model offloading and intelligent quantification. Fig. 9 The device is described in detail.

[0117] Fig. 9 A structural block diagram of an end-cloud collaborative reasoning device for model offloading and intelligent quantization according to an embodiment of the present disclosure is shown.

[0118] like Fig. 9 As shown, the end-cloud collaborative reasoning device 900 for model offloading and intelligent quantization of this embodiment includes a division module 910, a processing module 920, a matching module 930, an adjustment module 940, an intermediate target processing module 950 and an reasoning result determination module 960.

[0119] The division module 910 is used to divide the deep neural network model into a terminal-side sub-model and a cloud-side sub-model according to the computational complexity and inter-layer dependencies of each network layer in the deep neural network model.

[0120] Processing module 920 is used to process the i-1th reasoning subtask based on the end-side sub-model, obtain the i-1th intermediate reasoning sub-result, and start the i-th reasoning subtask. The end-side device where the end-side sub-model is located caches i intermediate reasoning sub-results. The reasoning task includes multiple reasoning subtasks, i>1.

[0121] The matching module 930 is used to match the semantic center of the i-1th intermediate reasoning sub-result with the data features of the reasoning task to obtain the i-1th data separation degree, the semantic center represents the matrix features of the i-1th intermediate reasoning sub-result, and the data features represent the semantic attributes of the input data related to the reasoning task.

[0122] The adjustment module 940 is used to adjust the accuracy of the i-1th intermediate reasoning sub-result based on the i-1th data separation degree to obtain the i-1th intermediate target reasoning sub-result.

[0123] The intermediate target processing module 950 is used to process the i-1th intermediate target reasoning sub-result based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task.

[0124] The reasoning result determination module 960 is used to determine the reasoning result of the reasoning task based on multiple reasoning sub-results.

[0125] According to an embodiment of the present disclosure, the deep neural network model is divided into a terminal-side sub-model and a cloud-side sub-model according to the computational complexity of each network layer and the inter-layer dependency in the deep neural network model. The segmentation of the model realizes the refined management and scheduling of the model. The i-1th reasoning sub-task is processed based on the terminal-side sub-model to obtain the i-1th intermediate reasoning sub-result, and the i-th reasoning sub-task is started. The reasoning task includes multiple reasoning sub-tasks; the semantic center of the i-1th intermediate reasoning sub-result is matched with the data feature of the reasoning task to obtain the i-1th data separation degree; the i-1th data separation degree is adaptively adjusted based on the i-1th data separation degree. The accuracy of the i-1th intermediate reasoning sub-result is improved to obtain the i-1th intermediate target reasoning sub-result, so as to reduce the impact of pipeline bubbles and network fluctuations on the system reasoning task throughput and latency, which can further reduce the transmission delay while ensuring the accuracy of reasoning; the i-1th intermediate target reasoning sub-result is processed based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task; based on multiple reasoning sub-results, the reasoning result of the reasoning task is determined. Such a segmented execution strategy not only optimizes the data transmission and processing process, but also can preliminarily process sensitive data locally, thereby enhancing data security.

[0126] According to an embodiment of the present disclosure, the end-cloud collaborative reasoning device 900 for model offloading and intelligent quantization of this embodiment further includes: an execution module.

[0127] An execution module is used to start executing the i-th transmission task at the i-1th transmission end time of the i-1th transmission task, wherein the i-1th transmission task is used to transmit the i-1th intermediate target reasoning sub-result from the terminal side device to the cloud side device according to the target transmission accuracy, and the i-th transmission task is used to transmit the i-th intermediate target reasoning sub-result from the terminal side device to the cloud side device according to the target transmission accuracy, and the target transmission accuracy represents the transmission rate of the intermediate target reasoning sub-result from the terminal side device to the cloud measurement device.

[0128] According to an embodiment of the present disclosure, the execution module includes: a directed acyclic graph determination unit, a first-level subgraph determination unit, an updating unit and a processing unit.

[0129] The directed acyclic graph determination unit is used to determine a directed acyclic graph representing the end-side sub-model and the cloud-side sub-model, wherein the nodes of the directed acyclic graph represent the network layer of the end-side sub-model or the network layer of the cloud-side sub-model, and the edge relationship between the nodes represents the data flow between the network layers.

[0130] The first-level subgraph determination unit is used to determine a first-level subgraph with a first-level parallel data stream transmission mode from a directed acyclic graph, and the first-level subgraph includes multiple first-level nodes with a first-level parallel data stream transmission mode.

[0131] The updating unit is used to use the first-level subgraph as the first-level virtual node and update the directed acyclic graph into a chain structure graph including the first-level virtual node and other nodes.

[0132] The processing unit is used to process the data transmission accuracy of the chain structure diagram based on the dichotomy method to obtain the target transmission accuracy, and the data transmission accuracy represents the data flow transmission rate between at least two network layers.

[0133] According to an embodiment of the present disclosure, a directed acyclic graph includes multiple level 2 nodes having a level 2 parallel data stream transmission mode, a topological structure representing the parallel data transmission mode between the multiple level 2 nodes is determined as a level 2 subgraph, and the level 1 subgraph includes a level 2 virtual node representing the level 2 subgraph.

[0134] According to an embodiment of the present disclosure, the processing unit includes:

[0135] The second-level target transmission accuracy obtaining subunit is used to process the second-level data transmission accuracy of the second-level subgraph based on the bisection method to obtain the second-level target transmission accuracy. The second-level data transmission accuracy characterizes the inter-layer data flow transmission efficiency of parallel data flow transmission from the same network layer in the second-level subgraph to different next-level network layers.

[0136] The first-level parallel transmission node determination subunit is used to determine, from the first-level subgraph, a first-level parallel transmission node having the same start node and in-degree node as the second-level virtual node.

[0137] The first-level data transmission accuracy determination subunit is used to determine the first-level data transmission accuracy of the first-level parallel transmission node based on the second-level target transmission accuracy.

[0138] The target transmission accuracy obtaining subunit is used for processing the first level data transmission accuracy based on the dichotomy method to obtain the target transmission accuracy.

[0139] According to an embodiment of the present disclosure, the second-level target transmission accuracy obtaining subunit includes: transmitting data to the second-level subgraph with a first preset transmission accuracy and a second preset transmission accuracy respectively, wherein the first preset transmission accuracy is greater than the second preset transmission accuracy, the first preset transmission accuracy corresponds to the first transmission loss rate, and the second preset transmission accuracy corresponds to the second transmission loss rate. The first preset transmission accuracy and the second preset transmission accuracy are processed by binary division to obtain a first intermediate transmission accuracy, and the first intermediate transmission accuracy corresponds to the first intermediate transmission loss rate. From the first intermediate transmission loss rate, the first transmission loss rate, and the second transmission loss rate, the first target transmission loss rate and the second target transmission loss rate that meet the preset conditions are determined. The first target accuracy corresponding to the first target transmission loss rate and the second target accuracy corresponding to the second target transmission loss rate are processed by binary division to obtain a second intermediate transmission accuracy, and the second intermediate transmission accuracy corresponds to the second intermediate transmission loss rate. The second-level target transmission loss rate that meets the preset loss rate threshold is determined from the first target transmission loss rate, the second target transmission loss rate, and the second intermediate transmission loss rate. The transmission accuracy corresponding to the second-level target transmission loss rate is determined as the second target transmission accuracy.

[0140] According to an embodiment of the present disclosure, the inference result determination module 960 includes: an early exit operation unit and an inference result determination unit.

[0141] The early exit operation unit is used to take an early exit operation on the i+1th reasoning subtask when the accuracy of the i-th intermediate target reasoning subtask is greater than a preset reliability threshold.

[0142] The reasoning result determination unit is used to determine the reasoning result of the reasoning task based on the first i reasoning sub-results.

[0143] According to an embodiment of the present disclosure, any multiple modules of the partitioning module 910, the processing module 920, the matching module 930, the adjustment module 940, the intermediate target processing module 950 and the reasoning result determination module 960 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the partitioning module 910, the processing module 920, the matching module 930, the adjustment module 940, the intermediate target processing module 950 and the reasoning result determination module 960 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Or at least one of the partitioning module 910, the processing module 920, the matching module 930, the adjustment module 940, the intermediate target processing module 950 and the reasoning result determination module 960 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0144] Fig.10 A block diagram of an electronic device suitable for implementing an end-cloud collaborative reasoning method for model offloading and intelligent quantization according to an embodiment of the present disclosure is shown.

[0145] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 to a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0146] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0147] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the I / O interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0148] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0149] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0150] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the end-cloud collaborative reasoning method of model offloading and intelligent quantization provided by the embodiment of the present disclosure.

[0151] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1001. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0152] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0153] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0154] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0155] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for end-cloud collaborative reasoning based on model offloading and intelligent quantization, characterized in that: The method comprises: Dividing the deep neural network model into a terminal-side sub-model and a cloud-side sub-model according to the computational complexity of each network layer and the inter-layer dependency relationship in the deep neural network model; Processing the i-1th reasoning subtask based on the end-side sub-model, obtaining the i-1th intermediate reasoning sub-result, and starting the i-th reasoning subtask, the end-side device where the end-side sub-model is located has i intermediate reasoning sub-results cached, the reasoning task includes multiple reasoning subtasks, i>1; Matching the semantic center of the i-1th intermediate inference sub-result with the data features of the inference task to obtain the i-1th data separation degree, wherein the semantic center represents the matrix features of the i-1th intermediate inference sub-result, and the data features represent the semantic attributes of the input data related to the inference task; Adjusting the accuracy of the i-1th intermediate inference sub-result based on the i-1th data separation degree to obtain the i-1th intermediate target inference sub-result; Processing the i-1th intermediate target reasoning sub-result based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task; and Based on the multiple inference sub-results, a reasoning result of the reasoning task is determined.

2. The method according to claim 1, characterized in that The method further comprises: At the i-1th transmission end time of the i-1th transmission task, start executing the i-1th transmission task, wherein the i-1th transmission task is used to transmit the i-1th intermediate target inference sub-result from the end-side device to the cloud-side device according to the target transmission accuracy, and the i-th transmission task is used to transmit the i-th intermediate target inference sub-result from the end-side device to the cloud-side device according to the target transmission accuracy, and the target transmission accuracy represents the transmission rate of the intermediate target inference sub-result from the end-side device to the cloud-measuring device.

3. The method according to claim 2, characterized in that The target transmission accuracy is determined based on the following operations: Determine a directed acyclic graph representing the terminal-side sub-model and the cloud-side sub-model, wherein the nodes of the directed acyclic graph represent the network layer of the terminal-side sub-model or the network layer of the cloud-side sub-model, and the edge relationship between the nodes represents the data flow between the network layers; Determining a first-level subgraph having a first-level parallel data stream transmission mode from the directed acyclic graph, wherein the first-level subgraph includes a plurality of first-level nodes having a first-level parallel data stream transmission mode; Using the first-level subgraph as a first-level virtual node, updating the directed acyclic graph to a chain structure graph including the first-level virtual node and other nodes; The data transmission accuracy of the chain structure diagram is processed based on the dichotomy method to obtain the target transmission accuracy, and the data transmission accuracy represents the data flow transmission rate between at least two of the network layers.

4. The method according to claim 3, characterized in that: The directed acyclic graph includes multiple level 2 nodes with a level 2 parallel data stream transmission mode, and the topological structure representing the parallel data transmission mode between the multiple level 2 nodes is determined as a level 2 subgraph, and the level 1 subgraph includes a level 2 virtual node representing the level 2 subgraph.

5. The method according to claim 4, characterized in that The processing of the data transmission accuracy of the chain structure diagram based on the dichotomy method to obtain the target transmission accuracy includes: Processing the second-level data transmission accuracy of the second-level subgraph based on the dichotomy method to obtain the second-level target transmission accuracy, wherein the second-level data transmission accuracy represents the inter-layer data flow transmission efficiency of parallel data flow transmission from the same network layer in the second-level subgraph to different next-level network layers; From the first-level subgraph, determining a first-level parallel transmission node having the same start node and in-degree node as the second-level virtual node; Determining the first-level data transmission accuracy of the first-level parallel transmission node based on the second-level target transmission accuracy; The first-level data transmission accuracy is processed based on the dichotomy method to obtain the target transmission accuracy.

6. The method according to claim 5, characterized in that The step of processing the second-level data transmission accuracy of the second-level subgraph based on the dichotomy method to obtain the second-level target transmission accuracy includes: Transmitting data to the second-level subgraph with a first preset transmission accuracy and a second preset transmission accuracy respectively, wherein the first preset transmission accuracy is greater than the second preset transmission accuracy, the first preset transmission accuracy corresponds to a first transmission loss rate, and the second preset transmission accuracy corresponds to a second transmission loss rate; Processing the first preset transmission accuracy and the second preset transmission accuracy by using the dichotomy method to obtain a first intermediate transmission accuracy, wherein the first intermediate transmission accuracy corresponds to a first intermediate transmission loss rate; Determining a first target transmission loss rate and a second target transmission loss rate that meet a preset condition from the first intermediate transmission loss rate, the first transmission loss rate, and the second transmission loss rate; Processing a first target accuracy corresponding to the first target transmission loss rate and a second target accuracy corresponding to the second target transmission loss rate by using the dichotomy method to obtain a second intermediate transmission accuracy, wherein the second intermediate transmission accuracy corresponds to the second intermediate transmission loss rate; Determine a second-level target transmission loss rate that meets a preset loss rate threshold from the first target transmission loss rate, the second target transmission loss rate, and the second intermediate transmission loss rate; The transmission accuracy corresponding to the second-level target transmission loss rate is determined as the second target transmission accuracy.

7. The method according to claim 1, characterized in that The determining the reasoning result of the reasoning task based on the multiple reasoning sub-results includes: When the accuracy of the result of the i-th intermediate target reasoning subtask is greater than the preset reliability threshold, the i+1-th reasoning subtask is retired early; Based on the first i reasoning sub-results, a reasoning result of the reasoning task is determined.

8. A device-cloud collaborative reasoning device based on model offloading and intelligent quantization, characterized in that: The device comprises: A partitioning module, configured to partition the deep neural network model into a terminal-side sub-model and a cloud-side sub-model according to the computational complexity of each network layer and the inter-layer dependency relationship in the deep neural network model; a processing module, configured to process the i-1th reasoning subtask based on the end-side sub-model, obtain the i-1th intermediate reasoning sub-result, and start the i-th reasoning subtask, wherein the end-side device where the end-side sub-model is located has i intermediate reasoning sub-results cached, and the reasoning task includes multiple reasoning subtasks, i>1; A matching module, used for matching the semantic center of the i-1th intermediate reasoning sub-result with the data features of the reasoning task to obtain the i-1th data separation degree, wherein the semantic center represents the matrix features of the i-1th intermediate reasoning sub-result, and the data features represent the semantic attributes of the input data related to the reasoning task; an adjustment module, configured to adjust the accuracy of the i-1th intermediate inference sub-result based on the i-1th data separation degree to obtain the i-1th intermediate target inference sub-result; an intermediate target processing module, configured to process the i-1th intermediate target reasoning sub-result based on the cloud-side sub-model to obtain the i-1th reasoning sub-result corresponding to the i-1th reasoning sub-task; and The reasoning result determination module is used to determine the reasoning result of the reasoning task based on multiple reasoning sub-results.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images