Recovery method, controller and recovery system for cloud side end DNN collaborative reasoning interruption

Abstract

The invention provides a cloud side end DNN collaborative reasoning interruption-oriented recovery method, a controller and a recovery system, and the method comprises the steps: if resource service interruption of an edge computing component and / or a cloud computing component is monitored in the reasoning process of a collaborative reasoning system for a DNN model, on the basis of the principle of preferentially selecting the edge computing component and the cloud computing component, the resource service interruption of the edge computing component and / or the cloud computing component is monitored; selecting a component of which the current resource service is not interrupted in the collaborative reasoning system as a current main control component; and on the basis of DNN model data pre-stored in the main control component, constructing a target reasoning recovery strategy for the current available resource quantity of the main control component, so as to recover reasoning for the DNN model in the collaborative reasoning system on the basis of the strategy. According to the method, automatic recovery of cooperative reasoning can be realized for interrupted cloud side end DNN cooperative reasoning, the efficiency and reliability of reasoning recovery can be effectively improved, and the reasoning reliability of reasoning can be ensured.

Description

Technical Field

[0001] The present application relates to the field of edge computing technology, and in particular to a recovery method, controller, and recovery system for cloud-edge DNN collaborative reasoning interruptions. Background Art

[0002] As a key technology of artificial intelligence, Deep Neural Network (DNN) has been widely used to realize image recognition, audio and video processing, natural language processing, etc. With the continuous integration of IoT and artificial intelligence technologies, it has become a trend to carry out DNN reasoning in device terminals to realize intelligent applications, such as autonomous driving and smart assistants. However, due to the limited computing power of device terminals, it is difficult for device terminals themselves to support DNN reasoning with high computing power requirements. Collaborative reasoning technology based on edge computing and cloud computing resources has emerged. This technology splits the reasoning calculation into multiple parts, and collaboratively utilizes the computing resources of device terminals, edges, and clouds to achieve low-latency reasoning.

[0003] However, existing collaborative reasoning solutions mainly focus on how to obtain low-latency collaborative reasoning strategies, and do not fully consider the impact of resource service interruptions of edge computing components and cloud computing components on reasoning reliability. When the resource services of edge computing components and / or cloud computing components are interrupted, some or all of the remote resources in the edge computing components and cloud computing components cannot be used, resulting in the failure of collaborative reasoning, and the failure of the reasoning tasks corresponding to the sample reasoning in the device terminal component, resulting in a decrease in the reliability and efficiency of collaborative reasoning.

[0004] Based on this, there is an urgent need to design a method that can automatically and quickly resume reasoning using available resources when cloud-edge DNN collaborative reasoning is interrupted. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a recovery method, controller and recovery system for cloud-edge DNN collaborative reasoning interruptions to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present application provides a method for recovering interrupted cloud-edge DNN collaborative reasoning, including:

[0007] If, during the reasoning process of the DNN model performed by the collaborative reasoning system composed of the device terminal component, the edge computing component and the cloud computing component, the resource service interruption of the edge computing component and / or the cloud computing component is detected, then based on the priority selection principle, a component whose current resource service is not interrupted is selected in the collaborative reasoning system as the current main control component; wherein the priority selection principle includes: the edge computing component or the cloud computing component is given priority, and the device terminal component is second;

[0008] Based on the DNN model data pre-stored in the main control component, a target reasoning recovery strategy for the collaborative reasoning system is constructed according to the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system whose current resource services are not interrupted based on the target reasoning recovery strategy.

[0009] In some embodiments of the present application, the target reasoning recovery strategy for the collaborative reasoning system is constructed based on the DNN model data pre-stored in the main control component according to the current available resource amount of the main control component, including:

[0010] If the main control component is the device terminal component, the compression model corresponding to the DNN model is extracted from the DNN model library included in the basic capability library preset in the main control component for providing DNN model data, and a target reasoning recovery strategy is generated that uses the compression model to replace the DNN model for reasoning.

[0011] In some embodiments of the present application, the target reasoning recovery strategy for the collaborative reasoning system is constructed based on the DNN model data pre-stored in the main control component according to the current available resource amount of the main control component, including:

[0012] If the main control component is the edge computing component or the cloud computing component, a preset progressive collaborative reasoning strategy optimization algorithm is adopted to construct a target reasoning recovery strategy for the collaborative reasoning system based on the basic capability library preset in the main control component for providing DNN model data, the current available resources of the main control component, the preset resource adjustment amount and the preset reasoning deadline threshold.

[0013] In some embodiments of the present application, if the main control component is the edge computing component or the cloud computing component, a preset progressive collaborative reasoning strategy optimization algorithm is adopted to construct a target reasoning recovery strategy for the collaborative reasoning system according to a basic capability library preset in the main control component for providing DNN model data, the current available resource amount of the main control component, the preset resource adjustment amount, and the preset reasoning deadline threshold, including:

[0014] Extracting each exit branch of the DNN model and the compression model corresponding to the DNN model from the DNN model library included in the basic capability library preset in the main control component, and extracting the DNN layer delay prediction model library contained therein from the basic capability library;

[0015] and, determining a corresponding minimum resource quota and a maximum resource quota according to the currently available resource amount of the main control component;

[0016] Branch selection step: selecting one of the exit branches of the DNN model as the current target exit branch in the order from the last exit branch to the first exit branch of the exit branches of the DNN model;

[0017] Executing a preset progressive collaborative reasoning strategy optimization step for the current target exit branch, so as to obtain corresponding output first result data, second result data or third result data through the progressive collaborative reasoning strategy optimization step;

[0018] If it is known from the first result data output by the progressive collaborative reasoning strategy optimization step that the current reasoning accuracy of the target exit branch is less than the reasoning accuracy of the preset compression model, a target reasoning recovery strategy is generated that uses the compression model to replace the DNN model for reasoning.

[0019] In some embodiments of the present application, the progressive collaborative reasoning strategy optimization step includes:

[0020] Obtaining the branch reasoning accuracy of the current target exit branch, and determining whether the branch reasoning accuracy of the target exit branch is less than the preset reasoning accuracy of the compression model;

[0021] If yes, outputting first result data indicating that the branch reasoning accuracy of the target exit branch is less than the preset reasoning accuracy of the compression model;

[0022] If not, the minimum resource quota is used as the initial target resource allocation amount, and a preset increment step is performed: the target resource allocation amount is incremented with the resource adjustment amount as a unit to form a current target resource allocation amount;

[0023] For the current target resource allocation, the partition point corresponding to the current target exit branch is solved with the goal of minimizing the total reasoning delay;

[0024] Solve the total inference delay corresponding to the collaborative inference strategy composed of the current target resource allocation, target exit branch and partition point based on the DNN layer delay prediction model library;

[0025] Determine whether the total inference delay corresponding to the current collaborative inference strategy is greater than the inference deadline threshold;

[0026] If the total reasoning delay is less than or equal to the reasoning deadline threshold, then output second result data indicating that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch, and the partition point is less than or equal to the reasoning deadline threshold;

[0027] If the total inference delay is greater than the inference deadline threshold, determining whether the current target resource allocation is less than the maximum resource quota;

[0028] If the current target resource allocation is less than the maximum resource quota, return to execute the increment step;

[0029] If the current target resource allocation amount is equal to or greater than the maximum resource quota, the output is used to indicate that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch, and the partition point is greater than the third result data of the reasoning deadline threshold.

[0030] In some embodiments of the present application, the recovery method for cloud-edge DNN collaborative reasoning interruption also includes:

[0031] If the second result data output by the progressive collaborative reasoning strategy optimization step shows that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch and the partition point is less than or equal to the reasoning deadline threshold, then the collaborative reasoning strategy is determined as the target reasoning recovery strategy.

[0032] In some embodiments of the present application, the recovery method for cloud-edge DNN collaborative reasoning interruption also includes:

[0033] If it is known from the third result data output by the progressive collaborative reasoning strategy optimization step that the reasoning accuracy of the current target exit branch is equal to or greater than the reasoning accuracy of the preset compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch and the partition point is greater than the reasoning deadline threshold, then it is determined whether the current target exit branch is the first exit branch. If not, return to execute the branch selection step; if so, generate a target reasoning recovery strategy that uses the compression model to replace the DNN model for reasoning.

[0034] A second aspect of the present application provides a recovery device for cloud-edge DNN collaborative reasoning interruption, including:

[0035] A main control component selection unit is used for, if in the process of performing reasoning on a DNN model by a collaborative reasoning system composed of a device terminal component, an edge computing component and a cloud computing component, a resource service interruption of the edge computing component and / or the cloud computing component is detected, then, based on a priority selection principle, selecting a component whose current resource service is not interrupted in the collaborative reasoning system as the current main control component, wherein the priority selection principle includes: the edge computing component or the cloud computing component is given priority, and the device terminal component is second;

[0036] A target strategy construction unit is used to construct a target reasoning recovery strategy for the collaborative reasoning system based on the DNN model data pre-stored in the main control component and the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system whose current resource services are not interrupted based on the target reasoning recovery strategy.

[0037] The third aspect of the present application provides a controller, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the recovery method for cloud-edge DNN collaborative reasoning interruption when executing the computer program.

[0038] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the recovery method for cloud-edge DNN collaborative reasoning interruptions.

[0039] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the recovery method for cloud-edge DNN collaborative reasoning interruptions.

[0040] A sixth aspect of the present application provides a recovery system for cloud-edge DNN collaborative reasoning interruption, wherein the recovery system for cloud-edge DNN collaborative reasoning interruption is respectively arranged in a device terminal component, an edge computing component, and a cloud computing component;

[0041] The recovery system for cloud-edge DNN collaborative reasoning interruption includes: a controller, a basic capability library and a collaborative reasoning execution module respectively connected to the controller in communication;

[0042] The controller is used to execute the recovery method for cloud-edge DNN collaborative reasoning interruption; the basic capability library is used to store DNN model data; the collaborative reasoning execution module is used to execute reasoning for the DNN model based on the target reasoning recovery strategy generated by the controller.

[0043] In some embodiments of the present application, the recovery system for cloud-edge DNN collaborative reasoning interruption in the device terminal component is also provided with a resource monitoring module that is communicatively connected to the controller, and the resource monitoring module is used to monitor in real time whether there is a resource service interruption of the edge computing component and / or cloud computing component in the collaborative reasoning system, and if so, the corresponding resource service interruption message is sent to the controller in the device terminal component;

[0044] The controller in the device terminal component is used to determine the cloud computing component as the current master control component if the resource service interruption of the edge computing component is learned according to the resource service interruption message, and send a notification message indicating the resource service interruption of the edge computing component and the cloud computing component as the current master control component to the controller in the cloud computing component, so that the controller in the cloud computing component selects the cloud computing component in which it is located as the current master control component based on the notification message;

[0045] The controller in the device terminal component is used to determine the edge computing component as the current master control component if the resource service interruption of the cloud computing component is learned according to the resource service interruption message, and send a notification message indicating the resource service interruption of the cloud computing component and the edge computing component as the current master control component to the controller in the edge computing component, so that the controller in the edge computing component selects the edge computing component where it is located as the current master control component based on the notification message;

[0046] The controller in the device terminal component is used to select the device terminal component where it is located as the current main control component if it is learned from the resource service interruption message that the resource services of the cloud computing component and the edge computing component are both interrupted;

[0047] Among them, the controller in the main control component is used to construct a target reasoning recovery strategy for the collaborative reasoning system based on the DNN model data pre-stored in the main control component for the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system where all current resource services are available and uninterrupted based on the target reasoning recovery strategy.

[0048] The present application provides a recovery method for interrupted cloud-edge DNN collaborative reasoning. If, during the reasoning process of a collaborative reasoning system composed of a device terminal component, an edge computing component and a cloud computing component, a resource service interruption of the edge computing component and / or the cloud computing component is monitored, then based on the principle of priority selection, a component whose current resource service is not interrupted is selected in the collaborative reasoning system as the current main control component; based on the DNN model data pre-stored in the main control component, a target reasoning recovery strategy for the collaborative reasoning system is constructed according to the current available resource amount of the main control component, so that based on the target reasoning recovery strategy, the reasoning for the DNN model is restored between one or more components whose current resource services are not interrupted in the collaborative reasoning system, so as to realize automatic recovery of collaborative reasoning for interrupted cloud-edge DNN collaborative reasoning, and effectively improve the efficiency and reliability of reasoning recovery, thereby ensuring the reasoning reliability of cloud-edge DNN collaborative reasoning.

[0049] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0050] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0052] Figure 1 Schematic diagram of DNN partitioning technology.

[0053] Figure 2 Schematic diagram of early withdrawal technique.

[0054] Figure 3 This is a first flow chart of a method for recovering interruptions of cloud-edge DNN collaborative reasoning in one embodiment of the present application.

[0055] Figure 4This is a second flow chart of a method for recovering interruptions of cloud-edge DNN collaborative reasoning in one embodiment of the present application.

[0056] Figure 5 This is a third flow chart of the recovery method for cloud-edge DNN collaborative reasoning interruption in one embodiment of the present application.

[0057] Figure 6 It is a flowchart of the corresponding progressive collaborative reasoning strategy optimization steps implemented based on the progressive collaborative reasoning strategy optimization algorithm in one embodiment of the present application.

[0058] Figure 7 A flowchart of the recovery process for cloud-edge DNN collaborative reasoning interruptions under different resource interruption scenarios in an application example of the present application.

[0059] Figure 8 This is a structural diagram of a recovery device for cloud-edge DNN collaborative reasoning interruptions in one embodiment of the present application.

[0060] Fig. 9 This is a structural diagram of a recovery system for cloud-edge DNN collaborative reasoning interruptions in one embodiment of the present application.

[0061] Fig.10 This is a schematic diagram of the architecture of a recovery system for cloud-edge DNN collaborative reasoning interruptions in an application example of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0063] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0064] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0065] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0066] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0067] With the development of artificial intelligence technology, DNN-based IoT applications continue to emerge. However, due to the limited computing power of IoT devices, IoT devices themselves are difficult to support DNN reasoning with high computing power requirements. Collaborative reasoning technology came into being. The core idea of ​​collaborative reasoning is to divide the DNN model into multiple parts. The reasoning calculations related to the shallow network part are completed in the device based on local computing resources, and the reasoning calculations related to the deep network part are offloaded to the edge computing components and / or cloud computing components based on remote computing resources. By making full use of the powerful computing resources in the edge computing components and cloud computing components, collaborative reasoning technology enables the device terminal components to complete all reasoning calculations with lower latency and obtain reasoning results. In one or more embodiments of the present application, the device terminal component can be referred to as a device, a device component or a terminal, the edge computing component can be referred to as an edge or an edge component, and the cloud computing component can be referred to as a cloud or a cloud component.

[0068] The two most commonly used techniques in collaborative reasoning are DNN partition and early-exit.

[0069] Schematic diagram of DNN partitioning technology Figure 1 As shown. DNN partitioning technology uses partitioning points to split the DNN model into multiple parts, and different parts are scheduled to different resources for execution. Assuming that a DNN model with an L-layer network is split into device, edge, and cloud execution, two partitioning points P1 and P2 need to be solved according to the optimization goal. After the DNN model is divided, the network layers 1 to P1-1 are executed in the device, the network layers P1 to P2-1 are executed in the edge, and the network layers P2 to L are executed in the cloud. The values ​​of the partitioning points are different under different optimization goals. Similarly, if the DNN model is split into device, edge, or cloud execution, only one partitioning point needs to be solved according to the optimization goal. The optimization goal is usually to minimize the total inference latency.

[0070] Schematic diagram of early withdrawal technique Figure 2As shown. The early exit technology sets multiple exit branches in the DNN model network, and adaptively decides whether to end the reasoning of the current sample in advance according to the reasoning results at the exit of each exit branch. For example, a DNN model with an L-layer network sets three exit branches, the first branch is set to exit at the B1 layer, the second branch is set to exit at the B2 layer, and the third branch is set to exit at the L layer. In each reasoning process of the sample, it is judged at the exit of each branch in the order from shallow to deep whether the sample reasoning result is not lower than the predefined reasoning result threshold of the branch. If the threshold requirement is met, the sample reasoning ends at the corresponding branch. Obviously, the earlier the sample reasoning ends, the less calculation is required. Therefore, combining DNN partitioning with early exit technology can further reduce the total reasoning delay of an reasoning task in collaborative reasoning.

[0071] Although collaborative reasoning technology enables devices to effectively perform DNN reasoning, existing solutions focus on how to use edge and cloud resources for collaborative reasoning, without fully considering the reliability of reasoning. When the resource service of the edge or cloud is interrupted, some or all of the remote resources in the edge and cloud are unavailable, resulting in the failure of collaborative reasoning, the failure of the reasoning task corresponding to the sample reasoning in the device, and the reduction of reasoning reliability.

[0072] Based on this, the embodiments of the present application respectively provide a recovery method for cloud-edge DNN collaborative reasoning interruption, a recovery device for cloud-edge DNN collaborative reasoning interruption for executing the recovery method for cloud-edge DNN collaborative reasoning interruption, a controller, a computer-readable storage medium and a computer product, and a recovery system for cloud-edge DNN collaborative reasoning interruption, which can adopt the "resource service interruption-controller selection-strategy formulation-reasoning recovery" mechanism to realize automatic recovery of reasoning for interrupted cloud-edge DNN collaborative reasoning.

[0073] The details are described in detail through the following examples.

[0074] Based on this, the embodiment of the present application provides a method for recovering cloud-edge DNN collaborative reasoning interruption, which can be implemented by a recovery device for cloud-edge DNN collaborative reasoning interruption. Figure 3 The recovery method for cloud-edge DNN collaborative reasoning interruption specifically includes the following contents:

[0075] Step 100: If, during the reasoning process of the DNN model performed by the collaborative reasoning system composed of the device terminal component, the edge computing component and the cloud computing component, a resource service interruption of the edge computing component and / or the cloud computing component is detected, then based on the priority selection principle, a component whose current resource service is not interrupted is selected in the collaborative reasoning system as the current main control component, wherein the priority selection principle includes: edge computing components or cloud computing components are given priority, followed by device terminal components.

[0076] Specifically, this application adopts the mechanism of "resource service interruption-controller selection-strategy formulation-inference recovery" to continuously monitor the resource status of devices, edges and clouds. When a resource service interruption is detected, it determines which type of interruption situation it is; then, according to the interruption situation, in the order of edge computing components or cloud computing components, device terminal components, a currently available component is selected as the main control component; then the main control component formulates a new reasoning strategy, namely the target reasoning recovery strategy, according to the current resource status, DNN model and DNN network layer delay prediction model; according to the new reasoning strategy, reasoning is restored based on the current resources, thereby ensuring the reasoning reliability of collaborative reasoning.

[0077] It can be understood that the recovery device for cloud-edge DNN collaborative reasoning interruption can be a functional module set in the controller, and the controller can also be called a reliability controller. Reliability controllers are provided in the device terminal component, edge computing component and cloud computing component.

[0078] Wherein, if the controller in the device terminal component learns that the resource service of the edge computing component is interrupted, the controller determines the cloud computing component as the current master control component, and sends a notification message indicating the resource service interruption of the edge computing component and that the cloud computing component is the current master control component to the controller in the cloud computing component, so that the controller in the cloud computing component selects the cloud computing component in which it is located as the current master control component based on the notification message;

[0079] The controller in the device terminal component is used to determine the edge computing component as the current master control component if it is known that the resource service of the cloud computing component is interrupted, and send a notification message indicating the resource service interruption of the cloud computing component and that the edge computing component is the current master control component to the controller in the edge computing component, so that the controller in the edge computing component selects the edge computing component where it is located as the current master control component based on the notification message;

[0080] The controller in the device terminal component is used to select the device terminal component where it is located as the current main control component if it is learned that the resource services of the cloud computing component and the edge computing component are both interrupted;

[0081] Among them, the controller in the main control component is used to construct a target reasoning recovery strategy for the collaborative reasoning system based on the DNN model data pre-stored in the main control component for the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system where all current resource services are available and uninterrupted based on the target reasoning recovery strategy.

[0082] Step 200: Based on the DNN model data pre-stored in the main control component, a target reasoning recovery strategy for the collaborative reasoning system is constructed according to the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system whose current resource services are not interrupted based on the target reasoning recovery strategy.

[0083] In one or more embodiments of the present application, the target reasoning recovery strategy may include at least two categories, one of which is: the target exit branch of the DNN model, the target resource allocation amount and partition point corresponding to the target exit branch; the other is: using the compression model to replace the DNN model for reasoning.

[0084] From the above description, it can be seen that the recovery method for interrupted cloud-edge DNN collaborative reasoning provided in the embodiment of the present application can realize automatic recovery of reasoning for interrupted cloud-edge DNN collaborative reasoning, and can effectively improve the efficiency and reliability of reasoning recovery, thereby ensuring the reasoning reliability of cloud-edge DNN collaborative reasoning.

[0085] In order to further improve the effectiveness and reliability of constructing a target reasoning recovery strategy for the collaborative reasoning system based on the current available resources of the main control component, so as to further improve the efficiency and reliability of reasoning recovery, in a recovery method for cloud-edge DNN collaborative reasoning interruption provided in an embodiment of the present application, see Figure 4 , step 200 in the recovery method for cloud-edge DNN collaborative reasoning interruption specifically includes the following contents:

[0086] Step 210: If the main control component is the device terminal component, extract the compression model corresponding to the DNN model from the DNN model library included in the basic capability library preset in the main control component for providing DNN model data, and generate a target reasoning recovery strategy that uses the compression model to replace the DNN model for reasoning.

[0087] In one or more embodiments of the present application, the basic capability library may include a DNN model library and a DNN layer delay prediction model library. The DNN model library contains different versions of the DNN model used for reasoning, such as: a version that supports early retirement, a version that supports quantization and compression, etc. The DNN network layer delay prediction model library is used to predict the computational delay of each network layer of the DNN model under different resource allocations, and is used to formulate collaborative reasoning strategies.

[0088] In order to achieve collaborative reasoning reliability assurance when collaborative reasoning fails due to resource interruption, in addition to restoring reasoning, it should also ensure that the restored reasoning task can be completed within the deadline. Therefore, the reasoning strategy obtained by the main controller using the reliability assurance algorithm requires, on the one hand, to restore reasoning based on the existing available resources, and on the other hand, to ensure that the total reasoning delay of the reasoning task is no higher than the deadline requirement. At the same time, considering the effectiveness of the reasoning results, the reasoning strategy should also achieve the highest possible reasoning accuracy.

[0089] However, to obtain high inference accuracy, a large amount of inference calculation is usually required, and the total inference delay is correspondingly high; to obtain a low total inference delay, it is usually necessary to reduce the amount of inference calculation, and the inference accuracy is correspondingly low. For example: a compression model with low calculation amount can complete reasoning independently in the device, but the reasoning accuracy will be lower than that of the native model; in the collaborative reasoning implementation based on DNN partitioning and early exit, the total inference delay is reduced when all reasoning tasks end early in shallow branches, but the reasoning accuracy will also be reduced. In addition, the amount of resources allocated to the device by the edge or cloud will also affect the total inference delay. The larger the amount of resource allocation, the smaller the total inference delay. To this end, the embodiment of the present application also proposes a progressive collaborative reasoning strategy optimization algorithm with the goal of restoring reasoning, meeting deadline requirements, and obtaining the highest possible reasoning accuracy. The reliability controllers of the device terminal component, edge computing component, and cloud computing component all use this algorithm as the reliability assurance algorithm by default. Therefore, in a recovery method for cloud-edge DNN collaborative reasoning interruption provided in an embodiment of the present application, see Figure 4 , step 200 in the recovery method for cloud-edge DNN collaborative reasoning interruption specifically includes the following contents:

[0090] Step 220: If the main control component is the edge computing component or the cloud computing component, a preset progressive collaborative reasoning strategy optimization algorithm is adopted to construct a target reasoning recovery strategy for the collaborative reasoning system according to the basic capability library preset in the main control component for providing DNN model data, the current available resources of the main control component, the preset resource adjustment amount and the preset reasoning deadline threshold.

[0091] It can be understood that the progressive collaborative reasoning strategy optimization algorithm refers to an algorithm that optimizes the collaborative reasoning strategy in a progressive manner. Its core idea is to restore reasoning and reach the reasoning deadline threshold by progressively reducing the reasoning accuracy to ensure the reliability of reasoning. In the case where both edge and cloud resources are interrupted, the algorithm chooses to use the compression model to restore reasoning in the device. The reasoning accuracy of the default compression model adopted is relatively low, but the total reasoning delay can meet the reasoning deadline threshold. In the case of edge or cloud interruption, the reasoning deadline threshold is met by gradually optimizing the last exit branch, edge or cloud resource allocation, partitioning points and other decisions, and gradually reducing the reasoning accuracy, increasing resources and optimizing the partitioning points.

[0092] In order to further improve the effectiveness and reliability of constructing a target reasoning recovery strategy for the collaborative reasoning system, and to further improve the efficiency and reliability of reasoning recovery, in a recovery method for cloud-edge DNN collaborative reasoning interruption provided in an embodiment of the present application, see Figure 5 , step 220 in the recovery method for cloud-edge DNN collaborative reasoning interruption specifically includes the following contents:

[0093] Step 221: Extract each exit branch of the DNN model and the compression model corresponding to the DNN model from the DNN model library included in the basic capability library preset in the main control component, and extract the DNN layer delay prediction model library contained therein from the basic capability library.

[0094] And, step 222: determining the corresponding minimum resource quota and maximum resource quota according to the current available resource amount of the main control component.

[0095] Step 223: Branch selection step: In the order from the last exit branch to the first exit branch of the exit branches of the DNN model, select one from the exit branches of the DNN model as the current target exit branch.

[0096] Step 224: Execute a preset progressive collaborative reasoning strategy optimization step for the current target exit branch to obtain the corresponding output first result data, second result data or third result data through the progressive collaborative reasoning strategy optimization step.

[0097] Step 225: If it is known from the first result data output by the progressive collaborative reasoning strategy optimization step that the current reasoning accuracy of the target exit branch is less than the preset reasoning accuracy of the compression model, a target reasoning recovery strategy is generated that uses the compression model to replace the DNN model for reasoning.

[0098] In order to further improve the execution effectiveness and reliability of step 224, and to further improve the efficiency and reliability of reasoning recovery, in a recovery method for cloud-edge DNN collaborative reasoning interruption provided in an embodiment of the present application, see Figure 6 The corresponding progressive collaborative reasoning strategy optimization step implemented based on the progressive collaborative reasoning strategy optimization algorithm in the recovery method for cloud-edge DNN collaborative reasoning interruption specifically includes the following contents:

[0099] Step 001: Obtain the branch reasoning accuracy of the current target exit branch, and determine whether the branch reasoning accuracy of the target exit branch is less than the preset reasoning accuracy of the compression model; if so, execute step 002; if not, execute step 003.

[0100] Step 002: Output first result data for indicating that the branch reasoning accuracy of the target exit branch is less than the preset reasoning accuracy of the compression model.

[0101] Step 003: Using the minimum resource quota as the initial target resource allocation amount.

[0102] Step 004: Execute a preset increment step: taking the resource adjustment amount as a unit, increment the target resource allocation amount to form a current target resource allocation amount.

[0103] Step 005: For the current target resource allocation, the partition point corresponding to the current target exit branch is solved with the goal of minimizing the total reasoning delay.

[0104] Step 006: Based on the DNN layer delay prediction model library, solve the total inference delay corresponding to the collaborative reasoning strategy composed of the current target resource allocation, target exit branch and partition point.

[0105] Step 007: Determine whether the total inference delay corresponding to the current collaborative inference strategy is greater than the inference deadline threshold; if not, execute step 008; if yes, execute step 009.

[0106] Step 008: Output second result data indicating that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch, and the partition point is less than or equal to the reasoning deadline threshold.

[0107] Step 009: Determine whether the current target resource allocation is less than the maximum resource quota; if so, return to step 004; if not, execute step 010;

[0108] Step 010: Output third result data indicating that the reasoning accuracy of the current target exit branch is equal to or greater than the reasoning accuracy of the preset compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch, and the partition point is greater than the reasoning deadline threshold.

[0109] In order to further improve the effectiveness and reliability of constructing a target reasoning recovery strategy for the collaborative reasoning system, and to further improve the efficiency and reliability of reasoning recovery, in a recovery method for cloud-edge DNN collaborative reasoning interruption provided in an embodiment of the present application, see Figure 5 , step 220 in the recovery method for cloud-edge DNN collaborative reasoning interruption further specifically includes the following contents:

[0110] Step 226: If the second result data output by the progressive collaborative reasoning strategy optimization step indicates that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch and the partition point is less than or equal to the reasoning deadline threshold, then the collaborative reasoning strategy is determined as the target reasoning recovery strategy.

[0111] In order to further improve the effectiveness and reliability of constructing a target reasoning recovery strategy for the collaborative reasoning system, and to further improve the efficiency and reliability of reasoning recovery, in a recovery method for cloud-edge DNN collaborative reasoning interruption provided in an embodiment of the present application, see Figure 5 , step 220 in the recovery method for cloud-edge DNN collaborative reasoning interruption further specifically includes the following contents:

[0112] Step 227: If the third result data outputted from the progressive collaborative reasoning strategy optimization step indicates that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy consisting of the target exit branch, the target resource allocation amount corresponding to the target exit branch, and the partition point is greater than the reasoning deadline threshold, then determine whether the current target exit branch is the first exit branch. If not, return to execute the branch selection step target reasoning recovery strategy. If so, execute step 228:

[0113] Step 228: Generate a target reasoning recovery strategy that uses the compression model to replace the DNN model for reasoning.

[0114] In order to further illustrate the progressive collaborative reasoning strategy optimization algorithm in the above-mentioned recovery method for cloud-edge DNN collaborative reasoning interruption, the present application also provides a specific application example of the recovery method for cloud-edge DNN collaborative reasoning interruption, which can be called a collaborative reasoning reliability assurance method for edge-cloud resource service interruption. In this application example, the collaborative reasoning failure caused by edge or cloud resource interruption has the following situations:

[0115] (1) Case 1: The edge is interrupted, but cloud resource services are available. For the reasoning tasks generated on the device side, the cloud can provide collaborative reasoning support, but the edge cannot. In this case, DNN partitioning and early exit techniques can be used to restore collaborative reasoning.

[0116] (2) Case 2: The cloud is interrupted, but edge resource services are available. For the reasoning tasks generated on the device side, the edge can provide collaborative reasoning support, but the cloud cannot. In this case, DNN partitioning and early exit techniques can be used to restore collaborative reasoning.

[0117] (3) Case 3: Both edge and cloud are interrupted. For the inference tasks generated on the device side, neither the edge nor the cloud can provide collaborative inference support. In this case, collaborative inference cannot be carried out, but model compression technology can be used to use the device's own resources to use the DNN quantized compression model to complete the inference task.

[0118] That is to say, in the case of different resource interruptions, the recovery process for cloud-edge DNN collaborative reasoning interruptions provided by this application instance is as follows: Figure 7 shown.

[0119] The symbols used in the progressive reliability assurance algorithm are as follows:

[0120] (1) The inference deadline threshold is denoted as T dl ;The total reasoning delay corresponding to the collaborative reasoning strategy is denoted as ft;

[0121] (2) The inference accuracy of the compression model used by the device terminal component is denoted as A comp ; The branch reasoning accuracy of the target exit branch is A b ;

[0122] (3) The minimum resource quota determined based on the current available resources of the master control component is denoted as X min , the maximum resource quota is recorded as X max ; The preset resource adjustment amount is recorded as X 0 ;

[0123] (4) Collaborative reasoning uses a DNN model based on DNN partitioning and early exit. Assume that the DNN model has a total of B exit branches, and the set of B exit branches is denoted as For the b-th exit branch, 1≤b≤B, B is a positive integer greater than 2, the set Indicates that the b-th exit branch needs to execute all network layers. The union of the sets of network layers that all exit branches need to execute is

[0124] (5) Let the variable mode∈{compression,de-cooperation,dc-cooperation} represent the reasoning recovery mode, where compression, de-cooperation and dc-cooperation represent compression model, end-edge collaborative reasoning and end-cloud collaborative reasoning respectively. The compression model compression is the target reasoning recovery strategy for using the compression model to replace the DNN model for reasoning; the end-edge collaborative reasoning de-cooperation is the target reasoning recovery strategy obtained when the end-edge collaborative reasoning is used in the case of cloud interruption; the end-cloud collaborative reasoning dc-cooperation is the target reasoning recovery strategy obtained when the end-cloud collaborative reasoning is used in the case of edge interruption;

[0125] (6) Let the decision variables x, b and p b They represent the resource allocation of the edge or cloud, the final exit branch (i.e., the target exit branch mentioned above), and the partition point corresponding to the final exit branch. b The constructed collaborative reasoning strategy is the target reasoning recovery strategy when the aforementioned end-edge collaborative reasoning mode or end-cloud collaborative reasoning mode is adopted.

[0126] For edge-end or cloud-end collaborative reasoning, the total reasoning latency includes three parts: local reasoning latency, intermediate result transmission latency, and remote reasoning latency. Since the amount of inference result data is usually very small, the transmission latency of the inference result is not included in the total reasoning latency. b Next, let the function T l (b,p b ) indicates the use of exit branch b and partition point p b The local inference delay at time t is the function T tr (b,p b ,X type ) indicates that the resource type is X type Use exit branch b, partition point p b The intermediate result transmission delay is T r (x,b,p b ,X type ) indicates that the resource type is X type∈{edge,cloud} uses resource allocation x, exit branch b, partition point p b The total inference delay ft of an inference task can be written as T(x,b,p b ,X type ), which can be specifically expressed as:

[0127] T(x,b,p b ,X type )=T l (b,p b )+T tr (b,p b ,X type )+T r (x,b,p b ,X type )

[0128] For different resource interruption situations, the progressive reliability assurance algorithm first determines to use one of the compression model, end-edge collaborative reasoning, or end-cloud collaborative reasoning modes to restore the execution of the reasoning task. Then, for the end-edge collaborative reasoning or end-cloud collaborative reasoning mode, the collaborative reasoning strategy composed of decisions such as optimizing the edge or cloud resource allocation, the final exit branch, and the partitioning point is used to minimize the total reasoning delay to ensure that the reasoning task is completed within the reasoning deadline threshold.

[0129] The pseudo code of the progressive reliability assurance algorithm is shown in Algorithm 1 corresponding to Table 1. The input information includes the branch set of the DNN model and the collection of all branch network layers The output information includes the reasoning recovery mode mode and the collaborative reasoning strategy x, b, p b If mode=compression in the output, no collaborative reasoning strategy is required, that is: x=NULL, b=NULL, p b = NULL. Algorithm 1 uses the progressive collaborative reasoning strategy optimization algorithm to obtain collaborative reasoning strategies that ensure reasoning reliability under different resource types.

[0130] Table 1

[0131]

[0132]

[0133] In Table 1, rows 1 to 3 handle the situation where both the edge and the cloud are interrupted, and use the compression model to restore reasoning. Rows 4 to 7 handle the situation where the edge is interrupted, and call Algorithm 2 to obtain the end-cloud collaborative reasoning strategy. Rows 8 to 12 handle the situation where the cloud is interrupted, and call Algorithm 2 to obtain the end-edge collaborative reasoning strategy.

[0134] Specifically, the pseudo code of the progressive collaborative reasoning strategy optimization algorithm is shown in Algorithm 2 corresponding to Table 2.

[0135] Table 2

[0136]

[0137] The goal of Algorithm 2 is to meet the inference deadline threshold with the highest possible inference accuracy and the least possible edge or cloud resource allocation. Collaborative inference based on DNN partitioning and early exit technology has the following characteristics: the inference accuracy of deep branches is higher than that of shallow branches; the larger the resource allocation, the shorter the total inference delay. Therefore, the core idea of ​​Algorithm 2 is: first, optimize the exit branch, resource allocation, and partition point layer by layer until the total inference delay meets the inference deadline threshold; then, if there is no collaborative inference strategy that meets the inference deadline threshold, use the compressed model to resume inference.

[0138] In Algorithm 2, the input information includes the branch set of the DNN model The collection of all branch network layers Resource Type X type , minimum resource quota X min , Maximum resource quota X max , Resource Adjustment Amount X 0 The output information includes the reasoning recovery mode mode and the collaborative reasoning strategy x, b, p b Algorithm 2 is also applicable to the formulation of edge-end and cloud-end collaborative reasoning strategies. When formulating cloud-end collaborative reasoning strategies, set parameters according to cloud resource information, as shown in line 5 of Algorithm 1, that is, type =cloud,X min =X min,cloud ,X max =X max,cloud ,X 0 =X 0,cloud When formulating the edge-end collaborative reasoning strategy, set the parameters according to the edge resource information, as shown in line 9 of Algorithm 1, that is, type =edge,X min =X min,edge ,X max =X max,edge ,X 0 =X 0,egge .

[0139] In Table 2: The layer-by-layer optimization implementation in Algorithm 2 is shown in rows 1 to 22. The exit branch b to be optimized is determined in the order from the deepest branch B to the shallowest branch 1. After determining branch b, if the branch reasoning accuracy A b Less than the inference accuracy of the compressed model A comp, then set the inference recovery mode to the compression model and end the optimization, as shown in lines 2 to 4; otherwise, continue to optimize the resource allocation and partition points. The resource allocation x is initialized to the minimum resource quota X min , and then adjust the resource amount X 0 After determining the current resource allocation x, the optimal partition point p is solved with the goal of minimizing the total inference delay. b , calculate the current total inference latency ft, as shown in lines 8 to 9. If the inference deadline threshold is not reached, increase the resource allocation and continue optimization, as shown in lines 10 to 11. If the inference deadline threshold is reached, determine whether the inference recovery mode is end-edge or end-cloud collaborative inference based on the resource identifier, and set the current x,b,p b It is returned as the best collaborative reasoning strategy and the optimization is terminated, as shown in lines 12 to 20. If the collaborative reasoning strategies corresponding to all branches cannot meet the reasoning deadline threshold, the reasoning recovery mode is set to the compression model and the optimization is terminated, as shown in lines 23 to 24.

[0140] It should be noted that the symbols in Table 1 and Table 2 are as follows:

[0141] T dl : Indicates the reasoning deadline requirement of the reasoning task.

[0142] The set of B exit branches of the DNN model.

[0143] The bth exit branch needs to execute the set of all network layers.

[0144] The intersection of all B exit branch network layer sets.

[0145] mode: indicates the inference recovery mode. A value of compression indicates a compression model. A value of de-cooperation indicates device-edge collaborative DNN inference. A value of dc-cooperation indicates device-cloud collaborative DNN inference.

[0146] x: indicates the decision on the amount of edge or cloud resources to be allocated for DNN collaborative reasoning. A value of NULL indicates that no edge or cloud resources are required.

[0147] b: indicates the exit branch decision of DNN collaborative reasoning. When the value is NULL, it means that the DNN model is executed locally and no collaborative reasoning is performed.

[0148] p b : Indicates the partition point decision of DNN collaborative reasoning. When the value is NULL, it means that the DNN model is executed locally and no collaborative reasoning is performed.

[0149] X type : The resource type used for DNN collaborative reasoning remote reasoning calculations. When the value is cloud, it indicates cloud resources; when the value is edge, it indicates edge resources.

[0150] X min : indicates the lower limit of remote resources that can be used for remote reasoning computing, that is, the minimum amount of resources. Depending on the resource type, it can take a value of X min,cloud or X min,edge .

[0151] X max : Indicates the upper limit of remote resources that can be used for remote reasoning computing, that is, the maximum amount of resources. Depending on the resource type, the value can be X max,cloud or X max,edge .

[0152] X min,cloud : Indicates the lower limit of cloud resources that can be used for remote reasoning computing, that is, the minimum amount of cloud resources.

[0153] X max,clout : Indicates the upper limit of cloud resources that can be used for remote reasoning computing, that is, the maximum amount of cloud resources.

[0154] X min,edge : Indicates the lower limit of edge resources that can be used for remote reasoning computing, that is, the minimum amount of edge resources.

[0155] X max,edge : Indicates the upper limit of edge resources that can be used for remote reasoning computing, that is, the maximum amount of edge resources.

[0156] X 0 : Indicates the unit resource adjustment amount. Depending on the resource type, it can take values ​​of X 0,clout or X 0,edge .

[0157] X 0,clout : Indicates the unit resource adjustment amount of cloud resources.

[0158] X 0,edge : Indicates the unit resource adjustment amount of edge resources.

[0159] From the software level, the present application also provides a recovery device for cloud-edge DNN collaborative reasoning interruption for executing all or part of the recovery method for cloud-edge DNN collaborative reasoning interruption, see Figure 8 The recovery device for cloud-edge DNN collaborative reasoning interruption specifically includes the following contents:

[0160] The main control component selection unit 10 is used for, if in the process of reasoning for the DNN model by the collaborative reasoning system composed of the device terminal component, the edge computing component and the cloud computing component, if the resource service interruption of the edge computing component and / or the cloud computing component is detected, then based on the priority selection principle, select a component whose current resource service is not interrupted in the collaborative reasoning system as the current main control component, wherein the priority selection principle includes: the edge computing component or the cloud computing component is given priority, and the device terminal component is second;

[0161] The target strategy construction unit 20 is used to construct a target reasoning recovery strategy for the collaborative reasoning system based on the DNN model data pre-stored in the main control component and the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system whose current resource services are not interrupted based on the target reasoning recovery strategy.

[0162] The embodiment of the recovery device for cloud-edge DNN collaborative reasoning interruption provided in the present application can be specifically used to execute the processing flow of the embodiment of the recovery method for cloud-edge DNN collaborative reasoning interruption in the above-mentioned embodiment. Its functions are not repeated here, and reference can be made to the detailed description of the embodiment of the recovery method for cloud-edge DNN collaborative reasoning interruption.

[0163] In one or more embodiments of the present application, the components include a device terminal component, an edge computing component, and a cloud computing component.

[0164] The part of the recovery device for cloud-edge DNN collaborative reasoning interruption that performs recovery for cloud-edge DNN collaborative reasoning interruption can be completed in the device terminal component, edge computing component or cloud computing component. The specific selection can be based on the processing capability of the component and the limitations of the user's usage scenario. This application is not limited to this. If all operations are completed in the client device, the client device may also include a processor for specific processing of recovery for cloud-edge DNN collaborative reasoning interruption.

[0165] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0166] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0167] From the above description, it can be seen that the recovery device for interrupted cloud-edge DNN collaborative reasoning provided by the embodiment of the present application can realize automatic recovery of collaborative reasoning for interrupted cloud-edge DNN collaborative reasoning, and can effectively improve the efficiency and reliability of reasoning recovery, thereby ensuring the reasoning reliability of reasoning.

[0168] The embodiment of the present application also provides an electronic device, which may be a controller, and the controller may include a processor, a memory, a receiver, and a transmitter, and the processor is used to execute the recovery method for cloud-edge DNN collaborative reasoning interruption mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, taking the bus connection as an example. The receiver may be connected to the processor and the memory via a wired or wireless manner.

[0169] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0170] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules corresponding to the method for recovering from cloud-edge DNN collaborative reasoning interruptions in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method for recovering from cloud-edge DNN collaborative reasoning interruptions in the above method embodiment.

[0171] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0172] The one or more modules are stored in the memory, and when executed by the processor, perform the recovery method for cloud-edge DNN collaborative reasoning interruption in the embodiment.

[0173] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0174] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0175] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0176] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned recovery method for cloud-edge DNN collaborative reasoning interruption are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0177] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the aforementioned method for recovering interruptions of cloud-edge DNN collaborative reasoning.

[0178] In addition, based on the aforementioned embodiments of the recovery method, device, and controller for cloud-edge DNN collaborative reasoning interruption, the present application also provides an embodiment of a recovery system for cloud-edge DNN collaborative reasoning interruption, see Fig. 9 The recovery system for cloud-edge DNN collaborative reasoning interruption specifically includes the following contents:

[0179] The recovery system for cloud-edge DNN collaborative reasoning interruption is respectively arranged in the device terminal component, the edge computing component and the cloud computing component;

[0180] The recovery system for cloud-edge DNN collaborative reasoning interruption includes: a controller 1, a basic capability library 2 respectively connected to the controller 1 for communication, and a collaborative reasoning execution module 3;

[0181] The controller 1 is used to execute the aforementioned recovery method for cloud-edge DNN collaborative reasoning interruption; the basic capability library 2 is used to store DNN model data; the collaborative reasoning execution module 3 is used to execute reasoning for the DNN model based on the target reasoning recovery strategy generated by the controller.

[0182] From the above description, it can be seen that the recovery system for cloud-edge DNN collaborative reasoning interruptions provided by the embodiment of the present application can realize automatic recovery of collaborative reasoning for interrupted cloud-edge DNN collaborative reasoning, and can effectively improve the efficiency and reliability of reasoning recovery, thereby ensuring the reasoning reliability of cloud-edge DNN collaborative reasoning.

[0183] Among them, the recovery system for cloud-edge DNN collaborative reasoning interruption in the device terminal component is also provided with a resource monitoring module 4 that is communicatively connected to the controller 1, and the resource monitoring module 4 is used to monitor in real time whether there is a resource service interruption of the edge computing component and / or cloud computing component in the collaborative reasoning system. If so, the corresponding resource service interruption message is sent to the controller 1 in the device terminal component;

[0184] The controller 1 in the device terminal component is used to determine the cloud computing component as the current main control component if the resource service interruption of the edge computing component is learned according to the resource service interruption message, and send a notification message indicating the resource service interruption of the edge computing component and the cloud computing component as the current main control component to the controller 1 in the cloud computing component, so that the controller 1 in the cloud computing component selects the cloud computing component in which it is located as the current main control component based on the notification message;

[0185] The controller 1 in the device terminal component is used to determine the edge computing component as the current main control component if the resource service interruption of the cloud computing component is learned according to the resource service interruption message, and send a notification message indicating the resource service interruption of the cloud computing component and the edge computing component as the current main control component to the controller in the edge computing component, so that the controller 1 in the edge computing component selects the edge computing component where it is located as the current main control component based on the notification message;

[0186] The controller 1 in the device terminal component is used to select the device terminal component where it is located as the current main control component if it is learned from the resource service interruption message that the resource services of the cloud computing component and the edge computing component are both interrupted;

[0187] Among them, the controller 1 in the main control component is used to construct a target reasoning recovery strategy for the collaborative reasoning system based on the DNN model data pre-stored in the main control component for the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system where all current resource services are available and uninterrupted based on the target reasoning recovery strategy.

[0188] In order to further illustrate the recovery system for cloud-edge DNN collaborative reasoning interruption proposed in this application, this application also provides a specific application example of the recovery system for cloud-edge DNN collaborative reasoning interruption, see Fig.10 , the controller 1 can be called a reliability controller, the resource monitoring module 4 can be abbreviated as resource monitoring, and the collaborative reasoning execution module 3 can be abbreviated as collaborative reasoning execution. Specifically:

[0189] Collaborative reasoning based on the edge-cloud (i.e., device-edge-cloud) involves computing resources of the device, edge, and cloud. When resource services in the device, edge, and cloud are all available, the device generates samples and reasoning tasks corresponding to sample reasoning. Using collaborative reasoning technology, the reasoning tasks are split into devices, edges, and clouds to complete all reasoning calculations and obtain reasoning results. When resource service interruptions occur on one or both of the edge and cloud, the collaborative reasoning strategy based on the edge-cloud fails and the reasoning task fails. In response to the problem of collaborative reasoning failure caused by resource service interruptions, the collaborative reasoning reliability assurance system for edge-cloud resource service interruptions proposed in this application (hereinafter referred to as the system) adopts the mechanism of "resource service interruption-controller selection-strategy formulation-reasoning recovery", and uses a variety of technologies such as DNN partitioning, early exit, and model compression to restore the execution of reasoning tasks based on the remaining available resources to ensure reasoning reliability. The core idea of ​​model compression technology is to simplify the model through pruning, weight quantization, and other technologies to reduce the amount of calculation and memory usage. Some medium and large models can run in IoT devices after model compression, but model compression usually sacrifices a certain degree of reasoning accuracy.

[0190] The recovery system for cloud-edge DNN collaborative reasoning interruption is deployed in a distributed manner in devices, edges, and clouds, and consists of device terminal components, edge computing components, and cloud computing components. The functional modules of the device terminal components include: reliability controller, collaborative reasoning execution, resource monitoring, basic capability library, etc. The functional modules of the edge computing components include: reliability controller, collaborative reasoning execution, basic capability library, etc. The functional modules of the cloud computing components include: reliability controller, collaborative reasoning execution, basic capability library, etc.

[0191] In the "resource service interruption-controller selection-strategy formulation-inference recovery" mechanism adopted by the recovery system for cloud-edge DNN collaborative reasoning interruptions, the system continuously monitors the resource status of devices, edges, and clouds. When a resource service interruption is detected, it determines which type of interruption situation it belongs to; based on the interruption situation and the edge-cloud priority principle, the system selects a reliability controller of an available component as the main controller in the order of edge computing components, cloud computing components, and device terminal components; the main controller selected by the system formulates a new reasoning strategy based on the current resource status, DNN model, and DNN network layer delay prediction model; the system recovers reasoning based on the current resources according to the new reasoning strategy, thereby ensuring the reliability of collaborative reasoning.

[0192] The functions implemented by each functional module in the device terminal component are described as follows:

[0193] (1) Reliability controller: Makes a master controller selection decision based on resource interruption conditions; sends the master controller decision to the currently available edge computing component or cloud computing component; and when acting as the master controller, runs the reliability assurance algorithm to develop a new strategy for recovery reasoning.

[0194] (2) Collaborative reasoning execution: Perform reasoning according to the reasoning strategy and perform local reasoning calculations for the reasoning task. If edge-end or cloud-end collaborative reasoning is used, the intermediate data is sent to the edge or cloud to continue the reasoning calculations and receive the reasoning results. If the reasoning has been completed in the device, the reasoning results are processed.

[0195] (3) Resource monitoring: Monitor the resource status of itself, the edge, and the cloud; determine whether a resource service interruption occurs based on the resource status of itself, the edge, and the cloud. If an interruption occurs, notify the reliability controller of the device terminal component.

[0196] (4) Basic capability library: including DNN model library and DNN layer delay prediction model library. The DNN model library contains different versions of DNN models used for reasoning, such as: versions that support early retirement, versions that support quantization and compression, etc. The DNN network layer delay prediction model library can predict the computing delay of each network layer of the DNN model under different resource capabilities, which is used to support the formulation of collaborative reasoning strategies.

[0197] The functions implemented by each functional module in the edge computing component are described as follows:

[0198] (1) Reliability controller: When selected as the primary controller, it runs the reliability assurance algorithm to formulate a new strategy for recovery reasoning.

[0199] (2) Collaborative reasoning execution: Collaborative reasoning is performed according to the reasoning strategy, intermediate data sent by the device terminal component is received, and the reasoning calculation of the reasoning task at the edge is completed. If the reasoning is not completed, the intermediate data after the calculation is completed is sent to the cloud. If the reasoning is completed, the reasoning result is sent to the device terminal component.

[0200] (3) Basic capability library: The function is consistent with the basic capability library in the device terminal component.

[0201] The functions implemented by each functional module in the cloud computing component are described as follows:

[0202] (1) Reliability controller: When selected as the primary controller, it runs the reliability assurance algorithm to formulate a new strategy for recovery reasoning.

[0203] (2) Collaborative reasoning execution: Collaborative reasoning is carried out according to the reasoning strategy, intermediate data sent by the edge computing component is received, and the reasoning calculation of the reasoning task is completed in the cloud. After the reasoning is completed, the reasoning result is sent to the device terminal component.

[0204] (3) Basic capability library: The function is consistent with the basic capability library in the device terminal component.

[0205] That is to say, the application example of this application is used to restore reasoning and meet the reasoning deadline threshold when the edge or cloud resource service is interrupted to ensure reliability; it can quickly restore reasoning under different resource interruption conditions to ensure that the total delay of reasoning meets the reasoning deadline threshold and ensure reasoning reliability. The recovery system for cloud-edge DNN collaborative reasoning interruption consists of device terminal components, edge computing components and cloud computing components, which are distributedly deployed in devices, edges and clouds. The functional modules of the device terminal components include: reliability controller, collaborative reasoning execution, resource monitoring, basic capability library, etc. The functional modules of the edge computing components include: reliability controller, collaborative reasoning execution, basic capability library, etc. The functional modules of the cloud computing components include: reliability controller, collaborative reasoning execution, basic capability library, etc. That is, the "resource service interruption-controller selection-strategy formulation-reasoning recovery" mechanism is adopted to achieve reliability assurance. For the formulation of reasoning recovery strategy in the system, this application proposes a progressive reliability assurance algorithm to restore reasoning and meet the reasoning deadline threshold with the highest possible reasoning accuracy and the least possible edge or cloud resources to meet high reliability requirements. The goal of the progressive reliability assurance algorithm is to restore reasoning and meet the reasoning deadline threshold with the highest possible reasoning accuracy and the least possible edge or cloud resources to achieve high reliability. For the case where both the edge and the cloud are interrupted, the progressive reliability assurance algorithm uses a compression model to restore reasoning and ensure reliability. For the case of edge or cloud interruption, on the basis of the progressive reliability assurance algorithm, a progressive collaborative reasoning strategy optimization algorithm is designed to obtain a collaborative reasoning strategy that meets the high reliability requirements. The core idea of ​​the progressive collaborative reasoning strategy optimization algorithm proposed in this application is: first, the exit branch, resource allocation amount, and partition point are optimized layer by layer until the total reasoning delay meets the reasoning deadline threshold; then, if there is no collaborative reasoning strategy that meets the reasoning deadline threshold, the compression model is used to restore reasoning. In terms of resource allocation decision optimization, the progressive collaborative reasoning strategy optimization algorithm designs a resource adjustment amount. By gradually increasing the resource allocation amount, it avoids the resource allocation amount greatly exceeding the amount of resources required for the reasoning deadline threshold, resulting in resource waste.

[0206] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0207] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0208] In the present application, features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0209] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for recovering interrupted cloud-edge DNN collaborative reasoning, characterized in that: include: If, during the reasoning process of the DNN model performed by the collaborative reasoning system composed of the device terminal component, the edge computing component and the cloud computing component, the resource service interruption of the edge computing component and / or the cloud computing component is detected, then based on the priority selection principle, a component whose current resource service is not interrupted is selected in the collaborative reasoning system as the current main control component; wherein the priority selection principle includes: the edge computing component or the cloud computing component is given priority, and the device terminal component is second; Based on the DNN model data pre-stored in the main control component, a target reasoning recovery strategy for the collaborative reasoning system is constructed according to the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system whose current resource services are not interrupted based on the target reasoning recovery strategy.

2. The method for recovering interrupted cloud-edge DNN collaborative reasoning according to claim 1 is characterized in that: The method of constructing a target reasoning recovery strategy for the collaborative reasoning system based on the DNN model data pre-stored in the main control component and according to the current available resource amount of the main control component comprises: If the main control component is the device terminal component, the compression model corresponding to the DNN model is extracted from the DNN model library included in the basic capability library preset in the main control component for providing DNN model data, and a target reasoning recovery strategy is generated that uses the compression model to replace the DNN model for reasoning.

3. The method for recovering interrupted cloud-edge DNN collaborative reasoning according to claim 1 is characterized in that: The method of constructing a target reasoning recovery strategy for the collaborative reasoning system based on the DNN model data pre-stored in the main control component and according to the current available resource amount of the main control component comprises: If the main control component is the edge computing component or the cloud computing component, a preset progressive collaborative reasoning strategy optimization algorithm is adopted to construct a target reasoning recovery strategy for the collaborative reasoning system based on the basic capability library preset in the main control component for providing DNN model data, the current available resources of the main control component, the preset resource adjustment amount and the preset reasoning deadline threshold.

4. The method for recovering interrupted cloud-edge DNN collaborative reasoning according to claim 3 is characterized in that: If the main control component is the edge computing component or the cloud computing component, a preset progressive collaborative reasoning strategy optimization algorithm is used to construct a target reasoning recovery strategy for the collaborative reasoning system according to a basic capability library preset in the main control component for providing DNN model data, the current available resource amount of the main control component, the preset resource adjustment amount, and the preset reasoning deadline threshold, including: Extracting each exit branch of the DNN model and the compression model corresponding to the DNN model from the DNN model library included in the basic capability library preset in the main control component, and extracting the DNN layer delay prediction model library contained therein from the basic capability library; and, determining a corresponding minimum resource quota and a maximum resource quota according to the currently available resource amount of the main control component; Branch selection step: selecting one of the exit branches of the DNN model as the current target exit branch in the order from the last exit branch to the first exit branch of the exit branches of the DNN model; Executing a preset progressive collaborative reasoning strategy optimization step for the current target exit branch, so as to obtain corresponding output first result data, second result data or third result data through the progressive collaborative reasoning strategy optimization step; If it is known from the first result data output by the progressive collaborative reasoning strategy optimization step that the current reasoning accuracy of the target exit branch is less than the reasoning accuracy of the preset compression model, a target reasoning recovery strategy is generated that uses the compression model to replace the DNN model for reasoning.

5. The method for recovering interrupted cloud-edge DNN collaborative reasoning according to claim 4 is characterized in that: The step of optimizing the progressive collaborative reasoning strategy includes: Obtaining the branch reasoning accuracy of the current target exit branch, and determining whether the branch reasoning accuracy of the target exit branch is less than the preset reasoning accuracy of the compression model; If yes, outputting first result data indicating that the branch reasoning accuracy of the target exit branch is less than the preset reasoning accuracy of the compression model; If not, the minimum resource quota is used as the initial target resource allocation amount, and a preset increment step is performed: the target resource allocation amount is incremented with the resource adjustment amount as a unit to form a current target resource allocation amount; For the current target resource allocation, the partition point corresponding to the current target exit branch is solved with the goal of minimizing the total reasoning delay; Solve the total inference delay corresponding to the collaborative inference strategy composed of the current target resource allocation, target exit branch and partition point based on the DNN layer delay prediction model library; Determine whether the total reasoning delay corresponding to the current collaborative reasoning strategy is greater than the reasoning deadline threshold; If the total reasoning delay is less than or equal to the reasoning deadline threshold, then output second result data indicating that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch, and the partition point is less than or equal to the reasoning deadline threshold; If the total inference delay is greater than the inference deadline threshold, determining whether the current target resource allocation is less than the maximum resource quota; If the current target resource allocation is less than the maximum resource quota, return to execute the increment step; If the current target resource allocation amount is equal to or greater than the maximum resource quota, the output is used to indicate that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch, and the partition point is greater than the third result data of the reasoning deadline threshold.

6. The method for recovering interrupted cloud-edge DNN collaborative reasoning according to claim 4 is characterized in that: Also includes: If the second result data output by the progressive collaborative reasoning strategy optimization step shows that the reasoning accuracy of the current target exit branch is equal to or greater than the preset reasoning accuracy of the compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch and the partition point is less than or equal to the reasoning deadline threshold, then the collaborative reasoning strategy is determined as the target reasoning recovery strategy.

7. The method for recovering interrupted cloud-edge DNN collaborative reasoning according to claim 4 is characterized in that: Also includes: If the third result data outputted from the progressive collaborative reasoning strategy optimization step indicates that the reasoning accuracy of the current target exit branch is equal to or greater than the reasoning accuracy of the preset compression model, and the total reasoning delay corresponding to the collaborative reasoning strategy composed of the target exit branch, the target resource allocation amount corresponding to the target exit branch, and the partition point is greater than the reasoning deadline threshold, then determine whether the current target exit branch is the first exit branch, and if not, return to execute the branch selection step; If so, a target reasoning recovery strategy is generated that uses the compression model to replace the DNN model for reasoning.

8. A controller comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the recovery method for cloud-edge DNN collaborative reasoning interruption as described in any one of claims 1 to 7.

9. A recovery system for cloud-edge DNN collaborative reasoning interruption, characterized in that: The recovery system for cloud-edge DNN collaborative reasoning interruption is respectively arranged in the device terminal component, the edge computing component and the cloud computing component; The recovery system for cloud-edge DNN collaborative reasoning interruption includes: a controller, a basic capability library and a collaborative reasoning execution module respectively connected to the controller in communication; The controller is used to execute the recovery method for cloud-edge DNN collaborative reasoning interruption as described in any one of claims 1 to 7; the basic capability library is used to store DNN model data; the collaborative reasoning execution module is used to execute reasoning for the DNN model based on the target reasoning recovery strategy generated by the controller.

10. The recovery system for cloud-edge-end DNN collaborative reasoning interruption according to claim 9 is characterized in that: The recovery system for cloud-edge DNN collaborative reasoning interruption in the device terminal component is also provided with a resource monitoring module that is communicatively connected to the controller, and the resource monitoring module is used to monitor in real time whether there is a resource service interruption of the edge computing component and / or cloud computing component in the collaborative reasoning system, and if so, the corresponding resource service interruption message is sent to the controller in the device terminal component; The controller in the device terminal component is used to determine the cloud computing component as the current master control component if the resource service interruption of the edge computing component is learned according to the resource service interruption message, and send a notification message indicating the resource service interruption of the edge computing component and the cloud computing component as the current master control component to the controller in the cloud computing component, so that the controller in the cloud computing component selects the cloud computing component in which it is located as the current master control component based on the notification message; The controller in the device terminal component is used to determine the edge computing component as the current master control component if the resource service interruption of the cloud computing component is learned according to the resource service interruption message, and send a notification message indicating the resource service interruption of the cloud computing component and the edge computing component as the current master control component to the controller in the edge computing component, so that the controller in the edge computing component selects the edge computing component where it is located as the current master control component based on the notification message; The controller in the device terminal component is used to select the device terminal component where it is located as the current main control component if it is learned from the resource service interruption message that the resource services of the cloud computing component and the edge computing component are both interrupted; Among them, the controller in the main control component is used to construct a target reasoning recovery strategy for the collaborative reasoning system based on the DNN model data pre-stored in the main control component for the current available resource amount of the main control component, so as to restore the reasoning for the DNN model between one or more components in the collaborative reasoning system where all current resource services are available and uninterrupted based on the target reasoning recovery strategy.

Images