Robot cloud and end collaborative computing processing method and device, and storage medium

Abstract

The application discloses a kind of robot cloud and end collaborative computing processing method, equipment and storage medium, each robot is deployed one local computing processing model and collaborative model, cloud server is deployed cloud computing processing model, comprising: step 1, the computing accuracy required by estimating robot computing task;Step 2, if by cloud server to complete the current state of robot computing task, estimate network delay;Step 3, by multi-agent deep reinforcement learning algorithm with computing accuracy, computing data and network delay training each robot collaborative model obtains optimal collaborative model, step 4, with the optimal collaborative model obtained, according to the current network and computing task state of cloud robot system determines the computing processing object of processing computing task, and the computing processing object is robot local computing processing model or cloud computing processing model of cloud server.The method realizes optimal robot cloud and end collaboration, effectively improves the overall performance of cloud robot system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computing processing of cloud robot system, and particularly relates to a robot cloud and terminal collaborative computing processing method and device and storage medium. BACKGROUND

[0002] Robots usually need to complete a variety of complex computing tasks in the application process, including positioning, perception, planning and target detection, etc. With the development of artificial intelligence and robot technology, more and more robot systems use computationally expensive models, such as large deep neural networks (DNNs). However, due to the limited computing resources on board of the robot itself, there is usually not enough computing power (such as power-hungry GPU cards) or power reserves (mobile robots are usually battery-powered) to run high-precision neural network models. In order to solve this problem, the robot usually offloads complex computing to the cloud server, forming a cloud robot system with cloud and terminal collaborative computing.

[0003] The problem of using robot cloud and end collaboration for complex calculation is that the robot end and the cloud server need to communicate through wireless network, and too much calculation shunting will cause network congestion and even delay or loss of calculation results. The cloud and end collaboration strategy is the core of the cloud robot system, and plays a crucial role in the performance improvement of the cloud robot system. However, the existing cloud and end collaboration mechanism related solutions are mostly for selecting which communication method to complete the collaborative shunting of the calculation task, and few are about improving the collaboration calculation strategy itself. In fact, due to network bandwidth, communication loss and unpredictable system delay, the stability and effectiveness of the collaborative shunting calculation mechanism cannot be fully guaranteed. At the same time, the communication data transmission during the execution of the calculation shunting operation also brings certain energy consumption, which undoubtedly also puts pressure on the limited on-board energy storage (battery) of the robot. When the remaining energy of the robot is too low, it will cause the robot to shut down offline and unable to respond to work, and if the robot also bears important roles such as communication relay or task collaboration, it is likely to cause the collapse of the entire cloud robot system. The existing solutions do not fully weigh the model utility and network transmission cost. A simple but currently widely used strategy is to perform collaborative calculation with the cloud at a fixed frequency, for example: the smart speaker needs to communicate with the cloud server frequently, and the fixed frequency cloud and end collaboration calculation method, although it also considers the actual bandwidth limitation of the network bandwidth, does not consider the actual network condition during each collaborative transmission, and does not maximize the advantages of the cloud model. In addition, there are some more detailed manual decision methods that consider the resource condition limitation, which are based on fixed experience rules and do not consider the model utility and network transmission cost. Due to network limitations, the manually designed cloud and end collaboration calculation method can only be used in specific fields and lacks flexibility. Designing such an algorithm needs to consider many real-world scenarios and trade-offs in various aspects, so it is difficult and may not achieve good results.

[0004] In view of the shortcomings of the above method, how to find a cloud and end collaboration calculation processing method that fully considers the calculation requirements and network communication capabilities, can handle different network conditions, and flexibly trade off calculation requirements and network conditions to effectively play the advantages of the cloud robot system is a problem to be solved.

[0005] In view of the above, the present application is proposed. SUMMARY

[0006] The purpose of the present application is to provide a robot cloud and end collaboration calculation processing method, device and storage medium, which can handle different network conditions, flexibly trade off calculation requirements and network conditions, make the calculation processing of the cloud robot system meet the needs of various application scenarios, and thus solve the above technical problems existing in the prior art.

[0007] The object of the present application is achieved by the following technical solutions:

[0008] A robot cloud and end collaborative computing processing method, for collaborative processing of robot computing tasks in communication between robots and a cloud server of a cloud robot system, each robot is deployed with a local computing processing model and a collaboration model, and the cloud server is deployed with a cloud computing processing model, comprising:

[0009] Step 1, estimating the computing accuracy required by the computing task under the current state of the robot;

[0010] Step 2, if the computing data of the robot is cooperated to the cloud server, the network delay in the transmission process and the processing computing task process is estimated when the cloud server completes the computing task under the current state of the robot;

[0011] Step 3, the optimal collaboration model is obtained by training the collaboration model of each robot using the computing accuracy, computing data and network delay through a multi-agent deep reinforcement learning algorithm, the optimal collaboration model can select a computing processing object for processing the computing task when processing the computing task state, the computing processing object is a local computing processing model of the robot or a cloud computing processing model of the cloud server;

[0012] Step 4, when the robot processes the computing task, the optimal collaboration model trained is applied to determine the computing processing object for processing the computing task according to the current network and computing task state of the cloud robot system, and the corresponding computing task is processed by the determined computing processing object.

[0013] A processing device, comprising:

[0014] At least one memory for storing one or more programs;

[0015] At least one processor capable of executing one or more programs stored in the memory, when the one or more programs are executed by the processor, the processor can implement the method described in the present application.

[0016] A readable storage medium storing a computer program, when the computer program is executed by a processor, the method described in the present application can be implemented.

[0017] Compared with the prior art, the robot cloud and end collaborative computing processing method, device and storage medium provided by the present application have the following advantages:

[0018] The robot cloud and end collaborative computing processing strategy is trained by using a multi-agent deep reinforcement learning algorithm, and the strategy can be used to select a processing object as the robot or the cloud server according to the current state of the robot computing task, so as to realize optimal utilization of computing resources, avoid challenges caused by manual design of a collaborative algorithm, and realize optimal robot cloud and end collaboration while ensuring normal load of a restricted network, thereby effectively improving the overall performance of the cloud robot system. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 The flowchart of the robot cloud and end collaborative computing processing method based on multi-agent reinforcement learning provided by the embodiments of the present application.

[0021] Figure 2 The schematic diagram of the robot cloud and end collaborative computing processing method based on multi-agent reinforcement learning provided by the embodiments of the present application.

[0022] Figure 3 The schematic diagram of the deep reinforcement learning algorithm of the robot cloud and end collaborative computing processing method based on multi-agent reinforcement learning provided by the embodiments of the present application.

[0023] Figure 4 The schematic diagram of the multi-robot cloud and end collaborative face detection using the robot cloud and end collaborative computing processing method of the present application provided by the embodiment 2 of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the specific contents of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments, which do not constitute a limitation to the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0025] Firstly, the terms possibly used in the present text are explained as follows:

[0026] The term "and / or" means either of the two or both at the same time, for example, X and / or Y means three cases including "X" or "Y" or "X and Y".

[0027] The terms "comprising", "containing", "including", "having" or any other similar semantic descriptions should be interpreted to be non-exclusive inclusion. For example, the inclusion of a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as including not only the explicitly listed technical feature element, but also other technical feature elements not explicitly listed but known in the art.

[0028] The term "consisting of" means excluding any technical feature element not explicitly listed. If this term is used in a claim, the term will make the claim closed, so that it does not contain technical feature elements other than those explicitly listed, except for conventional impurities associated therewith. If the term only appears in a certain clause of the claim, it only limits the elements explicitly listed in that clause, and the elements described in other clauses are not excluded from the overall claim.

[0029] The multi-agent reinforcement learning-based robot cloud and edge collaborative computing processing method provided by the present application is described in detail below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. If no specific conditions are specified in the embodiments of the present application, the conventional conditions or the conditions recommended by the manufacturer are used. If no manufacturer of the reagent or instrument used in the embodiments of the present application is specified, it is a conventional product that can be purchased on the market.

[0030] As shown in Figure 1 The embodiments of the present application provide a robot cloud and edge collaborative computing processing method, which is used for the communication between robots and cloud servers in a cloud robot system for collaborative processing of robot computing tasks. Each robot is deployed with a local computing processing model and a collaboration model, and the cloud server is deployed with a cloud computing processing model, which comprises:

[0031] Step 1, estimating the computing accuracy required by the computing task under the current state of the robot;

[0032] Step 2, if the computing data of the robot is cooperated to the cloud server, the network delay in the transmission process and the processing computing task process is estimated when the cloud server completes the computing task under the current state of the robot;

[0033] Step 3, the optimal collaboration model is obtained by training the collaboration model of each robot using the computing accuracy, computing data and network delay through a multi-agent deep reinforcement learning algorithm. The optimal collaboration model can select a computing processing object for processing the computing task when processing the computing task state, and the computing processing object is the local computing processing model of the robot or the cloud computing processing model of the cloud server.

[0034] Step 4, when the robot processes the computing task, the optimal collaborative model trained is applied, the computing processing object for processing the computing task is determined according to the current network and computing task state of the cloud robot system, and the corresponding computing task is processed by the determined computing processing object.

[0035] Referring to Figure 2 , in step 3 of the above method, the cloud and end collaborative computing strategy of the robot and the cloud server is modeled as a Markov decision process as follows:

[0036] M offload =(S offload ,A offload ,R offload ,P offload ,T)

[0037] Wherein, S offload is a state space, representing information for decision-making by the multi-agent deep reinforcement learning algorithm, each state space in it Information for decision-making includes: computing data x t , computing accuracy f t and network latency n t , that is A offload is a set of action spaces, each action space a offload in it represents an action selected by the multi-agent deep reinforcement learning algorithm, and each collaborative model can select two actions: one is to select a local computing processing model on the robot for computing, and the other is to offload computing data to the cloud server and select a cloud computing processing model for computing, that is:

[0038]

[0039] P offload is a reward function, representing the goal to be achieved by the optimal collaborative model in the multi-agent deep reinforcement learning algorithm, which needs to balance the reward robot computing resource limit and network load limit to achieve optimal cloud and end collaborative computing;

[0040]

[0041] Wherein, is a reward function, representing the reward given by the environment after this action; α acc and β cost are two hyperparameters, respectively used to adjust the computing resource limit and network load limit of the robot under different conditions, wherein represents the resources consumed for processing the current computing task, and cost(a t ) represents the reward function of the robot according to at The network resources consumed by selecting a local computing processing model or a cloud computing processing model; P offload The system represents the dynamic acquisition of input data by sensors and the change in the current bandwidth occupation of the restricted network; T is the time change.

[0042] In the above method, the calculation data x t If it is a high-dimensional vector, it is calculated from the calculation data x t The extracted features Characterize the calculation data.

[0043] In step 3 of the above method, the multi-agent deep reinforcement learning algorithm adopts a reinforcement mutual learning algorithm, and its processing steps are as follows:

[0044] Step 31, receiving the environment state information of the collaborative model of each robot, the size of the experience replay buffer M, the training number N, the target network parameter update frequency C and the soft update coefficient τ as the input of the multi-agent deep reinforcement learning algorithm;

[0045] Step 32, randomly initialize the actor network parameters θ and the critic network parameters ω of the collaborative model of each robot, let θ - = θ, ω - = ω, empty the experience replay buffer M; let t from 1 to the maximum round, iterate;

[0046] Step 33, in the environment o i , each robot's collaborative model selects an action According to the current policy Where the random number N t represents the exploration of the robot's collaborative model to the environment;

[0047] Step 34, the system composed of the collaborative models of each robot executes the action a = (a1, a2, …, a n ), obtains the reward value r, and at the same time the system is transferred to the new state s t+1 ;

[0048] Step 35, store (s t , a, r, s t+1 ) in the experience replay buffer M, and assign N b = N b + 1;

[0049] Step 36, update the state s t = s t+1 of the system;

[0050] Step 37, for each robot's collaborative model i from 1 to the training number N, randomly take Ns a sample (s j , a j , r j , s′ j ) updates the network of the collaborative model according to the following steps;

[0051] Step 38, let

[0052] Step 39, update the reviewer network parameter ω of the collaborative model according to the loss function ;

[0053] Step 310, update the actor network parameter θ of the collaborative model according to the policy gradient of the sample ;

[0054] Step 311, update the target network parameter θ' of the collaborative model of each robot: θ' ← τθ i +(1-τ)θ′ i ; i ;

[0055] Step 312, determine whether the iteration number is equal to the training number N, if yes, execute step 313, if not, repeat steps 33 to 312;

[0056] Step 313, output the optimal parameters θ of the actor network and the optimal parameters ω of the reviewer network of the collaborative model learned by the multi-agent deep reinforcement learning algorithm.

[0057] In step 31 of the above method, the environment state information of the collaborative model of each robot includes:

[0058] The number of robots, network limited bandwidth, the number of cloud servers and the calculation speed of the cloud computing processing model.

[0059] In the above method, the collaborative models of the plurality of robots are homogeneous, and the centralized Q network is used for training during training.

[0060] Each collaborative model uses a deep reinforcement learning neural network model whose input is the current environment input and whose output is the selected action, the first hidden layer of the deep neural network model uses a long short-term memory network capable of extracting historical states in a short period of time, and the long short-term memory network can extract environment input features;

[0061] The actor network of the deep reinforcement learning neural network model is composed of a long short-term memory network with 64 units, a fully connected layer with 256 units and an output layer of a normalized exponential function, the activation function of the fully connected layer uses a relu function, the specific width and depth can be defined according to actual needs, and different actions are selected according to the output of the output layer.

[0062] As can be seen from the above, the method of the embodiment of the present application can use the multi-agent deep reinforcement learning algorithm to train the optimal collaborative model that can give the cloud and end collaborative computing processing strategy, can use the optimal collaborative model to select whether the processing computing processing object is the robot or the cloud server according to the current state of the robot computing task, realize the optimal utilization of computing resources, can avoid the challenge brought by manual design of collaborative algorithm, and at the same time, ensure the normal load of the restricted network, realize the optimal robot cloud and end collaboration, thereby effectively improving the overall performance of the robot system.

[0063] In order to more clearly show the technical solutions provided by the present application and the technical effects generated, the robot cloud and end collaborative computing processing method provided by the embodiment of the present application is described in detail below with specific embodiments.

[0064] Embodiment 1

[0065] The embodiment of the present application provides a robot cloud and end collaborative computing processing method, which uses a multi-agent deep reinforcement learning algorithm to learn the optimal collaborative computing processing strategy, avoids manual design of complex rules, and at the same time, reduces the number of training on the entity robot, is suitable for a cloud robot system composed of multiple robot ends and cloud servers, that is, a large-scale application scenario composed of multiple robots and cloud servers in communication, each robot has an on-board local computing processing model and a deployed collaborative model.

[0066] In this embodiment, the collaborative model of the robot gives a cloud and end collaborative computing shunt decision under a specific state, wherein the state refers to the computing data of the robot under a specific network, and the decision refers to using the cloud computing processing model of the cloud server or using the local computing processing model of the robot on board for computing. The main difference between the two is that using the cloud computing processing model can improve the accuracy of the calculation, thereby improving the performance of the entire method, but there will be a certain network delay; using the local computing processing model can reduce the network load, but due to the limitation of the local computing processing model, the accuracy of the calculation will be reduced.

[0067] As Figure 1 shown, the method of the present application mainly includes the following steps:

[0068] Step 1, estimate the required calculation accuracy of the computing task under the current state; for example, how high the accuracy needs to be for identifying the category of an object, or whether the position of the object in space needs to be obtained, etc.

[0069] Step 2, if the computing data is cooperated to the cloud server, estimate the network delay in the transmission process and the processing process when the cloud server completes the corresponding computing task;

[0070] Step 3, the optimal coordination model is obtained by training the coordination model of each robot deployment through a deep reinforcement learning algorithm, which can give the optimal cloud and end coordination computing processing strategy, and the optimal coordination model can select the computing processing object under the computing task state, that is, determine whether to calculate through the local computing processing model locally or to flow to the cloud server for calculation by the cloud computing processing model;

[0071] Step 4, when the robot processes the computing task, the trained optimal coordination model is applied to determine whether to flow the computing task of the robot to the cloud server for processing according to the current network and task state of the cloud robot system.

[0072] Referring to Figure 2 In the embodiment, Figure 1 The cloud coordination computing problem shown in the figure can be modeled as a Markov Decision Process (MDP) as follows:

[0073] M offload =(S offload ,A offload ,R offload ,P off l oad ,T);

[0074] Where, the state space S offload represents the information contained in the decision of the multi-agent deep reinforcement learning algorithm. In the present application, each state space contains: computing data, computing accuracy and network delay, that is:

[0075]

[0076] In practical applications, the computing data x t is likely to be a high-dimensional vector, and direct use will result in a too large state space, so feature extraction is required The extracted low-dimensional features are used to represent the computing data, thereby reducing the dimension of the state space.

[0077] The action space a offload represents the action selection of the multi-agent deep reinforcement learning algorithm. In the cloud and end coordination computing problem, the coordination model of each robot can generally select two actions: selecting the on-board local computing processing model for calculation (lower accuracy) and selecting the cloud computing processing model for calculation (higher accuracy) by flowing the computing data to the cloud server.

[0078]

[0079] The reward function R offloadrepresents the goal that the cloud and edge collaborative computing system in the multi-agent deep reinforcement learning algorithm needs to achieve. In the present application, the function needs to balance the reward robot computing resource limit and network load limit, so as to achieve the optimization of cloud and edge collaborative computing, that is:

[0080]

[0081] wherein, alpha acc and beta cost are two hyperparameters, respectively used to adjust the airborne resource limit and network load limit in different cases.

[0082] Considering the scenario of multiple robots working at the same time, sending computing data when the robot and the cloud collaborative computing will significantly improve the common network load condition. Therefore, the multi-agent deep reinforcement learning algorithm is selected, and the resources shared by multiple robots are modeled, which is more in line with the actual situation, and can improve the performance of the cloud and edge collaborative computing processing method.

[0083] In the multi-agent deep reinforcement learning algorithm, the present application establishes an explicit communication rule between the collaborative models (i.e. agents), and each collaborative model as an agent needs to make decisions according to the information transmitted by other collaborative models. Since the collaborative models perform the same task, the collaborative models can be considered to be isomorphic, and a centralized Q network is used for training, reducing the training cost.

[0084] The multi-agent deep reinforcement learning algorithm is as shown in the following formula: Figure 3

[0085] The inputs of the algorithm include: multi-agent environment, i.e. the state information of multiple collaborative model environments (number of robots, bandwidth limited network, number of cloud servers, computing speed of cloud computing processing model), size of experience replay buffer (ReplayBuffer) M, training times N, target network parameter update frequency C, and soft update (SoftUpdate) coefficient tau.

[0086] The output of the algorithm is the optimal network parameters theta of the actor (Actor) of the collaborative model representing the policy (Policy) learned by the deep reinforcement learning algorithm and the optimal network parameters omega of the critic (Critic).

[0087] The main steps of the algorithm are as follows:

[0088] Step 31, receiving inputs, including: state information of multiple collaborative model environments (number of robots, bandwidth limited network, number of cloud servers, computing speed of cloud computing processing model), size of experience replay buffer (ReplayBuffer) M, training times N, target network parameter update frequency C, and soft update (SoftUpdate) coefficient tau.​

[0089] Step 32, the algorithm randomly initializes the actor network parameters θ and critic network parameters ω of each collaborative model, let θ = θ, ω = ω, clear the experience replay buffer M; let t from 1 to the maximum round, iteration; - -

[0090] Step 33, for each collaborative model, in the environment o i , the collaborative model selects an action a according to the current policy where the random number N t represents exploration of the environment;

[0091] Step 34, the system composed of each collaborative model performs the action a = (a1, a2, …, a n ), obtains the reward value r, and the system is transferred to the new state s t+1 ;

[0092] Step 35, store (s t , a, r, s t+1 ) in the experience replay buffer M, let N b ← N b + 1;

[0093] Step 36, update the system state s t = s t+1 ;

[0094] Step 37, for each collaborative model i from 1 to N, randomly take N s samples (s j , a j , r j , s′ j ) from the experience replay buffer M to update the networks of the collaborative model as follows;

[0095] Step 38, let

[0096] Step 39, update the critic network of the collaborative model according to the loss function ;

[0097] Step 310, update the actor network of the collaborative model according to the policy gradient of the sample ;

[0098] Step 311, update the target network parameters θ′ of each collaborative model: θ′ i ← τθ i + (1-τ)θ′ i ;​​

[0099] Step 312, judge whether the iteration number is equal to the training number N, if yes, execute step 313, if not, repeat steps 33 to 312;

[0100] Step 313, output the optimal actor network parameter θ and the optimal critic network parameter ω representing the policy learned by the deep reinforcement learning algorithm.

[0101] Embodiment 2

[0102] Referring to Figure 4 , this embodiment provides a cloud and end collaborative computing processing method based on multi-agent deep reinforcement learning algorithm, which is based on a face detection scene using vision and deep learning. The computing task of this scene is object detection from image, which is an important part of robot perception. First, each robot selects a NVIDIA Jetson Tx2 embedded computer, which is a commonly used on-board device for service robots. The robot collects real-time video stream through the camera on the Jetson, and each frame collected is encoded into a 128-dimensional byte vector. On the local computing processing model carried by the robot, a lightweight open source package OpenCV is selected for video frame processing, and a trained SVM classifier is used to output face detection results. This on-board local computing processing model has low demand for computing resources, and the corresponding detection accuracy is also low. In the cloud server, a FaceNet deep neural network with higher face detection accuracy and higher demand for computing resources (such as GPU) is built as a cloud computing processing model.

[0103] In the current application, if the robot only knows the current state, the input data can present time-related features, and the system formed by the collaborative model under the multi-agent deep reinforcement learning algorithm becomes unstable. In order to solve this problem, the collaborative model of the embodiment adopts a deep reinforcement learning neural network model, and selects a long short term memory (LSTM) as the first hidden layer in the neural network of the deep reinforcement learning neural network model to extract the historical state in a short time; and the actor network of the deep reinforcement learning neural network model is composed of a 64-unit LSTM, a 256-unit fully connected layer and a normalized exponential function (Softmax) output layer. In the process of training the collaborative model, video data of human faces in different scenes can be collected for training, including multiple training videos and a small amount of test videos. In order to distinguish between the robot-side SVM classifier and the neural network as a cloud computing processing model on the cloud server, the SVM classifier carried on the robot only selects a small part of data for training in the training process. In the model use stage, when the robot local computing processing model has low confidence (recognizes an untrained person) or multiple people appear in the video, the collaborative algorithm always tends to distribute the computing data to the cloud computing processing model of the cloud server, which is as expected.

[0104] In the embodiment of the application, the "reinforced mutual learning algorithm" is preferably used. Those skilled in the art can know that if other types of multi-agent deep reinforcement learning algorithms, such as differentiable inter-agent learning (MADDPG) and the like, can achieve the method of the application, they should not be considered as the main difference from the application.

[0105] In the embodiment of the application, the reward function is designed according to the trade-off between local resource limitations and network bandwidth limitations. In the example of the application, a linear function is selected to describe the trade-off between robot resources and network bandwidth. Those skilled in the art can know that using other types of functions, such as power functions, exponential functions, logarithmic functions, etc., to describe resource trade-offs should not be considered as the main difference from the application.

[0106] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by programs instructing related hardware. The programs can be stored in a computer-readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0107] The above description is merely that of the preferred embodiments of the application, and is not intended to limit its scope. Any changes or modifications that can be made by one with ordinary skill in the art without departing from the spirit of the application are intended to be included within the scope of the application. Therefore, the scope of the present application should be determined by the following claims. The information disclosed in the Background section is merely for the purpose of enhancing the understanding of the overall background of the application and should not be taken as admitting that such information forms a prior art that is already known to those of ordinary skill in the art.

Claims

1. A method for robot cloud and edge collaborative computing processing, characterized in that, In the robot-cloud server communication for collaborative processing of a robot computing task in a cloud robot system, each robot is deployed with a local computing processing model and a collaboration model, and the cloud server is deployed with a cloud computing processing model, including: Step 1, estimating the computing accuracy required by the computing task under the current state of the robot; Step 2, if the computing data of the robot is cooperated to the cloud server, the network delay in the transmission process and the processing computing task process is estimated when the cloud server completes the computing task under the current state of the robot; Step 3, the optimal collaboration model is obtained by training the collaboration model of each robot using the computing accuracy, computing data and network delay through a multi-agent deep reinforcement learning algorithm, which can select the computing processing object for processing the computing task in the processing computing task state, and the computing processing object is the local computing processing model of the robot or the cloud computing processing model of the cloud server; the processing steps are as follows: Step 31, receive the environment state information of the coordination model of each robot, the size of the experience replay buffer M, the training times N, the target network parameter update frequency C and the soft update coefficient as the input of the multi-agent deep reinforcement learning algorithm; Step 32, randomly initialize actor network parameters of the coordination model of each robot and critic network parameters , let = , = , empty the experience replay buffer M; let t from 1 to maximum rounds, iterate; Step 33, in the environment Below, the collaborative model of each robot is based on the current strategy. Choose an action ( ) + , where random number This represents the collaborative model of robots exploring the environment; Step 34, the system of coordinated models of each robot performs an action a = ( ), obtains a reward value r, and the system moves to a new state ; Step 35, to into the experience playback cache M and is assigned; Step 36, updating the state of the system ; Step 37, for each robot's cooperative model i from 1 to the number of training N, randomly take one sample from the experience replay buffer M according to the following steps; Step 38, let ; Step 39, updating the reviewer network parameters of the collaborative model according to the loss function updating the reviewer network parameters of the collaborative model ; Step 310, updating the actor network parameters of the co-model according to the policy gradient of the sample updating the actor network parameters of the co-model ; Step 311, updating the target network parameters of the coordination model of each robot: ; Step 312, determining whether the iteration number is equal to the training number N, if yes, executing step 313, if not, repeating steps 33 to 312; Step 313, output the optimal parameters of the actor network of the collaborative model learned by the multi-agent deep reinforcement learning algorithm and the optimal parameters of the critic network ; Step 4, when the robot processes the computing task, the optimal collaboration model trained is applied to determine the computing processing object for processing the computing task according to the current network and computing task state of the cloud robot system, and the corresponding computing task is processed by the determined computing processing object. 2.The method of claim 1, wherein, In step 3, the cloud and end collaborative computing process of the robot and the cloud server is modeled as a Markov decision process as follows: ； wherein, is a state space, representing information used for decision-making by the multi-agent deep reinforcement learning algorithm; each state space in Information used for decision-making includes: calculation data , calculation accuracy f t and network delay n t , that is ; is a set of action spaces, each of which represents an action selected by the multi-agent deep reinforcement learning algorithm, and each collaborative model can select two actions: one is to select a local calculation processing model carried by the robot to perform calculation, and the other is to offload the calculation data to a cloud server and select a cloud calculation processing model to perform calculation, that is: ； The reward function is a target to be achieved by an optimal collaborative model in a multi-agent deep reinforcement learning algorithm; the reward function needs to balance the reward robot computing resource limitations and network load limitations to achieve optimal cloud and edge collaborative computing; ； wherein, is a reward function, representing the reward given by the environment after the action, and are two hyperparameters, respectively used to adjust the computational resource limit and network load limit of the robot in different situations, wherein represents the resources consumed in processing the current computing task; represents the network resources consumed in selecting the local computing processing model or the cloud computing processing model according to ; represents the change in the current bandwidth occupied by the sensor dynamically acquired input data and the limited network; T is the time change. 3.The method of claim 2, wherein, The computed data If it is a high-dimensional vector, then use the computed data The extracted features Characterize the computed data. 4.The method of claim 1, wherein, In step 31, the collaboration model environment state information of the robot includes: The number of robots, bandwidth-limited network, the number of cloud servers and the processing speed of the cloud computing processing model. 5.The method of claim 4, wherein, The collaboration models of multiple robots are homogeneous, and a centralized Q network is used for training. 6.The method of claim 4, wherein, Each collaboration model uses a deep reinforcement learning neural network model with current environment input as input and selected action as output, the first hidden layer of the deep neural network model uses a long short-term memory network that can extract historical states in a short time, and the long short-term memory network can extract environment input features; The actor network of the deep reinforcement learning neural network model is composed of a long short-term memory network with 64 units, a fully connected layer with 256 units and an output layer of a normalized exponential function, the activation function of the fully connected layer uses a relu function, and different actions are selected according to the output of the output layer.

7. A processing device, characterized by Including: At least one memory for storing one or more programs; At least one processor capable of executing one or more programs stored in the memory, when the one or more programs are executed by the processor, the processor can implement the method of any one of claims 1-6.

8. A readable storage medium, storing a computer program, characterized in that, When the computer program is executed by the processor, the method of any one of claims 1-6 can be implemented.

Images