A feeding and unloading robot and a dynamic path optimization method thereof

Abstract

The present invention discloses a feeding and unloading robot and a dynamic optimization method for its path, which belongs to the technical field of intelligent robots. The key points of its technical solution include: obtaining images of multiple unloading areas within a first preset range, and selecting a first area and a first node according to the image; judging whether there is an adjacent robot according to the image, and if so, obtaining a second area and a second node, updating the first area and the first node according to the second area and the second node, and obtaining a current unloading area and a current unloading node; the second area and the second node are the first area and the first node of the adjacent robot, and the adjacent robot is an unloading and feeding robot located within the second preset range; moving to the current unloading node, and completing the unloading work of the current unloading area according to an action sequence. The present invention configures an independently running algorithm for each robot, overcomes the bottleneck of traditional centralized control, and senses the operating status of adjacent robots in real time, and communicates and negotiates the optimal path.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and more particularly to a feeding and unloading robot and a dynamic path optimization method thereof. Background Art

[0002] Traditional unloading operations rely heavily on manual labor or centralized control by a central controller. Manual operations are inefficient and susceptible to fatigue and human error. Furthermore, with a central controller, algorithm complexity and computational effort increase exponentially as the number of robots increases, resulting in poor scalability and high maintenance costs.

[0003] Traditional path planning methods often rely on simple environmental models, making them difficult to adapt to the complex and changing environment of the unloading area. Furthermore, these algorithms fail to adequately consider the interactions between robots when handling multi-robot collaborative operations, which can easily lead to path conflicts and further reduce the smoothness of the operation. Therefore, existing technologies have shortcomings. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide a material unloading and feeding robot and a dynamic optimization method for its path. By configuring an independently running algorithm for each robot, the bottleneck of traditional centralized control is overcome, and the optimal unloading path is selected through communication and negotiation with adjacent robots.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The present invention provides a method for dynamically optimizing the path of a feeding and unloading robot. The feeding and unloading robots are multiple and located in the same storage area. The storage area includes multiple unloading areas, each of which corresponds to multiple nodes. The method for dynamically optimizing the path of the feeding and unloading robot is executed by one of the feeding and unloading robots. The method for dynamically optimizing the path of the feeding and unloading robot includes:

[0007] Acquire images of a plurality of unloading areas within a first preset range, and select a first area and a first node according to the images;

[0008] Determine whether there is an adjacent robot. If so, obtain a second area and a second node, and update the first area and the first node according to the second area and the second node to obtain a current unloading area and a current unloading node; the second area and the second node are the first area and the first node of the adjacent robot, and the adjacent robot is a feeding and unloading robot located within a second preset range;

[0009] Move to the current unloading node and complete the unloading work in the current unloading area according to the action sequence, and the action sequence is obtained through the deep reinforcement learning model.

[0010] As a further improvement of the present invention, the selecting the first area and the first node according to the image includes:

[0011] determining the depth of the material within the plurality of discharge areas based on the image;

[0012] Obtaining the driving distances between the plurality of unloading areas and the vehicle itself;

[0013] Obtaining an evaluation value corresponding to each unloading area according to the travel distance, the depth of the material and a preset evaluation index;

[0014] The unloading area with the highest evaluation value is used as the first area, and a node closest to the own position among the nodes corresponding to the first area is used as the first node.

[0015] As a further improvement of the present invention, the updating of the first area and the first node according to the second area and the second node to obtain the current unloading area and the current unloading node includes:

[0016] Determining whether the second region and the second node are the same as the first region and the first node;

[0017] If the second area is the same as the first area, updating the first area and the first node according to the evaluation value corresponding to each unloading area to obtain the current unloading area and the current unloading node;

[0018] If the second area is different from the first area, and the second node is the same as the first node, the first area is used as the current unloading area, and the first node is updated to obtain the current unloading node.

[0019] As a further improvement of the present invention, the updating of the first area and the first node according to the evaluation value corresponding to each unloading area to obtain the current unloading area and the current unloading node includes:

[0020] Obtaining an evaluation value corresponding to the second area;

[0021] If the evaluation value corresponding to the second area is greater than the evaluation value corresponding to the first area, the first area and the first node are updated to obtain the current unloading area and the current unloading node.

[0022] As a further improvement of the present invention, the updating of the first area and the first node to obtain the current unloading area and the current unloading node includes:

[0023] Eliminating the evaluation value corresponding to the first area from the evaluation values ​​corresponding to each unloading area to obtain a remaining evaluation value;

[0024] The unloading area corresponding to the highest evaluation value among the remaining evaluation values ​​is used as the current unloading area, and a node closest to the own position among the nodes corresponding to the current unloading area is used as the current unloading node.

[0025] As a further improvement of the present invention, the action sequence is obtained through a deep reinforcement learning model, including:

[0026] Acquire the posture information after moving to the current unloading node, the posture information including the position coordinates of the manipulator arm, the posture angle of the manipulator arm, and the position coordinates of the material to be unloaded;

[0027] Perform an iterative operation, the iterative operation including: inputting the posture information into the deep reinforcement learning model, obtaining a quality value for each action in a preset action space, selecting an action corresponding to the highest quality value, placing it in an action sequence, updating the posture information until the action of putting down the material is placed in the action sequence, and outputting the current action sequence.

[0028] As a further improvement of the present invention, the deep reinforcement learning model is obtained through training, including:

[0029] Initializing the parameters of the deep reinforcement learning model;

[0030] Generate multiple samples in sequence based on environmental interactions, and store each sample in the experience replay pool in sequence;

[0031] A plurality of samples are selected from the experience replay pool, and parameters of the deep reinforcement learning model are updated according to the plurality of samples until the deep reinforcement learning model converges.

[0032] As a further improvement of the present invention, the selecting a plurality of samples from the experience replay pool includes:

[0033] Calculating the current quality value of each sample according to a preset main quality network, and calculating the target quality value of each sample according to a preset target quality network;

[0034] Calculating a time difference error of each sample according to the current quality value and the target quality value;

[0035] Determining the priority of each sample according to the time difference error;

[0036] A plurality of samples are selected from the experience replay pool according to the priority.

[0037] As a further improvement of the present invention, updating the parameters of the deep reinforcement learning model according to the multiple samples includes:

[0038] Get the current quality value and target quality value of each sample;

[0039] Calculating a loss value based on the current quality value, the target quality value, and a preset loss function;

[0040] Update the parameters of the deep reinforcement learning model based on the loss value and the back-propagation algorithm.

[0041] The present invention provides an unloading and feeding robot, which is applied to the above-mentioned unloading and feeding robot path dynamic optimization method, and the unloading and feeding robot includes: a binocular camera, an image processing module, a computing module, a communication module and a mechanical arm;

[0042] The binocular camera is used to acquire images of multiple unloading areas within a first preset range;

[0043] The image processing module is used to select a first area and a first node according to the image;

[0044] The communication module is used to obtain the second area and the second node;

[0045] The calculation module is used to update the first area and the first node according to the second area and the second node to obtain the current unloading area and the current unloading node;

[0046] The communication module is used to obtain the second area and the second node;

[0047] The robotic arm is used to complete the unloading work in the current unloading area according to the action sequence.

[0048] The present invention first uses image recognition technology to select the first area and the first node, and determines whether there are adjacent robots. Then, the position is exchanged with the adjacent robots in real time through a communication protocol to avoid collisions. In addition, the method provided by the present invention can be executed by each robot individually, so that each robot can make independent decisions based on the information of real-time interaction, overcoming the problem of high algorithm complexity in traditional centralized control. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a schematic diagram of the steps of a method for dynamic optimization of the path of a feeding and unloading robot according to the present invention;

[0050] Figure 2 This is a schematic diagram of the storage area;

[0051] Figure 3 is a schematic diagram of a first preset range;

[0052] Figure 4 is a schematic diagram of a second preset range;

[0053] Figure 5is a schematic diagram of the first node;

[0054] Figure 6 This is a schematic diagram of the scenario when the first area needs to be updated;

[0055] Figure 7 This is a schematic diagram of the scenario when the first node needs to be updated.

[0056] Figure numerals: 1. unloading area; 2. conveyor belt; 3. moving channel; 4. node; 5. projection area; 6. first preset range; 7. second preset range. DETAILED DESCRIPTION

[0057] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations of the technical solution of the present invention.

[0058] Identical components are denoted by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "upper," and "lower" used in the following description refer to directions in the accompanying drawings, and the terms "bottom," "top," "inner," and "outer" refer to directions toward or away from the geometric center of a particular component, respectively.

[0059] The term "and / or" in the following text simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects.

[0060] like Figure 1 As shown, the present invention provides a method for dynamically optimizing the path of a feeding and unloading robot, comprising:

[0061] Acquire images of multiple unloading areas within a first preset range, and select a first area and a first node based on the images;

[0062] Determine whether there is an adjacent robot. If so, obtain the second area and the second node, update the first area and the first node according to the second area and the second node, and obtain the current unloading area and the current unloading node; the second area and the second node are the first area and the first node of the adjacent robot, and the adjacent robot is an unloading robot located within the second preset range;

[0063] Move to the current unloading node and complete the unloading work in the current unloading area according to the action sequence. The action sequence is obtained through the deep reinforcement learning model.

[0064] Among them, there are multiple unloading and feeding robots located in the same storage area. The dynamic optimization method of the unloading and feeding robot path is executed by each unloading and feeding robot separately, and each unloading area is only completed by one robot to complete the unloading work, and a robot is parked at each unloading node.

[0065] like Figure 2 As shown, the storage area includes a plurality of square unloading areas 1 arranged in a grid shape. Between the unloading areas are conveyor belts 2 for placing the unloaded materials. Above the conveyor belts is a moving channel 3 for the robot. The structure of the moving channel is the same as that of the conveyor belt. Figure 2 The virtual image wireframe in the figure is obtained by projecting the unloading area upward, which is called the projection area. It can be used to illustrate the corresponding relationship between the conveyor belt and the moving channel. Each unloading area corresponds to multiple nodes 4, and the nodes are located on the moving channel. If the nodes are projected onto the conveyor belt, the projected nodes are located on the median line of each edge of the unloading area. Figure 2 Only the nodes corresponding to the two unloading areas are shown. It can be seen that the nodes between the two adjacent unloading areas overlap.

[0066] like Figure 3 As shown, the first preset range of each unloading and feeding robot can be set according to actual work needs, usually set to a circular area centered on itself (node ​​A), and the radius of the circular area is It is twice the distance between two diagonally adjacent nodes. At this time, the multiple unloading areas within the first preset range are the unloading areas corresponding to projection areas No. 1 to No. 8.

[0067] like Figure 4 As shown, assuming that the unloading area corresponding to projection area No. 7 is the first area, the second preset range is a circular range with the first area as the center and the distance between itself and the first area as the radius, wherein the distance between itself and the first area is specifically the distance between itself and the center of the projection area corresponding to the first area, that is, the distance between node A in the figure and the center of projection area No. 7. The first area and the first node selected by the robot within the second preset range are more likely to be the same as the first area and the first node selected by itself, which leads to a collision accident. Therefore, it is used as an adjacent robot for subsequent analysis.

[0068] If there is no adjacent robot within the second preset range, the first area will be used as the current unloading area, and the first node will be used as the current unloading node. The path planning algorithm will then be used to determine the path to the current unloading node, thereby completing the unloading work.

[0069] This embodiment configures an independently running algorithm for each robot to overcome the bottleneck of traditional centralized control, and updates and determines the current unloading area and current unloading node through communication with adjacent robots, ultimately completing the unloading work.

[0070] Furthermore, an embodiment of the present application provides a step of selecting a first region and a first node according to an image, including:

[0071] Determine the depth of materials in multiple discharge areas based on the images;

[0072] Get the driving distance between multiple unloading areas and the vehicle;

[0073] According to the travel distance, material depth and preset evaluation indicators, the evaluation value corresponding to each unloading area is obtained;

[0074] The unloading area with the highest evaluation value is taken as the first area, and the node closest to the own position among the nodes corresponding to the first area is taken as the first node.

[0075] The above images are collected by a binocular camera. Then, based on the triangle similarity principle and the camera's geometric model, the parallax can be used to determine the depth of the material in the unloading area. Since each unloading area corresponds to multiple nodes, the driving distance between each unloading area and itself is the average of the driving distances between each node corresponding to the unloading area and itself.

[0076] Specifically, the evaluation indicators are:

[0077] ;

[0078] in, Indicates the The evaluation value corresponding to each unloading area, Indicates the The depth of the material in the discharge area, Indicates the The driving distance between the unloading area and the vehicle itself, and are the weight coefficients of depth and travel distance respectively, is the maximum value of the material depth in all unloading areas, The maximum value of the driving distance in all unloading areas is calculated by the ratio to the maximum value to achieve the effect of normalization, so that the depth and driving distance can be calculated at the same scale. The weight coefficient is determined according to the actual work needs. If the current work focuses more on the unloading volume, a higher depth weight is selected. If the current work focuses more on transportation and time costs, a higher driving distance weight is selected.

[0079] Then the unloading area with the highest evaluation value is taken as the first area, and the node closest to the own position in the first area is taken as the first node. Figure 5As shown, since the robot can only move in a straight line along the moving channel, there are usually multiple nodes closest to its own position in the nodes corresponding to the first area (such as Figure 5 Node C and node D in ), at this time, one node is randomly selected from multiple nodes as the first node.

[0080] This embodiment selects the first area and the first node according to the depth and the driving distance, thereby reducing unnecessary transportation time and driving time and improving work efficiency.

[0081] Furthermore, this embodiment provides a step of updating the first area and the first node according to the second area and the second node to obtain the current unloading area and the current unloading node, including:

[0082] Determining whether the second region and the second node are the same as the first region and the first node;

[0083] If the second area is the same as the first area, update the first area and the first node according to the evaluation value corresponding to each unloading area to obtain the current unloading area and the current unloading node;

[0084] If the second area is different from the first area, and the second node is the same as the first node, the first area is used as the current unloading area, and the first node is updated to obtain the current unloading node.

[0085] There may be multiple adjacent robots, so there may be multiple second areas and second nodes. The above judgment steps need to be performed for each second area and second node.

[0086] Furthermore, the embodiment of the present application provides a step of updating the first area and the first node according to the evaluation value corresponding to each unloading area to obtain the current unloading area and the current unloading node, including:

[0087] Obtaining an evaluation value corresponding to the second area;

[0088] If the evaluation value corresponding to the second area is greater than the evaluation value corresponding to the first area, the first area and the first node are updated to obtain the current unloading area and the current unloading node.

[0089] Furthermore, an embodiment of the present application provides a step of updating the first area and the first node to obtain the current unloading area and the current unloading node, including:

[0090] Eliminate the evaluation value corresponding to the first area from the evaluation values ​​corresponding to each unloading area to obtain a remaining evaluation value;

[0091] The unloading area corresponding to the highest evaluation value among the remaining evaluation values ​​is used as the current unloading area, and the node closest to the own position among the nodes corresponding to the current unloading area is used as the current unloading node.

[0092] Specifically, if there is an area that is the same as the first area in the plurality of second areas, such as Figure 6 As shown in , the adjacent robot is located at node B, the first area (second area) corresponding to the adjacent robot is the unloading area corresponding to projection area No. 7, and the second node is node E. At this time, it is necessary to interact and communicate with the adjacent robot corresponding to the second area, obtain the evaluation value corresponding to the second area, and send the evaluation value corresponding to the first area to the adjacent robot. If the evaluation value corresponding to the second area is greater than the evaluation value corresponding to the first area, it means that it is more efficient to select an adjacent robot to complete the unloading work in this area. Therefore, it is necessary to update the first area and the first node, that is, the unloading area corresponding to the highest evaluation value among the remaining evaluation values ​​is used as the current unloading area, and a node closest to its own position among the nodes corresponding to the current unloading area is selected as the current unloading node; if the evaluation value corresponding to the second area is less than the evaluation value corresponding to the first area, there is no need to update the first area and the first node, and the update step is performed by the adjacent robot; if the evaluation value corresponding to the second area is equal to the evaluation value corresponding to the first area, continue to communicate with the adjacent robot, obtain the highest evaluation value among the remaining evaluation values ​​corresponding to the adjacent robot, and compare it with the highest evaluation value among its own remaining evaluation values. If the highest evaluation value corresponding to the adjacent robot is less than its own highest evaluation value, the adjacent robot performs the update step. If it is greater than, it performs the update step by itself. If it is equal, communicate again and repeat the above steps.

[0093] Furthermore, if the second area is different from the first area, and the second node is the same as the first node, it means that the two robots have selected different unloading areas but occupy the same node, e.g. Figure 7 As shown, the first area corresponding to itself is the unloading area corresponding to the projection area No. 7, the first node is node C, the adjacent robot is located at node B, the second area is the unloading area corresponding to the projection area No. 8, and the second node is node C. At this time, the first node is updated by the robot closer to node C, that is, the node D closest to its own position is selected from the remaining nodes corresponding to the first area as the current unloading node.

[0094] Afterwards, a driving path with the current position as the starting point and the current unloading node as the end point can be generated through a path planning algorithm, wherein the path planning algorithm can use the A* algorithm, the Dijkstra algorithm, etc. Specifically, in order to avoid collisions with adjacent robots during driving, different priorities can be set for different robots. For example, after the robot and the adjacent robots have completed the determination of the current unloading area and the current unloading node, each robot sends the evaluation value of its corresponding current unloading area to other robots. After each robot receives the evaluation value, it sorts all robots including itself in descending order according to the evaluation value, and each robot generates its own driving path in sequence.

[0095] In addition, each robot needs to send the generated driving path to the subsequent robot that needs to generate the path. The subsequent robot that needs to generate the path will set the nodes in the driving path as inaccessible nodes to avoid overlap between paths. Specifically, for the A* algorithm, the evaluation function values ​​of these nodes can be set to maximum values, and for the Dijkstra algorithm, these nodes can be marked as inaccessible nodes.

[0096] After detecting a neighboring robot, this embodiment considers collisions with the neighboring robot from two perspectives: first, collisions during unloading operations, i.e., when the current unloading area or node is the same; and second, collisions during driving. For the first scenario, the robot communicates with the neighboring robot to obtain its area's evaluation value for the neighboring robot. By comparing the evaluation values, the robot determines whether to change areas. For the second scenario, collisions during driving are avoided by configuring the algorithm and generation sequence.

[0097] Through the method of this embodiment, each robot can make independent decisions through real-time interactive information. When adding a new robot, it only needs to be connected to the communication network and enabled to interact with other robots and make independent decisions. Compared with the method of unified scheduling by a central controller, this embodiment does not require large-scale modifications to the planning algorithm and architecture of the entire robot system. In addition, the robots in this embodiment only interact with other robots when necessary, and do not need to frequently send status information and receive instructions to the central server as in the unified planning mode of the central server, thereby reducing communication bandwidth requirements and energy consumption.

[0098] Furthermore, the embodiment of the present application provides a step of obtaining an action sequence through a deep reinforcement learning model, including:

[0099] Obtain the posture information after moving to the current unloading node, which includes the position coordinates of the robot arm, the posture angle of the robot arm, and the position coordinates of the material to be unloaded;

[0100] Perform iterative operations, which include: inputting posture information into the deep reinforcement learning model, obtaining the quality value of each action in the preset action space, selecting the action corresponding to the highest quality value, placing it in the action sequence, updating the posture information until the action of putting down the material is placed in the action sequence, and outputting the current action sequence.

[0101] Among them, the position coordinates of the robotic arm are specifically the two-dimensional position coordinates of the end effector of the robotic arm; the posture angle of the robotic arm refers to the angle between the robotic arm and the horizontal direction; the preset action space includes: the robotic arm moves forward 0.5 meters, the robotic arm moves backward 0.5 meters, the robotic arm translates 0.2 meters to the left, the robotic arm translates 0.2 meters to the right, the robotic arm rotates 15 degrees clockwise, the robotic arm rotates 15 degrees counterclockwise, grabbing materials and putting down materials, etc.

[0102] Exemplarily, the posture information is input into the main quality network in the deep reinforcement learning model. The network will output the Q value (quality value) of each action. At this time, the action corresponding to the highest quality value is selected, for example, "the robotic arm translates 0.2 meters to the left" is selected and placed in the action sequence. Then, the posture information after executing this action is calculated, and the next action is input again and selected. The above steps are repeated until the action of putting down the material is placed in the action sequence. The actions selected each time in the action sequence are arranged in the order of selection, and the current action sequence is output. For example, the action sequence of "the robotic arm translates 0.2 meters to the left, the robotic arm moves 0.5 meters backward, grabs the material, the robotic arm rotates 15 degrees clockwise, and puts down the material" is output. Then, the robot completes the unloading work according to this action sequence.

[0103] This embodiment obtains the action sequence through a deep reinforcement learning model. The model directly uses the original posture information as input and ultimately outputs the unloading action sequence. There is no need to manually design intermediate feature extraction and processing links, which reduces human intervention and errors and improves the automation and intelligence level of the entire unloading system.

[0104] Furthermore, the embodiment of the present application provides a step of obtaining a deep reinforcement learning model through training, including:

[0105] Initialize the parameters of the deep reinforcement learning model;

[0106] Generate multiple samples in sequence based on environmental interactions, and store each sample in the experience replay pool in sequence;

[0107] Select multiple samples from the experience replay pool and update the parameters of the deep reinforcement learning model based on the multiple samples until the deep reinforcement learning model converges.

[0108] The parameters of the deep reinforcement learning model include: Q network parameters (quality network parameters), learning rate, discount factor, exploration rate, and the capacity of the experience replay pool. Specifically, the Q network parameters include weights and biases, whose initial values ​​are generated by a random number generator; the learning rate is set to 0.001 to control the step size of each parameter update; the discount factor is set to 0.9, reflecting the emphasis on future rewards; the exploration rate is set to 0.8; a higher exploration rate can avoid falling into local optima; and the capacity of the experience replay pool is set to 1000.

[0109] The specific steps for sequentially generating multiple samples based on environmental interaction are as follows: first, a virtual environment is set up. Within this virtual environment, a robot's posture information is randomly generated (this posture information is called the old state). An action is selected based on the exploration rate. The posture information after executing this action (this posture information is called the new state) and the reward value obtained are calculated. For example, if the new state is closer to the material to be unloaded than the old state, the reward value is +5; if it is farther away, the reward value is -5. Finally, the (old state, action, reward value, new state) is taken as a sample and stored in the experience replay pool. These steps are repeated until the number of samples in the experience replay pool reaches the preset capacity.

[0110] Furthermore, this embodiment provides a step of selecting multiple samples from the experience replay pool, including:

[0111] Calculate the current quality value of each sample based on the preset main quality network, and calculate the target quality value of each sample based on the preset target quality network;

[0112] According to the current quality value and the target quality value, the time difference error of each sample is calculated;

[0113] Determine the priority of each sample based on the time difference error;

[0114] Select multiple samples from the experience replay pool based on priority.

[0115] Specifically, the Q network in this embodiment includes a main Q network (main quality network) and a target Q network (target quality network). Both the main Q network and the target Q network are neural networks, for example, they can be set as a multi-layer perceptron or a convolutional neural network. Each sample can be represented as ,in Indicates the old state, Indicates action, Represents the reward value, Indicates a new state.

[0116] Will The input is sent to the main Q network, and after the multiplication of the weight matrix and the input data and the addition of the bias term, the nonlinear transformation is performed through the activation function, and the result is passed to the next layer. The calculation is done layer by layer, and the result is finally output by the output layer. Each neuron in the output layer corresponds to an action in the action space, so the final output result is a vector with the same length as the number of actions. The corresponding element, as the current quality value of the sample . Then Input to the main Q network, select the action corresponding to the maximum value from the elements of the output vector, recorded as , and Input to the target Q network, select the output vector with The corresponding element is recorded as , and calculate the target quality value of the sample based on the element for:

[0117] ;

[0118] in Represents the discount factor. Repeat the above steps for each sample to obtain the current quality value and target quality value of each sample.

[0119] For each sample, the time difference error of each sample is obtained by subtracting the corresponding current quality value from the target quality value. . Get the priority of each sample based on the time difference error in is a very small positive number, which is used to avoid the situation where the priority is zero when the error is zero, and ensure that each sample has a certain probability of being selected. The probability of a sample being selected is , where N represents the total number of samples in the experience replay pool, Indicates the The priority of the samples, is a hyperparameter used to control the influence of priority. The closer it is to 1, the greater the probability that a high-priority sample will be selected. The number of samples selected is determined according to actual needs, and the method of selecting samples in this embodiment is sampling with replacement.

[0120] This embodiment improves the traditional Q network. The Q network in the prior art only includes a main Q network. When the same network is used for action selection and action evaluation, as errors accumulate, the output value is prone to being too high, affecting the accuracy of the results. This embodiment decouples action selection and action evaluation and uses different networks. That is, the main Q network is used to select actions, and the target Q network is used to evaluate actions. Since the parameters in the target Q network are updated slowly, the frequency of parameter updates is greatly reduced compared to the main Q network, which can effectively avoid the transmission and accumulation of errors, thereby avoiding the above-mentioned problems.

[0121] In addition, this embodiment selects samples based on their priority, which can enable the deep reinforcement learning model to learn from high-priority samples more effectively compared to the traditional random selection method. High-priority samples usually contain more important and representative information. By prioritizing the learning of these samples, the model can capture key patterns and regularities in the environment more quickly, thereby more effectively adjusting its own parameters and accelerating the model convergence.

[0122] Furthermore, an embodiment of the present application provides a step of updating parameters of a deep reinforcement learning model based on multiple samples, including:

[0123] Get the current quality value and target quality value of each sample;

[0124] Calculate the loss value based on the current quality value, target quality value and preset loss function;

[0125] Update the parameters of the deep reinforcement learning model based on the loss value and the backpropagation algorithm.

[0126] Specifically, the mean square error between the current quality value and the target quality value can be used as the loss function. It can be expressed as:

[0127] ;

[0128] in It represents the number of selected samples, and then the parameters in the main Q network are updated through the loss value and the back propagation algorithm. This step is a prior art and will not be described in detail in this embodiment.

[0129] The above steps of selecting samples and updating parameters are then repeated for a preset number of repetitions. When the preset number of repetitions is reached, the parameters of the main Q network are copied to the target Q network to ensure that the parameters of the target Q network are relatively stable while gradually updating in line with the learning of the main Q network. The exploration rate is then updated by multiplying the current exploration rate by a preset decay factor. This preset decay factor is greater than zero and less than 1, and its specific value is determined based on the actual training situation. The above steps of selecting samples, updating parameters, and updating the exploration rate are then repeated until the model converges.

[0130] An embodiment of the present application provides a method for dynamic optimization of the path of an unloading and feeding robot, in which each robot independently runs a path decision algorithm and, through interactive communication between robots, updates the unloading area and unloading node when a collision occurs to improve work efficiency. Subsequently, a deep reinforcement learning method is used to further determine the unloading action sequence to ensure the smooth completion of the unloading work.

[0131] Furthermore, an embodiment of the present application provides an unloading and feeding robot, which is applied to the above-mentioned unloading and feeding robot path dynamic optimization method, and the unloading and feeding robot includes: a binocular camera, an image processing module, a computing module, a communication module and a robotic arm;

[0132] The binocular camera is used to obtain images of multiple unloading areas within a first preset range;

[0133] The image processing module is used for selecting a first region and a first node according to the image;

[0134] The calculation module is used to update the first area and the first node according to the second area and the second node to obtain the current unloading area and the current unloading node;

[0135] The communication module is used to obtain the second area and the second node;

[0136] The robotic arm is used to complete the unloading work in the current unloading area according to the action sequence.

[0137] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for dynamic optimization of the path of a feeding and unloading robot, characterized in that: There are multiple unloading and feeding robots located in the same storage area, the storage area includes multiple unloading areas, each unloading area corresponds to multiple nodes, and the unloading and feeding robot path dynamic optimization method is executed by one of the unloading and feeding robots. The unloading and feeding robot path dynamic optimization method includes: Acquire images of multiple unloading areas within a first preset range, and select a first area and a first node based on the images, wherein the first preset range is a circular area centered on itself, and the radius of the circular area is twice the distance between two diagonally adjacent nodes; Determine whether there is an adjacent robot; if so, obtain a second area and a second node; update the first area and the first node according to the second area and the second node to obtain a current unloading area and a current unloading node; the second area and the second node are the first area and the first node of the adjacent robot, and the adjacent robot is an unloading robot located within a second preset range, wherein the second preset range is a circular range centered on the first area and determined by the distance between the adjacent robot and the first area as a radius; Move to the current unloading node and complete the unloading work in the current unloading area according to the action sequence, wherein the action sequence is obtained through the deep reinforcement learning model; The selecting the first region and the first node according to the image includes: determining the depth of the material within the plurality of discharge areas based on the image; Obtaining the driving distances between the plurality of unloading areas and the vehicle itself; Obtaining an evaluation value corresponding to each unloading area according to the travel distance, the depth of the material and a preset evaluation index; The unloading area with the highest evaluation value is used as the first area, and a node closest to the own position among the nodes corresponding to the first area is used as the first node.

2. A method for dynamic optimization of the path of a feeding and unloading robot according to claim 1, characterized in that: The updating of the first area and the first node according to the second area and the second node to obtain the current unloading area and the current unloading node includes: Determining whether the second region and the second node are the same as the first region and the first node; If the second area is the same as the first area, updating the first area and the first node according to the evaluation value corresponding to each unloading area to obtain the current unloading area and the current unloading node; If the second area is different from the first area, and the second node is the same as the first node, the first area is used as the current unloading area, and the first node is updated to obtain the current unloading node.

3. A method for dynamic optimization of the path of a feeding and unloading robot according to claim 2, characterized in that: The updating of the first area and the first node according to the evaluation value corresponding to each unloading area to obtain the current unloading area and the current unloading node includes: Obtaining an evaluation value corresponding to the second area; If the evaluation value corresponding to the second area is greater than the evaluation value corresponding to the first area, the first area and the first node are updated to obtain the current unloading area and the current unloading node.

4. A method for dynamic optimization of the path of a feeding and unloading robot according to claim 3, characterized in that: The updating of the first area and the first node to obtain the current unloading area and the current unloading node includes: Eliminating the evaluation value corresponding to the first area from the evaluation values ​​corresponding to each unloading area to obtain a remaining evaluation value; The unloading area corresponding to the highest evaluation value among the remaining evaluation values ​​is used as the current unloading area, and a node closest to the own position among the nodes corresponding to the current unloading area is used as the current unloading node.

5. The method for dynamic optimization of the path of a feeding and unloading robot according to claim 1, characterized in that: The action sequence is obtained through a deep reinforcement learning model, including: Acquire the posture information after moving to the current unloading node, the posture information including the position coordinates of the manipulator arm, the posture angle of the manipulator arm, and the position coordinates of the material to be unloaded; Perform an iterative operation, the iterative operation including: inputting the posture information into the deep reinforcement learning model, obtaining a quality value for each action in a preset action space, selecting an action corresponding to the highest quality value, placing it in an action sequence, updating the posture information until the action of putting down the material is placed in the action sequence, and outputting the current action sequence.

6. A method for dynamic optimization of the path of a feeding and unloading robot according to claim 5, characterized in that: The deep reinforcement learning model is obtained through training, including: Initializing the parameters of the deep reinforcement learning model; Generate multiple samples in sequence based on environmental interactions, and store each sample in the experience replay pool in sequence; A plurality of samples are selected from the experience replay pool, and parameters of the deep reinforcement learning model are updated according to the plurality of samples until the deep reinforcement learning model converges.

7. A method for dynamic optimization of the path of a feeding and unloading robot according to claim 6, characterized in that: The selecting a plurality of samples from the experience replay pool includes: Calculating the current quality value of each sample according to a preset main quality network, and calculating the target quality value of each sample according to a preset target quality network; Calculating a time difference error of each sample according to the current quality value and the target quality value; Determining the priority of each sample according to the time difference error; A plurality of samples are selected from the experience replay pool according to the priority.

8. A method for dynamic optimization of the path of a feeding and unloading robot according to claim 7, characterized in that: Updating the parameters of the deep reinforcement learning model according to the multiple samples includes: Get the current quality value and target quality value of each sample; Calculating a loss value based on the current quality value, the target quality value, and a preset loss function; Update the parameters of the deep reinforcement learning model based on the loss value and the back-propagation algorithm.

9. A feeding and unloading robot, applied to a method for dynamic optimization of a feeding and unloading robot path according to any one of claims 1 to 8, characterized in that: The unloading and feeding robot includes: a binocular camera, an image processing module, a computing module, a communication module and a mechanical arm; The binocular camera is used to acquire images of multiple unloading areas within a first preset range; The image processing module is used to select a first area and a first node according to the image; The communication module is used to obtain the second area and the second node; The calculation module is used to update the first area and the first node according to the second area and the second node to obtain the current unloading area and the current unloading node; The robotic arm is used to complete the unloading work in the current unloading area according to the action sequence.

Images