Robot delivery process control method, delivery robot and storage medium

Abstract

The invention provides a robot delivery process control method, a delivery robot and a computer readable storage medium. The robot delivery process control method comprises the following steps: when the staying duration of a delivery robot at a delivery address of a target order is greater than a preset duration threshold, controlling the delivery robot to return to a distribution address of the target order; when the delivery robot arrives at the distribution address, acquiring surrounding figure images of the distribution address; performing identification based on the surrounding figure image, and determining a first relative distance between the delivery robot and a distribution person of the target order; and when the first relative distance is smaller than a preset distance threshold value, controlling the delivery robot to broadcast a delivery abnormity prompt of the target order. The problem that the delivery robot waits for a long time when delivery of the target order is abnormal can be avoided, resource waste of the delivery robot is reduced, and the delivery time efficiency of the delivery robot is improved to a certain extent.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot delivery process control method, a delivery robot, and a computer-readable storage medium. Background Art

[0002] As robot technology matures and consumers’ demand for automated services increases, robots are being used in more and more scenarios, such as hotels, retail, warehousing and logistics, etc. Order delivery is an important application of robots. At present, it is mainly the delivery personnel who complete the delivery of the orders for this shift for the delivery robot. The delivery robot will deliver the orders for this shift according to the delivery address of the orders for this shift. The delivery process of the order is completed after the delivery robot delivers the orders for this shift and the consignee of the orders for this shift successfully picks up the goods.

[0003] However, the inventors of the present application discovered during the actual research and development process that during the delivery process, the delivery robot will inevitably fail to deliver the goods due to reasons such as the absence of the recipient of the order. This will cause the delivery robot to wait for a long time when the order delivery is abnormal, resulting in a waste of resources of the delivery robot, and further leading to a low delivery efficiency of the delivery robot (for example, in a hotel meal delivery scenario, if the resident is not in the room after the robot delivers the meal to the target room, waiting for the resident to pick up the meal for a long time will result in a waste of resources on the machine side, and other orders will not be delivered in time). Summary of the invention

[0004] The present application provides a robot delivery process control method, a delivery robot and a computer-readable storage medium, which can avoid the problem of long waiting time of the delivery robot when the delivery of the target order is abnormal, reduce the waste of resources of the delivery robot, and improve the delivery timeliness of the delivery robot to a certain extent.

[0005] In a first aspect, the present application provides a robot delivery process control method, the method comprising:

[0006] When the delivery robot stays at the delivery address of the target order for longer than a preset time threshold, controlling the delivery robot to return to the delivery address of the target order;

[0007] When the delivery robot arrives at the delivery address, obtaining images of people around the delivery address;

[0008] Based on the surrounding person images, the first relative distance between the delivery robot and the delivery person of the target order is determined;

[0009] When the first relative distance is less than a preset distance threshold, the delivery robot is controlled to broadcast a delivery abnormality prompt of the target order.

[0010] In some embodiments, the method further comprises:

[0011] When the first relative distance is greater than or equal to a preset distance threshold, controlling the delivery robot to move toward the delivery person;

[0012] When the second relative distance between the moved position of the delivery robot and the delivery personnel is less than a preset distance threshold, a delivery abnormality prompt of the target order is broadcast.

[0013] In some embodiments, controlling the delivery robot to move toward the delivery person includes:

[0014] Acquire a moving image sequence of the cargo picker;

[0015] Recognize based on the moving image sequence, and determine the first moving direction of the delivery person;

[0016] Planning is performed according to the first moving direction and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot;

[0017] According to the second moving direction, the delivery robot is controlled to move toward the delivery personnel.

[0018] In some embodiments, the moving image sequence includes a first frame image and a second frame image, and the identifying based on the moving image sequence to determine the first moving direction of the delivery person includes:

[0019] Performing optical flow calculation based on the first frame image and the second frame image to obtain an optical flow map between the first frame image and the second frame image;

[0020] Acquire a delivery personnel area in the first frame of image;

[0021] The first moving direction of the picker is acquired according to the picker region in the first frame image and the optical flow map.

[0022] In some embodiments, the planning according to the first moving direction and the current spatial position of the delivery robot to obtain the second moving direction of the delivery robot includes:

[0023] Determining an estimated meeting position of the delivery robot and the cargo picker according to the first moving direction and the current spatial position of the delivery robot;

[0024] A second moving direction of the delivery robot is determined based on the estimated encounter position and the current spatial position of the delivery robot.

[0025] In some embodiments, the method further comprises:

[0026] Recognize based on the moving image sequence, and determine the first moving speed of the delivery person;

[0027] In some embodiments, the planning according to the first moving direction and the current spatial position of the delivery robot to obtain the second moving direction of the delivery robot includes:

[0028] Planning is performed according to the first moving direction, the first moving speed and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot.

[0029] In some embodiments, the step of identifying the surrounding person images and determining the first relative distance between the delivery robot and the delivery person of the target order includes:

[0030] Determine a target person image including the goods picker based on a comparison between the person image of the goods picker during goods pickering and the surrounding person images;

[0031] Obtaining the robot posture when the target person image is collected;

[0032] Determine the current spatial position of the picker based on the target person image and the robot posture;

[0033] The first relative distance is determined according to the current spatial position of the delivery personnel and the current spatial position of the delivery robot.

[0034] In some embodiments, the method further comprises:

[0035] Control the delivery robot to send delivery exception information of the target order to the back end.

[0036] In a second aspect, the present application also provides a delivery robot, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it executes any one of the robot delivery process control methods provided in the present application.

[0037] In a third aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the robot delivery process control method.

[0038] In the present application, on the one hand, when the delivery robot stays at the delivery address of the target order for a longer time than a preset time threshold, the delivery robot is controlled to return to the distribution address of the target order to avoid the problem of the delivery robot waiting for a long time when the delivery of the target order is abnormal, so that the delivery of other orders can be processed in time, thereby reducing the waste of resources of the delivery robot and improving the delivery timeliness of the delivery robot to a certain extent; on the other hand, by controlling the delivery robot to return to the distribution address of the target order, images of people around the distribution address are obtained to determine a first relative distance between the distribution personnel and the delivery robot. When the first relative distance is less than the preset distance threshold, the delivery robot is controlled to broadcast a delivery abnormality prompt of the target order, so that the distribution personnel can promptly know the failure of the delivery of the target order, so that the distribution personnel can re-arrange the delivery of the target order according to the actual situation, thereby improving the overall delivery timeliness of the delivery robot to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 is a schematic structural block diagram of a delivery robot provided in an embodiment of the present application;

[0041] Figure 2 It is a flow chart of a robot delivery process control method provided in an embodiment of the present application;

[0042] Figure 3 is another flow chart of the robot delivery process control method provided in the embodiment of the present application;

[0043] Figure 4 is a schematic flow chart of an embodiment of step 205 provided in an embodiment of the present application;

[0044] Figure 5 is another embodiment flow chart of step 205 provided in the embodiments of the present application;

[0045] Figure 6 It is a schematic diagram illustrating the movement of a delivery robot toward a delivery person provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0047] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0048] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0049] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.

[0050] Embodiments of the present application provide a robot delivery process control method, a delivery robot, and a computer-readable storage medium.

[0051] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0052] Figure 1 It is a schematic block diagram of the structure of a delivery robot provided in an embodiment of the present application.

[0053] like Figure 1 As shown, the delivery robot 100 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected via a bus 103, such as a PCIe (Peripheral Component Interconnect Express) bus.

[0054] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire delivery robot 100. The processor 101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0055] Specifically, the memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0056] Those skilled in the art will understand that Figure 1 The structure shown in the figure is merely a block diagram of a partial structure related to the embodiment scheme of the present application, and does not constitute a limitation on the delivery robot to which the embodiment scheme of the present application is applied. The specific delivery robot may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0057] The processor 101 is used to run the computer program stored in the memory 102, and implement any one of the robot delivery process control methods provided in the embodiments of the present application when executing the computer program. For example, the processor 101 is used to run the computer program stored in the memory 102, and can implement the following steps when executing the computer program:

[0058] When the delivery robot stays at the delivery address of the target order for longer than a preset time threshold, controlling the delivery robot to return to the delivery address of the target order;

[0059] When the delivery robot arrives at the delivery address, obtaining images of people around the delivery address;

[0060] Based on the surrounding person images, the first relative distance between the delivery robot and the delivery person of the target order is determined;

[0061] When the first relative distance is less than a preset distance threshold, the delivery robot is controlled to broadcast a delivery abnormality prompt of the target order.

[0062] In some embodiments, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, the following steps may be implemented:

[0063] When the first relative distance is greater than or equal to a preset distance threshold, controlling the delivery robot to move toward the delivery person;

[0064] When the second relative distance between the moved position of the delivery robot and the delivery personnel is less than a preset distance threshold, a delivery abnormality prompt of the target order is broadcast.

[0065] In some embodiments, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, the following steps may be implemented:

[0066] Acquire a moving image sequence of the cargo picker;

[0067] Recognize based on the moving image sequence, and determine the first moving direction of the delivery person;

[0068] Planning is performed according to the first moving direction and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot;

[0069] According to the second moving direction, the delivery robot is controlled to move toward the delivery personnel.

[0070] In some embodiments, the moving image sequence includes a first frame image and a second frame image, and the processor 101 is used to run a computer program stored in the memory 102, and can implement the following steps when executing the computer program:

[0071] Performing optical flow calculation based on the first frame image and the second frame image to obtain an optical flow map between the first frame image and the second frame image;

[0072] Acquire a delivery personnel area in the first frame of image;

[0073] The first moving direction of the picker is acquired according to the picker region in the first frame image and the optical flow map.

[0074] In some embodiments, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, the following steps may be implemented:

[0075] Determining an estimated meeting position of the delivery robot and the cargo picker according to the first moving direction and the current spatial position of the delivery robot;

[0076] A second moving direction of the delivery robot is determined based on the estimated encounter position and the current spatial position of the delivery robot.

[0077] In some embodiments, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, the following steps may be implemented:

[0078] Recognition is performed based on the moving image sequence to determine a first moving speed of the picker.

[0079] In some embodiments, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, the following steps may be implemented:

[0080] Planning is performed according to the first moving direction, the first moving speed and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot.

[0081] In some embodiments, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, the following steps may be implemented:

[0082] Determine a target person image including the goods picker based on a comparison between the person image of the goods picker during goods pickering and the surrounding person images;

[0083] Obtaining the robot posture when the target person image is collected;

[0084] Determine the current spatial position of the picker based on the target person image and the robot posture;

[0085] The first relative distance is determined according to the current spatial position of the delivery personnel and the current spatial position of the delivery robot.

[0086] In some embodiments, the processor 101 is used to run a computer program stored in the memory 102, and when executing the computer program, the following steps may be implemented:

[0087] Control the delivery robot to send delivery exception information of the target order to the back end.

[0088] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the delivery robot described above can refer to the corresponding process in the following robot delivery process control method embodiment, and will not be repeated here.

[0089] In the following, Figure 1 The delivery robot shown in the figure is taken as an example as the execution subject of the robot delivery process control method, and the robot delivery process control method provided in the embodiment of the present application is described in detail. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments.

[0090] See also Figure 2 , Figure 2 201 to 204, wherein:

[0091] 201. When the delivery robot stays at the delivery address of the target order for longer than a preset time threshold, control the delivery robot to return to the delivery address of the target order.

[0092] In this embodiment, the order delivery process includes three parts: order matching, order delivery and order pickup: after the matching personnel complete the order matching for the delivery robot, the delivery robot performs the order delivery process according to the delivery address of the order of this shift, and after the delivery robot arrives at the delivery address of the order of this shift (the robot opens the hatch), the user picks up the goods and completes the entire order delivery process. Among them, the target order can be an express order, a takeaway order, etc., and the specific form of the target order is not limited here. For example, in the hotel delivery scenario, the target order can be an order to deliver daily necessities to each household, and in the hotel or office coffee delivery scenario, the target order can be an order to deliver coffee to each user.

[0093] There are many ways to implement step 201, illustratively including:

[0094] (1) In some embodiments, a delivery robot can perform the delivery process of one order in one shift. When the delivery robot stays at the delivery address of the target order for a time longer than a preset time threshold, the delivery robot can be controlled to return to the delivery address of the target order. For example, a compartment is set in the cargo compartment of the delivery robot, and the delivery process of one order is performed at a time. At this time, in step 201, after the delivery robot arrives at the delivery address of the target order, the timing starts, and the length of time the delivery robot stays at the delivery address of the target order is counted. When the length of time is longer than a preset time threshold (for example, 5 minutes), the delivery robot is controlled to return to the delivery address of the target order.

[0095] (2) In some embodiments, a delivery robot can also perform the delivery process of multiple orders in one shift. When the delivery robot stays at the delivery address of the target order for a time longer than a preset time threshold and completes the delivery of the remaining orders of this shift, the delivery robot is controlled to return to the delivery address of the target order. For example, the cargo compartment of the delivery robot is provided with multiple compartments, and the delivery process of multiple orders is performed at a time. At this time, in step 201, after the delivery robot arrives at the delivery address of the target order, the timing is started, and the length of time the delivery robot stays at the delivery address of the target order is counted. When the length of time is longer than the preset time threshold (for example, 5 minutes), the delivery robot no longer waits, but continues to perform the delivery process of the remaining orders of this shift. When the remaining orders are delivered, the delivery robot is controlled to return to the delivery address of the target order.

[0096] 202. When the delivery robot arrives at the delivery address, obtain images of people around the delivery address.

[0097] Exemplarily, a camera may be provided on the delivery robot. When delivering the target order, the camera provided on the delivery robot may be used to capture the image of the delivery personnel of the target order. After the delivery robot returns to the delivery address of the target order, the camera of the delivery robot is turned on to preview the area around the delivery address. When a person is detected in the surrounding environment, the camera of the delivery robot may be used to capture the image of the person around the delivery address.

[0098] 203. Determine a first relative distance between the delivery robot and a delivery person for the target order based on recognition of the surrounding person images.

[0099] The first relative distance refers to the distance between the delivery robot and the delivery personnel detected before the delivery robot is controlled to move toward the delivery personnel.

[0100] There are many ways to implement step 203, which include, for example:

[0101] (1) In some embodiments, the image of the surrounding people is acquired by a depth camera, and the first relative distance is determined based on the depth information of the pixel position of the delivery personnel in the image. In this case, step 203 may specifically include 2031A to 2032A:

[0102] 2031A. Determine a target person image including the goods picker based on a comparison between the person image of the goods picker during goods pickering and the surrounding person images.

[0103] The implementation of step 2031A is similar to that of step 2031B. For details, please refer to the relevant instructions below and will not be repeated here.

[0104] 2032A. Determine a first relative distance between the delivery robot and the delivery personnel of the target order based on the depth information of the pixel position of the delivery personnel in the target person image.

[0105] Exemplarily, the target person image is a depth image, and each pixel in the target person image contains depth information, which is used to indicate the distance between the object corresponding to each pixel and the depth camera. In step 202, the depth camera of the delivery robot collects surrounding person images, and after the target person image is identified and determined in step 2031A, the depth value of the pixel position corresponding to the delivery person in the target person image is obtained as the first relative distance between the delivery robot and the delivery person.

[0106] (2) In some embodiments, the first relative distance may be determined based on the current spatial position of the delivery robot and the current spatial position of the delivery personnel. In this case, step 203 may specifically include the following steps 2031B to 2034B:

[0107] 2031B. Determine a target person image including the goods picker based on a comparison between the person image of the goods picker during goods pickering and the surrounding person images.

[0108] Exemplarily, the delivery robot may collect multiple surrounding person images, such as collecting one surrounding person image in front of, behind, on the left side, and on the right side of the stop position of the delivery robot, and performing the following processing on each collected surrounding person image i: performing human body detection on the surrounding person image i through the target detection model to obtain the portrait area in the surrounding person image i; comparing each portrait area in the surrounding person image i with the person image when picking goods, if the similarity between a portrait area in the surrounding person image i and the person image when picking goods is greater than a preset similarity threshold, then the surrounding person image i is used as the target person image. When the target person image is detected, other surrounding person images may not be processed (for example, the first, second, third, and fourth surrounding person images are collected in front of, behind, on the left side, and on the back side of the stop position of the delivery robot, and the first surrounding person image is identified to find that it does not contain the picker, then the second surrounding person image is continued to be identified, and if the second surrounding person image is identified to find that it contains the picker, then the second surrounding person image is used as the target person image, and the third and fourth surrounding person images are no longer identified, and so on), so as to save processing resources.

[0109] 2032B. Obtain the robot posture when the target person image is collected.

[0110] Among them, the robot posture refers to the posture of the delivery robot when collecting the image of the target person.

[0111] In some embodiments, the robot posture when the target person image is collected can be obtained by visual odometer. Through visual odometer, first, feature matching can be performed on adjacent image pairs collected by the delivery robot (the current collected image frame and the previous image frame can constitute an adjacent image pair), and the relative displacement of the delivery robot between the position when the current collected image frame is collected (the latest position) and the position when the previous image frame is collected (i.e., the initial position) is estimated; then, according to the relative displacement of the delivery robot between (the position when the current collected image frame is collected and the position when the previous image frame is collected), and the position when the previous image frame is collected (i.e., the initial position), the posture of the delivery robot when the current collected image frame is collected can be calculated, and so on, the posture of the delivery robot when the target person image is collected can be calculated in step 2031B as the robot posture.

[0112] In some embodiments, the robot posture when the target person image is collected can be obtained by SLAM (Simultaneous Localization and Mapping) algorithm. The robot posture is obtained by building an environmental map in real time and estimating the position of the delivery robot in the environmental map.

[0113] In some embodiments, a wheel encoder can be installed on the delivery robot. The wheel encoder can be used to infer the position and orientation of the delivery robot when the target person image is captured based on the spatial position of the delivery robot's starting point, the rotation angle and speed of the delivery robot's wheels, thereby obtaining the robot's posture when the target person image is captured.

[0114] 2033B. Determine the current spatial position of the delivery person based on the target person image and the robot posture.

[0115] Exemplarily, the pixel position (u, v) of the picker is obtained (i.e., the pixel position (u, v) of the picker in the target person image); then, according to the pixel position (u, v) of the picker, the depth information of the pixel position (u, v), the preset conversion relationship between the image coordinate system and the camera coordinate system, and the preset conversion relationship between the camera coordinate system and the robot coordinate system, the pixel position (u, v) of the picker is converted to the robot coordinate system to obtain the robot coordinate system position (X r ,Y r ,Z r ); Finally, according to the robot posture and the robot coordinate system position (X r ,Y r ,Z r ), and the preset conversion relationship between the robot coordinate system and the three-dimensional space coordinate system, calculate the three-dimensional space coordinate system position (X w ,Yw ,Z w ) as the current spatial location of the picker.

[0116] 2034B. Determine the first relative distance according to the current spatial position of the delivery personnel and the current spatial position of the delivery robot.

[0117] 204. When the first relative distance is less than a preset distance threshold, control the delivery robot to broadcast a delivery abnormality prompt of the target order.

[0118] For example, when the first relative distance is less than a preset distance threshold (such as 1.5 meters), the delivery robot is controlled to broadcast a delivery abnormality prompt of the target order, "The delivery of the order for room XX failed, and the reason for the failure was that the goods were not picked up for a long time after the order was delivered." In this way, the delivery abnormality prompt of the target order can be broadcast when the relative distance between the delivery robot and the delivery personnel of the target order is close, so that the delivery personnel can be informed of the delivery failure of the target order in time, so that the delivery personnel can re-arrange the delivery of the target order according to the actual situation (for example, after all orders are delivered, after a certain interval, and after confirming the time with the delivery personnel of the target order, re-deliver), thereby avoiding the problem of long waiting time caused by the abnormal delivery of the target order after the delivery robot delivers the target order, reducing the waste of delivery robot resources, and enabling other orders to be delivered in time, thereby improving the overall delivery timeliness of the delivery robot to a certain extent.

[0119] Furthermore, in order to avoid the situation where the delivery personnel are far away from the delivery robot, the delivery personnel cannot receive the delivery abnormality prompt of the target order in time, such as Figure 3 As shown, the robot delivery process control method may further include the following steps 205-206:

[0120] 205. When the first relative distance is greater than or equal to a preset distance threshold, control the delivery robot to move toward the delivery personnel.

[0121] Exemplarily, when the first relative distance is greater than or equal to a preset distance threshold and the abnormal delivery status of the target order has not been processed for more than a specified time, the delivery robot is controlled to move toward the delivery personnel.

[0122] There are multiple ways to control the delivery robot to move toward the delivery personnel in step 205. For example, the following methods are included: ①, ② and ③:

[0123] ① In some embodiments, based on the target person image including the delivery person, the current spatial position of the delivery person is identified, and the delivery robot is controlled to move directly from the current spatial position of the delivery robot to the current spatial position of the delivery person. Figure 4As shown, step 205 may specifically include the following steps 2051A to 2052A:

[0124] 2051A. Obtain the current spatial position of the delivery person.

[0125] In some embodiments, step 203 is determined using steps 2031B to 2034B. In this case, step 2051A may directly use the result of step 2033B as the current spatial position of the delivery person.

[0126] 2052A. Control the delivery robot to move from the current spatial position of the delivery robot toward the current spatial position of the delivery personnel.

[0127] ② In some embodiments, in order to prevent the delivery personnel from moving, resulting in a large distance between the delivery robot's moving spatial position and the delivery personnel's moving spatial position, the delivery robot can be identified and controlled to move toward the delivery personnel according to the delivery personnel's first moving direction. Figure 5 As shown, step 205 may specifically include the following steps 2051B to 2054B:

[0128] 2051B. Acquire a moving image sequence of the cargo picker.

[0129] 2052B. Perform recognition based on the moving image sequence to determine the first moving direction of the delivery personnel.

[0130] Exemplarily, step 2052B may specifically include: performing optical flow calculation based on the first frame image and the second frame image to obtain an optical flow map between the first frame image and the second frame image; obtaining a picker area in the first frame image; and obtaining a first moving direction of the picker based on the picker area in the first frame image and the optical flow map.

[0131] The calculation of the optical flow map is based on the brightness consistency assumption, that is, it is assumed that the brightness of the object is constant between two adjacent frames, such as the first frame image I1 (x, y) and the second frame image I2 (x, y), that is, the pixel value of the object remains unchanged at different times. In this embodiment, this principle is used to assume that the brightness of the corresponding pixel of the goods picker is constant between the first frame image I1 (x, y) and the second frame image I2 (x, y), that is, when the first frame image I1 (x, y) is collected and when the second frame image I2 (x, y) is collected, the pixel value of the corresponding pixel of the goods picker remains unchanged, as shown in the following formula 1:

[0132] I1(x,y,t)=I2(x+Δx,y+Δy,t+Δt) Formula 1

[0133] In Formula 1, Δx and Δy represent the displacement of the corresponding pixels of the picker in the horizontal and vertical directions of the image, respectively, and Δt represents the time difference between the first frame image I1(x, y) and the second frame image I2(x, y).

[0134] Then, based on this assumption, the horizontal gradient and vertical gradient of each pixel in the first frame image I1(x,y) are calculated according to the first frame image I1(x,y) and the second frame image I2(x,y); and the time gradient of the image is calculated according to the time difference between the first frame image I1(x,y) and the second frame image I2(x,y).

[0135] Next, the horizontal gradient, vertical gradient, and time gradient of each pixel in the first frame image I1(x,y) are substituted into the optical flow calculation formula shown in Formula 2 below to obtain the optical flow vector (u,v) of each pixel in the first frame image I1(x,y); using the optical flow vector (u,v) of each pixel in the first frame image I1(x,y), an optical flow map containing the optical flow vector (u,v) of each pixel in the first frame image I1(x,y) can be obtained as the optical flow map between the first frame image and the second frame image. At this time, the resolution of the obtained optical flow map is the same as that of the first frame image, and the optical flow vector of each pixel in the optical flow map represents the movement direction and speed of the pixel. Formula 2 is as follows:

[0136] I x u+I y v+I t =0 Formula 2

[0137] In formula 2, I x ,I y represents the horizontal gradient and vertical gradient of each pixel in the first frame image I1(x, y), u and v represent the horizontal motion component and vertical motion component of each pixel in the first frame image I1(x, y), respectively. t Represents the temporal gradient of an image.

[0138] Next, the target detection model is used to detect the portrait areas in the first frame image I1(x, y); then, the portrait areas in the first frame image I1(x, y) are compared with the image of the person at the time of goods distribution, and the portrait areas in the first frame image I1(x, y) whose similarity with the image of the person at the time of goods distribution is greater than a preset similarity threshold are used as the distribution personnel areas.

[0139] Finally, according to the picker area of ​​the first frame image I1(x, y), the optical flow information of the pixel position corresponding to the picker area is extracted from the optical flow map, and the average optical flow vector of the pixel position corresponding to the picker area is calculated; the direction of the average optical flow vector is used as the overall movement direction of the picker, so that the first movement direction of the picker can be obtained.

[0140] 2053B. Plan according to the first moving direction and the current spatial position of the delivery robot to obtain the second moving direction of the delivery robot.

[0141] Exemplarily, step 2053B may specifically include: determining an estimated meeting position between the delivery robot and the delivery person according to the first moving direction and the current spatial position of the delivery robot; and determining a second moving direction of the delivery robot according to the estimated meeting position and the current spatial position of the delivery robot. Figure 6 As shown, assuming that the current spatial position of the picker is point A, according to the first moving direction (such as Figure 6 direction 1) and the current spatial position of the delivery robot (e.g. Figure 6 ), the estimated meeting position of the delivery robot and the picker is determined to be C; and based on the estimated meeting position C, the current spatial position B of the delivery robot and the second speed of the delivery robot, the second moving direction of the delivery robot is determined to be direction 2; finally, the delivery robot can be controlled to move according to direction 2, so that the delivery robot continuously moves from the current spatial position B of the delivery robot toward the estimated meeting position C, and continuously approaches the picker.

[0142] 2054B. According to the second moving direction, control the delivery robot to move toward the delivery personnel.

[0143] ③ In some embodiments, in order to prevent the delivery personnel from moving, resulting in a large distance between the spatial position of the delivery robot after movement and the spatial position of the delivery personnel after movement, the delivery robot can be controlled to move toward the delivery personnel according to the first moving direction and the first moving speed of the delivery personnel. At this time, step 205 can specifically include the following steps 2051C to 2055C:

[0144] 2051C. Acquire a moving image sequence of the cargo picker.

[0145] 2052C. Perform recognition based on the moving image sequence to determine the first moving direction of the delivery personnel.

[0146] The implementation of steps 2051C to 2052C is similar to that of steps 2051B to 2052B. For details, please refer to the relevant descriptions in the previous text, which will not be repeated here.

[0147] 2053C. Determine a first moving speed of the delivery personnel based on recognition of the moving image sequence.

[0148] In some embodiments, an optical flow calculation can be performed based on the first frame image and the second frame image to obtain an optical flow map between the first frame image and the second frame image; obtain a picker area in the first frame image; and obtain a first moving speed of the picker based on the picker area in the first frame image and the optical flow map. Exemplarily, the optical flow map between the first frame image and the second frame image can be determined with reference to the method in step 2052B, and based on the picker area of ​​the first frame image I1(x,y), the optical flow information of the pixel position corresponding to the picker area is extracted from the optical flow map, and the average optical flow vector of the pixel position corresponding to the picker area is calculated; the speed of the average optical flow vector is used as the overall moving speed of the picker, so that the first moving speed of the picker can be obtained.

[0149] 2054C. Planning is performed according to the first moving direction, the first moving speed and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot.

[0150] 2055C. According to the second moving direction, control the delivery robot to move toward the delivery personnel.

[0151] 206. When the second relative distance between the moved position of the delivery robot and the delivery personnel is less than a preset distance threshold, broadcast a delivery abnormality prompt of the target order.

[0152] The second relative distance refers to the distance between the delivery robot and the delivery personnel detected after the delivery robot is controlled to move toward the delivery personnel.

[0153] Specifically, the delivery robot is controlled to move toward the picker in a second moving direction, and a second relative distance between the delivery robot's moved position and the picker is detected. Until it is detected that the second relative distance between the delivery robot's moved position and the picker is less than a preset distance threshold, a delivery abnormality prompt of the target order is broadcast, and the delivery robot can be controlled to stop moving toward the picker.

[0154] In this way, when the relative distance between the delivery robot and the picker of the target order is far, the delivery robot can be controlled to move towards the picker, and when the relative distance between the delivery robot and the picker of the target order is close, the delivery abnormality prompt of the target order is broadcast, so that the picker can be informed of the delivery failure of the target order in time.

[0155] Furthermore, the robot delivery process control method also includes: controlling the delivery robot to send the delivery exception information of the target order to the back end, so that the back end sends the delivery exception information of the target order to the front end, so that the picking personnel can receive the delivery exception information of the target order through the front end and make further disposal. At this time, after the target order that failed to be delivered returns to the delivery address, the picking personnel can choose to take the target order that failed to be delivered out of the grid first, continue to pick other orders and complete the delivery, so as to avoid the delivery process of the target order occupying the grid resources of the delivery robot for a long time, thereby improving the delivery efficiency of the delivery robot to a certain extent.

[0156] From the above content, it can be seen that, on the first hand, when the delivery robot stays at the delivery address of the target order for a longer time than the preset time threshold, the delivery robot is controlled to return to the distribution address of the target order to avoid the problem of the delivery robot waiting for a long time when the delivery of the target order is abnormal, so that the delivery of other orders can be handled in time, thereby reducing the waste of resources of the delivery robot and improving the delivery efficiency of the delivery robot to a certain extent; on the second hand, by controlling the delivery robot to return to the distribution address of the target order, the image of the people around the distribution address is obtained to determine the first relative distance between the distribution personnel and the delivery robot. When the first relative distance is less than the preset distance threshold, the delivery robot is controlled to broadcast the delivery abnormality prompt of the target order. , so that the delivery personnel can be informed of the failure of delivery of the target order in time, so that the delivery personnel can re-arrange the delivery of the target order according to the actual situation, and improve the overall delivery timeliness of the delivery robot to a certain extent; thirdly, by obtaining the image of the surrounding people of the delivery address, it is used to determine the first relative distance between the delivery personnel and the delivery robot. When the first relative distance is less than the preset distance threshold, the delivery robot is controlled to broadcast the delivery abnormality prompt of the target order, so as to avoid the problem that the delivery personnel is busy and cannot be informed of the return of the delivery robot to the delivery address in time. Therefore, after the delivery robot returns to the delivery address, the delivery personnel can deal with the delivery abnormality of the target order in time or continue to pick the goods, thereby improving the overall delivery efficiency of the delivery robot.

[0157] A person of ordinary skill in the art will appreciate that all or part of the steps in the above-mentioned robot delivery process control method may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0158] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute any one of the robot delivery process control methods provided in the embodiment of the present application. For example, the computer program can be loaded by a processor to execute the following steps:

[0159] When the delivery robot stays at the delivery address of the target order for longer than a preset time threshold, controlling the delivery robot to return to the delivery address of the target order;

[0160] When the delivery robot arrives at the delivery address, obtaining images of people around the delivery address;

[0161] Based on the surrounding person images, the first relative distance between the delivery robot and the delivery person of the target order is determined;

[0162] When the first relative distance is less than a preset distance threshold, the delivery robot is controlled to broadcast a delivery abnormality prompt of the target order.

[0163] In some embodiments, the computer program can be loaded by a processor to perform the following steps:

[0164] When the first relative distance is greater than or equal to a preset distance threshold, controlling the delivery robot to move toward the delivery person;

[0165] When the second relative distance between the moved position of the delivery robot and the delivery personnel is less than a preset distance threshold, a delivery abnormality prompt of the target order is broadcast.

[0166] In some embodiments, the computer program can be loaded by a processor to perform the following steps:

[0167] Acquire a moving image sequence of the cargo picker;

[0168] Recognize based on the moving image sequence, and determine the first moving direction of the delivery person;

[0169] Planning is performed according to the first moving direction and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot;

[0170] According to the second moving direction, the delivery robot is controlled to move toward the delivery personnel.

[0171] In some embodiments, the moving image sequence includes a first frame image and a second frame image, and the computer program can be loaded by a processor to perform the following steps:

[0172] Performing optical flow calculation based on the first frame image and the second frame image to obtain an optical flow map between the first frame image and the second frame image;

[0173] Acquire a delivery personnel area in the first frame of image;

[0174] The first moving direction of the picker is acquired according to the picker region in the first frame image and the optical flow map.

[0175] In some embodiments, the computer program can be loaded by a processor to perform the following steps:

[0176] Determining an estimated meeting position of the delivery robot and the cargo picker according to the first moving direction and the current spatial position of the delivery robot;

[0177] A second moving direction of the delivery robot is determined based on the estimated encounter position and the current spatial position of the delivery robot.

[0178] In some embodiments, the computer program can be loaded by a processor to perform the following steps:

[0179] Recognition is performed based on the moving image sequence to determine a first moving speed of the picker.

[0180] In some embodiments, the computer program can be loaded by a processor to perform the following steps:

[0181] Planning is performed according to the first moving direction, the first moving speed and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot.

[0182] In some embodiments, the computer program can be loaded by a processor to perform the following steps:

[0183] Determine a target person image including the goods picker based on a comparison between the person image of the goods picker during goods pickering and the surrounding person images;

[0184] Obtaining the robot posture when the target person image is collected;

[0185] Determine the current spatial position of the picker based on the target person image and the robot posture;

[0186] The first relative distance is determined according to the current spatial position of the delivery personnel and the current spatial position of the delivery robot.

[0187] In some embodiments, the computer program can be loaded by a processor to perform the following steps:

[0188] Control the delivery robot to send delivery exception information of the target order to the back end.

[0189] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0190] In the above robot delivery process control method, computer readable storage medium, and delivery robot embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described computer readable storage medium, delivery robot and its corresponding units can refer to the description of the robot delivery process control method in the above embodiment, and will not be repeated here.

[0191] The above is a detailed introduction to a robot delivery process control method, a delivery robot and a computer-readable storage medium provided in an embodiment of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A robot delivery process control method, characterized in that: The method comprises: When the delivery robot stays at the delivery address of the target order for longer than a preset time threshold, controlling the delivery robot to return to the delivery address of the target order; When the delivery robot arrives at the delivery address, obtaining images of people around the delivery address; Based on the surrounding person images, the first relative distance between the delivery robot and the delivery person of the target order is determined; When the first relative distance is less than a preset distance threshold, the delivery robot is controlled to broadcast a delivery abnormality prompt of the target order.

2. The robot delivery process control method according to claim 1, characterized in that: The method further comprises: When the first relative distance is greater than or equal to a preset distance threshold, controlling the delivery robot to move toward the delivery person; When the second relative distance between the moved position of the delivery robot and the delivery personnel is less than a preset distance threshold, a delivery abnormality prompt of the target order is broadcast.

3. The robot delivery process control method according to claim 2, characterized in that: The controlling the delivery robot to move toward the cargo picker comprises: Acquire a moving image sequence of the cargo picker; Recognize based on the moving image sequence, and determine the first moving direction of the delivery person; Planning is performed according to the first moving direction and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot; According to the second moving direction, the delivery robot is controlled to move toward the delivery personnel.

4. The robot delivery process control method according to claim 3, characterized in that: The moving image sequence includes a first frame image and a second frame image, and the identifying based on the moving image sequence to determine the first moving direction of the goods picker includes: Performing optical flow calculation based on the first frame image and the second frame image to obtain an optical flow map between the first frame image and the second frame image; Acquire a delivery personnel area in the first frame of image; The first moving direction of the picker is acquired according to the picker region in the first frame image and the optical flow map.

5. The robot delivery process control method according to claim 3, characterized in that: The planning according to the first moving direction and the current spatial position of the delivery robot to obtain the second moving direction of the delivery robot includes: Determining an estimated meeting position of the delivery robot and the cargo picker according to the first moving direction and the current spatial position of the delivery robot; A second moving direction of the delivery robot is determined based on the estimated encounter position and the current spatial position of the delivery robot.

6. The robot delivery process control method according to claim 3, characterized in that: The method further comprises: Recognize based on the moving image sequence, and determine the first moving speed of the delivery person; The planning according to the first moving direction and the current spatial position of the delivery robot to obtain the second moving direction of the delivery robot includes: Planning is performed according to the first moving direction, the first moving speed and the current spatial position of the delivery robot to obtain a second moving direction of the delivery robot.

7. The robot delivery process control method according to claim 1, characterized in that: The step of identifying the surrounding person images and determining a first relative distance between the delivery robot and the delivery person of the target order includes: Determine a target person image including the goods picker based on a comparison between the person image of the goods picker during goods pickering and the surrounding person images; Obtaining the robot posture when the target person image is collected; Determine the current spatial position of the picker based on the target person image and the robot posture; The first relative distance is determined according to the current spatial position of the delivery personnel and the current spatial position of the delivery robot.

8. The robot delivery process control method according to claim 1, characterized in that: The method further comprises: Control the delivery robot to send delivery exception information of the target order to the back end.

9. A delivery robot, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the robot delivery process control method as claimed in any one of claims 1 to 8 is executed.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the robot delivery process control method as described in any one of claims 1 to 8.

Images