Control device and method of sorting robot

Abstract

The invention relates to the technical field of robot control, in particular to a control device and method for a sorting robot, and the method comprises a mechanical arm which is used for responding to a control instruction and executing a sorting action corresponding to sorting of articles to be sorted; the image acquisition equipment is used for acquiring a panoramic image of the target to-be-sorted area; the mechanical main control subsystem is used for generating a control instruction for controlling the mechanical arm to sort the to-be-sorted articles according to the panoramic image; the visual main control subsystem is used for identifying and positioning a sorting target object in the panoramic picture according to a target sorting requirement; the operation equipment is used for displaying the panoramic image and sending a high-priority instruction of an operator for capturing any one or any type of targets; and the remote terminal subsystem is used for monitoring the motion information of the mechanical arm so as to generate a mechanical arm parameter adjusting instruction according to the motion information. Therefore, the problem that cost control is difficult to carry out under the condition that certain key technical indexes such as control precision, processing speed and operation stability are guaranteed is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot control, and in particular to a control device and method for a sorting robot. Background Art

[0002] With the rapid development of e-commerce and intelligent manufacturing industries, product sorting faces new pressures. The traditional manual sorting method cannot meet the growing demand for logistics sorting in terms of efficiency, and the market is in urgent need of automated and intelligent sorting robots. At the same time, the development of high-tech technologies such as artificial intelligence and machine vision also provides technical support for the research and development and deployment of sorting robots. The application of these technologies enables sorting robots to more accurately identify item information and improve sorting efficiency and accuracy. On the other hand, due to the cost pressure and environmental protection policies generally faced by the industry, the low-cost realization of sorting robots has become the mainstream market demand. The technical difficulty and core component of the sorting robot is its control system. The control system should include functions such as kinematic control of the robotic arm servo, identification and positioning of the sorting target, etc. The mechanical control accuracy, target recognition accuracy, and human-computer interaction intelligence of the system determine the overall performance of the sorting robot.

[0003] Although the development of high-reliability, high-intelligence, and low-cost sorting robot control systems has become an important research direction, achieving cost control while ensuring certain key technical indicators such as control accuracy, processing speed, and operating stability is still a recognized difficulty in the industry. Summary of the invention

[0004] The present invention provides a control device and method for a sorting robot, so as to solve the problem that it is difficult to carry out cost control while ensuring certain key technical indicators such as control accuracy, processing speed, and operation stability.

[0005] A first aspect of the present invention provides a control device for a sorting robot, comprising: a mechanical arm, the mechanical arm is arranged between a target area to be sorted and a sorting area, so as to respond to a control instruction and perform a sorting action corresponding to sorting items to be sorted; an image acquisition device, the image acquisition device is arranged in the target area to be sorted, so as to acquire a panoramic image of the target area to be sorted; a mechanical main control subsystem, the mechanical main control subsystem is respectively connected to the mechanical arm and the image acquisition device, so as to generate a control instruction for controlling the mechanical arm to sort the items to be sorted according to the panoramic image; a visual main control subsystem, the visual main control subsystem is connected to the mechanical main control subsystem, and is used to identify and locate the sorting target object in the panoramic picture according to the target sorting requirement; an operating device, the operating device is connected to the visual main control subsystem, and is used to display the panoramic image and send a high priority instruction of an operator to grab any one or any type of target; a remote terminal subsystem, the remote terminal subsystem is connected to the mechanical main control subsystem, and is used to monitor the motion information of the mechanical arm, so as to generate a mechanical arm parameter adjustment instruction according to the motion information.

[0006] Optionally, the mechanical main control subsystem includes:

[0007] A visual preprocessing module, used for receiving and processing the panoramic image of the image acquisition device to obtain video stream data, and preparing the video stream data in HDMI protocol for transmission to the visual main control subsystem;

[0008] The integrated communication A module is used to verify the information of the target to be sorted between the mechanical main control subsystem and the visual main control subsystem to obtain the target information;

[0009] The robot arm control module is used to receive the target information, generate a control instruction for controlling the robot arm to perform sorting according to the target information, and map the motion information of the robot arm to the remote terminal subsystem.

[0010] Optionally, the integrated communication A module is provided with three synchronization lines, namely, F1 thread, F2 thread and F3 thread, wherein:

[0011] The F1 thread is used to verify the to-be-sorted target information sent by the visual main control subsystem using UART protocol communication to obtain the target information, and send the target information to the robotic arm control module;

[0012] The F2 thread is used to communicate to the visual main control subsystem that the current robot arm has completed sorting and needs to update the sorting target object, otherwise it communicates the visual main control subsystem approval information;

[0013] The F3 thread is used to receive the robot arm parameter adjustment instruction sent by the remote terminal subsystem.

[0014] Optionally, the target information includes inspection information, categories of target objects to be sorted, horizontal plane lateral coordinates of the target objects to be sorted, horizontal plane longitudinal coordinates of the target objects to be sorted, and horizontal rotation angles of the target objects to be sorted relative to an expected angle after sorting is completed.

[0015] Optionally, the visual main control subsystem includes:

[0016] A visual solution module, used for receiving the video stream data of the mechanical main control subsystem, identifying the sorting target object in the video stream data by using a neural network model, and calibrating the sorting target object according to hand-eye to generate a target data queue;

[0017] The integrated communication B module is used to provide a communication interface for the visual main control subsystem and the mechanical main control subsystem to transmit communication content and the target information;

[0018] The human-computer interaction module is used to display the real-time sorting situation and send a high-priority instruction to grab any one or any type of target instructed by the operator through the operating device.

[0019] Optionally, the remote terminal subsystem includes:

[0020] An integrated communication C module, used to cooperate with the integrated communication A module of the mechanical main control subsystem;

[0021] A digital twin module, used to build a simulated robotic arm in a virtual scene to simulate the movement of the robotic arm using real physical parameters;

[0022] The parameter adjustment module is used to correct the hand-eye calibration of the robotic arm, obtain the robotic arm parameter adjustment instruction, and adjust the visual parameters of the image acquisition device according to the light conditions of the deployment environment of the robotic arm.

[0023] A second aspect of the present invention provides a control method for a sorting robot, comprising:

[0024] Using an image acquisition device to acquire a panoramic image of the target area to be sorted, and sending the panoramic image to the mechanical main control subsystem;

[0025] The panoramic image is sent to the visual main control subsystem through the mechanical main control subsystem, the sorting target object in the panoramic image is identified and located through the visual main control subsystem, and the panoramic image is sent to the operating device;

[0026] Using the mechanical main control subsystem to control the mechanical arm to grab the sorting target object to the sorting area, and sending the movement information of the mechanical arm to the remote terminal subsystem;

[0027] Analyzing the motion information through the remote terminal subsystem to generate a robot arm parameter adjustment instruction;

[0028] The panoramic image is displayed by the operating device, and a high priority instruction of an operator to capture any one or any type of target is sent.

[0029] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the control method of the sorting robot as described in the above embodiment.

[0030] A fourth aspect of the present invention provides a computer program product, which implements the above-mentioned control method of the sorting robot when the computer program / instructions are executed by a processor.

[0031] A fifth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the control method of the sorting robot as described above.

[0032] The control device and method for a sorting robot proposed in an embodiment of the present invention adopt an FPGA platform to deploy a mechanical main control subsystem, give play to the advantages of parallel processing, and effectively solve the problem of the running speed of the robot control system; adopt an NPU device to complete the visual main control subsystem, give play to its acceleration effect on the neural network model, and improve the target recognition speed of the robot control system; deploy a remote terminal subsystem on a general-purpose computer, and enhance the deployment flexibility of the system; the three core subsystems work together to achieve low-cost, highly intelligent completion of target recognition and sorting tasks.

[0033] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0035] Figure 1 A schematic diagram of the structure of a control device for a sorting robot provided by an embodiment of the present invention;

[0036] Figure 2 A physical connection diagram of a control device of a sorting robot provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a deployment scenario provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the working effect provided by an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of hardware deployment provided by an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of the specific structural connection of the control device of the sorting robot provided by an embodiment of the present invention;

[0041] Figure 7 A working block diagram of a visual preprocessing module provided in an embodiment of the present invention;

[0042] Figure 8 A working block diagram of the integrated communication A module provided in an embodiment of the present invention;

[0043] Fig. 9 A working block diagram of a robotic arm control module provided by an embodiment of the present invention;

[0044] Fig.10 A working block diagram of the integrated communication B module provided in an embodiment of the present invention;

[0045] Fig.11 A working block diagram of a visual solution module provided in an embodiment of the present invention;

[0046] Fig.12 A flowchart of the model training provided by an embodiment of the present invention;

[0047] Fig.13 A working block diagram of a human-computer interaction module provided in an embodiment of the present invention;

[0048] Fig.14 A working block diagram of a remote terminal subsystem provided by an embodiment of the present invention;

[0049] Fig.15 A real picture of the digital twin module provided in an embodiment of the present invention;

[0050] Fig.16 A real picture of the parameter adjustment module provided in an embodiment of the present invention;

[0051] Fig.17 A flow chart of a control method for a sorting robot provided by an embodiment of the present invention;

[0052] Fig.18 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0053] Description of reference numerals:

[0054] 1-Robotic arm, 2-Mechanical main control subsystem, 3-Visual main control subsystem, 4-Operation equipment, 5-Image acquisition equipment, 6-Remote terminal subsystem, 7-Digital twin module, 8-Serial bus No. 01, 9-HDMI cable No. 08, 10-04 Bluetooth wireless, 11-14 HDMI cable No. 12-15 Robotic arm control cable No. 13-13 HDMI cable No. 14-12 Touch signal cable No. 15-07 Parallel signal cable. DETAILED DESCRIPTION

[0055] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0056] The control device and method of the sorting robot according to the embodiment of the present invention will be described below with reference to the accompanying drawings.

[0057] Figure 1 A schematic structural diagram of a control device for a sorting robot provided in an embodiment of the present invention.

[0058] like Figure 1 As shown, the control device of the sorting robot includes three core subsystems: a mechanical main control subsystem 2, a visual main control subsystem 3 and a remote terminal subsystem 6, and four peripheral devices: a mechanical arm 1, an image acquisition device 5, and an operating device 4.

[0059] Among them, Figure 2-4 As shown, the robot arm 1 is arranged between the target area to be sorted and the sorting area, so as to respond to the control instruction and perform the sorting action corresponding to the sorting items to be sorted. The image acquisition device 5 can be a camera, which is arranged in the target area to be sorted to collect a panoramic image of the target area to be sorted. The mechanical main control subsystem 2 is connected to the robot arm 1 and the image acquisition device 5 respectively, so as to generate a control instruction for controlling the robot arm 1 to sort the items to be sorted according to the panoramic image. The visual main control subsystem 3 is connected to the mechanical main control subsystem 2, and is used to identify and locate the sorting target object in the panoramic picture according to the target sorting requirements. The operating device 4 can be a touch screen, and the touch screen 4 is connected to the visual main control subsystem 3, and is used to display the panoramic image and send the operator's high priority instruction to grab any one or any type of target. The remote terminal subsystem 6 is connected to the mechanical main control subsystem 2, and is used to monitor the motion information of the robot arm, so as to generate the robot arm parameter adjustment instruction according to the motion information.

[0060] Specifically, Figure 5As shown, the robot 1 is connected to the mechanical main control subsystem 2 through the robot control line 12 No. 15, and the robot 1 should be placed between the to-be-sorted area and the sorting target area; the mechanical main control subsystem 2 is connected to the camera 5 through the parallel signal line 15 No. 07, and the camera 5 is placed on one side of the to-be-sorted area and takes a full view of the to-be-sorted area; the mechanical main control subsystem 2 is connected to the visual main control subsystem 3 through the serial bus 8 No. 01 and the HDMI line 9 No. 08; the visual main control subsystem 3 is connected to the touch screen 4 through the HDMI line 13 No. 13 and the touch signal line 14 No. 12; the mechanical main control subsystem 2 is connected to the remote terminal subsystem 6 through the Bluetooth wireless 10 No. 04; the remote terminal subsystem 6 is connected through the HDMI line 11 No. 14.

[0061] In some embodiments, the mechanical main control subsystem 2 includes:

[0062] A visual preprocessing module, used to receive and process the panoramic image of the image acquisition device 5 to obtain video stream data, and prepare the video stream data into HDMI protocol for transmission to the visual main control subsystem 3;

[0063] The integrated communication A module is used to verify the information of the target to be sorted between the mechanical main control subsystem 2 and the visual main control subsystem 3 to obtain the target information;

[0064] The robot arm control module is used to receive target information, to generate control instructions for controlling the robot arm 1 to perform sorting according to the target information, and to map the motion information of the robot arm 1 to the remote terminal subsystem 6.

[0065] Specifically, Figure 6 As shown, the mechanical control subsystem 2 is provided with a mechanical arm control module, an integrated communication A module and a visual preprocessing module, wherein the integrated communication A module is the core communication module of the mechanical control subsystem 2, which receives the information of the target to be sorted sent by the integrated communication B module of the visual main control subsystem 3 through the serial bus No. 01 8 and sends the hand-eye calibration instruction and the current mechanical arm status notification, sends the target data to be sorted to the mechanical arm control module through the parallel internal signal line No. 02, receives the working status notification of the mechanical arm control module through the serial internal signal line No. 03, and receives the remote terminal subsystem 6 through the Bluetooth wireless 10 No. 04. The integrated communication C module receives parameter adjustment instructions and sends digital twin instructions, transmits visual preprocessing parameter adjustment information to the visual preprocessing module through serial internal signal line No. 05, and receives the digital twin control signal of the robot 1 through serial internal signal line No. 06; in addition, the visual preprocessing module connects to the peripheral device camera 5 through parallel signal line No. 07 15 to send camera initialization parameters and receive panoramic images, and transmits video stream data to the visual solution module of the visual main control subsystem 3 through HDMI line No. 08 9, and the robot control module is connected to the peripheral device robot 1 through robot arm control line No. 15 12.

[0066] like Figure 7 As shown, when the system starts, the visual preprocessing module of the mechanical main control subsystem 2 adopts the SCCB protocol, and configures the camera parameters through the No. 07 parallel signal line 15. In the initialization parameters, the resolution of camera 1 is set to 1280×720, the frame rate is 30fps, and the image format is RGB565; after the configuration of camera 5 is completed, the image data is input through the No. 07 parallel signal line 15 in the DVP protocol. The visual preprocessing module parses the line field synchronization signal and converts the panoramic image from RGB format to HSV, and adjusts the image brightness, chroma and saturation in the HSV color space to adapt to the changes in ambient light. These adjustments are made based on the visual preprocessing parameter adjustment information sent by the F3 thread of the integrated communication A module through the No. 05 serial internal signal line; one adjustment instruction receives a signal U consisting of 3 decimal numbers S , U T , U, using the step adjustment method. The first number U of the received signal S Control adjustment parameters, where U S =1 represents brightness adjustment, U S =2 represents the adjustment of chromaticity, U S =3 represents the adjustment of saturation, the second number U T Specifies the adjustment direction, where U T =15 represents upward addition, U T =10 represents downward subtraction, and the third number U represents the step size of addition and subtraction, and its value is in the closed interval of 0 to 255. These adjustments ensure that the system can work under a wide range of light conditions. The adjusted image data is converted into RGB format. Since the above operation takes 7 clock cycles, the visual preprocessing module performs timing synchronization operations to avoid image dislocation and caches a frame of image. At the same time, it generates the line field synchronization signal required by the HDMI protocol to generate an HDMI format video stream, and sends the preprocessed video stream to the visual main control subsystem 3 through the HDMI line 9 of No. 08.

[0067] like Figure 8As shown, the integrated communication module A is provided with three synchronously working threads, namely, F1 thread, F2 thread and F3 thread, to ensure the immediacy of communication. The three threads run in parallel and will not interrupt each other. The F1 thread processes the target information to be sorted sent by the visual main control subsystem 3 through the No. 01 serial bus 8, and adopts the UART protocol for communication. One communication interaction includes 5 communication frames, and each frame contains an 8-bit binary number. The integrated communication module A first checks the information of the first frame (first frame). If its content is 0xAA, it continues to process the subsequent information through the check, otherwise it continues to wait for the correct information; after the first frame check passes, the integrated communication module A checks the lower 4 bits of the 8-bit data contained in the fourth frame. If it is 0xA, it continues to process, otherwise it waits for the correct information again; the above After both checks are passed, the F1 thread parses the second frame of the five communication frames into the horizontal coordinate of the target to be sorted, which is marked as a decimal number X, the third frame into the vertical coordinate of the target to be sorted, which is marked as a decimal number Y, the high 4 bits of the 8-bit data contained in the fourth frame are parsed into the category of the target to be sorted, which is recorded as a decimal number L, and the fifth frame is parsed into the horizontal rotation angle of the target to be sorted relative to the expected angle after sorting is completed, which is recorded as a decimal number C; these data of the target to be sorted are sent to the robot control module through the parallel internal signal line 02. Figure 5 As shown, the F2 thread checks the working status notification sent by the robot control module through the serial internal signal line 03. If it is 0xBB, the visual main control subsystem 3 is notified through the serial bus 8 of 01 that the current robot 1 has completed sorting and updated the sorting target object. Otherwise, the visual main control subsystem 3 is notified of the approval information. Figure 5 As shown, the F3 thread receives the parameter adjustment command sent by the remote terminal through the Bluetooth wireless communication No. 04 10. The F3 thread decodes the Bluetooth data according to the 115200 baud rate, 8 data bits and 1 end bit, even parity mode and verifies the validity of the command, that is, whether it is composed of 4 hexadecimal digits, the first bit is recorded as U U If the instruction is valid, the instruction type is determined, that is, U U = C or D is the hand-eye calibration adjustment instruction, and the rest are chroma, brightness, and saturation adjustment instructions; the chroma, brightness, and saturation adjustment instructions are translated and distributed to the visual preprocessing module through the serial internal signal line 05. The translation rule is: the first bit is converted to decimal and recorded as U S , the second digit is converted to decimal and recorded as U T , the third digit is converted to decimal and recorded as U1, the fourth digit is converted to decimal and recorded as U2, U=U1×16+U2, U S , U T , U is distributed to the visual preprocessing module in sequence. The hand-eye calibration adjustment instruction is translated and sent to the visual main control subsystem 3 via serial bus 01. The translation rules are as above.

[0068] like Fig. 9As shown, the robot control module receives the target data to be sorted sent from the integrated communication A module through the parallel internal signal line No. 02. The module stores this target data as the target information to be grasped and verifies the validity of the information. The validity is verified based on whether the coordinate position of the grasped target is in the working range of the robot, that is, whether X is in the interval [0, 255] and whether Y is in the interval [0, 150]. If the verification is passed, the grasping task will continue to be executed, otherwise the integrated communication A module will be informed through the serial internal signal line No. 03 that the visual main control subsystem 3 needs to request the approval information. After the verification is passed, the module notifies the robot 1 of the calibration command through the robot control line 12 No. 15. All subsequent communications with the robot 1 are completed with the help of the robot control line 12 No. 15. If there is no special need, it will not be explained separately. The calibration command is a group of 13 binary 8-bit numbers arranged in ASCII code, and 10 groups are issued to correct the robot 1 to the specified position and wait for the next instruction. The coding method of the calibration command and all subsequent communications with robot arm 1 is as follows: the first number is 01000000 (decoded as @ in ASCII encoding); the second number is the number of the robot arm joint controlled by the command and it should be in the interval [0,9]; the third number is 01001011 (decoded as K in ASCII encoding), and the fourth, fifth, sixth and seventh numbers are recorded as t, h, r and s in decimal. According to the calculation formula p=t×1000+h×100+r×10+s, p is calculated. p is in the closed interval of 0 to 2500, and p represents the angle of the servo. , p=0 represents 0°, p=2500 represents 270°, counterclockwise; the 8th bit is 01010100 (decoded as T in ASCII encoding); the 9th, 10th, 11th and 12th bits are recorded as t2, h2, r2 and s2 respectively, and p2 is calculated according to the formula p2=t2×1000+h2×100+r2×10+s2, p2 is in the closed interval of 100 to 1500, and p2 represents the time taken by the servo to execute the action, in units of ms (milliseconds); the 13th number is 00100011 (decoded as # in ASCII encoding) to indicate the end of the instruction. The number of calibration instruction groups is determined according to the degrees of freedom of the manipulator, each degree of freedom has two directions of motion, and each direction of motion has a set of calibration instructions, which helps the manipulator, especially the low-cost manipulator, to eliminate the systematic errors caused by motion inertia and gear clearance. In this embodiment, the manipulator has 5 degrees of freedom, so there are a total of 10 groups. After the correction is completed, the robot control module completes the angle coordinate system correction according to the above-mentioned target information to ensure that the target angle after sorting is consistent with the expected angle. The correction formula is: ∝ = -β + θ, where ∝ is the rotation angle executed by the servo at the end of the robot, β is the rotation angle of the robot during the sorting and grasping task, and θ is the horizontal rotation angle of the target to be grasped relative to the expected angle after sorting is completed. The robot control module retrieves the pre-stored grab codebook according to the target position in the target information to be grabbed. The grab codebook contains the servo action instructions required to grab the target at any position (X, Y) within the working range of the robot 1. These codebooks are stored in the ROM memory of the mechanical control subsystem 2, so that when the grab operation is executed, there is no need to perform real-time kinematic solution. Only codebook retrieval is needed to quickly drive the robot 1 to complete the grab action, which provides a solution to the problem of insufficient computing power under low-cost conditions. The codebook has a total of 38,400 entries. At the same time, the robot control module will calculate the servo motion plan for placing the target based on the target type information L in the target information to be grabbed. Each L value corresponds to a specific angle correction value σ. The formula for calculating the rotation angle of the terminal servo when placing the target is γ=-α-β+σ, where γ is the rotation angle of the terminal servo when placing the target. To ensure the smoothness of sorting, the robot arm 1 keeps the sorting target horizontal to the ground throughout the whole process. To achieve this goal, the formula δ=270°-μ is used, where μ is the sum of the rotation angles of each servo required to grab the target in the vertical direction specified by the grab code book, and δ is the correction angle required to keep the target horizontal. After completing all the above calculations and code book access, the module compiles the instructions and uniformly compiles them into 25 groups of instructions in a group of 8 hexadecimal numbers. These instructions guide the robot arm to complete the tasks of target grabbing, horizontal maintenance, angle correction, and target placement. They are uniformly sent to the robot arm 1, and the robot arm work instructions are always sent according to the principle that horizontal movement precedes vertical movement. At the same time, the air pump work instructions are also sent to the robot arm 1, and the air pump drives the grab suction cup to complete the picking and placement of the target. The compilation method of the air pump work instructions is the same as the above robot arm instructions. After all control instructions are sent, the robot control module waits for the robot 1 to reply. When the action is completed, it sends 0xBB to the integrated communication A module through the serial internal signal line 03. The integrated communication A module notifies the visual main control subsystem 3 that the target information can be updated through the serial bus 8 of No. 01. At the same time, after the instruction assembly is completed, a copy will be made in the order of the instructions as a digital twin control signal and sent to the integrated communication A module through the serial internal signal line 06.

[0069] like Figure 8 As shown, the F3 thread receives the digital twin control signal sent by the robot control module through the serial internal bus No. 06. The digital twin control signal is a group of 8 hexadecimal numbers, each group indicates an action of a servo and the speed of the action. The encoding method is as described above. These control signals are sent to the remote terminal when Bluetooth is idle to simulate the real robot movement in a virtual environment.

[0070] In some embodiments, the visual master control subsystem 3 includes:

[0071] A visual solution module is used to receive video stream data from the mechanical main control subsystem 2, identify sorting target objects in the video stream data using a neural network model, and calibrate the sorting target objects according to hand-eye calibration to generate a target data queue;

[0072] The integrated communication B module is used to provide a communication interface for the visual main control subsystem 3 and the mechanical main control subsystem 2 to transmit communication content and target information;

[0073] The human-computer interaction module is used to display the real-time sorting situation and send high-priority instructions to grab any or any type of target instructed by the operator through the operating device.

[0074] Specifically, Figure 6 As shown, the visual main control subsystem 3 is internally provided with an integrated communication B module, a human-computer interaction module and a visual solution module; the internal parallel signal line No. 09 updates the to-be-sorted target data of the visual solution module to the integrated communication B module in real time, and at the same time, the internal parallel signal line No. 09 transmits the hand-eye calibration parameter adjustment information from the integrated communication B module to the visual solution module, the internal parallel signal line No. 10 transmits the interaction instruction of the human-computer interaction module to the integrated communication B module, the internal parallel signal line No. 11 transmits the grabbing queue information solved by the visual solution module and the real-time image of the to-be-grabbed area to the human-computer interaction module, and the human-computer interaction module is connected to the peripheral device touch screen 4 via the touch signal line No. 12 14 and the HDMI line 13 13, receives the touch instruction and pushes the integrated rendering video stream.

[0075] like Fig.10 As shown, the integrated communication B module set in the visual main control subsystem 3 receives the hand-eye calibration parameter adjustment information sent by the mechanical main control subsystem 2 through the serial bus 8 No. 01. The parameters are transmitted in the UART protocol. One hexadecimal number is one frame, and four frames form a group. The first bit is converted to decimal and recorded as U S , the second digit is converted to decimal and recorded as U T , the third digit is converted to decimal and recorded as U1, the fourth digit is converted to decimal and recorded as U2, U=U1×16+U2, if U T =1 then U=U, if U T =2 then U = -U; U S =12 represents the horizontal lateral coordinate linear difference P between the eye coordinate system where the camera 5 is located and the hand coordinate system where the robot arm 1 is located U =U,U S =13 represents the linear difference N between the horizontal and vertical coordinates of the eye coordinate system where the camera 5 is located and the hand coordinate system where the robot arm 1 is located U =U; integrated communication module B will P U and N U The data is passed to the visual solution module for correction. Figure 2 and Figure 3 As shown, the camera 5 should be placed above the area to be sorted during setting, and the camera 5 field of view should fully include the entire area to be sorted. However, in this embodiment, the vibration caused by the movement of the robot 1, the influence of the camera 5's own gravity and other forces will cause the camera 5 to move slightly during long-term continuous work. If the operator finds this trend, the hand-eye calibration parameter adjustment information can be given through the above-mentioned remote terminal subsystem 6. The integrated communication B module will receive and translate the instruction, and inform the visual solution module through the internal parallel signal line No. 09, avoiding interruption of system operation and redeployment, thereby improving work efficiency.

[0076] like Fig.11 As shown, the visual solution module receives and captures the video stream data sent by the mechanical main control subsystem 2 through the HDMI line 9 No. 08, and receives the hand-eye calibration parameter adjustment information sent by the above-mentioned integrated communication B module through the internal parallel signal line No. 09. The capture and reception of these two data use parallel multi-threading technology and are executed synchronously. The hand-eye calibration correction instruction includes the horizontal lateral offset P U and horizontal longitudinal offset N U Two parameters, the visual solution module locks through the foot point recognition algorithm Figure 2 , Figure 3 The black boundary of the area to be captured is shown as a reference for the coordinate system. Figure 3 The coordinate reference point shown is the position coordinate (x1, y1) of the right bottom point of the rectangle in the image as the origin of the reference system, so the origin of the corrected image reference system is (x1+P U ,y1+N U ). The visual solution module runs the pre-trained neural network model to complete the target recognition. The embodiment uses a deep learning method to train the model and lightweight processing, but this model is not unique and should be trained differently according to different sorting targets. The model training process used in this embodiment is shown in Fig.12 ,according to Fig.12A generalized model training method is provided; firstly, data is collected, and images of the sorting targets of the sorting robot are collected. Meanwhile, images are collected by changing the stacking mode of the targets, the light source environment and other factors. Generally, no less than 100 images are collected for each target; then, the collected data set is labeled by category, and the data set is optimized by adding noise, randomly adding interference factors and other measures; then, the model is designed and deployed, and parameters such as the number of model iterations, the number of categories, and the training weight are set; then, the data set is divided into a validation set and a training set according to a ratio of 3:7, the model is deployed to complete the training, and the model parameters are further adjusted through result analysis, the recognition effect is optimized, and the clustering prior is modified as needed; the model is adapted and lightweighted according to the type of computing power used by the deployed model to improve the running speed of the model; finally, the model is debugged and deployed. In this embodiment, the model reserves 5 ports, including: an input port in RGB565 format, an output port in RGB888 format, a target position output port, a target posture output port, and a target recognition status output port; after the above training process, the model training and deployment can be completed and other parts of the present invention can be adapted. After the model is run, the boundary coordinates of the identified target are generated. The visual solution module segments the image according to the boundary coordinate information and converts the image into HSV format. The watershed algorithm is used to further segment the target boundary. At the same time, the target structure integrity is analyzed in HSV format. The structural integrity analysis is based on the target area and the typical area ratio of the target of the type it describes, as well as the number of its foot points and the perimeter area ratio parameters. According to the integrity analysis results, the stacking relationship of the target to be sorted can be determined, eliminating the binocular camera originally required for grasping three-dimensional stacked targets, which helps solve the cost problem of the sorting robot. The visual solution module calculates the horizontal rotation angle θ of the target relative to the expected angle after sorting is completed through Hough transform, determines the target center point as the grasping point by generating the minimum circumscribed moment of the target, generates the target coordinate information (X, Y) and generates the grasping logic. The natural language expression of the grasping logic is: prioritize grasping the targets stacked on the upper layer. After completing the grasping logic planning, the coordinate information, type information, and angle information of all targets are stored, and the grasping queue is formed according to the grasping logic. The target at the top of the queue is sent to the integrated communication B module through the internal parallel signal line No. 09, recorded as A signal. At the same time, the grasping queue information and the real-time image of the area to be grasped are synchronously updated to the human-computer interaction module.

[0077] like Fig.13 As shown, the human-computer interaction module has two input ports, wherein the internal parallel signal line 11 receives the capture queue of the visual solution module and the real-time image of the area to be captured and caches it; the target in the real-time image is selected by the minimum external rectangular frame and the coordinates, priority, and category information of the target are displayed on the screen; at the same time, the rendering interaction button is output to the touch screen through the HDMI line 13, as shown in FIG. Figure 2As shown, the operator can view the target situation in the area to be sorted in real time, and can click the left button to require the robot to give priority to grabbing a certain type of target or click the right button to require the robot to give priority to grabbing a specific target; these touch signals are transmitted to the human-computer interaction module by the touch signal line 14 No. 12. The human-computer interaction module determines the sorting instruction according to the touch area, and retrieves the grabbing queue according to the instruction. The grabbing queue is the grabbing queue sent from the visual solution module and cached. If the queue is retrieved successfully, that is, the operator-specified target exists in the area to be sorted, then the target or the target type information is sent to the integrated communication B module, recorded as B signal. If it fails, it indicates that there is no such target in the area.

[0078] like Fig.10 As shown, the integrated communication B module receives the B signal sent from the human-machine interaction module through the No. 10 internal parallel signal line, and the A signal sent from the visual settlement module through the No. 09 internal parallel signal line. If the B signal is valid, the information of the target to be sorted contained in the B signal is encoded unconditionally and preferentially, that is, the instructions issued by the operator through the touch screen are executed preferentially; otherwise, the information of the target to be sorted contained in the A signal is automatically encoded. During encoding, one communication interaction contains 5 communication frames, each frame contains an 8-bit binary number, the first frame adds the check code 0xAA, the last 4 bits of the 8-bit data of the fourth frame add the check code 0xA, the second frame writes the horizontal coordinate X of the sorting target, the third frame writes the horizontal coordinate Y of the sorting target, the first 4 bits of the 8-bit data of the fourth frame write the category L of the sorting target, and the fifth frame writes the horizontal rotation angle of the current sorting target relative to the expected angle after the sorting is completed. Waiting for the mechanical main control subsystem 2 to send a request to update the sorting target instruction through the No. 01 serial bus 8, and send the target information to be sorted.

[0079] In some embodiments, the remote terminal subsystem 6 includes:

[0080] The integrated communication C module is used to cooperate with the integrated communication A module of the mechanical main control subsystem 2;

[0081] Digital twin module 7, used to build a simulated robotic arm in a virtual scene to simulate the movement of the robotic arm using real physical parameters;

[0082] The parameter adjustment module is used to correct the hand-eye calibration of the robot arm 1, obtain the robot arm parameter adjustment instruction, and adjust the visual parameters of the image acquisition device according to the light conditions of the deployment environment of the robot arm 1.

[0083] Specifically, Figure 6 As shown, the remote terminal subsystem 6 is internally provided with an integrated communication C module, a digital twin module 7 and a parameter adjustment module, wherein the parameter adjustment module pushes the digital twin rendering image to the peripheral device monitor via HDMI line No. 14 11 .

[0084] like Fig.14 As shown, the remote terminal subsystem 6 can be flexibly deployed on a general-purpose computer, which receives the digital twin command sent by the mechanical main control subsystem 2 through the No. 04 Bluetooth wireless 10, and sends the parameter adjustment command given by the parameter adjustment module to the mechanical main control subsystem 2 through the No. 04 Bluetooth wireless 10. Among them, the integrated communication C module performs the encoding and decoding operation of Bluetooth communication. In this embodiment, 115200 baud rate, 8 data bits, 1 stop bit and even parity method are used for Bluetooth communication. The integrated communication C module continuously receives the digital twin instructions sent by the mechanical central control subsystem 2. The instruction encoding method is the same as the format of sending instructions to the robot 1 through the No. 15 robot control line 12. The integrated communication C module decodes the instruction information and checks whether the data contained in the instruction contains valid information. If valid, it is transmitted to the digital twin module 7. The valid information verification rules are as follows: Each group of instructions consists of 13 8-bit binary numbers. When the first number is 01000000 (decoded as @ in ASCII encoding) and the second number is decoded as decimal in ASCII encoding and is in the closed interval from 0 to 9, and the third number is 01001011 (decoded as K), and the 4th, 5th, 6th and 7th numbers are respectively decoded into decimal according to ASCII encoding and recorded as t, h, r and s, and p is calculated according to the calculation formula p=t×1000+h×100+r×10+s, and p is in the closed interval of 0 to 2500, and the 8th bit is 01010100 (decoded as T according to ASCII encoding), and the 9th, 10th, 11th and 12th bits are respectively recorded as t2, h2, r2 and s2, and p2 is calculated according to the calculation formula p2=t2×1000+h2×100+r2×10+s2, and p2 is in the closed interval of 100 to 1500, and the 13th number is 00100011 (decoded as # according to ASCII encoding). After receiving the valid instruction, the digital twin module 7 performs command mapping, that is, mapping the instruction to the corresponding virtual robot arm joint according to the second digit mentioned above, mapping the above p value to the movement angle of the robot arm joint servo, angle θ=p×0.108, and mapping the above p2 to the movement time of the robot arm joint servo, time T=p2ms. The digital twin module 7 is provided with physical modeling of each part of the robot arm 1, and virtually assembled into a digital twin robot arm identical to the real robot arm according to the real physical laws. The virtual servo will drive the digital twin robot arm to work according to the above mapping results, and always maintain the same physical posture as the real robot arm. The digital twin robot arm is placed in the virtual work scene and rendered in 3D according to the light and material, and the effect is as follows: Fig.15 As shown in the figure, the real-time picture after 3D rendering will be sent to the digital twin monitor through the 14-gauge HDMI line 11. The parameter adjustment module first renders the human-computer interaction interface, which becomes the operation interface visible to the operator. The effect is as follows Fig.16As shown, the parameter adjustment module will wait for the operator's adjustment instructions for brightness, chromaticity, saturation and hand-eye calibration parameters. The instruction format is a 4-bit hexadecimal number, where the first bit indicates the adjustment object, the second bit indicates the adjustment direction, and the third and fourth bits indicate the adjustment value. The integrated communication C module will assemble the instructions into the sentences specified by the Bluetooth protocol and send them in the Bluetooth idle state.

[0085] In summary, the control device of the sorting robot proposed in the embodiment of the present invention adopts the FPGA platform to deploy the mechanical main control subsystem, takes advantage of parallel processing, and effectively solves the problem of the running speed of the robot control system; adopts the NPU device to complete the visual main control subsystem, takes advantage of its acceleration effect on the neural network model, and improves the target recognition speed of the robot control system; deploys the remote terminal subsystem on a general-purpose computer, which enhances the deployment flexibility of the system; the three core subsystems work together to achieve low-cost and highly intelligent completion of target recognition and sorting tasks.

[0086] Next, the control method of the sorting robot proposed according to the embodiment of the present invention is described with reference to the accompanying drawings.

[0087] Fig.17 It is a flow chart of the control method of the sorting robot according to the embodiment of the present invention.

[0088] like Fig.17 As shown, the control method of the sorting robot includes the following steps:

[0089] In step S1701, a panoramic image of the target area to be sorted is captured by an image acquisition device, and the panoramic image is sent to the mechanical main control subsystem.

[0090] In step S1702, the panoramic image is sent to the visual main control subsystem through the mechanical main control subsystem, the sorting target object in the panoramic image is identified and located through the visual main control subsystem, and the panoramic image is sent to the operating device.

[0091] In step S1703, the mechanical main control subsystem is used to control the mechanical arm to grab the sorting target object to the sorting area, and the movement information of the mechanical arm is sent to the remote terminal subsystem.

[0092] In step S1704, the motion information is analyzed by the remote terminal subsystem to generate a robot arm parameter adjustment instruction.

[0093] In step S1705, the panoramic image is displayed by the operating device, and a high-priority instruction of the operator to capture any one or any type of target is sent.

[0094] It should be noted that the aforementioned explanation of the embodiment of the control device of the sorting robot is also applicable to the control method of the sorting robot of this embodiment, which will not be repeated here.

[0095] According to the control method of the sorting robot proposed in the embodiment of the present invention, the FPGA platform is used to deploy the mechanical main control subsystem, which takes advantage of parallel processing and effectively solves the problem of the running speed of the robot control system; the NPU device is used to complete the visual main control subsystem, which takes advantage of its acceleration effect on the neural network model and improves the target recognition speed of the robot control system; the remote terminal subsystem is deployed on a general-purpose computer to enhance the deployment flexibility of the system; the three core subsystems work together to achieve low-cost and highly intelligent completion of target recognition and sorting tasks.

[0096] Fig.18 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0097] A memory 1801 , a processor 1802 , and a computer program stored in the memory 1801 and executable on the processor 1802 .

[0098] When the processor 1802 executes the program, the control method of the sorting robot provided in the above embodiment is implemented.

[0099] Furthermore, the electronic device further comprises:

[0100] The communication interface 1803 is used for communication between the memory 1801 and the processor 1802 .

[0101] The memory 1801 is used to store computer programs that can be executed on the processor 1802 .

[0102] The memory 1801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0103] If the memory 1801, the processor 1802 and the communication interface 1803 are implemented independently, the communication interface 1803, the memory 1801 and the processor 1802 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.18 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0104] Optionally, in a specific implementation, if the memory 1801, the processor 1802 and the communication interface 1803 are integrated on a chip, the memory 1801, the processor 1802 and the communication interface 1803 can communicate with each other through an internal interface.

[0105] The processor 1802 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0106] An embodiment of the present invention further provides a computer program product, which implements the above control method of the sorting robot when the computer program / instructions are executed by a processor.

[0107] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the control method of the sorting robot as described above when the program is executed by a processor.

[0108] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0109] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0110] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0111] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0112] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0114] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0115] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A control device for a sorting robot, characterized in that: include: A mechanical arm, the mechanical arm is arranged between the target to-be-sorted area and the sorting area, so as to respond to the control instruction and perform a sorting action corresponding to sorting the to-be-sorted items; An image acquisition device, wherein the image acquisition device is arranged in the target area to be sorted to acquire a panoramic image of the target area to be sorted; A mechanical main control subsystem, the mechanical main control subsystem is connected to the mechanical arm and the image acquisition device respectively, so as to generate a control instruction for controlling the mechanical arm to sort the objects to be sorted according to the panoramic image; A visual main control subsystem, which is connected to the mechanical main control subsystem and is used to identify and locate the sorting target object in the panoramic picture according to the target sorting requirements; An operating device, connected to the visual main control subsystem, for displaying the panoramic image and sending a high-priority instruction of an operator to capture any one or any type of target; A remote terminal subsystem is connected to the mechanical main control subsystem and is used to monitor the motion information of the mechanical arm to generate a mechanical arm parameter adjustment instruction according to the motion information.

2. The control device of the sorting robot according to claim 1, characterized in that: The mechanical main control subsystem includes: A visual preprocessing module, used for receiving and processing the panoramic image of the image acquisition device to obtain video stream data, and preparing the video stream data in HDMI protocol for transmission to the visual main control subsystem; The integrated communication A module is used to verify the information of the target to be sorted between the mechanical main control subsystem and the visual main control subsystem to obtain the target information; The robot arm control module is used to receive the target information, generate a control instruction for controlling the robot arm to perform sorting according to the target information, and map the motion information of the robot arm to the remote terminal subsystem.

3. The control device of the sorting robot according to claim 2, characterized in that: The integrated communication A module is provided with three synchronization lines: F1 thread, F2 thread and F3 thread, wherein: The F1 thread is used to verify the to-be-sorted target information sent by the visual main control subsystem using UART protocol communication to obtain the target information, and send the target information to the robotic arm control module; The F2 thread is used to communicate to the visual main control subsystem that the current robot arm has completed sorting and needs to update the sorting target object, otherwise it communicates the visual main control subsystem approval information; The F3 thread is used to receive the robot arm parameter adjustment instruction sent by the remote terminal subsystem.

4. The control device of the sorting robot according to claim 3, characterized in that: The target information includes inspection information, categories of target objects to be sorted, horizontal plane lateral coordinates of the target objects to be sorted, horizontal plane longitudinal coordinates of the target objects to be sorted, and horizontal rotation angles of the target objects to be sorted relative to the expected angles after sorting is completed.

5. The control device of the sorting robot according to claim 1, characterized in that: The visual main control subsystem includes: A visual solution module, used for receiving the video stream data of the mechanical main control subsystem, identifying the sorting target object in the video stream data by using a neural network model, and calibrating the sorting target object according to hand-eye to generate a target data queue; The integrated communication B module is used to provide a communication interface for the visual main control subsystem and the mechanical main control subsystem to transmit communication content and the target information; The human-computer interaction module is used to display the real-time sorting situation and send a high-priority instruction to grab any one or any type of target instructed by the operator through the operating device.

6. The control device of the sorting robot according to claim 1, characterized in that: The remote terminal subsystem includes: An integrated communication C module, used to cooperate with the integrated communication A module of the mechanical main control subsystem; A digital twin module, used to build a simulated robotic arm in a virtual scene to simulate the movement of the robotic arm using real physical parameters; The parameter adjustment module is used to correct the hand-eye calibration of the robotic arm, obtain the robotic arm parameter adjustment instruction, and adjust the visual parameters of the image acquisition device according to the light conditions of the deployment environment of the robotic arm.

7. A control method for a sorting robot, characterized in that: The control system of the sorting robot according to any one of claims 1 to 6 comprises the following steps: Using an image acquisition device to acquire a panoramic image of the target area to be sorted, and sending the panoramic image to the mechanical main control subsystem; The panoramic image is sent to the visual main control subsystem through the mechanical main control subsystem, the sorting target object in the panoramic image is identified and located through the visual main control subsystem, and the panoramic image is sent to the operating device; Using the mechanical main control subsystem to control the mechanical arm to grab the sorting target object to the sorting area, and sending the movement information of the mechanical arm to the remote terminal subsystem; Analyzing the motion information through the remote terminal subsystem to generate a robot arm parameter adjustment instruction; The panoramic image is displayed through the operating device, and a high-priority instruction of an operator to capture any one or any type of target is sent.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the control method of the sorting robot as claimed in claim 7.

9. A computer program product, characterized in that When the computer program / instructions are executed by a processor, the control method of the sorting robot described in claim 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the control method of the sorting robot as claimed in claim 7.

Images