Control device and method for sorting robot

Abstract

The present invention relates to the field of robot control technology, and more particularly to a control device and method for a sorting robot, wherein the method comprises: a robotic arm for executing sorting actions corresponding to sorting items to be sorted in response to control instructions; an image acquisition device for acquiring a panoramic image of a target area to be sorted; a mechanical main control subsystem for generating control instructions for controlling the robotic arm to sort the items to be sorted based on the panoramic image; a visual main control subsystem for identifying and locating sorting target objects in the panoramic image based on target sorting requirements; an operating device for displaying the panoramic image and sending high-priority instructions to an operator to grab any or any type of target; and a remote terminal subsystem for monitoring the motion information of the robotic arm to generate robotic arm parameter adjustment instructions based on the motion information. This solves the problem of difficulty in cost control while ensuring certain key technical indicators such as control accuracy, processing speed, and operational stability.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a control device and method for a sorting robot. Background Art

[0002] With the rapid development of e-commerce and smart manufacturing, product sorting faces new pressures. Traditional manual sorting methods are unable to meet the growing demand for logistics sorting in terms of efficiency, and the market is urgently demanding automated and intelligent sorting robots. Simultaneously, the development of advanced technologies such as artificial intelligence and machine vision has provided technical support for the development and deployment of sorting robots. The application of these technologies enables sorting robots to more accurately identify item information, improving sorting efficiency and accuracy. Furthermore, due to widespread cost pressures and environmental protection policies facing the industry, low-cost implementation of sorting robots has become a mainstream market demand. The technical difficulty and core component of a sorting robot lies in its control system, which should include functions such as kinematic control of the robotic arm servos and identification and positioning of sorting targets. The system's mechanical control precision, target recognition accuracy, and intelligent human-machine interaction determine the overall performance of the sorting robot.

[0003] Although the development of highly reliable, highly intelligent, and low-cost sorting robot control systems has become an important research direction, achieving cost control while ensuring certain key technical indicators such as control accuracy, processing speed, and operational stability is still a recognized difficulty in the industry. Summary of the Invention

[0004] The present invention provides a control device and method for a sorting robot to solve the problem of difficulty in cost control while ensuring certain key technical indicators such as control accuracy, processing speed, and operation stability.

[0005] A first embodiment of the present invention provides a control device for a sorting robot, comprising: a robotic arm, the robotic arm being arranged between a target area to be sorted and a sorting area, and executing a sorting action corresponding to sorting items to be sorted in response to a control instruction; an image acquisition device, the image acquisition device being arranged in the target area to be sorted, and acquiring a panoramic image of the target area to be sorted; a mechanical main control subsystem, the mechanical main control subsystem being connected to the robotic arm and the image acquisition device, respectively, and generating a control instruction for controlling the robotic arm to sort the items to be sorted according to the panoramic image; a visual main control subsystem, the visual main control subsystem being connected to the mechanical main control subsystem, and configured to identify and locate sorting target objects in the panoramic image according to target sorting requirements; an operating device, the operating device being connected to the visual main control subsystem, and configured to display the panoramic image and send a high-priority instruction to an operator to grab any one or any type of target; and a remote terminal subsystem, the remote terminal subsystem being connected to the mechanical main control subsystem, and configured to monitor motion information of the robotic arm, and generate a robotic arm parameter adjustment instruction according to the motion information.

[0006] Optionally, the mechanical main control subsystem includes:

[0007] A visual preprocessing module, configured to receive and process the panoramic image from the image acquisition device to obtain video stream data, and prepare the video stream data in HDMI protocol for transmission to the visual master control subsystem;

[0008] The integrated communication A module is used to verify the information of the target to be sorted between the mechanical main control subsystem and the visual main control subsystem to obtain the target information;

[0009] The robot arm control module is used to receive the target information, generate a control instruction for controlling the robot arm to perform sorting according to the target information, and map the motion information of the robot arm to the remote terminal subsystem.

[0010] Optionally, the integrated communication A module is provided with three synchronization lines: F1 thread, F2 thread and F3 thread, wherein:

[0011] The F1 thread is used to verify the to-be-sorted target information sent by the visual main control subsystem using UART protocol communication to obtain the target information, and send the target information to the robotic arm control module;

[0012] The F2 thread is used to communicate to the visual main control subsystem that the current robot arm has completed sorting and needs to update the sorting target object; otherwise, it communicates the visual main control subsystem approval information;

[0013] The F3 thread is used to receive the robot arm parameter adjustment instruction sent by the remote terminal subsystem.

[0014] Optionally, the target information includes inspection information, category of the target object to be sorted, horizontal coordinates of the target object to be sorted, horizontal coordinates of the target object to be sorted, and horizontal rotation angles of the target object to be sorted relative to an expected angle after sorting is completed.

[0015] Optionally, the visual master control subsystem includes:

[0016] a visual solution module, configured to receive video stream data from the mechanical main control subsystem, identify sorting target objects in the video stream data using a neural network model, and calibrate the sorting target objects based on hand-eye recognition to generate a target data queue;

[0017] An integrated communication B module is used to provide a communication interface for the visual main control subsystem and the mechanical main control subsystem to transmit communication content and the target information;

[0018] The human-computer interaction module is used to display the real-time status of sorting and send high-priority instructions for grabbing any one or any type of target instructed by the operator through the operating device.

[0019] Optionally, the remote terminal subsystem includes:

[0020] An integrated communication C module, configured to cooperate with the integrated communication A module of the mechanical main control subsystem;

[0021] A digital twin module, used to build a simulated robotic arm in a virtual scene to simulate the movement of the robotic arm using real physical parameters;

[0022] The parameter adjustment module is used to correct the hand-eye calibration of the robotic arm, obtain the robotic arm parameter adjustment instruction, and adjust the visual parameters of the image acquisition device according to the light conditions of the deployment environment of the robotic arm.

[0023] A second embodiment of the present invention provides a control method for a sorting robot, comprising:

[0024] Using an image acquisition device to capture a panoramic image of the target area to be sorted, and sending the panoramic image to the mechanical main control subsystem;

[0025] The panoramic image is sent to the visual main control subsystem through the mechanical main control subsystem, the sorting target object in the panoramic image is identified and located through the visual main control subsystem, and the panoramic image is sent to the operating device;

[0026] Utilizing the mechanical main control subsystem to control the mechanical arm to grab the sorting target object to the sorting area, and sending the movement information of the mechanical arm to the remote terminal subsystem;

[0027] Analyzing the motion information through the remote terminal subsystem to generate a robot arm parameter adjustment instruction;

[0028] The panoramic image is displayed by the operating device, and a high-priority instruction of an operator to capture any one or any type of target is sent.

[0029] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the control method of the sorting robot as described in the above embodiment.

[0030] A fourth aspect of the present invention provides a computer program product, which implements the above-mentioned control method for the sorting robot when the computer program / instructions are executed by a processor.

[0031] A fifth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the control method of the sorting robot as described above.

[0032] The control device and method for a sorting robot proposed in an embodiment of the present invention adopt an FPGA platform to deploy the mechanical main control subsystem, leveraging the advantages of parallel processing to effectively solve the problem of operating speed of the robot control system. An NPU device is used to complete the visual main control subsystem, leveraging its acceleration effect on the neural network model to improve the target recognition speed of the robot control system. The remote terminal subsystem is deployed on a general-purpose computer to enhance the deployment flexibility of the system. The three core subsystems work together to achieve low-cost, highly intelligent completion of target recognition and sorting tasks.

[0033] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0035] Figure 1 A schematic structural diagram of a control device for a sorting robot provided by an embodiment of the present invention;

[0036] Figure 2 A physical connection diagram of a control device for a sorting robot provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a deployment scenario provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the working effect provided by an embodiment of the present invention;

[0039] Figure 5 A schematic diagram of hardware deployment provided by an embodiment of the present invention;

[0040] Figure 6 A schematic diagram of the specific structural connection of the control device of the sorting robot provided by an embodiment of the present invention;

[0041] Figure 7 This is a working block diagram of the visual preprocessing module provided in an embodiment of the present invention;

[0042] Figure 8 This is a working block diagram of the integrated communication A module provided in an embodiment of the present invention;

[0043] Figure 9 This is a working block diagram of the robotic arm control module provided by an embodiment of the present invention;

[0044] Figure 10 This is a working block diagram of the integrated communication B module provided in an embodiment of the present invention;

[0045] Figure 11 This is a working block diagram of the visual solution module provided in an embodiment of the present invention;

[0046] Figure 12 A flowchart of the model training provided by an embodiment of the present invention;

[0047] Figure 13 A working block diagram of the human-computer interaction module provided in an embodiment of the present invention;

[0048] Figure 14 A working block diagram of the remote terminal subsystem provided by an embodiment of the present invention;

[0049] Figure 15 This is a real picture of the digital twin module provided by an embodiment of the present invention;

[0050] Figure 16 This is a real picture of the parameter adjustment module provided in an embodiment of the present invention;

[0051] Figure 17 A flow chart of a control method for a sorting robot provided by an embodiment of the present invention;

[0052] Figure 18 The present invention provides a schematic structural diagram of an electronic device.

[0053] Description of reference numerals:

[0054] 1-Robotic arm, 2-Mechanical main control subsystem, 3-Visual main control subsystem, 4-Operation equipment, 5-Image acquisition equipment, 6-Remote terminal subsystem, 7-Digital twin module, 8-Serial bus No. 01, 9-HDMI cable No. 08, 10-Bluetooth wireless No. 04, 11-HDMI cable No. 14, 12-Robotic arm control cable No. 15, 13-HDMI cable No. 13, 14-12 Touch signal cable No. 15, Parallel signal cable No. 07. DETAILED DESCRIPTION

[0055] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0056] The control device and method for a sorting robot according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0057] Figure 1 This is a schematic structural diagram of a control device for a sorting robot provided by an embodiment of the present invention.

[0058] like Figure 1 As shown, the control device of the sorting robot includes three core subsystems: a mechanical main control subsystem 2, a visual main control subsystem 3 and a remote terminal subsystem 6, and four peripheral devices: a robotic arm 1, an image acquisition device 5 and an operating device 4.

[0059] Among them, such as Figure 2-4 As shown, the robotic arm 1 is positioned between the target area to be sorted and the sorting area to respond to control instructions and execute sorting actions corresponding to the items to be sorted. The image acquisition device 5 can be a camera, which is positioned in the target area to be sorted to capture a panoramic image of the target area to be sorted. The robotic main control subsystem 2 is connected to the robotic arm 1 and the image acquisition device 5, respectively, to generate control instructions for controlling the robotic arm 1 to sort the items to be sorted based on the panoramic image. The visual main control subsystem 3 is connected to the robotic main control subsystem 2 and is used to identify and locate the sorting target objects in the panoramic image based on the target sorting requirements. The operating device 4 can be a touch screen, which is connected to the visual main control subsystem 3 and is used to display the panoramic image and send high-priority instructions from the operator to grab any or any type of target. The remote terminal subsystem 6 is connected to the robotic main control subsystem 2 and is used to monitor the motion information of the robotic arm and generate robotic arm parameter adjustment instructions based on the motion information.

[0060] Specifically, if Figure 5As shown, the robot arm 1 is connected to the mechanical main control subsystem 2 through the robot arm control line 12 No. 15, and the robot arm 1 should be placed between the to-be-sorted area and the sorting target area; the mechanical main control subsystem 2 is connected to the camera 5 through the parallel signal line 15 No. 07, and the camera 5 is placed on one side of the to-be-sorted area and takes a full view of the to-be-sorted area; the mechanical main control subsystem 2 is connected to the visual main control subsystem 3 through the serial bus 8 No. 01 and the HDMI line 9 No. 08; the visual main control subsystem 3 is connected to the touch screen 4 through the HDMI line 13 No. 13 and the touch signal line 14 No. 12; the mechanical main control subsystem 2 is connected to the remote terminal subsystem 6 through the Bluetooth wireless 10 No. 04; and the remote terminal subsystem 6 is connected through the HDMI line 11 No. 14.

[0061] In some embodiments, the mechanical main control subsystem 2 includes:

[0062] A visual pre-processing module is used to receive and process the panoramic image from the image acquisition device 5 to obtain video stream data, and prepare the video stream data into HDMI protocol for transmission to the visual main control subsystem 3;

[0063] The integrated communication module A is used to verify the information of the target to be sorted between the mechanical main control subsystem 2 and the visual main control subsystem 3 to obtain the target information;

[0064] The robot arm control module is used to receive target information, generate control instructions for controlling the robot arm 1 to perform sorting according to the target information, and map the motion information of the robot arm 1 to the remote terminal subsystem 6.

[0065] Specifically, if Figure 6 As shown, the mechanical control subsystem 2 is provided with a manipulator control module, an integrated communication A module and a visual preprocessing module. Among them, the integrated communication A module is the core communication module of the mechanical control subsystem 2. It receives the information of the target to be sorted from the integrated communication B module of the visual main control subsystem 3 through the serial bus 8 No. 01 and sends the hand-eye calibration instruction and the current manipulator status notification. It sends the target data to be sorted to the manipulator control module through the parallel internal signal line No. 02, receives the working status notification of the manipulator control module through the serial internal signal line No. 03, and receives the remote terminal subsystem 6 through the Bluetooth wireless 10 No. 04. The integrated communication C module sends parameter adjustment instructions and digital twin instructions, transmits the visual preprocessing parameter adjustment information to the visual preprocessing module through the serial internal signal line No. 05, and receives the digital twin control signal of the robot arm 1 through the serial internal signal line No. 06; in addition, the visual preprocessing module connects to the peripheral device camera 5 through the parallel signal line No. 07 15 to send camera initialization parameters and receive panoramic images, and transmits the video stream data to the visual solution module of the visual main control subsystem 3 through the HDMI line No. 08 9, and the robot arm control module is connected to the peripheral device robot arm 1 through the robot arm control line No. 15 12.

[0066] like Figure 7 As shown, when the system starts up, the visual preprocessing module of the mechanical main control subsystem 2 uses the SCCB protocol to configure camera parameters via parallel signal line 15, number 07. The initialization parameters set the resolution of camera 1 to 1280×720, the frame rate to 30fps, and the image format to RGB565. After camera 5 is configured, it inputs image data using the DVP protocol via parallel signal line 15, number 07. The visual preprocessing module parses the horizontal and vertical synchronization signals and converts the panoramic image from RGB format to HSV. In the HSV color space, it adjusts the image brightness, hue, and saturation to adapt to changes in ambient light. These adjustments are made based on the visual preprocessing parameter adjustment information sent by the F3 thread of the integrated communication module A via the internal serial signal line 05. Each adjustment command receives a signal consisting of three decimal numbers, U. S 、U T , U, adopts step adjustment method. The first number U of the received signal S Control adjustment parameters, where U S =1 means adjust brightness, U S =2 represents the adjustment of chroma, U S =3 represents the adjustment of saturation, the second number U T Specifies the adjustment direction, where U T =15 represents upward addition, U T =10 represents downward subtraction, and the third number U represents the increment and decrement step size, with a value between 0 and 255. These adjustments ensure that the system can operate under a wide range of lighting conditions. The adjusted image data is converted to RGB format. Since this operation takes seven clock cycles, the visual preprocessing module performs timing synchronization to prevent image misalignment and caches one frame. It also generates the horizontal and vertical synchronization signals required by the HDMI protocol to produce an HDMI-formatted video stream. The preprocessed video stream is then transmitted to the visual master control subsystem 3 via HDMI cable 9 (#08).

[0067] like Figure 8As shown, the integrated communication module A is provided with three synchronous threads, namely F1 thread, F2 thread and F3 thread, to ensure the immediacy of communication. The three threads run in parallel and will not interrupt each other. The F1 thread processes the target information to be sorted sent by the visual main control subsystem 3 through the No. 01 serial bus 8, and adopts the UART protocol for communication. One communication interaction contains 5 communication frames, and each frame contains an 8-bit binary number. The integrated communication module A first checks the information of the first frame (the first frame). If its content is 0xAA, it will continue to process the subsequent information after the check, otherwise it will continue to wait for the correct information. After the first frame check passes, the integrated communication module A checks the lower 4 bits of the 8-bit data contained in the fourth frame. If it is 0xA, it will continue to process, otherwise it will wait for the correct information again. The above After both checks are passed, the F1 thread parses the second frame of the five communication frames into the horizontal coordinate of the target to be sorted, which is marked as a decimal number X. The third frame is parsed into the vertical coordinate of the target to be sorted, which is marked as a decimal number Y. The high 4 bits of the 8-bit data contained in the fourth frame are parsed into the category of the target to be sorted, which is recorded as a decimal number L. The fifth frame is parsed into the horizontal rotation angle of the target to be sorted relative to the expected angle after sorting is completed, which is recorded as a decimal number C. These data of the target to be sorted are sent to the robot control module through the parallel internal signal line 02. Figure 5 As shown, the F2 thread checks the working status notification sent by the robot control module through the serial internal signal line 03. If it is 0xBB, it notifies the visual main control subsystem 3 through the serial bus 8 01 that the current robot 1 has completed sorting and updates the sorting target object. Otherwise, it notifies the visual main control subsystem 3 of the approval information. Figure 5 As shown, the F3 thread receives the parameter adjustment command sent by the remote terminal through the Bluetooth wireless communication No. 04 10. The F3 thread decodes the Bluetooth data according to the 115200 baud rate, 8 data bits and 1 end bit, even parity mode and verifies the validity of the command, that is, whether it is composed of 4 hexadecimal digits, the first bit of which is recorded as U U If the instruction is valid, the instruction type is determined, that is, U U = C or D is the hand-eye calibration adjustment instruction, and the others are chroma, brightness, and saturation adjustment instructions; the chroma, brightness, and saturation adjustment instructions are translated and distributed to the visual preprocessing module through the serial internal signal line 05. The translation rule is: the first bit is converted to decimal and recorded as U S , the second digit is converted to decimal and recorded as U T , the third digit is converted to decimal and recorded as U1, the fourth digit is converted to decimal and recorded as U2, U=U1×16+U2, and U S , U T , U is distributed to the visual pre-processing module in sequence. The hand-eye calibration adjustment instruction is translated and sent to the visual main control subsystem 3 via serial bus 01. The translation rules are as above.

[0068] like Figure 9As shown, the robot control module receives the target data to be sorted sent from the integrated communication module A through the parallel internal signal line 02. The module stores this target data as the target information to be grasped and verifies the validity of the information. The validity test is based on whether the coordinate position of the grasped target is within the working range of the robot arm, that is, whether X is within the interval [0, 255] and whether Y is within the interval [0, 150]. If the test passes, the grasping task will continue to be executed. Otherwise, the integrated communication module A will be notified through the serial internal signal line 03 that the visual main control subsystem 3 needs to request approval of the information. After the test passes, the module notifies the robot arm 1 of the calibration command through the robot arm control line 12 No. 15. All subsequent communications with the robot arm 1 are completed through the robot arm control line 12 No. 15. Unless otherwise specified, no further explanation will be given. The calibration command consists of 13 binary 8-bit bits arranged in ASCII code, and 10 groups are issued to correct the robot arm 1 to the specified position and wait for the next instruction. The encoding method for the calibration command and all subsequent communications with robot arm 1 is as follows: the first number is 01000000 (decoded as @ in ASCII encoding); the second number is the number of the robot arm joint controlled by the command and it should be in the interval [0,9]; the third number is 01001011 (decoded as K in ASCII encoding), and the fourth, fifth, sixth and seventh numbers are recorded as t, h, r and s in decimal. According to the calculation formula p = t×1000+h×100+r×10+s, p is calculated. p is in the closed interval of 0 to 2500, and p represents the angle of the servo. , with p=0 representing 0° and p=2500 representing 270°, counterclockwise. The eighth bit is 01010100 (decoded as T in ASCII encoding). The ninth, tenth, eleventh, and t2, h2, r2, and s2, respectively, are calculated using the formula p2=t2×1000+h2×100+r2×10+s2. p2 is in the closed interval between 100 and 1500. p2 represents the time it takes the servo to execute the action, in milliseconds. The thirteenth number, 00100011 (decoded as # in ASCII encoding), indicates the end of the command. The number of calibration instruction groups is determined by the robot's degrees of freedom. Each degree of freedom has two directions of motion, and each direction has one set of calibration instructions. This helps the robot, especially a low-cost robot, eliminate systematic errors caused by motion inertia and gear backlash. In this embodiment, the robot has 5 degrees of freedom, so a total of 10 groups are used. After the correction is completed, the robot arm control module completes the angle coordinate system correction based on the above-mentioned target information to ensure that the target angle after sorting is consistent with the expected angle. The correction formula is: ∝ = -β + θ, where ∝ is the rotation angle executed by the servo at the end of the robot arm, β is the rotation angle of the robot arm itself during the sorting and grasping task, and θ is the horizontal rotation angle of the target to be grasped relative to the expected angle after sorting is completed. The robotic arm control module retrieves a pre-stored grab codebook based on the target position in the target information to be grasped. This codebook contains the servo motion instructions required to grasp a target at any (X, Y) position within the working range of robotic arm 1. These codebooks are stored in the ROM memory of mechanical control subsystem 2. This eliminates the need for real-time kinematic calculations during the grasping operation. Instead, a codebook search is required to quickly drive robotic arm 1 to complete the grasping action, providing a low-cost solution to the problem of insufficient computing power. The codebook contains 38,400 entries. Simultaneously, the robotic arm control module calculates the servo motion plan for placing the target based on the target type information L in the target information to be grasped. Each L value corresponds to a specific angle correction value σ. The formula for calculating the rotation angle of the terminal servo when placing the target is γ = -α - β + σ, where γ is the rotation angle of the terminal servo when placing the target. To ensure smooth sorting, Robot Arm 1 maintains the target horizontally to the ground throughout the entire sorting process. To achieve this, the formula δ = 270° - μ is used, where μ is the sum of the servo rotation angles required to grasp the target vertically, as specified in the grab codebook, and δ is the correction angle required to maintain the target horizontally. After completing all the aforementioned calculations and codebook access, the module compiles the instructions into 25 groups of eight hexadecimal numbers. These instructions guide the Robot Arm in completing the tasks of target grasping, horizontal maintenance, angle correction, and target placement. These instructions are sent uniformly to Robot Arm 1, and the Robot Arm's operating instructions are always sent in the principle that horizontal movement precedes vertical movement. Simultaneously, the operating instructions for the air pump are also sent to Robot Arm 1, which drives the grabbing suction cup to pick up and place the target. The air pump operating instructions are compiled in the same way as the Robot Arm instructions described above. After all control commands are sent, the robot control module waits for a response from robot 1. Upon receiving the action completion command, it sends 0xBB to the integrated communication module A via serial internal signal line 03. Integrated communication module A then notifies the visual master control subsystem 3 via serial bus 8 that the target information can be updated. Furthermore, after the command assembly is complete, a copy is made in sequence as a digital twin control signal and sent to the integrated communication module A via serial internal signal line 06.

[0069] like Figure 8 As shown in the figure, the F3 thread receives the digital twin control signal sent by the robot arm control module through the serial internal bus 06. The digital twin control signal is a group of 8 hexadecimal numbers, each group indicating an action of a servo and the speed at which the action is in place. The encoding method is as described above. These control signals are sent to the remote terminal when the Bluetooth is idle to simulate the real robot arm movement in the virtual environment.

[0070] In some embodiments, the visual control subsystem 3 includes:

[0071] A visual solution module is used to receive video stream data from the mechanical main control subsystem 2, identify sorting target objects in the video stream data using a neural network model, and calibrate the sorting target objects based on hand-eye calibration to generate a target data queue;

[0072] The integrated communication module B is used to provide a communication interface for the visual main control subsystem 3 and the mechanical main control subsystem 2 to transmit communication content and target information;

[0073] The human-computer interaction module is used to display the real-time status of sorting and send high-priority instructions to grab any or any type of target instructed by the operator through the operating device.

[0074] Specifically, if Figure 6 As shown, the visual main control subsystem 3 is internally provided with an integrated communication B module, a human-computer interaction module and a visual solution module; the internal parallel signal line No. 09 updates the target data to be sorted of the visual solution module to the integrated communication B module in real time, and at the same time, the internal parallel signal line No. 09 transmits the hand-eye calibration parameter adjustment information from the integrated communication B module to the visual solution module, the internal parallel signal line No. 10 transmits the interaction instruction of the human-computer interaction module to the integrated communication B module, the internal parallel signal line No. 11 transmits the grabbing queue information solved by the visual solution module and the real-time image of the area to be grabbed to the human-computer interaction module, and the human-computer interaction module is connected to the peripheral device touch screen 4 via the touch signal line No. 12 and the HDMI line 13, receives the touch instruction and pushes the integrated rendering video stream.

[0075] like Figure 10 As shown, the integrated communication B module set in the visual main control subsystem 3 receives the hand-eye calibration parameter adjustment information sent by the mechanical main control subsystem 2 through the serial bus 8 No. 01. The parameters are transmitted in the UART protocol. One hexadecimal number is one frame, and four frames form a group. The first bit is converted to decimal and recorded as U S , the second digit is converted to decimal and recorded as U T , the third digit is converted to decimal and recorded as U1, the fourth digit is converted to decimal and recorded as U2, U=U1×16+U2, if U T =1 then U=U, if U T =2 then U = -U; U S =12 represents the horizontal coordinate linear difference P between the eye coordinate system where the camera 5 is located and the hand coordinate system where the robot arm 1 is located. U =U,U S =13 represents the linear difference N between the horizontal and vertical coordinates of the eye coordinate system where the camera 5 is located and the hand coordinate system where the robot arm 1 is located. U =U; integrated communication module B will P U and N U The data is passed to the visual solution module for correction. Figure 2 and Figure 3 As shown, the camera 5 should be placed above the area to be sorted during setup, and the field of view of the camera 5 should fully include the entire area to be sorted. However, in this embodiment, the vibration caused by the movement of the robot arm 1, the influence of the camera 5's own gravity and other forces will cause the camera 5 to move slightly during long-term continuous operation. If the operator finds this trend, he can provide hand-eye calibration parameter adjustment information through the above-mentioned remote terminal subsystem 6. The integrated communication B module will receive and translate the instruction and inform the visual solution module through the internal parallel signal line No. 09, avoiding interruption of system operation and redeployment, thereby improving work efficiency.

[0076] like Figure 11 As shown, the visual solution module receives and captures the video stream data sent by the mechanical main control subsystem 2 through the HDMI line 9 No. 08, and receives the hand-eye calibration parameter adjustment information sent by the above-mentioned integrated communication B module through the internal parallel signal line No. 09. The capture and reception of these two data are executed synchronously using parallel multi-threading technology. The hand-eye calibration correction instruction includes the horizontal lateral offset P U and horizontal longitudinal offset N U Two parameters, the visual solution module locks through the foot point recognition algorithm Figure 2 、 Figure 3 The black boundary of the area to be grasped is shown as a reference for the coordinate system. Figure 3 The coordinate reference point shown is the position coordinate (x1, y1) of the right bottom point of the rectangle in the image as the origin of the reference system, so the origin of the corrected image reference system is (x1+P U ,y1+N U ). The visual solution module runs the pre-trained neural network model to complete the target recognition. The embodiment uses a deep learning method to train the model and lightweight processing, but this model is not unique and should be trained differently according to the different sorting targets. The model training process used in this embodiment is shown in Figure 12 ,according to Figure 12A generalized model training method is provided. First, data is collected, and images of the sorting targets of the sorting robot are collected. At the same time, images should be collected by changing factors such as the stacking method of the targets and the light source environment. Generally, no less than 100 images of each target are collected. Then, the collected data set is labeled and optimized by adding noise, randomly adding interference factors, and other measures. Then, the model is designed and deployed, and parameters such as the number of model iterations, the number of categories, and the training weight are set. Then, the data set is divided into a validation set and a training set in a ratio of 3:7, the model is deployed to complete the training, and the model parameters are further adjusted through result analysis to optimize the recognition effect and modify the clustering prior as needed. The model is adapted and lightweighted according to the type of computing power used by the deployed model to improve the running speed of the model. Finally, the model is debugged and deployed. In this embodiment, the model reserves 5 ports, including: an input port in RGB565 format, an output port in RGB888 format, a target position output port, a target posture output port, and a target recognition status output port. After the above training process, the model training and deployment can be completed and other parts of the present invention can be adapted. After the model runs, it generates the boundary coordinates of the identified target. The visual solution module segments the image based on this boundary coordinate information and converts it into HSV format. It then uses the watershed algorithm to further clearly segment the target boundary. It also analyzes the target's structural integrity in HSV format. This structural integrity analysis is based on the ratio of the target area to the typical area of ​​its type, as well as its number of foot points and perimeter-to-area ratio parameters. The integrity analysis results can be used to determine the stacking relationship of the objects to be sorted, eliminating the need for the binocular camera originally required for grasping three-dimensional stacked objects and helping to address the cost of sorting robots. The visual solution module uses a Hough transform to calculate the horizontal rotation angle θ of the target relative to the expected angle after sorting is completed. By generating the target's minimum circumscribed moment, the target's center point is determined as the grasping point. The target coordinate information (X, Y) is generated, along with the grasping logic. The natural language representation of the grasping logic is: prioritize grasping objects stacked on top. After completing the grasping logic planning, the coordinate information, type information, and angle information of all targets are stored, and a grasping queue is formed according to the grasping logic. The target at the top of the queue is sent to the integrated communication B module through the internal parallel signal line No. 09, recorded as A signal. At the same time, the grasping queue information and the real-time image of the area to be grasped are synchronously updated to the human-computer interaction module.

[0077] like Figure 13 As shown, the human-computer interaction module has two input ports, among which the internal parallel signal line No. 11 receives the capture queue of the above-mentioned visual solution module and the real-time image of the area to be captured and caches it; the target in the real-time image is selected by the minimum external rectangle and the coordinates, priority and category information of the target are displayed on the screen, and the rendering interaction button is output to the touch screen through the HDMI line No. 13, as shown. Figure 2As shown, the operator can view the status of the targets in the area to be sorted in real time, and can click the left button to require the robot to prioritize grabbing a certain type of target or click the right button to require the robot to prioritize grabbing a specific target; these touch signals are transmitted to the human-computer interaction module via the touch signal line 14 No. 12. The human-computer interaction module determines the sorting instruction based on the touch area and retrieves the grabbing queue according to the instruction. The grabbing queue is the grabbing queue sent and cached from the visual solution module. If the retrieval of the queue is successful, that is, the operator-specified target exists in the area to be sorted, then the target or target type information is sent to the integrated communication B module and recorded as B signal. If it is unsuccessful, it indicates that there is no such target in the area.

[0078] like Figure 10 As shown, the integrated communication module B receives the B signal sent from the human-machine interaction module via the internal parallel signal line No. 10, and the A signal sent from the visual settlement module via the internal parallel signal line No. 09. If the B signal is valid, the information of the target to be sorted contained in the B signal is unconditionally and preferentially encoded, that is, the instructions issued by the operator through the touch screen are preferentially executed; otherwise, the information of the target to be sorted contained in the A signal is automatically encoded. During encoding, a communication interaction includes 5 communication frames, each frame contains an 8-bit binary number. The first frame adds the check code 0xAA, and the last 4 bits of the 8-bit data of the fourth frame add the check code 0xA. The second frame writes the horizontal coordinate X of the sorting target, the third frame writes the horizontal coordinate Y of the sorting target, the first 4 bits of the 8-bit data of the fourth frame write the category L of the sorting target, and the fifth frame writes the horizontal rotation angle of the current sorting target relative to the expected angle after sorting is completed. Waiting for the mechanical main control subsystem 2 to send a request to update the sorting target instruction through the No. 01 serial bus 8, and send the target information to be sorted.

[0079] In some embodiments, the remote terminal subsystem 6 includes:

[0080] Integrated communication C module, used to cooperate with the integrated communication A module of the mechanical main control subsystem 2;

[0081] Digital twin module 7, used to build a simulated robotic arm in a virtual scene to simulate the movement of the robotic arm using real physical parameters;

[0082] The parameter adjustment module is used to correct the hand-eye calibration of the robot arm 1, obtain the robot arm parameter adjustment instruction, and adjust the visual parameters of the image acquisition device according to the light conditions of the deployment environment of the robot arm 1.

[0083] Specifically, if Figure 6 As shown, the remote terminal subsystem 6 is internally provided with an integrated communication C module, a digital twin module 7 and a parameter adjustment module, wherein the parameter adjustment module pushes the digital twin rendering image to the peripheral device monitor through HDMI cable No. 14 11.

[0084] like Figure 14 As shown, the remote terminal subsystem 6 can be flexibly deployed on a general-purpose computer. It receives digital twin instructions from the mechanical main control subsystem 2 via Bluetooth wireless 10 No. 04, and sends parameter adjustment instructions given by the parameter adjustment module to the mechanical main control subsystem 2 via Bluetooth wireless 10 No. 04. Among them, the integrated communication C module performs Bluetooth communication encoding and decoding operations. In this embodiment, Bluetooth communication is performed using a baud rate of 115200, 8 data bits, 1 stop bit, and even parity. The integrated communication C module continuously receives the digital twin instructions sent by the mechanical central control subsystem 2. The instruction encoding method is the same as the format of sending instructions to the robot 1 through the No. 15 robot control line 12. The integrated communication C module decodes the instruction information and checks whether the data contained in the instruction contains valid information. If valid, it is transmitted to the digital twin module 7. The valid information verification rules are as follows: Each group of instructions consists of 13 8-bit binary numbers. When the first number is 01000000 (decoded as @ in ASCII encoding) and the second number is in the closed interval from 0 to 9 after being decoded into decimal in ASCII encoding, and the third number is 01001011 (decoded as K), and the 4th, 5th, 6th and 7th numbers are respectively decoded into decimal according to ASCII encoding and recorded as t, h, r and s, and p is calculated according to the calculation formula p=t×1000+h×100+r×10+s, and p is in the closed interval of 0 to 2500, and the 8th bit is 01010100 (decoded as T according to ASCII encoding), and the 9th, 10th, 11th and 12th bits are respectively recorded as t2, h2, r2 and s2, and p2 is calculated according to the calculation formula p2=t2×1000+h2×100+r2×10+s2, and p2 is in the closed interval of 100 to 1500, and the 13th number is 00100011 (decoded as # according to ASCII encoding). After receiving a valid instruction, the digital twin module 7 performs command mapping, that is, mapping the instruction to the corresponding virtual robotic arm joint according to the second digit mentioned above, mapping the above p value to the movement angle of the robotic arm joint servo, angle θ = p × 0.108, and mapping the above p2 to the movement time of the robotic arm joint servo, time T = p2ms. The digital twin module 7 is provided with physical modeling of each part of the robotic arm 1, and is virtually assembled into a digital twin robotic arm identical to the real robotic arm according to the real physical laws. The virtual servo will drive the digital twin robotic arm to work according to the above mapping results, and always maintain the same physical posture as the real robotic arm. The digital twin robotic arm is placed in the virtual work scene and rendered in 3D according to the light and material, and the effect is as follows. Figure 15 As shown, the real-time picture after 3D rendering will be sent to the digital twin monitor via HDMI line 11. The parameter adjustment module first renders the human-computer interaction interface, which becomes the operation interface visible to the operator. The effect is as follows Figure 16As shown, the parameter adjustment module awaits operator instructions for adjusting brightness, hue, saturation, and hand-eye calibration parameters. These instructions are formatted as a 4-digit hexadecimal number, where the first digit indicates the adjustment target, the second digit indicates the adjustment direction, and the third and fourth digits indicate the adjustment value. The integrated communication C module assembles these instructions into Bluetooth protocol-specified segments and transmits them while Bluetooth is idle.

[0085] In summary, the control device of the sorting robot proposed in the embodiment of the present invention adopts an FPGA platform to deploy the mechanical main control subsystem, giving play to the advantages of parallel processing, and effectively solving the operation speed problem of the robot control system; adopts an NPU device to complete the visual main control subsystem, giving play to its acceleration effect on the neural network model, and improving the target recognition speed of the robot control system; deploys the remote terminal subsystem on a general-purpose computer, enhancing the deployment flexibility of the system; the three core subsystems work together to achieve low-cost, highly intelligent completion of target recognition and sorting tasks.

[0086] Next, the control method of the sorting robot proposed according to the embodiment of the present invention will be described with reference to the accompanying drawings.

[0087] Figure 17 4 is a flow chart of a control method for a sorting robot according to an embodiment of the present invention.

[0088] like Figure 17 As shown, the control method of the sorting robot includes the following steps:

[0089] In step S1701, a panoramic image of the target area to be sorted is captured by an image capture device, and the panoramic image is sent to the mechanical main control subsystem.

[0090] In step S1702, the panoramic image is sent to the visual main control subsystem via the mechanical main control subsystem, the visual main control subsystem identifies and locates the sorting target object in the panoramic image, and sends the panoramic image to the operating device.

[0091] In step S1703, the mechanical main control subsystem is used to control the mechanical arm to grab the sorting target object to the sorting area, and the movement information of the mechanical arm is sent to the remote terminal subsystem.

[0092] In step S1704, the motion information is analyzed by the remote terminal subsystem to generate a robot arm parameter adjustment instruction.

[0093] In step S1705 , the panoramic image is displayed by the operating device, and a high-priority instruction of the operator to capture any one or any type of target is sent.

[0094] It should be noted that the aforementioned explanation of the embodiment of the control device of the sorting robot is also applicable to the control method of the sorting robot of this embodiment, and will not be repeated here.

[0095] According to the control method for a sorting robot proposed in an embodiment of the present invention, an FPGA platform is used to deploy the mechanical main control subsystem, leveraging the advantages of parallel processing to effectively solve the problem of the operating speed of the robot control system. An NPU device is used to complete the visual main control subsystem, leveraging its acceleration effect on the neural network model to improve the target recognition speed of the robot control system. A remote terminal subsystem is deployed on a general-purpose computer to enhance the deployment flexibility of the system. The three core subsystems work together to achieve low-cost, highly intelligent completion of target recognition and sorting tasks.

[0096] Figure 18 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0097] Memory 1801 , processor 1802 , and computer programs stored in the memory 1801 and executable on the processor 1802 .

[0098] When the processor 1802 executes the program, the control method of the sorting robot provided in the above embodiment is implemented.

[0099] Furthermore, the electronic device further includes:

[0100] The communication interface 1803 is used for communication between the memory 1801 and the processor 1802 .

[0101] The memory 1801 is used to store computer programs that can be run on the processor 1802 .

[0102] The memory 1801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0103] If the memory 1801, processor 1802, and communication interface 1803 are implemented independently, the communication interface 1803, memory 1801, and processor 1802 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 18 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0104] Optionally, in a specific implementation, if the memory 1801, the processor 1802 and the communication interface 1803 are integrated on a chip, the memory 1801, the processor 1802 and the communication interface 1803 can communicate with each other through an internal interface.

[0105] The processor 1802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0106] An embodiment of the present invention further provides a computer program product, which implements the above-mentioned control method of the sorting robot when the computer program / instructions are executed by a processor.

[0107] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned control method for the sorting robot when the program is executed by a processor.

[0108] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0109] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0110] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0111] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0112] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0114] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0115] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A control device for a sorting robot, characterized in that: include: A robotic arm is disposed between the target to-be-sorted area and the sorting area, and is configured to perform a sorting action corresponding to sorting the to-be-sorted items in response to a control instruction; An image acquisition device, the image acquisition device being arranged in the target area to be sorted, so as to acquire a panoramic image of the target area to be sorted; A mechanical main control subsystem, which is connected to the mechanical arm and the image acquisition device respectively to generate control instructions for controlling the mechanical arm to sort the items to be sorted based on the panoramic image, wherein the mechanical main control subsystem includes: A visual preprocessing module is used to receive and process the panoramic image of the image acquisition device to obtain video stream data, and prepare the video stream data into HDMI protocol for transmission to the visual main control subsystem; The integrated communication A module is used to verify the information of the target to be sorted between the mechanical main control subsystem and the visual main control subsystem to obtain the target information; A manipulator control module is configured to receive the target information, generate control instructions for controlling the manipulator to perform sorting according to the target information, and map the motion information of the manipulator to a remote terminal subsystem; A visual main control subsystem, connected to the mechanical main control subsystem, for identifying and locating sorting target objects in the panoramic image according to target sorting requirements; An operating device, connected to the visual main control subsystem, for displaying the panoramic image and sending a high-priority instruction from an operator to capture any object or any type of object; A remote terminal subsystem is connected to the mechanical main control subsystem and is used to monitor the motion information of the mechanical arm to generate a mechanical arm parameter adjustment instruction according to the motion information.

2. The control device of the sorting robot according to claim 1, characterized in that: The integrated communication A module is provided with three synchronization lines: F1 thread, F2 thread and F3 thread, wherein: The F1 thread is used to verify the to-be-sorted target information sent by the visual main control subsystem using UART protocol communication to obtain the target information, and send the target information to the robotic arm control module; The F2 thread is used to communicate to the visual main control subsystem that the current robot arm has completed sorting and needs to update the sorting target object; otherwise, it communicates the visual main control subsystem approval information; The F3 thread is used to receive the robot arm parameter adjustment instruction sent by the remote terminal subsystem.

3. The control device of the sorting robot according to claim 2, characterized in that: The target information includes inspection information, category of the target object to be sorted, horizontal coordinates of the target object to be sorted, vertical coordinates of the target object to be sorted, and horizontal rotation angle of the target object to be sorted relative to the expected angle after sorting is completed.

4. The control device of the sorting robot according to claim 1, characterized in that: The visual main control subsystem includes: a visual solution module, configured to receive video stream data from the mechanical main control subsystem, identify sorting target objects in the video stream data using a neural network model, and calibrate the sorting target objects based on hand-eye recognition to generate a target data queue; An integrated communication B module is used to provide a communication interface for the visual main control subsystem and the mechanical main control subsystem to transmit communication content and the target information; The human-computer interaction module is used to display the real-time status of sorting and send high-priority instructions for grabbing any one or any type of target instructed by the operator through the operating device.

5. The control device of the sorting robot according to claim 1, characterized in that: The remote terminal subsystem includes: An integrated communication C module, configured to cooperate with the integrated communication A module of the mechanical main control subsystem; A digital twin module, used to build a simulated robotic arm in a virtual scene to simulate the movement of the robotic arm using real physical parameters; The parameter adjustment module is used to correct the hand-eye calibration of the robotic arm, obtain the robotic arm parameter adjustment instruction, and adjust the visual parameters of the image acquisition device according to the light conditions of the deployment environment of the robotic arm.

6. A control method for a sorting robot, characterized in that: The control device of the sorting robot according to any one of claims 1 to 5 comprises the following steps: Using an image acquisition device to capture a panoramic image of the target area to be sorted, and sending the panoramic image to the mechanical main control subsystem; The panoramic image is sent to the visual main control subsystem through the mechanical main control subsystem, the sorting target object in the panoramic image is identified and located through the visual main control subsystem, and the panoramic image is sent to the operating device; Utilizing the mechanical main control subsystem to control the mechanical arm to grab the sorting target object to the sorting area, and sending the movement information of the mechanical arm to the remote terminal subsystem; Analyzing the motion information through the remote terminal subsystem to generate a robot arm parameter adjustment instruction; The panoramic image is displayed by the operating device, and a high-priority instruction of an operator to capture any one or any type of target is sent.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the control method of the sorting robot according to claim 6.

8. A computer program product, characterized in that When the computer program / instructions are executed by a processor, the control method of the sorting robot according to claim 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the control method of the sorting robot according to claim 6.

Images