Intelligent robot control method and system for depalletizing

Abstract

The present application relates to an intelligent robot control method and system for depalletizing, which is applied to the field of intelligent robot technology. The method and system include initializing the intelligent robot and its accessory equipment through a driver integrated in the intelligent robot system to obtain an initial parameter file; receiving a mode selection instruction from an administrator terminal, and determining an execution plan according to the mode execution instruction; issuing a continuous imaging instruction to the 3D visual imaging device to obtain a target image to be measured; identifying the target image to be measured, marking the operating area in the target image to be measured, and generating a marking frame; automatically locating the center position of the marking frame according to a preset graphic center recognition algorithm; and controlling the intelligent robot to grasp the target object and perform depalletizing according to the execution plan, the marking frame, the target image to be measured, and the center position.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent robots, and in particular to an intelligent robot control method and system for depalletizing. Background Art

[0002] In the manufacturing industry, the advantages of robots, such as long working hours, high repetition accuracy, high efficiency and high consistency, have brought huge value to the manufacturing industry.

[0003] In the scene of palletizing and unpacking, before the appearance of robots, the work has always been done manually, with high work intensity, harsh environment, time-consuming and labor-intensive, and low efficiency. The quality of unpacking and unpacking is not high, and the execution quality is uneven. After the appearance of robots, this phenomenon has been greatly improved, but whether it is a four-axis or six-axis industrial robot, a collaborative palletizing workstation, a non-standard palletizer, or a truss robot, they can only work through fixed program logic and fixed execution actions, and do not collect data on the work object. Therefore, the current robot palletizing and unpacking products have no ability to respond to the replacement of product classification and palletizing at the work site, changes in the work process, and unexpected situations during the work process. This has caused manual work to be off-duty but not off-duty due to function changes (unpacking and unpacking conversion), production changes, mixed production, accidental equipment failures, and unexpected production accidents. The further popularization of robot palletizing and unpacking has created a great problem.

[0004] With respect to the above-mentioned related technologies, it is believed that the above-mentioned palletizing and unpacking robots lack vision and spatial perception, and are not convenient for coping with changes in the operation process, which affects the scope of application of the palletizing and unpacking robots. Summary of the invention

[0005] In order to improve the problem that the depalletizing and stacking robot lacks vision and spatial perception, is inconvenient to cope with changes in the operating process, and affects the application scope of the depalletizing and stacking robot, the present application provides an intelligent robot control method and system for depalletizing and stacking.

[0006] In the first aspect, the present application provides an intelligent robot control method for depalletizing, which adopts the following technical solution: including:

[0007] Initialize the intelligent robot and its accessory devices through the driver integrated in the intelligent robot system to obtain the initial parameter file;

[0008] Receiving a mode selection instruction from an administrator terminal, and determining an execution plan according to the mode execution instruction;

[0009] Sending continuous imaging instructions to the 3D visual imaging device to obtain an image of the target to be measured;

[0010] Identify the target image to be measured, mark the operation area in the target image to be measured, and generate a marking frame;

[0011] Automatically locate the center position of the annotation frame according to a preset graphic center recognition algorithm;

[0012] According to the execution scheme, the marking frame, the target image to be measured and the center position, the intelligent robot is controlled to grasp the target object and perform the depalletizing and stacking work.

[0013] Optionally, the intelligent robot and its accessory devices are initialized through a driver integrated into the intelligent robot system to obtain an initial parameter file, including:

[0014] Perform initial parameter debugging on the robot assembly through the robot and supporting equipment programs integrated in the intelligent robot system, including: initializing and monitoring the robot status, resetting the robot assembly to the initial position, initializing and monitoring the visual gimbal status, resetting the gimbal to the initial position, and resetting each device to the initial state;

[0015] Calibrate and synchronize the visual coordinate system of the 3D visual imaging device with the coordinates of the robot or the end of the robot suction fixture according to the preset machine vision calibration algorithm;

[0016] The intelligent robot system reads the preset executable solution file list, automatically loads the current executable solution task, obtains parameter configuration loading, and calls out pre-stored operation parameters, covering cargo type, pallet specifications, destacking target location, and destacking sequence rules, and determines the task operation logic based on the parameters.

[0017] Optionally, the operation area includes: an identification area, a safety area, and an arrival area;

[0018] After identifying the target image to be measured, marking the operation area in the target image to be measured, and generating a marking frame, the method further includes:

[0019] Through 2D / 3D image synchronization and fusion technology, the technology corrects the deviation of the spatial coordinate system of the identified object caused by the deviation of the viewing angle;

[0020] Through the human-computer interaction technology of image annotation and the computing technology of image annotation and spatial data synchronization, the safety zone, identification zone and in-place zone corresponding to the area where the identified object is located are divided;

[0021] With the help of laser sensors, the status of incoming and outgoing materials can be identified and monitored;

[0022] Through image filtering preprocessing, region of interest segmentation, mean operation, depth threshold limitation, and image area data comparison, it is determined whether the identified object is completely within the identification area.

[0023] Optionally, after identifying the target image to be measured, marking the operation area in the target image to be measured, and generating a marking frame, the method further includes:

[0024] The 2D color image is fused with the 3D depth data, the camera internal and external parameters are calibrated by the coordinate system conversion algorithm, and the coordinate system of the depth camera is converted to the color camera coordinate system;

[0025] The depth image is reprojected through the image reprojection algorithm to ensure that each pixel can be accurately mapped to the corresponding position of the RGB image;

[0026] Through data alignment and synchronization, the region of interest is cropped from the RGB image, and it is ensured that the region of interest corresponds to the corresponding point in the depth image one by one;

[0027] Through point cloud generation and posture calculation, the selected depth image data is converted into a point cloud format, and then the position and posture of the target in the three-dimensional space are calculated.

[0028] Optionally, after converting the selected depth image data into a point cloud format through point cloud generation and posture calculation, and then calculating the position and posture of the target in the three-dimensional space, the method further includes:

[0029] Select the cutting axis along the X-axis, Y-axis or Z-axis to cut according to your needs;

[0030] Set the cutting range according to the minimum and maximum values ​​in the specified axis to define the cutting limit;

[0031] Automatically filter out qualified point cloud data according to the specified axis and the cutting range;

[0032] The points in the point cloud data are grouped into a plurality of clusters according to certain similarity standards through a point cloud clustering algorithm, a spatial distance calculation algorithm, a cluster formation algorithm, and an object extraction algorithm.

[0033] Optionally, after the points in the point cloud data are grouped into a plurality of clusters according to a certain similarity standard by using the point cloud clustering algorithm, the spatial distance calculation algorithm, the cluster formation algorithm, and the object extraction algorithm, the method further includes:

[0034] Through the threshold definition procedure, a distance threshold is pre-set to determine whether the points belong to the same plane;

[0035] By random sampling and plane equation calculation: randomly select three points to construct an initial plane, and calculate the distance from the point to the initial plane;

[0036] Project all points onto the calculated plane and measure the actual distance of each point to the initial plane;

[0037] Through the interior point determination and iterative optimization algorithm, the points whose distances from the screening points to the initial plane meet the requirements below a preset threshold are set as interior points;

[0038] Through continuous iterative optimization, the plane with the maximum number of internal points is found;

[0039] Through covariance matrix calculation and feature analysis, the covariance matrix of the point cloud data is calculated, and the eigenvalue decomposition is performed to determine the main direction of the point cloud;

[0040] Then, a new coordinate system is constructed based on the eigenvectors through the coordinate system transformation algorithm to ensure that the new coordinate axis is consistent with the main direction of the point cloud;

[0041] By calculating the boundary value in the new coordinate system, the maximum and minimum values ​​of the point cloud along each axis are solved respectively;

[0042] The minimum volume cube of the above boundary value is determined by bounding box calculation and inverse transformation algorithm, and converted back to the original coordinate system to complete the calculation of the minimum oriented bounding box.

[0043] In a second aspect, the present application provides an intelligent robot control device for depalletizing, which adopts the following technical solutions, including:

[0044] The initial module is used to initialize the intelligent robot and its accessory devices through the driver integrated in the intelligent robot system and obtain the initial parameter file;

[0045] An interactive module, used to receive a mode selection instruction from an administrator terminal, and determine an execution plan according to the mode execution instruction;

[0046] An image module is used to send continuous image acquisition instructions to the 3D visual imaging device to obtain an image of a target to be measured;

[0047] A recognition module, used for recognizing the target image to be measured, marking the operation area in the target image to be measured, and generating a marking frame;

[0048] A marking module, used for automatically locating the center position of the marking frame according to a preset graphic center recognition algorithm;

[0049] An execution module is used to control the intelligent robot to grasp the target object and perform depalletizing and stacking according to the execution scheme, the marking frame, the target image to be measured and the center position.

[0050] In a third aspect, the present application further provides a control device, the device comprising:

[0051] The invention comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and executes the intelligent robot control method for depalletizing as described above.

[0052] In a fourth aspect, the present application further provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above for the intelligent robot control method for depalletizing.

[0053] In summary, in this application, by calling the camera driver provided by the 2D / 3D industrial camera manufacturer, integration and optimization are carried out to realize the adaptation, communication, control, data extraction, and data optimization operations of the 2D / 3D industrial camera. 2D vision and AI intelligent recognition are used to realize accurate identification of the working object in a complex environment; 3D vision and 2D / 3D synchronization technology are used to realize the information collection of the appearance, shape, category, size, coordinates, and posture of the working object, and the point cloud in the object space is obtained based on the laser triangulation algorithm or the time of flight algorithm. The point cloud is cut and separated by the artificial intelligence visual recognition ability, and the actual size, shape, posture, and spatial coordinates of the object required by this application are finally obtained through the algorithm processing of the 3D point cloud. Then, the corresponding object is carried out. The depalletizing and stacking planning operation is converted into the execution trajectory of the robot, thereby improving the vision and spatial perception of the intelligent robot, and then it is convenient to determine the execution object, execution area, and execution trajectory according to the robot's vision and spatial perception. Finally, the robot performs depalletizing operations according to the execution trajectory to achieve intelligence, high flexibility, and can adapt to the depalletizing and stacking effects of different types and specifications of items, greatly increasing the scope of application of the depalletizing and stacking robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The present invention is a structural schematic diagram of an intelligent robot control system for palletizing.

[0055] Figure 2 The invention is a flow chart of an intelligent robot control method for depalletizing.

[0056] Figure 3 The invention is an operation flow chart of an intelligent robot control system for palletizing and disassembling.

[0057] Figure 4 It is an artificial intelligence flow chart of an intelligent robot control system for depalletizing and stacking.

[0058] Figure 5 The present invention is a structural block diagram of an intelligent robot control device for depalletizing.

[0059] Explanation of reference numerals: 510, initial module; 520, interaction module; 530, image module; 540, recognition module; 550, labeling module; 560, execution module. DETAILED DESCRIPTION

[0060] The following combination Figure 1 - Figure 5 This application is described in further detail.

[0061] Reference Figure 1 The execution system of this application is a robot vision intelligent recognition and intelligent palletizing system. It includes a robot 2D / 3D vision camera, auxiliary light source, visual rotating pan-tilt head, multifunctional machine vision bracket, conveying device, robot body and drive control, standard robot workbench (including base, control cabinet, debugging table, robot lifting mechanism, signal and system control center related equipment and accessories), artificial intelligence image recognition model, artificial intelligence image annotation software, artificial intelligence image training program, local server, local database, process palletizing software and human-computer interaction interface, server interaction system, database project management software. 2D vision camera, auxiliary vision auxiliary light source, artificial intelligence image annotation software, artificial intelligence image training program, in order to obtain artificial intelligence recognition model; 3D vision camera collects spatial information of palletizing items through the use of laser triangular valve to realize the recognition of the position and posture of the object in space; visual rotating pan-tilt head provides a continuous working angle of view for 2D / 3D camera; mechanical conveying device, robot body and integrated drive and control and standard robot workbench are the action actuators of the equipment; local server, local database, process palletizing software and human-computer interaction interface, server interaction system, database project management software provide management, service and application support for the system.

[0062] Compared with the existing technology, the beneficial effects of the present application are as follows: the present application adopts 2D vision and AI intelligent recognition to realize accurate identification of work objects in complex environments; 3D vision and 2D / 3D synchronization technology are used to realize the information collection of the appearance, shape, category, size, coordinates, and posture of the work object, and then the corresponding object depalletizing and stacking planning calculation is performed, which is converted into the execution trajectory of the robot, and finally the robot performs depalletizing and stacking operations according to the execution trajectory. It is intelligent and flexible, can adapt to the depalletizing and stacking of items of different types and specifications, is basically independent of manual labor, and has high application prospects and market value.

[0063] Reference Figure 2 , the embodiment of the present application at least includes step S10 to step S30.

[0064] S10, initializing the intelligent robot and its accessory devices through the driver integrated into the intelligent robot system to obtain an initial parameter file.

[0065] S20, receiving a mode selection instruction from the administrator terminal, and determining an execution plan according to the mode execution instruction.

[0066] S30, sending a continuous imaging instruction to the 3D visual imaging device to obtain the image of the target to be measured.

[0067] S40, identifying the target image to be measured, marking the operation area in the target image to be measured, and generating a marking frame.

[0068] S50: automatically locate the center position of the annotation frame according to a preset graphic center recognition algorithm.

[0069] S60, according to the execution plan, the marking box, the target image to be measured and the center position, the intelligent robot is controlled to grab the target object and perform the depalletizing and stacking work.

[0070] Specifically, by calling the camera driver provided by the 2D / 3D industrial camera manufacturer, integration and optimization are carried out to realize the adaptation, communication, control, data extraction, and data optimization operations of the 2D / 3D industrial camera. 2D vision and AI intelligent recognition are used to realize accurate identification of the working object in a complex environment; 3D vision and 2D / 3D synchronization technology are used to realize the information collection of the appearance, shape, category, size, coordinates, and posture of the working object, and the point cloud in the object space is obtained based on the laser triangulation algorithm or the time of flight algorithm. The point cloud is cut and separated through the artificial intelligence visual recognition ability, and the actual size, shape, posture, and spatial coordinates of the object required by this application are finally obtained through the algorithm processing of the 3D point cloud. Then, the corresponding object is carried out. The depalletizing and stacking planning operation is converted into the execution trajectory of the robot, thereby improving the visual and spatial perception of the intelligent robot, and then it is convenient to determine the execution object, execution area, and execution trajectory according to the robot's vision and spatial perception. Finally, the robot performs depalletizing operations according to the execution trajectory, so as to achieve intelligence, high flexibility, and can adapt to the depalletizing and stacking effects of different types and specifications of items, greatly increasing the scope of application of the depalletizing and stacking robot.

[0071] Reference Figure 3In fact, the intelligent code disassembly workstation system software includes the initial data, functional modules, main program, and communication module of the intelligent code disassembly workstation system software. The software program modules mainly include the main program, human-computer interaction interface program, initialization data access, initialization parameter adjustment program, visual hand-eye calibration program, robot hand-eye calibration program, program creation program, program management program, remote maintenance program, manual mode program, intelligent mode program, communication mode program, internal core equipment communication program, external equipment communication program, execution action guidance program, execution monitoring program, data statistics program, and personnel safety protection auxiliary program. By calling the camera driver provided by the 2D / 3D industrial camera manufacturer, integration and optimization are carried out to achieve the adaptation, communication, control, data extraction and data optimization of the 2D / 3D industrial camera. After integrating and debugging the camera driver, the control of the 2D / 3D industrial camera is realized. The robot assembly driver is integrated and debugged to achieve the control of the robot assembly. Then the system is initialized and the executable solution is called. After completing the above initialization work, the human-computer interaction interface of the intelligent palletizing robot system is entered to carry out the first creation of the solution (or confirm the factory default solution), mode selection (manual mode & communication mode & intelligent mode), and the method After the user selects the scheme (default execution scheme & manual selection scheme) and execution mode (fast mode & monitoring mode) through the administrator terminal, the mode execution command is sent to the system. The system drives the PTZ to move the camera field of view to each target position and determine the working area. After the system completes the acquisition and analysis of the previous node data, it uses the path planning algorithm to comprehensively consider the equipment's motion range, obstacle distribution, speed and acceleration limits for the round-trip path of the robot arm and palletizing operations, and finds the shortest, least time-consuming and obstacle-avoiding safe path. It recalculates the optimized path, sends the recognition object coordinates, transition position, and offset parameters to the robot assembly to reduce energy consumption and operation cycle. The robot assembly performs the palletizing operation according to the system instructions.

[0072] In some embodiments, step S10 includes at least the following steps: performing initial parameter debugging on the robot assembly through the robot and supporting equipment program integrated by the intelligent robot system, including: initializing and monitoring the robot status, resetting the robot assembly to the initial position, initializing and monitoring the visual gimbal status, resetting the gimbal to the initial position, and resetting each device to the initial state; calibrating and synchronizing the visual coordinate system of the 3D visual imaging device with the coordinates of the robot or the end of the robot suction fixture according to a preset machine vision calibration algorithm; reading the preset executable solution file list through the intelligent robot system, automatically loading the current executable solution task, obtaining parameter configuration loading, and calling pre-stored operation parameters, covering cargo type, pallet specifications, de-stacking target position, and de-stacking sequence rules, and determining the task operation logic according to the parameters.

[0073] Specifically, the 2D / 3D industrial camera driver integrated in the intelligent palletizing robot system is used to debug the initial parameters of the 2D / 3D industrial camera and monitor the status of the 2D / 3D industrial camera to generate an initial parameter file; the robot assembly is debugged and an initial parameter file is generated through the robot and supporting equipment program integrated in the intelligent palletizing robot system (initialize and monitor the robot status, reset the robot assembly to the initial position; initialize and monitor the visual gimbal status, reset the gimbal to the initial position; reset each device to the initial state; the 2D / 3D industrial camera and the robot assembly are calibrated by the hand-eye calibration software of the intelligent palletizing robot system and the robot hand-eye calibration program of the intelligent palletizing robot system. Hand-eye calibration, and generate executable calibration files; read the list of executable solution files stored in the system through the intelligent palletizing robot system, automatically load the current executable solution task, obtain parameter configuration loading, and call up pre-stored operation parameters, covering cargo type (size, weight, material, packaging style), pallet specifications, destacking target location (conveyor belt or pallet location in different cargo classification temporary storage areas), destacking sequence rules, and after confirmation, the system executes the task operation logic according to the parameters. If there is no solution in the executable solution file list, wait for the first creation of the solution according to the intelligent palletizing robot system; read the artificial intelligence recognition model file required by the current solution through the intelligent palletizing robot system as the model basis for the subsequent recognition function of the solution.

[0074] In practice, the coordinate system calibration software for the vision system and the robot actuator is mainly implemented by capturing the image of the calibration object through the camera at different positions and postures, and recording the position and posture of the end effector of the robot arm in the same coordinate system. The feature points of the calibration object are extracted from the camera image, usually using corner detection or other feature extraction algorithms. The posture of the camera and the robot arm is represented as an axis-angle representation, where the axis is a unit vector and the angle represents the magnitude of the rotation. The rotation can be converted into an axis-angle form by the logarithmic mapping of the rotation matrix. Form a linear equation system: The posture data in the axis-angle form is organized into a linear equation system in a certain form. The equation system can be expressed as AX=XB, where A and B are the axis-angle representations of the camera and the robot arm, respectively, and X is the transformation matrix to be calculated. The linear equation system is solved using the least squares method, that is, an optimal solution X is found such that AX≈XB. The optimal solution is obtained by converting the equation system into a matrix form and solving the equation system using the least squares method. The transformation matrix between the camera and the robot arm is extracted from the optimal solution X, and finally the unified data collection of the visual and robot execution coordinate systems is realized.

[0075] Optionally, considering the identification of the working area, the relevant steps are as follows: through 2D / 3D image synchronization and fusion technology, the technology of correcting the deviation of the spatial coordinate system of the identification object caused by the deflection of the viewing angle; through the human-computer interaction technology of image annotation and the computing technology of image annotation and spatial data synchronization, the safety zone, identification zone and arrival zone corresponding to the area where the identification object is located are divided; through the assistance of laser sensors, the status of incoming and outgoing materials is identified and monitored; through image filtering preprocessing, area of ​​interest segmentation, mean calculation, depth threshold limit, and image area data comparison, it is determined whether the identification object is completely within the identification area.

[0076] Among them, the working area includes: identification area, safety area and arrival area.

[0077] Specifically, through the depth map / RGB map obtained by the 3D vision device and the synchronization information of the depth map and RBG map, the spatial coordinate data based on the 3D vision device coordinate system corresponding to the pixel points of a single or multiple targets marked by the graphic annotation tool on the RGB map is read, and the coordinate set is converted into the robot coordinate system through the machine vision annotation algorithm, and the execution order and compensation parameters of each coordinate are attached. At this point, the purpose of specifying the robot coordinates through the visual field and forming the action execution logic is achieved.

[0078] In some embodiments, considering that the system target recognition requires the processing of visual algorithms, the corresponding steps are as follows: the 2D color image is fused with the 3D depth data, and the camera internal and external parameters obtained by calibration of the coordinate system conversion algorithm are converted to the color camera coordinate system; the depth image is reprojected through the image reprojection algorithm to ensure that each pixel can be accurately mapped to the corresponding position of the RGB image; through data alignment and synchronization, the focus area is cropped from the RGB image, and it is ensured that the focus area corresponds to the corresponding point in the depth image one by one; through point cloud generation and posture calculation, the selected depth image data is converted into a point cloud format, and then the position and posture of the target in three-dimensional space are calculated.

[0079] Specifically, the image data source is provided by a 2D / 3D industrial camera. After the 2D / 3D industrial camera is integrated, calibrated, initialized, and debugged, a stable, high-quality 2D / 3D image that meets the requirements of this application is obtained. The image is pre-processed through a point cloud generation algorithm and a 2D / 3D image point cloud synchronization calibration algorithm to obtain a color plane image of the object, construct a three-dimensional outline, and accurately measure the unit geometric dimensions, providing data support for the next step of image processing. The artificial intelligence model (trained and optimized with a large number of cargo image samples) and target detection obtained by the process operation based on artificial intelligence need to quickly distinguish different cargo categories and corresponding label positions. Therefore, this application uses a visual algorithm to fuse 2D color images with 3D depth data to achieve the effect of mapping the 2D image results cut out by the artificial intelligence model to the 3D point cloud and processing them.

[0080] Furthermore, the corresponding processing steps are as follows: select the cutting axis defined along the X-axis, Y-axis or Z-axis for cutting according to the needs; set the cutting range according to the minimum and maximum values ​​along the specified axis to define the cutting limit; automatically filter out the qualified point cloud data according to the specified axis and cutting range; group the points in the point cloud data into multiple clusters according to certain similarity standards through the point cloud clustering algorithm, spatial distance calculation algorithm, cluster formation algorithm and object extraction algorithm.

[0081] Furthermore, the corresponding steps are as follows: through the threshold definition program, a distance threshold is pre-set to determine whether the points belong to the same plane; through random sampling and plane equation calculation: three points are randomly selected to construct the initial plane, and the distance from the point to the initial plane is calculated; all points are projected onto the calculated plane, and the actual distance from each point to the initial plane is measured; through the interior point judgment and iterative optimization algorithm, the points whose distance from the point to the initial plane meets the preset threshold are screened and set as interior points; through continuous iterative optimization, until the plane with the largest number of interior points is found; through covariance matrix calculation and feature analysis, the covariance matrix of the point cloud data is calculated, and the eigenvalue decomposition is performed on it to determine the main direction of the point cloud; then through the coordinate system transformation algorithm, a new coordinate system is constructed based on the eigenvector to ensure that the new coordinate axis is consistent with the main direction of the point cloud; through boundary value calculation in the new coordinate system, the maximum and minimum values ​​of the point cloud along each axis are solved respectively; through the bounding box calculation and inverse transformation algorithm, the cube with the minimum volume of the above boundary value is determined, and it is converted back to the original coordinate system to complete the calculation of the minimum oriented bounding box.

[0082] Specifically, according to the above-mentioned 3D point cloud processing steps, accurate identification of work objects in complex environments is achieved; 3D vision and 2D / 3D synchronization technology are used to collect information on the appearance, shape, category, size, coordinates, and posture of the work objects, and then the corresponding object depalletizing and stacking planning operations are performed and converted into the robot's execution trajectory.

[0083] In order to improve the efficiency of image processing in this application and save computing resources of edge computing devices, it is also necessary to divide the three-dimensional space into several small cubes (voxels) through voxel grid division. Point cloud downsampling technology; through point selection strategy, all points in each voxel are selected using certain rules (such as average value, center point, etc.) to select a representative point; after processing by the above downsampling algorithm, a significantly reduced point cloud data set is output while maintaining the overall structure and characteristics of the original point cloud. Ultimately, the data volume is simplified without losing key features, achieving the goal of improving efficiency and saving edge computing power.

[0084] Reference Figure 4 The artificial intelligence of this application is through artificial intelligence training algorithms and software, artificial intelligence annotation software (manual annotation and intelligent annotation), artificial intelligence picture training software, and artificial intelligence model data preprocessing debugging software. By collecting a large number of 2D pictures of the application objects of this application in various situations, and using this artificial intelligence annotation software for manual or intelligent screening, annotation, and combination, and using this picture training software and artificial intelligence model parameter debugging software for 2D image training and parameter allocation, until a large artificial intelligence recognition model required for this application is obtained, the artificial intelligence visual recognition function of this application is realized;

[0085] After obtaining a high-quality artificial intelligence model through the above-mentioned artificial intelligence-related software tools developed by this application, the trained instance segmentation model is converted into the ONNX format and loaded using the ONNX Runtime GPU module to realize model deployment; secondly, necessary preprocessing operations are performed on the input image, such as scaling, normalization, etc., to meet the model requirements and complete the preprocessing of the model data; the preprocessed image is sent to the model, forward propagation is performed, and the prediction results are obtained to realize model reasoning; after the reasoning is completed, it is necessary to add a post-processing algorithm to post-process the data, filter low-quality prediction boxes by setting a confidence threshold, and apply the non-maximum suppression (NMS) algorithm to eliminate redundant detection boxes; the final output includes the bounding box, confidence score and classification label of each detected object, realizes instance segmentation, and adapts to monomer recognition and segmentation in complex backgrounds.

[0086] In the artificial intelligence implementation process of this application, the implementation of artificial intelligence is a professional, innovative, complex, and long-term work. This work requires the continuous optimization of the above-mentioned artificial intelligence training algorithms and software, artificial intelligence annotation software (manual annotation and intelligent annotation), artificial intelligence image training software, artificial intelligence model data preprocessing debugging software, and the acquisition of advanced, innovative, efficient, accurate, stable, and flexible artificial intelligence model creation tools. After repeated efforts in object image collection, manual screening of images, manual annotation, intelligent annotation, manual re-inspection, original annotation image screening, artificial intelligence training, model screening, model parameter adjustment, artificial intelligence model screening, artificial intelligence model merging, etc., practice, debugging, and testing efforts, a high-quality intelligent model is finally obtained, and finally a large artificial intelligence model in a subdivided field with ultra-high recognition ability, ultra-high compatibility, ultra-high accuracy, ultra-high precision, and ultra-high efficiency is obtained.

[0087] The implementation principle of the intelligent robot control method for palletizing in the embodiment of the present application is as follows: by calling the camera driver provided by the 2D / 3D industrial camera manufacturer, integration and optimization are carried out to realize the adaptation, communication, control, data extraction and data optimization operations of the 2D / 3D industrial camera, and 2D vision and AI intelligent recognition are used to realize accurate recognition of the working object in a complex environment; 3D vision and 2D / 3D synchronization technology are used to realize the information collection of the appearance, shape, category, size, coordinates and posture of the working object, and the point cloud in the object space is obtained based on the laser triangulation algorithm or the time of flight algorithm, and the point cloud is obtained through artificial The intelligent visual recognition capability cuts and separates the point cloud, and finally obtains the actual size, shape, posture, and spatial coordinates of the object required by this application through algorithm processing of the 3D point cloud. Then, the depalletizing and stacking planning calculation of the corresponding object is performed and converted into the execution trajectory of the robot, thereby improving the vision and spatial perception of the intelligent robot, and then facilitating the determination of the execution object, execution area, and execution trajectory according to the robot's vision and spatial perception. Finally, the robot performs depalletizing and stacking operations according to the execution trajectory, so as to achieve intelligence, high flexibility, and adaptability to the depalletizing and stacking of items of different types and specifications, greatly increasing the scope of application of the depalletizing and stacking robot.

[0088] Figure 2 FIG. 1 is a flow chart of an intelligent robot control method for depalletizing in one embodiment. It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 2At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0089] Based on the same technical concept, Figure 5 The embodiment of the present application also provides an intelligent robot control device for depalletizing, which adopts the following technical solution. The device includes:

[0090] The initialization module 510 is used to initialize the intelligent robot and its accessory devices through the driver integrated by the intelligent robot system and obtain an initial parameter file;

[0091] Interaction module 520, used to receive a mode selection instruction from an administrator terminal and determine an execution plan according to the mode execution instruction;

[0092] The imaging module 530 is used to send continuous imaging instructions to the 3D visual imaging device to obtain the image of the target to be measured;

[0093] The recognition module 540 is used to recognize the target image to be measured, mark the operation area in the target image to be measured, and generate a marking frame;

[0094] The marking module 550 is used to automatically locate the center position of the marking frame according to a preset graphic center recognition algorithm;

[0095] The execution module 560 is used to control the intelligent robot to grab the target object and perform the depalletizing and stacking work according to the execution plan, the marking box, the target image to be measured and the center position.

[0096] In some embodiments, the initial module 510 is specifically used to perform initial parameter debugging on the robot assembly through the robot and supporting equipment programs integrated in the intelligent robot system, including: initializing and monitoring the robot state, resetting the robot assembly to the initial position, initializing and monitoring the visual gimbal state, resetting the gimbal to the initial position, and resetting each device to the initial state;

[0097] Calibrate and synchronize the visual coordinate system of the 3D visual imaging device with the coordinates of the robot or the end of the robot suction fixture according to the preset machine vision calibration algorithm;

[0098] The intelligent robot system reads the preset executable solution file list, automatically loads the current executable solution task, obtains parameter configuration loading, and calls out pre-stored operation parameters, covering cargo type, pallet specifications, destacking target location, and destacking sequence rules, and determines the task operation logic based on the parameters.

[0099] In some embodiments, the operation area includes: an identification area, a safety area, and an arrival area;

[0100] The recognition module 540 is also used to correct the deviation of the recognition object space coordinate system caused by the deflection of the viewing angle through 2D / 3D image synchronization and fusion technology;

[0101] Through the human-computer interaction technology of image annotation and the computing technology of image annotation and spatial data synchronization, the safety zone, identification zone and in-place zone corresponding to the area where the identified object is located are divided;

[0102] With the help of laser sensors, the status of incoming and outgoing materials can be identified and monitored;

[0103] Through image filtering preprocessing, region of interest segmentation, mean operation, depth threshold limitation, and image area data comparison, it is determined whether the identified object is completely within the identification area.

[0104] In some embodiments, the recognition module 540 is further used to fuse the 2D color image with the 3D depth data, calibrate the acquired camera internal and external parameters through a coordinate system conversion algorithm, and convert the coordinate system of the depth camera to the color camera coordinate system;

[0105] The depth image is reprojected through the image reprojection algorithm to ensure that each pixel can be accurately mapped to the corresponding position of the RGB image;

[0106] Through data alignment and synchronization, the region of interest is cropped from the RGB image, and it is ensured that the region of interest corresponds to the corresponding point in the depth image one by one;

[0107] Through point cloud generation and posture calculation, the selected depth image data is converted into point cloud format, and then the position and posture of the target in three-dimensional space are calculated.

[0108] In some embodiments, the identification module 540 is further used to select a cutting axis defined along the X-axis, the Y-axis or the Z-axis for cutting according to the requirements;

[0109] Set the cutting range according to the minimum and maximum values ​​in the specified axis to define the cutting limit;

[0110] Automatically filter out qualified point cloud data according to the specified axis and cutting range;

[0111] Through point cloud clustering algorithm, spatial distance calculation algorithm, cluster formation algorithm and object extraction algorithm, the points in the point cloud data are grouped into multiple clusters according to certain similarity standards.

[0112] In some embodiments, the identification module 540 is further used to pre-set a distance threshold through a threshold definition procedure for determining whether the points belong to the same plane;

[0113] Through random sampling and plane equation calculation: randomly select three points to construct the initial plane, and calculate the distance from the point to the initial plane;

[0114] Project all points onto the calculated plane and measure the actual distance of each point to the initial plane;

[0115] Through the internal point determination and iterative optimization algorithm, the points whose distance from the screening point to the initial plane meets the preset threshold are set as internal points;

[0116] Through continuous iterative optimization, the plane with the maximum number of internal points is found;

[0117] Through covariance matrix calculation and feature analysis, the covariance matrix of the point cloud data is calculated, and the eigenvalue decomposition is performed to determine the main direction of the point cloud;

[0118] Then, a new coordinate system is constructed based on the eigenvectors through the coordinate system transformation algorithm to ensure that the new coordinate axis is consistent with the main direction of the point cloud;

[0119] By calculating the boundary value in the new coordinate system, the maximum and minimum values ​​of the point cloud along each axis are solved respectively;

[0120] The minimum volume cube of the above boundary value is determined by bounding box calculation and inverse transformation algorithm, and converted back to the original coordinate system to complete the calculation of the minimum oriented bounding box.

[0121] The embodiment of the present application also discloses a control device.

[0122] Specifically, the control device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned intelligent robot control method for depalletizing.

[0123] The embodiment of the present application also discloses a computer-readable storage medium.

[0124] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute the above-mentioned intelligent robot control method for depalletizing. The computer-readable storage medium includes, for example, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0125] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A control method for an intelligent robot for palletizing, characterized in that: include: Initialize the intelligent robot and its accessory devices through the driver integrated in the intelligent robot system to obtain the initial parameter file; receiving a mode selection instruction from an administrator terminal, and determining an execution plan according to the mode selection instruction; Sending continuous imaging instructions to the 3D visual imaging device to obtain the image of the target to be measured; Identify the target image to be measured, mark the operation area in the target image to be measured, and generate a marking frame; Automatically locate the center position of the annotation frame according to a preset graphic center recognition algorithm; According to the execution scheme, the marking frame, the target image to be measured and the center position, the intelligent robot is controlled to grasp the target object and perform the depalletizing and stacking work; Wherein, the operation area includes: an identification area, a safety area and an arrival area; After identifying the target image to be measured, marking the operation area in the target image to be measured, and generating a marking frame, the method further includes: Through 2D / 3D image synchronization and fusion technology, the deviation of the spatial coordinate system of the identified object caused by the deflection of the viewing angle is corrected; Through the human-computer interaction technology of image annotation and the computing technology of image annotation and spatial data synchronization, the safety zone, identification zone and in-place zone corresponding to the area where the identified object is located are divided; With the help of laser sensors, the status of incoming and outgoing materials can be identified and monitored; Through image filtering preprocessing, region of interest segmentation, mean operation, depth threshold limitation, and image area data comparison, it is determined whether the identified object is completely within the identification area.

2. The intelligent robot control method for depalletizing according to claim 1, characterized in that: The intelligent robot and its accessory devices are initialized by the driver integrated by the intelligent robot system to obtain the initial parameter file, including: Perform initial parameter debugging on the robot assembly through the robot and supporting equipment programs integrated in the intelligent robot system, including: initializing and monitoring the robot status, resetting the robot assembly to the initial position, initializing and monitoring the visual gimbal status, resetting the gimbal to the initial position, and resetting each device to the initial state; Calibrate and synchronize the visual coordinate system of the 3D visual imaging device with the coordinates of the robot or the end of the robot suction fixture according to the preset machine vision calibration algorithm; The intelligent robot system reads the preset executable solution file list, automatically loads the current executable solution task, obtains parameter configuration loading, and calls out pre-stored operation parameters, covering cargo type, pallet specifications, destacking target location, and destacking sequence rules, and determines the task operation logic based on the parameters.

3. The intelligent robot control method for depalletizing according to claim 1, characterized in that: After identifying the target image to be measured, marking the operation area in the target image to be measured, and generating a marking frame, the method further includes: The 2D color image is fused with the 3D depth data, the camera internal and external parameters are calibrated by the coordinate system conversion algorithm, and the coordinate system of the depth camera is converted to the color camera coordinate system; The depth image is reprojected through the image reprojection algorithm to ensure that each pixel can be accurately mapped to the corresponding position of the RGB image; Through data alignment and synchronization, the region of interest is cropped from the RGB image, and it is ensured that the region of interest corresponds to the corresponding point in the depth image one by one; Through point cloud generation and posture calculation, the selected depth image is converted into a point cloud format, and then the position and posture of the target in the three-dimensional space are calculated.

4. The intelligent robot control method for depalletizing according to claim 3 is characterized in that: After the selected depth image data is converted into a point cloud format through point cloud generation and posture calculation, and then the position and posture of the target in the three-dimensional space are calculated, the method further includes: Select the cutting axis along the X-axis, Y-axis or Z-axis to cut according to your needs; Set the cutting range according to the minimum and maximum values ​​in the specified axis to define the cutting limit; Automatically filter out qualified point cloud data according to the specified axis and the cutting range; The points in the point cloud data are grouped into a plurality of clusters according to certain similarity standards through a point cloud clustering algorithm, a spatial distance calculation algorithm, a cluster formation algorithm, and an object extraction algorithm.

5. The intelligent robot control method for depalletizing according to claim 4, characterized in that: After the points in the point cloud data are grouped into a plurality of clusters according to a certain similarity standard by using the point cloud clustering algorithm, the spatial distance calculation algorithm, the cluster formation algorithm, and the object extraction algorithm, the method further includes: Through the threshold definition procedure, a distance threshold is pre-set to determine whether the points belong to the same plane; By random sampling and plane equation calculation: randomly select three points to construct an initial plane, and calculate the distance from the point to the initial plane; Project all points onto the calculated plane and measure the actual distance of each point to the initial plane; Through the interior point determination and iterative optimization algorithm, the points whose distances from the screening points to the initial plane meet the requirements below a preset threshold are set as interior points; Through continuous iterative optimization, the plane with the maximum number of internal points is found; Through covariance matrix calculation and feature analysis, the covariance matrix of the point cloud data is calculated, and the eigenvalue decomposition is performed to determine the main direction of the point cloud; Then, a new coordinate system is constructed based on the eigenvectors through the coordinate system transformation algorithm to ensure that the new coordinate axis is consistent with the main direction of the point cloud; By calculating the boundary value in the new coordinate system, the maximum and minimum values ​​of the point cloud along each axis are solved respectively; The minimum volume cube of the above boundary value is determined by bounding box calculation and inverse transformation algorithm, and converted back to the original coordinate system to complete the calculation of the minimum oriented bounding box.

6. An intelligent robot control device for depalletizing, characterized in that: The device comprises: The initial module is used to initialize the intelligent robot and its accessory devices through the driver integrated in the intelligent robot system and obtain the initial parameter file; An interactive module, used to receive a mode selection instruction from an administrator terminal and determine an execution plan according to the mode selection instruction; An image module is used to send continuous image acquisition instructions to the 3D visual imaging device to obtain the image of the target to be measured; A recognition module, used for recognizing the target image to be measured, marking the operation area in the target image to be measured, and generating a marking frame; A marking module, used for automatically locating the center position of the marking frame according to a preset graphic center recognition algorithm; An execution module, used to control the intelligent robot to grab the target object and perform depalletizing according to the execution scheme, the marking frame, the target image to be measured and the center position; Wherein, the operation area includes: an identification area, a safety area and an arrival area; The recognition module is also used to correct the deviation of the spatial coordinate system of the recognition object caused by the deflection of the viewing angle through 2D / 3D image synchronization and fusion technology; Through the human-computer interaction technology of image annotation and the computing technology of image annotation and spatial data synchronization, the safety zone, identification zone and in-place zone corresponding to the area where the identified object is located are divided; With the help of laser sensors, the status of incoming and outgoing materials can be identified and monitored; Through image filtering preprocessing, region of interest segmentation, mean operation, depth threshold limitation, and image area data comparison, it is determined whether the identified object is completely within the identification area.

7. A control device, characterized in that: The device comprises: The method comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 5.

Images