AI Vision-Based Intelligent Palletizing Robot Safety Management System and Method

Abstract

The present invention relates to the technical field of intelligent robots, and specifically to a safety management system and method for an intelligent palletizing robot based on AI vision. The present invention obtains multi-modal data in the palletizing operation area to construct a three-dimensional model; uses deep learning algorithms for target detection and recognition; dynamically divides the safety area of the operation area according to the target information; constructs a mobile path planning strategy to control the safe operation of the robot; at the same time, detects the state of the palletized goods through an AI vision model, automatically alarms and takes safety measures when abnormalities are found. The present invention can improve the intelligent level and safety performance of the palletizing robot and has broad application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and particularly to a safety management system and method for an intelligent palletizing robot based on AI vision. Background Art

[0002] Palletizing robots, with their advantages of high efficiency, flexibility, and low cost, have been widely used in fields such as warehousing, logistics, and manufacturing, greatly improving the automation level of cargo handling and stacking. However, due to the complexity, dynamics, and uncertainty of the palletizing operation environment, traditional palletizing robots still have many deficiencies in terms of safety and intelligence, and it is difficult to meet the increasingly high operation requirements.

[0003] Traditional palletizing robots mainly rely on pre-set operation paths and fixed safety areas, lacking the ability of real-time perception and dynamic adaptation to the environment and target objects. They cannot accurately obtain and understand the real-time state of the operation site, and it is difficult to make timely responses and flexible adjustments to environmental changes and unexpected situations. When there are interference factors such as personnel intrusion, obstacles, and changes in the position of goods in the operation environment, the robot often cannot make correct decisions and countermeasures in a timely manner, and it is extremely easy to occur safety accidents such as collisions, pinches, and misoperations, bringing serious safety hazards to on-site operators and equipment facilities.

[0004] The state monitoring and quality inspection of traditional palletized goods also mostly rely on manual inspections and interventions, lacking intelligent solutions. Due to the variety of palletized goods, diverse shapes, and irregular stacking, it is difficult to effectively monitor them through conventional automated equipment. Operators need to frequently conduct visual inspections on the pallet stacks and manually identify and handle abnormal situations such as tilting, scattering, and damage. This inefficient and time-consuming manual supervision mode not only has a high labor intensity and high cost, but is also easily affected by human factors, resulting in problems such as missed inspections and misjudgments, bringing great uncertainty risks to the safety of goods and the quality of palletizing.

[0005] In view of this, the present invention proposes a safety management system and method for an intelligent palletizing robot based on AI vision. Summary of the Invention

[0006] To achieve the above object, the present invention provides a safety management system and method for an intelligent palletizing robot based on AI vision, and the specific technical solutions are as follows:

[0007] The safety management method for an intelligent palletizing robot based on AI vision includes:

[0008] Obtain multi-modal data of the palletizing operation area, perform three-dimensional fusion perception on the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration;

[0009] Build a deep learning-based object detection algorithm to perform real-time detection on 3D models and identify multiple types of objects;

[0010] Based on the object categories and location information identified in the palletizing operation area, dynamically divide the safety area of the palletizing operation area;

[0011] According to the results of the dynamic safety area division, construct a mobile path planning strategy to control the palletizing robot to perform mobile path planning in the palletizing work area;

[0012] When the palletizing robot obtains the status of the palletized goods in the palletizing operation area, detect the status of the palletized goods through the AI vision model. When detecting goods with abnormal palletizing, the palletizing robot automatically triggers an alarm and takes safety measures.

[0013] Preferably, the multi-modal data includes image data of the palletizing operation area captured by a depth camera and lidar point cloud data of the palletizing operation area scanned by a lidar; fix the depth camera and the lidar on the top of the palletizing robot respectively, and the depth camera and the lidar rotate on the top of the palletizing robot to obtain image data and lidar point cloud data within a 360° range of the palletizing robot;

[0014] Adopt a multi-view point cloud stitching algorithm to register and stitch the point cloud data collected by the lidar at different times, and synchronize the image data and the point cloud data in time;

[0015] Perform downsampling and voxel filtering on the fused point cloud to remove noise points and outliers in the fused point cloud;

[0016] The downsampling includes performing downsampling processing on the stitched point cloud, and using a voxelized grid method to divide the point cloud into uniform small voxels;

[0017] The voxel filtering includes using a voxel filtering algorithm to filter the point cloud, and based on the set voxel size, retain the centroid points within each voxel to obtain uniform point cloud data;

[0018] Register the color image obtained by the depth camera with the point cloud, and assign corresponding RGB color information to each point cloud point;

[0019] Use the greedy triangulation algorithm to perform surface reconstruction on the fused colored point cloud, and generate a triangular mesh according to the spatial distribution and normal vector information of the point cloud;

[0020] Perform simplification and smoothing processing on the reconstructed triangular mesh to reduce the number of mesh patches and obtain a 3D model of the palletizing operation area.

[0021] Preferably, the YOLOv5 algorithm is used to perform real-time object detection on the three-dimensional model of the palletizing operation area, and the object detection results are fused with the lidar point cloud to obtain the three-dimensional model coordinate positions of the objects through nearest neighbor matching; the nearest neighbor matching finds the best match by calculating the distance between the detected objects and the point cloud set;

[0022] The Deep SORT algorithm is used to achieve multi-object tracking; the Deep SORT algorithm uses the Hungarian algorithm to solve the object association problem and optimizes the association matrix by maximizing the similarity between the tracking objects and the detected objects;

[0023] After identifying multiple object types based on the three-dimensional model, different risk impact coefficients are assigned to each type of object for dynamic safety area division.

[0024] Preferably, the palletizing operation area is divided into uniform grids to form a grid map; in the grid map, the risk value of each grid is determined by the positions and influence ranges of the objects around the grid, and the grid risk value calculation formula is:

[0025]

[0026] where R(x, y) represents the risk value at the position (x, y), i is the index of the target object; (x i , y i ) are the position coordinates of the i-th target object; α i is the risk impact coefficient of the i-th target, indicating the degree of danger of the 9th target; σ i represents the parameter indicating the influence range size of the i-th target;

[0027] In the grid map, the grid map is divided into a safe area and a dangerous area according to the safety threshold. The safe area includes a low-risk area, and the dangerous area includes a medium-risk area and a high-risk area;

[0028] Set the grid map safety thresholds TR1 and TR2, and TR2 > TR1; when R(x, y) < TR1, the grid area is divided into a low-risk area; when TR1 ≤ R(x, y) ≤ TR2, the grid area is divided into a medium-risk area; when R(x, y) > TR2, the grid area is divided into a high-risk area;

[0029] Set a fixed time interval Δt. Whenever Δt time has passed, recalculate the risk values of the grid areas and perform safety area division according to the recalculated grid risk values.

[0030] Preferably, based on the divided safe areas and dangerous areas, an optimized A* algorithm is used to plan the hierarchical movement paths of the palletizing robot in the palletizing operation area; the hierarchical movement path planning includes the first-layer movement path planning, the second-layer movement path planning, and the third-layer movement path planning;

[0031] The first-layer movement path planning includes using the A* algorithm to search for the optimal path within the low-risk area, setting the starting point and the target point as the inputs of the A* algorithm, and taking the low-risk area as the passable area;

[0032] The second-layer movement path planning includes, if a feasible path cannot be found within the low-risk area, adding the medium-risk area to the passable area and using the A* algorithm to search for the path again;

[0033] The third-layer movement path planning includes, if a feasible path cannot be found within the low- and medium-risk areas, adding the high-risk area to the passable area and using the A* algorithm to search for the path again;

[0034] If a feasible path cannot be found within all the risk areas, a path planning failure alarm is triggered;

[0035] The optimized A* algorithm includes locally optimizing the initial path obtained by the A* algorithm and smoothing the path using Bezier curves. The optimization objective is:

[0036]

[0037] where (x a , y a ) is the coordinate of the ath sampling point on the path, is the coordinate of the hth control point, m and n are the number of sampling points and the number of control points respectively, and λ is the smoothness weight coefficient;

[0038] Whenever after a time interval of Δt, when the grid risk value of the grid map changes, the movement path of the palletizing robot is re-planned by the optimized A* algorithm.

[0039] Preferably, based on the image data obtained by a depth camera, the goods to be palletized are identified. When the goods to be palletized are recognized, the point cloud data obtained when the lidar scans the palletized goods is simultaneously acquired. The acquired point cloud data is marked as the goods point cloud, and the normal vector and curvature of the goods point cloud are calculated to obtain the normal vector and the curvature c p value of each point in the goods point cloud;

[0040] The normal vector and the curvature reference value c ref of the goods under normal conditions are set, as well as the abnormal curvature upper limit c max ;

[0041] Calculate the anomaly index A(p) for each point in the point cloud of the goods:

[0042]

[0043] where w n is the weight coefficient of the deviation of the normal vector, and w c is the weight coefficient of the high curvature;

[0044] Set the anomaly threshold T a , which is used to determine whether a single point is abnormal; when the anomaly index A(p) of the point is greater than T a , it is determined that the point is in an abnormal state. When the proportion of the number of points where A(p) is greater than T a exceeds the percentage threshold TPa, it is determined that the state of the goods stacking is abnormal.

[0045] Preferably, according to the distribution and clustering of the abnormal points in the point cloud of the goods, determine the abnormal type of the goods stacking. The abnormal type of the goods stacking includes tilting and scattering;

[0046] Downsample the point cloud of the goods to obtain uniformly distributed core points; calculate the covariance matrix C of the core points:

[0047]

[0048] where h is the number of core points, and p z is the coordinate of the z-th core point; is the central coordinate of the core points;

[0049] Perform eigenvalue decomposition on C: C = UΛU T ; obtain the eigenvectors where U is the eigenvector matrix, Λ is the eigenvalue matrix, and U T is the transpose of the eigenvector matrix;

[0050] Take the eigenvector as the main direction of the stacked goods, representing the pose of the stacked goods; calculate the central coordinate of the core points of the stacked goods as the position of the goods;

[0051] According to the pose and abnormal type of the goods, search for the corresponding correction strategy in the predefined strategy library; if a matching correction strategy is found, enter the automatic correction process; otherwise, trigger an alarm; the correction strategy includes translating and rotating the goods along the direction to adjust the pose of the abnormally stacked goods; calculate the translation vector of the abnormally stacked goods according to and the target position; convert the correction strategy into the motion control instruction of the stacking robot;

[0052] After the correction is completed, scan the goods and re-acquire the point cloud data of the goods; and detect the status and pose of the palletized goods after correction; if the abnormal status of the palletized goods is eliminated, the palletizing robot continues to operate normally within the palletizing work area; otherwise, an alarm is triggered.

[0053] An intelligent palletizing robot safety management system based on AI vision, which is used to implement an intelligent palletizing robot safety management method based on AI vision, including: a data acquisition module, a target detection module, a region division module, a path planning module, and an anomaly correction module;

[0054] The data acquisition module is used to acquire multi-modal data of the palletizing operation area, perform three-dimensional fusion perception on the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration;

[0055] The target detection module is used to construct a target detection algorithm based on deep learning, and perform real-time detection and recognition of multiple types of targets on the three-dimensional model;

[0056] The region division module is used to dynamically divide the safety region of the palletizing operation area based on the target category and location information identified in the palletizing operation area;

[0057] The path planning module is used to construct a mobile path planning strategy according to the dynamic safety region division result, and control the palletizing robot to perform mobile path planning within the palletizing work area;

[0058] The anomaly correction module is used to detect the status of the palletized goods in the palletizing operation area by the AI vision model when the palletizing robot obtains the status of the palletized goods in the palletizing operation area. When detecting palletized goods with anomalies, the palletizing robot automatically takes safety measures and triggers an alarm.

[0059] An electronic device includes: a processor and a memory, wherein, a computer program that can be called by the processor is stored in the memory; the processor executes the above-mentioned intelligent palletizing robot safety management method based on AI vision by calling the computer program stored in the memory.

[0060] A storage medium stores instructions, and when the instructions run on a computer, the computer is made to execute the above-mentioned intelligent palletizing robot safety management method based on AI vision.

[0061] Advantages of the present invention: The present invention constructs a three-dimensional model of the palletizing operation area, providing comprehensive and accurate environmental information for subsequent target detection, risk assessment, and path planning.

[0062] The present invention realizes real-time detection and recognition of multiple types of targets through deep learning algorithms, improving the robustness and generalization ability of the perception system.

[0063] The present invention dynamically divides safety areas, flexibly adjusts the working space of the robot according to the real-time changes at the operation site, and improves the operation efficiency while ensuring safety.

[0064] The present invention combines the results of safety area division to plan the moving path, selects the movement route with the lowest risk and the most energy saving, and improves the efficiency and safety of the palletizing operation.

[0065] The present invention uses an AI vision model to realize the intelligent detection of the state of palletized goods, automatically identifies abnormal situations and triggers an alarm, and takes safety measures in time to prevent palletizing operation accidents. Description of the Drawings

[0066] Figure 1 is a flowchart of the safety management method for an intelligent palletizing robot based on AI vision provided by the present invention;

[0067] Figure 2 is a structural diagram of the safety management system for an intelligent palletizing robot based on AI vision provided by the present invention;

[0068] Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention;

[0069] Figure 4 is a schematic structural diagram of a storage medium provided by an embodiment of the present invention. Detailed Embodiments

[0070] To better understand the present invention, more detailed descriptions of various aspects of the present invention will be made with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present invention, and do not limit the scope of the present invention in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0071] In the drawings, for ease of illustration, the sizes, dimensions, and shapes of the elements have been slightly adjusted. The drawings are only examples and are not drawn to an exact scale. As used herein, the terms "substantially", "about" and similar terms are used as terms indicating approximation, rather than terms indicating degree, and are intended to illustrate the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art. Additionally, in the present invention, the order of description of each step process does not necessarily represent the order in which these processes occur in actual operation, unless otherwise clearly defined or derivable from the context.

[0072] It should also be understood that expressions such as "comprising", "including", "having", "containing" and / or "including" are open rather than closed expressions in this specification, which means that there are the stated features, elements and / or components, but do not exclude the existence of one or more other features, elements, components and / or their combinations. In addition, when an expression such as "at least one of..." appears after a list of listed features, it modifies the entire list of features, rather than just individual elements in the list. In addition, when describing embodiments of the present invention, the use of "may" means "one or more embodiments of the present invention". And the term "exemplary" is intended to refer to an example or illustration.

[0073] Unless otherwise defined, all terms used herein (including engineering terms and technical terms) have the same meaning as commonly understood by those of ordinary skill in the art to which this invention belongs. It should also be understood that, unless clearly stated in the present invention, words defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense.

[0074] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.

[0075] Example 1

[0076] Refer to Figure 1 , for the first embodiment of the present invention, a safety management method for an intelligent palletizing robot based on AI vision is provided.

[0077] S1: Obtain multi-modal data of the palletizing operation area, perform three-dimensional fusion perception on the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration.

[0078] The multi-modal data includes image data of the palletizing operation area captured by a depth camera and laser point cloud data of the palletizing operation area scanned by a lidar; the depth camera and the lidar are respectively fixed on the top of the palletizing robot, and the depth camera and the lidar rotate on the top of the palletizing robot to obtain image data and laser point cloud data within a 360° range of the palletizing robot.

[0079] Use the Point Cloud Library (PCL) as the basic library for point cloud processing, adopt the multi-viewpoint cloud stitching algorithm provided by PCL to register and stitch the point cloud data collected by the lidar at different times, and perform time synchronization on the image data and the point cloud data.

[0080] Use the message_filters package in ROS to synchronize the image data of the palletizing operation area captured by the depth camera and the lidar point cloud data of the palletizing operation area scanned by the lidar in terms of time;

[0081] Perform downsampling and voxel filtering on the fused point cloud to remove the noise points and outliers in the fused point cloud.

[0082] The downsampling includes performing downsampling processing on the spliced point cloud, dividing the point cloud into uniform small voxels using the voxel grid method, and only retaining one point in each voxel to reduce the data volume of the point cloud.

[0083] The voxel filtering includes filtering the point cloud using the voxel filtering algorithm, retaining the centroid points in each voxel based on the set voxel size, and obtaining uniform point cloud data.

[0084] Register the color image obtained by the depth camera with the point cloud to assign corresponding RGB color information to each point cloud point.

[0085] Use the greedy triangulation algorithm to perform surface reconstruction on the fused color point cloud, and generate a triangular mesh according to the spatial distribution and normal vector information of the point cloud.

[0086] Perform simplification and smoothing processing on the reconstructed triangular mesh, reduce the number of mesh patches, improve the smoothness of the surface, and obtain a three-dimensional model of the palletizing operation area.

[0087] Through the multi-sensor fusion perception technology, the palletizing robot can comprehensively and accurately obtain the environmental information of the operation area, construct a three-dimensional scene model, and provide a reliable data basis for subsequent target detection, risk assessment, and path planning.

[0088] S2: Construct a deep learning-based target detection algorithm to perform real-time detection on the three-dimensional model and identify multiple types of targets.

[0089] Use the YOLOv5 algorithm to perform real-time target detection on the three-dimensional model of the palletizing operation area. The YOLOv5 algorithm realizes the detection of different targets by predicting the confidence of each target category.

[0090] Fuse the target detection results with the lidar point cloud to obtain the three-dimensional model coordinate position of the target through nearest neighbor matching; Nearest neighbor matching finds the best match by calculating the distance between the detected target and the point cloud set.

[0091] Use the Deep SORT algorithm to achieve multi-target tracking; The Deep SORT algorithm uses the Hungarian algorithm to solve the target association problem and optimizes the association matrix by maximizing the similarity between the tracking target and the detected target.

[0092] After identifying multiple target types based on the 3D model, different risk impact coefficients are assigned to targets of each type for dynamic safety area division.

[0093] Using deep learning algorithms for object detection and tracking, the palletizing robot can identify and locate key targets such as personnel, equipment, and goods in the working area in real time, grasp their dynamic changes, and provide a basis for formulating safety strategies.

[0094] S3: Based on the target categories and location information identified in the palletizing working area, perform dynamic safety area division on the palletizing working area.

[0095] Divide the palletizing working area into uniform grids, that is, divide the palletizing working area into a grid map; in the grid gradient grid map, the risk value of each grid is determined by the positions and influence ranges of the surrounding targets. The grid risk value calculation formula is:

[0096]

[0097] Among them, R(x,y) represents the risk value at the position (x,y), i is the index of the target object; (x i ,y i is the position coordinate of the i-th target object; α i is the risk impact coefficient of the i-th target, indicating the degree of danger of this target; σ i represents the parameter of the influence range size of the i-th target.

[0098] After identifying multiple target types in the palletizing working area, corresponding impact coefficients need to be assigned according to the risk levels of different targets. The risk impact coefficients can be obtained through expert experience or data statistics.

[0099] Exemplarily, for different types of targets such as personnel, vehicles, and goods, risk coefficients can be assigned from high to low respectively, such as λ1 = 1.0, λ2 = 0.8, λ3 = 0.6.

[0100] In addition, the parameter of the risk influence range size also needs to be determined according to the attributes of the target such as size and moving speed; the influence range parameter σ i is proportional to the average size and maximum speed of the target, and can be estimated by the following formula: σ i = k·max(r i ,v i ); among them, r i and v i are respectively the average radius and maximum speed of the i-th target, and k is the proportionality coefficient of the influence range, which can be set according to the safety margin, usually taking a value between 1.5 and 2.0.

[0101] When calculating the risk values of each grid, the risk superposition effect of multiple targets can be considered. When a grid is affected by multiple targets simultaneously, its risk value can be taken as the maximum value of the weighted sum of the influence factors.

[0102] In the grid map, the grid map is divided into a safe area and a dangerous area according to the safety threshold. The safe area includes a low-risk area, and the dangerous area includes a medium-risk area and a high-risk area.

[0103] Set the safety thresholds TR1 and TR2 of the grid map, and TR2 > TR1; when R(x, y) < TR1, the grid area is divided into a low-risk area; when TR1 ≤ R(x, y) ≤ TR2, the grid area is divided into a medium-risk area; when R(x, y) > TR2, the grid area is divided into a high-risk area.

[0104] When dynamically dividing the safe area of the palletizing operation area, it is first necessary to select appropriate thresholds TR1 and TR2 according to the task requirements and scene complexity; the setting of the safety threshold needs to be balanced between the flexibility of the robot's movement and safety assurance.

[0105] It should be noted that the higher the threshold, the larger the movement space of the robot, but the higher the collision risk it faces. On the contrary, the lower the threshold, the more restricted the movement of the robot, but the safety of itself and the environment can be better guaranteed. Therefore, the selection of the threshold needs to be combined with the specific application scenario and the optimal value determined through repeated experiments and evaluations.

[0106] Exemplarily, statistical analysis is performed on the distribution of the grid risk values in the scenario to obtain its numerical range and variation law, and then the threshold is set to a certain percentile, such as the 20th percentile and the 80th percentile, so as to be neither too conservative nor too radical. The threshold can also be adjusted in real time according to the change of the task progress or risk situation. For example, at the initial stage of the palletizing operation, since the goods are relatively neat and the scene risk is low, the threshold can be appropriately relaxed to allow the robot to move and operate more freely. At the later stage of the palletizing operation, since the pallet stack is getting higher and the environment is getting more crowded, the threshold can be correspondingly reduced to make the robot's movement more cautious and restrained. Another example is that when detecting that a person or other important target enters the operation area, the threshold can be immediately increased to limit the movement range of the robot until the target leaves.

[0107] Set a fixed time interval Δt. Whenever Δt time has passed, recalculate the risk value of the grid area, and perform the safe area division according to the recalculated grid risk value.

[0108] Based on the grid map and regional risk assessment, the palletizing robot can dynamically divide the safe area and dangerous area of the working area, quantify the risk levels of different areas, provide constraint conditions for the robot's motion decision-making, and effectively avoid potential safety hazards such as collisions and interferences.

[0109] S4: According to the results of the dynamic safety area division, construct a mobile path planning strategy to control the palletizing robot to perform mobile path planning within the palletizing working area.

[0110] Based on the divided safe area and dangerous area, use the optimized A* algorithm to perform hierarchical mobile path planning for the palletizing robot in the palletizing operation area; the hierarchical mobile path planning includes the first-layer mobile path planning, the second-layer mobile path planning, and the third-layer mobile path planning.

[0111] The first-layer mobile path planning includes using the A* algorithm to search for the optimal path within the low-risk area, setting the starting point and the target point as the input of the A* algorithm, and taking the low-risk area as the passable area.

[0112] The second-layer mobile path planning includes, if no feasible path can be found within the low-risk area, adding the medium-risk area to the passable area and using the A* algorithm to search for the path again;

[0113] The third-layer mobile path planning includes, if no feasible path can be found within the low- and medium-risk areas, adding the high-risk area to the passable area and using the A* algorithm to search for the path again.

[0114] If no feasible path can be found within all risk areas, trigger a path planning failure alarm.

[0115] The optimized A* algorithm includes locally optimizing the initial path obtained by the A* algorithm and smoothing the path using the Bezier curve. The optimization objective is:

[0116]

[0117] where, (x a ,y a ) is the coordinate of the ath sampling point on the path, is the coordinate of the hth control point, m and n are the number of sampling points and the number of control points respectively, and λ is the smoothness weight coefficient.

[0118] Whenever after Δt time, when the grid risk value of the grid map changes, re-plan the moving path of the palletizing robot through the optimized A* algorithm.

[0119] Using a hierarchical optimized A* algorithm for mobile path planning, the palletizing robot can autonomously search for the optimal feasible path based on the division result of the safety area, taking into account both motion efficiency and safety. At the same time, through the Bezier curve smoothing technology, the motion trajectory of the robot becomes more natural and smooth.

[0120] S5: When the palletizing robot obtains the state of the palletized goods in the palletizing operation area, it detects the state of the palletized goods through the AI vision model. When detecting goods with abnormal palletizing, the palletizing robot automatically triggers an alarm and takes safety measures.

[0121] Based on the depth camera to obtain image data, identify the palletized goods. When the palletized goods are identified, at the same time, obtain the point cloud data when the lidar scans the palletized goods, mark the obtained point cloud data as goods point cloud, and calculate the normal vector and curvature of the goods point cloud to obtain the normal vector of each point in the goods point cloud and curvature c p value.

[0122] According to prior knowledge, set the normal vector of the goods under normal conditions and the curvature reference value c ref , as well as the abnormal curvature upper limit c max .

[0123] Calculate the abnormal index A(p) of each point in the goods point cloud:

[0124]

[0125] where w n is the weight coefficient of the normal vector deviation, w n takes values in [0,1], the larger w n , the greater the contribution of the normal vector deviation to the abnormal index; w c is the weight coefficient of the high curvature; w c takes values in [0,1], the larger w c , the greater the contribution of the high curvature to the abnormal index.

[0126] Set the abnormal threshold T a , used to judge whether a single point is abnormal; when the abnormal index A(p) of the point is greater than T a , it is determined that the point is in an abnormal state. When the proportion of the number of points with A(p) greater than T a exceeds the percentage threshold TPa, it is determined that the state of the goods palletizing is abnormal.

[0127] According to the distribution and clustering of the abnormal points in the goods point cloud, determine the abnormal type of the goods palletizing. The abnormal type of the goods palletizing includes tilting and scattering.

[0128] Downsample the point cloud of the goods to obtain evenly distributed core points; calculate the covariance matrix C of the core points:

[0129]

[0130] where h is the number of core points, and p z is the coordinate of the z-th core point; is the central coordinate of the core points.

[0131] Perform eigenvalue decomposition on C: C = UΛU T ; obtain the eigenvectors where U is the eigenvector matrix, Λ is the eigenvalue matrix, and U T is the transpose of the eigenvector matrix.

[0132] Use the eigenvector as the main direction of the palletized goods to represent the pose of the palletized goods; calculate the central coordinate of the core points of the palletized goods as the position of the goods.

[0133] According to the pose and abnormal type of the goods, look up the corresponding correction strategy in the predefined strategy library; if a matching correction strategy is found, enter the automatic correction process; otherwise, trigger an alarm; the correction strategy includes translating and rotating the goods along the direction to adjust the pose of the abnormally palletized goods; calculate the translation vector of the abnormally palletized goods according to and the target position; convert the correction strategy into the motion control instructions of the palletizing robot.

[0134] After the correction is completed, rescan the goods to obtain the point cloud data; and detect the status and pose of the corrected palletized goods; if the abnormal status of the palletized goods is eliminated, the palletizing robot continues to operate normally in the palletizing work area; otherwise, trigger an alarm.

[0135] By introducing AI vision detection and abnormal diagnosis technologies, the palletizing robot can automatically monitor the status of goods palletizing, timely detect quality defects such as tilting and scattering, and perform automatic correction or alarm processing according to the predefined strategies, maximizing the stacking quality and safety of the goods.

[0136] Embodiment 2

[0137] Referring to Figure 2 , as the second embodiment of the present invention, an intelligent palletizing robot safety management system based on AI vision is provided.

[0138] The system includes: a data acquisition module, a target detection module, a region division module, a path planning module, and an abnormal correction module.

[0139] The data acquisition module is used to acquire multi-modal data of the palletizing operation area, perform three-dimensional fusion perception on the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration.

[0140] The target detection module is used to construct a target detection algorithm based on deep learning, and perform real-time detection and identification of multiple types of targets on the three-dimensional model.

[0141] The area division module is used to perform dynamic safety area division on the palletizing operation area based on the target categories and location information identified in the palletizing operation area.

[0142] The path planning module is used to construct a mobile path planning strategy according to the dynamic safety area division result, and control the palletizing robot to perform mobile path planning in the palletizing work area.

[0143] The anomaly correction module is used to detect the state of the palletized goods in the palletizing operation area by the AI vision model when the palletizing robot obtains the state of the palletized goods in the palletizing operation area. When detecting goods with palletizing anomalies, the palletizing robot automatically takes safety measures and triggers an alarm.

[0144] Embodiment 3

[0145] Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As Figure 3 shown, according to another aspect of the present invention, an electronic device 500 is also provided. The electronic device 500 may include one or more processors and one or more memories. Among them, computer-readable code is stored in the memory, and when the computer-readable code is run by one or more processors, it can execute the above-mentioned AI vision-based intelligent palletizing robot safety management method.

[0146] The method or system according to the embodiment of the present invention can also be implemented by means of Figure 3 the architecture of the electronic device shown.

[0147] As Figure 3 shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to the network, an input / output component 506, a hard disk 507, etc.

[0148] The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store the AI vision-based intelligent palletizing robot safety management method provided by the present invention.

[0149] An AI vision-based intelligent palletizing robot safety management method includes obtaining multi-modal data of the palletizing operation area, performing three-dimensional fusion perception on the palletizing operation area, and constructing a three-dimensional model of the palletizing operation area through data synchronization and registration; constructing a deep learning-based object detection algorithm to perform real-time detection and identify multiple types of objects on the three-dimensional model; dynamically dividing the safety area of the palletizing operation area based on the object categories and location information identified in the palletizing operation area; constructing a mobile path planning strategy according to the dynamic safety area division result, and controlling the palletizing robot to perform mobile path planning in the palletizing work area; when the palletizing robot obtains the state of the palletized goods in the palletizing operation area, detecting the state of the palletized goods through the AI vision model, and when detecting abnormal palletized goods, the palletizing robot automatically triggers an alarm and takes safety measures.

[0150] Furthermore, the electronic device 500 may also include a user interface 508. Of course, Figure 3 The architecture shown is only exemplary. When implementing different devices, one or more components in the Figure 3 shown electronic device may be omitted according to actual needs.

[0151] Embodiment 4

[0152] Figure 4 It is a schematic diagram of the storage medium structure provided by an embodiment of the present invention.

[0153] As Figure 4 shown, it is a storage medium 600 according to an embodiment of the present invention.

[0154] Computer-readable instructions are stored on the storage medium 600.

[0155] When the computer-readable instructions are run by a processor, the AI vision-based intelligent palletizing robot safety management method according to the embodiment of the present invention described with reference to the above drawings can be executed.

[0156] The storage medium 600 includes but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. Additionally, according to the embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs.

[0157] For example, the present invention provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be run by a processor to execute instructions corresponding to the method steps provided by the present invention. For example: acquiring multi-modal data of the palletizing operation area, performing three-dimensional fusion perception on the palletizing operation area, and constructing a three-dimensional model of the palletizing operation area through data synchronization and registration; constructing an object detection algorithm based on deep learning, performing real-time detection on the three-dimensional model and identifying multiple types of objects; based on the object categories and position information identified in the palletizing operation area, performing dynamic safety area division on the palletizing operation area; according to the dynamic safety area division result, constructing a mobile path planning strategy to control the palletizing robot to perform mobile path planning in the palletizing working area; when the palletizing robot acquires the state of the palletized goods in the palletizing operation area, detecting the state of the palletized goods through an AI vision model, and when detecting goods with abnormal palletizing, the palletizing robot automatically triggers an alarm and takes safety measures.

[0158] When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present invention are executed. The method, apparatus, and device of the present invention can be implemented in many ways. For example, the method, apparatus, and device of the present invention can be implemented through software, hardware, firmware, or any combination of software, hardware, and firmware.

[0159] The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the specific order described above, unless otherwise specifically stated.

[0160] In addition, in some embodiments, the present invention can also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0161] In addition, parts of the above technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive elaboration.

[0162] As described above in the specific embodiments, the purpose, technical solutions, and beneficial effects of the present invention are further described in detail. It should be understood that the above is only the specific embodiment of the present invention and is not used to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent palletizing robot safety management method based on AI vision, characterized in that, Including: Obtain multi-modal data of the palletizing operation area, conduct three-dimensional fusion perception of the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration; Adopt a multi-view point cloud stitching algorithm to register and stitch the point cloud data collected by lidar at different times, and synchronize the image data and point cloud data in time; downsample and voxel filter the fused point cloud to remove the noise points and outliers in the fused point cloud; Register the color image obtained by the depth camera with the point cloud, and assign corresponding RGB color information to each point cloud point; use the greedy triangulation algorithm to perform surface reconstruction on the fused color point cloud, and generate a triangular mesh according to the spatial distribution and normal vector information of the point cloud; simplify and smooth the reconstructed triangular mesh to reduce the number of mesh patches and obtain a three-dimensional model of the palletizing operation area; Construct an object detection algorithm based on deep learning to perform real-time detection and identify multiple types of objects on the three-dimensional model; Adopt the YOLOv5 algorithm to perform real-time object detection on the three-dimensional model of the palletizing operation area, fuse the object detection results with the lidar point cloud, and obtain the three-dimensional model coordinate position of the object through nearest neighbor matching; the nearest neighbor matching finds the best match by calculating the distance between the detected object and the point cloud set; Adopt the Deep SORT algorithm to achieve multi-object tracking; the Deep SORT algorithm uses the Hungarian algorithm to solve the object association problem and optimizes the association matrix by maximizing the similarity between the tracking object and the detected object; Based on the object category and position information identified in the three-dimensional model of the palletizing operation area, assign different risk impact coefficients to each type of object, and conduct dynamic safety area division for the palletizing operation area; Divide the palletizing operation area into uniform grids to form a grid map; In the grid map, the risk value of each grid is determined by the position and influence range of the objects around the grid. The calculation formula for the grid risk value is: Among them, R(x, y) represents the risk value at the position (x, y), and i is the index of the target object; (x i , y i ) are the position coordinates of the i-th target object; α i is the risk impact coefficient of the i-th target, indicating the degree of danger of the i-th target; σ i represents the parameter of the influence range size of the i-th target; In the grid map, divide the grid map into a safe area and a dangerous area according to the safety threshold. The safe area includes a low-risk area, and the dangerous area includes a medium-risk area and a high-risk area; Set the safety thresholds TR1 and TR2 of the grid map, and TR2 > TR1; when R(x,y) < TR1, divide the grid area into a low-risk area; when TR1 ≤ R(x,y) ≤ TR2, divide the grid area into a medium-risk area; when R(x,y) > TR2, divide the grid area into a high-risk area; Set a fixed time interval Δt. Whenever Δt time has passed, recalculate the risk value of the grid area, and conduct safety area division according to the recalculated grid risk value; According to the dynamic safety area division result, construct a mobile path planning strategy to control the palletizing robot to perform mobile path planning in the palletizing work area; When the palletizing robot obtains the status of the palletized goods in the palletizing operation area, detect the status of the palletized goods through the AI vision model. When detecting goods with abnormal palletizing, the palletizing robot automatically triggers an alarm and takes safety measures.

2. The safety management method of the intelligent palletizing robot based on AI vision according to claim 1, characterized in that The multimodal data includes image data of the palletizing operation area captured by a depth camera and lidar point cloud data of the palletizing operation area scanned by a lidar; the depth camera and the lidar are respectively fixed on the top of the palletizing robot, and the depth camera and the lidar rotate on the top of the palletizing robot to obtain image data and lidar point cloud data within a 360° range of the palletizing robot; The downsampling includes performing downsampling processing on the spliced point cloud, and dividing the point cloud into uniform small voxels using a voxel grid method; The voxel filtering includes filtering the point cloud using a voxel filtering algorithm, and based on the set voxel size, retaining the centroid point within each voxel to obtain uniform point cloud data.

3. The safety management method of the intelligent palletizing robot based on AI vision according to claim 2, wherein, Based on the divided safe area and dangerous area, an optimized A* algorithm is used to perform hierarchical movement path planning for the palletizing robot in the palletizing operation area; the hierarchical movement path planning includes the first-layer movement path planning, the second-layer movement path planning, and the third-layer movement path planning; The first-layer movement path planning includes searching for the optimal path using the A* algorithm in the low-risk area, setting the starting point and the target point as the input of the A* algorithm, and using the low-risk area as the passable area; The second-layer movement path planning includes, if a feasible path cannot be found in the low-risk area, adding the medium-risk area to the passable area and searching for the path using the A* algorithm again; The third-layer movement path planning includes, if a feasible path cannot be found in the low- and medium-risk areas, adding the high-risk area to the passable area and searching for the path using the A* algorithm again; If a feasible path cannot be found in all risk areas, a path planning failure alarm is triggered; The optimized A* algorithm includes locally optimizing the initial path obtained by the A* algorithm and smoothing the path using a Bezier curve. The optimization objective is: Among them, (x a , y a ) is the coordinate of the a-th sampling point on the path, is the coordinate of the h-th control point, m and n are the number of sampling points and the number of control points respectively, and λ is the smoothness weight coefficient; Whenever Δt time has passed and the grid risk value of the grid map changes, the movement path of the palletizing robot is re-planned using the optimized A* algorithm.

4. The safety management method of the intelligent palletizing robot based on AI vision according to claim 3, wherein, Based on the image data obtained by the depth camera, identify the palletized goods. When the palletized goods are identified, simultaneously obtain the point cloud data when the lidar scans the palletized goods, mark the obtained point cloud data as the goods point cloud, calculate the normal vector and curvature of the goods point cloud, and obtain the normal vector of each point in the goods point cloud and curvature c p value; Set the normal vector of the goods in the normal state and the curvature reference value c ref , and the abnormal curvature upper limit c max ; Calculate the anomaly index A(p) for each point in the cargo point cloud: Among them, w n is the weight coefficient of the deviation of the normal vector, and w c is the weight coefficient of the high curvature; Set the anomaly threshold T a , which is used to determine whether a single point is anomalous; When the anomaly index A(p) of a point is greater than T a it is determined that the point is in an abnormal state. When the proportion of the number of points where A(p) is greater than T a exceeds the percentage threshold TPa, it is determined that the state of the goods stacking is abnormal.

5. The safety management method of the intelligent palletizing robot based on AI vision according to claim 4, wherein, Based on the distribution and clustering of the abnormal points in the cargo point cloud, determine the types of cargo palletizing anomalies, and the types of cargo palletizing anomalies include tilting and scattering; Downsample the cargo point cloud to obtain uniformly distributed core points; calculate the covariance matrix C of the core points: Among them, h is the number of core points, p z is the coordinate of the z-th core point; is the central coordinate of the core point; Perform eigenvalue decomposition on C: C = UΛU T ; Obtain the eigenvectors where U is the eigenvector matrix, Λ is the eigenvalue matrix, and U T is the transpose of the eigenvector matrix; Take the feature vector as the main direction of the palletized goods to represent the pose of the palletized goods; calculate the central coordinates of the core point of the palletized goods as the position of the goods; According to the pose and abnormal type of the goods, search for the corresponding correction strategy in the predefined strategy library; if a matching correction strategy is found, enter the automatic correction process; otherwise, trigger an alarm; the correction strategy includes translating and rotating the goods along the direction to translate and rotate the goods, and adjust the pose of the abnormally palletized goods; according to and the target position, calculate the translation vector of the abnormally palletized goods; convert the correction strategy into the motion control instructions of the palletizing robot; After the correction is completed, scan the cargo to re-obtain the cargo point cloud data; and detect the status and pose of the palletized cargo after correction; if the abnormal status of the palletized cargo is eliminated, the palletizing robot continues to operate normally in the palletizing work area; otherwise, an alarm is triggered.

6. An intelligent palletizing robot safety management system based on AI vision, which is used to implement the safety management method of the intelligent palletizing robot based on AI vision according to any one of claims 1 to 5, characterized in that, Including: A data acquisition module, a target detection module, a region division module, a path planning module, and an anomaly correction module; The data acquisition module is used to acquire the multimodal data of the palletizing operation area, perform three-dimensional fusion perception on the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration; The target detection module is used to construct a target detection algorithm based on deep learning, and perform real-time detection and recognition of multiple types of targets on the three-dimensional model; An area division module, configured to perform dynamic safety area division on the palletizing operation area based on the identified target categories and position information within the palletizing operation area; A path planning module, configured to construct a mobile path planning strategy according to the dynamic safety area division result, and control the palletizing robot to perform mobile path planning within the palletizing working area; An anomaly correction module, configured to, when the palletizing robot obtains the state of the palletized goods within the palletizing operation area, detect the state of the palletized goods through an AI vision model, and when detecting goods with palletizing anomalies, the palletizing robot automatically takes safety measures and triggers an alarm.

7. An electronic device, characterized in that, Including: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; the processor executes the method for safety management of an intelligent palletizing robot based on AI vision according to any one of claims 1 to 5 by calling the computer program stored in the memory.

8. A storage medium, characterized in that, Stores instructions that, when run on a computer, cause the computer to execute the method for safety management of an intelligent palletizing robot based on AI vision according to any one of claims 1 to 5.

Images