Intelligent palletizing robot safety management system and method based on AI vision

Abstract

The invention relates to the technical field of intelligent robots, in particular to an intelligent palletizing robot safety management system and method based on AI vision. According to the method, the three-dimensional model is constructed by obtaining the multi-modal data of the stacking operation area; carrying out target detection and recognition by using a deep learning algorithm; performing dynamic safety area division on the operation area according to the target information; a moving path planning strategy is constructed, and the robot is controlled to operate safely; and meanwhile, the state of the stacked goods is detected through an AI visual model, and when abnormity is found, an alarm is automatically given, and safety measures are taken. The intelligent level and the safety performance of the palletizing robot can be improved, and wide application prospects are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a safety management system and method for intelligent palletizing robots based on AI vision. Background Art

[0002] Palletizing robots have been widely used in warehousing, logistics, manufacturing and other fields due to their high efficiency, flexibility and low cost, greatly improving the automation level of cargo loading and unloading and stacking. However, due to the complexity, dynamics and uncertainty of the palletizing operation environment, traditional palletizing robots still have many deficiencies in safety and intelligence, and are difficult to meet the increasing operational requirements.

[0003] Traditional palletizing robots mainly rely on pre-set working paths and fixed safety areas, and lack the ability to perceive the environment and target objects in real time and adapt dynamically. They cannot accurately obtain and understand the real-time status of the work site, and it is difficult for them to respond to environmental changes and unexpected situations in a timely manner and make flexible adjustments. When there are interference factors such as people intruding, obstacles appearing, and changes in the position of goods in the working environment, the robots are often unable to make correct decisions and countermeasures in time, and safety accidents such as collisions, pinching, and misoperation are very likely to occur, posing serious safety hazards to on-site workers and equipment facilities.

[0004] The status monitoring and quality inspection of traditional palletized goods also mostly rely on manual inspection and intervention, lacking intelligent solutions. Due to the wide variety, diverse shapes and irregular stacking of palletized goods, it is difficult to effectively monitor them through conventional automated equipment. Operators need to frequently conduct visual inspections of the cargo stacks, and need to manually identify and handle abnormal conditions such as tilting, disarray, and damage. This inefficient and time-consuming manual supervision mode is not only labor-intensive and costly, but also easily affected by human factors, resulting in problems such as missed inspections and misjudgments, which brings great uncertainty risks to cargo safety and palletizing quality.

[0005] In view of this, the present invention proposes an intelligent palletizing robot safety management system and method based on AI vision. Summary of the invention

[0006] To achieve the above objectives, the present invention provides an intelligent palletizing robot safety management system and method based on AI vision, and the specific technical solutions are as follows:

[0007] The safety management method of intelligent palletizing robots based on AI vision includes:

[0008] Acquire multimodal data of the palletizing operation area, perform 3D fusion perception of the palletizing operation area, and build a 3D model of the palletizing operation area through data synchronization and registration;

[0009] Build a deep learning-based target detection algorithm to detect 3D models in real time and identify multiple types of targets;

[0010] Based on the target categories and location information identified in the palletizing operation area, the palletizing operation area is dynamically divided into safety areas;

[0011] According to the dynamic safety area division results, a mobile path planning strategy is constructed to control the palletizing robot to plan the mobile path in the palletizing work area;

[0012] When the palletizing robot obtains the status of the palletized goods in the palletizing operation area, it detects the status of the palletized goods through the AI ​​vision model. When abnormal palletizing of goods is detected, the palletizing robot automatically triggers an alarm and takes safety measures.

[0013] Preferably, the multimodal data includes image data of the palletizing operation area taken by a depth camera and laser point cloud data of the palletizing operation area scanned by a laser radar; the depth camera and the laser radar are respectively fixed on the top of the palletizing robot, and the depth camera and the laser radar are rotated on the top of the palletizing robot to obtain image data and laser point cloud data within a 360° range of the palletizing robot;

[0014] A multi-view point cloud stitching algorithm is used to register and stitch the point cloud data collected by the lidar at different times, and the image data and point cloud data are synchronized in time;

[0015] Downsample and voxel filter the fused point cloud to remove noise and outliers;

[0016] The downsampling includes downsampling the spliced ​​point cloud and dividing the point cloud into uniform small voxels using a voxelized grid method;

[0017] The voxel filtering includes filtering the point cloud using a voxel filtering algorithm, retaining the center of gravity point in each voxel based on a set voxel size, and obtaining uniform point cloud data;

[0018] The color image obtained by the depth camera is registered with the point cloud, and the corresponding RGB color information is assigned to each point cloud point;

[0019] The surface of the fused color point cloud is reconstructed using a greedy triangulation algorithm, and a triangular mesh is generated based on the spatial distribution and normal vector information of the point cloud.

[0020] The reconstructed triangular mesh is simplified and smoothed to reduce the number of mesh faces and obtain a three-dimensional model of the palletizing operation area.

[0021] Preferably, the YOLOv5 algorithm is used to perform real-time object detection on the three-dimensional model of the palletizing operation area, and the object detection results are fused with the lidar point cloud to obtain the three-dimensional model coordinate positions of the objects through nearest neighbor matching; the nearest neighbor matching finds the best match by calculating the distance between the detected objects and the point cloud set;

[0022] The Deep SORT algorithm is used to achieve multi-object tracking; the Deep SORT algorithm uses the Hungarian algorithm to solve the object association problem and optimizes the association matrix by maximizing the similarity between the tracking objects and the detected objects;

[0023] After identifying multiple object types based on the three-dimensional model, different risk impact coefficients are assigned to each type of object for dynamic safety area division.

[0024] Preferably, the palletizing operation area is divided into uniform grids to form a grid map; in the grid map, the risk value of each grid is determined by the positions and influence ranges of the objects around the grid, and the grid risk value calculation formula is:

[0025]

[0026] where R(x,y) represents the risk value at the position (x,y), i is the index of the object; (x i ,y i ) is the position coordinate of the i-th object; α i is the risk impact coefficient of the i-th object, indicating the degree of danger of the 9th object; σ i represents the parameter indicating the influence range size of the i-th object;

[0027] In the grid map, the grid map is divided into a safe area and a dangerous area according to the safety threshold. The safe area includes a low-risk area, and the dangerous area includes a medium-risk area and a high-risk area;

[0028] Set the grid map safety thresholds TR1 and TR2, and TR2 > TR1; when R(x,y) < TR1, the grid area is divided into a low-risk area; when TR1 ≤ R(x,y) ≤ TR2, the grid area is divided into a medium-risk area; when R(x,y) > TR2, the grid area is divided into a high-risk area;

[0029] Set a fixed time interval Δt. Whenever Δt time has passed, recalculate the risk value of the grid area, and perform safety area division according to the recalculated grid risk value.

[0030] Preferably, based on the divided safe area and dangerous area, an optimized A* algorithm is used to perform hierarchical mobile path planning for the palletizing robot in the palletizing operation area; the hierarchical mobile path planning includes first-layer mobile path planning, second-layer mobile path planning and third-layer mobile path planning;

[0031] The first level of mobile path planning includes searching for an optimal path in a low-risk area using an A* algorithm, setting a starting point and a target point as inputs of the A* algorithm, and treating the low-risk area as a traversable area;

[0032] The second-level mobile path planning includes, if no feasible path can be found in the low-risk area, adding the medium-risk area to the passable area and searching for a path again using the A* algorithm;

[0033] The third-level mobile path planning includes, if no feasible path can be found in the low-risk and medium-risk areas, adding the high-risk area to the passable area and searching for the path again using the A* algorithm;

[0034] If no feasible path can be found in all risk areas, a path planning failure alarm is triggered;

[0035] The optimized A* algorithm includes locally optimizing the initial path obtained by the A* algorithm and smoothing the path using a Bezier curve. The optimization goal is:

[0036]

[0037] Among them, (x a ,y a ) is the coordinate of the ath sampling point on the path, is the coordinate of the hth control point, m and n are the number of sampling points and control points respectively, and λ is the smoothness weight coefficient;

[0038] Every time after Δt, when the grid risk value of the grid map changes, the moving path of the palletizing robot is replanned through the optimized A* algorithm.

[0039] Preferably, the image data is obtained based on the depth camera to identify the stacked goods. When the stacked goods are identified, the point cloud data of the laser radar scanning the stacked goods is obtained at the same time, and the obtained point cloud data is marked as the cargo point cloud. The normal vector and curvature of the cargo point cloud are calculated to obtain the normal vector of each point in the cargo point cloud. and curvature c p value;

[0040] Set the normal vector of the cargo in normal state and the curvature reference value c ref , and the upper limit of the anomalous curvature c max ;

[0041] Calculate the abnormal index A(p) of each point in the cargo point cloud:

[0042]

[0043] Among them, w n is the weight coefficient of the normal vector deviation, w c is the weight coefficient for high curvature;

[0044] Set the abnormal threshold T a , used to determine whether a single point is abnormal; when the abnormal index A(p) of the point is greater than T a When A(p) is greater than T a When the percentage of points exceeds the percentage threshold TPa, the state of the cargo palletizing is determined to be abnormal.

[0045] Preferably, the abnormal type of cargo stacking is determined according to the distribution and clustering of abnormal points in the cargo point cloud, and the abnormal type of cargo stacking includes tilting and scattering;

[0046] Downsample the cargo point cloud to obtain evenly distributed core points; calculate the covariance matrix C of the core points:

[0047]

[0048] Among them, h is the number of core points, p z is the coordinate of the zth core point; are the center coordinates of the core point;

[0049] Perform eigenvalue decomposition on C: C = UΛU T ; Get the feature vector where U is the eigenvector matrix, Λ is the eigenvalue matrix, and U T is the transpose of the eigenvector matrix;

[0050] The feature vector As the main direction of the stacked goods, it indicates the position of the stacked goods; calculates the center coordinates of the core point of the stacked goods as the location of the goods;

[0051] According to the position and abnormality type of the goods, the corresponding correction strategy is searched in the predefined strategy library; if a matching correction strategy is found, the automatic correction process is entered; otherwise, an alarm is triggered; the correction strategy includes Directionally translate and rotate goods to adjust the abnormal stacking posture of goods; and target position, calculate the translation vector of the abnormally stacked goods; convert the correction strategy into motion control instructions for the stacking robot;

[0052] After the correction is completed, the goods are scanned and the point cloud data of the goods is re-acquired; and the state and posture of the corrected palletized goods are detected; if the abnormal state of the palletized goods is eliminated, the palletizing robot continues to operate normally in the palletizing work area; otherwise, an alarm is triggered.

[0053] An AI vision-based intelligent palletizing robot safety management system, which is used to implement an AI vision-based intelligent palletizing robot safety management method, including: a data acquisition module, a target detection module, an area division module, a path planning module, and an abnormality correction module;

[0054] The data acquisition module is used to acquire multimodal data of the palletizing operation area, perform three-dimensional fusion perception of the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration;

[0055] The target detection module is used to build a deep learning-based target detection algorithm to detect 3D models in real time and identify multiple types of targets;

[0056] An area division module is used to divide the palletizing operation area into dynamic safety areas based on the target categories and location information identified in the palletizing operation area;

[0057] The path planning module is used to build a mobile path planning strategy based on the dynamic safety area division results, and control the palletizing robot to plan the mobile path within the palletizing work area;

[0058] The abnormality correction module is used to detect the status of the palletizing goods in the palletizing operation area through the AI ​​visual model when the palletizing robot obtains the status of the palletizing goods in the palletizing operation area. When abnormal palletizing goods are detected, the palletizing robot automatically takes safety measures and triggers an alarm.

[0059] An electronic device comprises: a processor and a memory, wherein the memory stores a computer program that can be called by the processor; the processor executes the AI ​​vision-based intelligent palletizing robot safety management method by calling the computer program stored in the memory.

[0060] A storage medium stores instructions, which, when executed on a computer, enable the computer to execute the AI ​​vision-based intelligent palletizing robot safety management method.

[0061] Beneficial effects of the present invention: The present invention constructs a three-dimensional model of the palletizing operation area to provide comprehensive and accurate environmental information for subsequent target detection, risk assessment and path planning.

[0062] The present invention realizes real-time detection and recognition of multiple types of targets through deep learning algorithms, thereby improving the robustness and generalization ability of the perception system.

[0063] The present invention dynamically divides the safety area and flexibly adjusts the robot's working space according to the real-time changes of the operation site, thereby improving the operation efficiency while ensuring safety.

[0064] The present invention combines the safety area division result to plan the moving path, selects the lowest risk and most energy-saving moving route, and improves the efficiency and safety of the palletizing operation.

[0065] The present invention utilizes AI visual models to realize intelligent detection of the status of palletized goods, automatically identifies abnormal situations and triggers alarms, and takes safety measures in time to prevent palletizing operation accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flow chart of the safety management method of the intelligent palletizing robot based on AI vision provided by the present invention;

[0067] Figure 2 This is a structural diagram of the AI ​​vision-based intelligent palletizing robot safety management system provided by the present invention;

[0068] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0069] Figure 4 It is a schematic diagram of the storage medium structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to better understand the present invention, a more detailed description will be made of various aspects of the present invention with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0071] In the accompanying drawings, the size, dimensions and shapes of the elements have been slightly adjusted for ease of illustration. The accompanying drawings are for illustration only and are not strictly drawn to scale. As used herein, the terms "substantially", "approximately" and similar terms are used as terms of approximation, not as terms of degree, and are intended to illustrate the inherent deviations in measurements or calculations that will be recognized by those of ordinary skill in the art. In addition, in the present invention, the order in which the steps are described does not necessarily represent the order in which these processes occur in actual operation, unless otherwise specified or can be derived from the context.

[0072] It should also be understood that expressions such as "comprises", "including", "having", "includes" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features rather than just the individual elements in the list. In addition, when describing embodiments of the present invention, "may" is used to mean "one or more embodiments of the present invention". And, the term "exemplary" is intended to refer to an example or illustration.

[0073] Unless otherwise defined, all words (including engineering terms and scientific and technological terms) used in this article have the same meaning as those commonly understood by ordinary technicians in the field to which the present invention belongs. It should also be understood that unless there is a clear explanation in the present invention, the words defined in the commonly used dictionary should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense.

[0074] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0075] Example 1

[0076] Reference Figure 1 , which is the first embodiment of the present invention, provides a safety management method for an intelligent palletizing robot based on AI vision.

[0077] S1: Acquire multimodal data of the palletizing operation area, perform three-dimensional fusion perception of the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration.

[0078] The multimodal data includes image data of the palletizing operation area taken by a depth camera and laser point cloud data of the palletizing operation area scanned by a laser radar; the depth camera and the laser radar are respectively fixed on the top of the palletizing robot, and the depth camera and the laser radar are rotated on the top of the palletizing robot to obtain image data and laser point cloud data within a 360° range of the palletizing robot.

[0079] Point Cloud Library (PCL) is used as the basic library for point cloud processing. The multi-view point cloud stitching algorithm provided by PCL is used to align and stitch the point cloud data collected by the lidar at different times, and the image data and point cloud data are synchronized in time.

[0080] Use the message_filters package of ROS to synchronize the image data of the palletizing work area taken by the depth camera and the laser point cloud data of the palletizing work area scanned by the lidar;

[0081] The fused point cloud is downsampled and voxel filtered to remove noise and outliers from the fused point cloud.

[0082] The downsampling includes performing downsampling processing on the spliced ​​point cloud, dividing the point cloud into uniform small voxels using a voxelized grid method, and retaining only one point in each voxel, thereby reducing the data volume of the point cloud.

[0083] The voxel filtering includes filtering the point cloud using a voxel filtering algorithm, retaining the center of gravity point in each voxel based on a set voxel size, and obtaining uniform point cloud data.

[0084] The color image obtained by the depth camera is registered with the point cloud, and the corresponding RGB color information is assigned to each point cloud point.

[0085] The greedy triangulation algorithm is used to reconstruct the surface of the fused color point cloud, and a triangular mesh is generated according to the spatial distribution and normal vector information of the point cloud.

[0086] The reconstructed triangular mesh is simplified and smoothed to reduce the number of mesh faces, improve the surface smoothness, and obtain a three-dimensional model of the palletizing operation area.

[0087] Through multi-sensor fusion perception technology, the palletizing robot can comprehensively and accurately obtain the environmental information of the work area, build a three-dimensional scene model, and provide a reliable data basis for subsequent target detection, risk assessment and path planning.

[0088] S2: Build a deep learning-based target detection algorithm to detect 3D models in real time and identify multiple types of targets.

[0089] The YOLOv5 algorithm is used to perform real-time target detection on the three-dimensional model of the palletizing operation area. The YOLOv5 algorithm realizes the detection of different targets by predicting the confidence of each target category.

[0090] The target detection results are fused with the lidar point cloud, and the 3D model coordinate position of the target is obtained through nearest neighbor matching; nearest neighbor matching finds the best match by calculating the distance between the detected target and the point cloud set.

[0091] The Deep SORT algorithm is used to achieve multi-target tracking; the Deep SORT algorithm uses the Hungarian algorithm to solve the target association problem and optimizes the association matrix by maximizing the similarity between the tracked target and the detected target.

[0092] After identifying multiple target types based on the three-dimensional model, different risk impact coefficients are assigned to each type of target to perform dynamic safety area division.

[0093] By using deep learning algorithms for target detection and tracking, the palletizing robot can identify and locate key targets such as personnel, equipment, and goods in the work area in real time, grasp their dynamic changes, and provide a basis for the formulation of safety strategies.

[0094] S3: Based on the target categories and location information identified in the palletizing operation area, the palletizing operation area is dynamically divided into safety areas.

[0095] The palletizing operation area is divided into uniform grids, that is, the palletizing operation area is divided into a grid map; in the grid gradient grid map, the risk value of each grid is determined by the location and influence range of the target around the grid, and the grid risk value calculation formula is:

[0096]

[0097] Where R(x,y) represents the risk value at the position (x,y), i is the index of the target object; (x i ,y i is the position coordinate of the i-th target object; α i is the risk impact coefficient of the i-th target, indicating the degree of danger of the target; σ i A parameter representing the size of the influence range of the i-th target.

[0098] After identifying multiple target types in the palletizing operation area, it is necessary to assign corresponding impact coefficients according to the risk level of different targets. The risk impact coefficient can be obtained through expert experience or data statistics.

[0099] For example, different types of targets such as people, vehicles, and cargo can be assigned risk coefficients from high to low, such as λ 1 =1.0,λ 2 =0.8,λ 3 =0.6.

[0100] In addition, the parameters of the risk impact range also need to be determined based on the size, movement speed and other attributes of the target; the impact range parameter σ i It is proportional to the average size and maximum speed of the target and can be estimated by the following formula: i = k·max(r i ,v i ), where r i and v i are the average radius and maximum speed of the i-th target respectively, and k is the proportional coefficient of the impact range, which can be set according to the safety margin and is usually between 1.5 and 2.0.

[0101] When calculating the risk values of each grid, the risk superposition effect of multiple targets can be considered. When a grid is affected by multiple targets simultaneously, its risk value can be taken as the maximum value of the weighted sum of the influence factors.

[0102] In the grid map, the grid map is divided into a safe area and a dangerous area according to a safety threshold. The safe area includes a low-risk area, and the dangerous area includes a medium-risk area and a high-risk area.

[0103] Set the safety thresholds TR1 and TR2 of the grid map, and TR2 > TR1; when R(x, y) < TR1, the grid area is divided into a low-risk area; when TR1 ≤ R(x, y) ≤ TR2, the grid area is divided into a medium-risk area; when R(x, y) > TR2, the grid area is divided into a high-risk area.

[0104] When dynamically dividing the safe area of the palletizing operation area, first, appropriate thresholds TR1 and TR2 need to be selected according to the task requirements and the complexity of the scenario; the setting of the safety threshold needs to be balanced between the flexibility of the robot's movement and safety assurance.

[0105] It should be noted that the higher the threshold, the larger the activity space of the robot, but the higher the collision risk it faces. On the contrary, the lower the threshold, the more restricted the movement of the robot, but the safety of itself and the environment can be better guaranteed. Therefore, the selection of the threshold needs to be combined with the specific application scenario and the optimal value determined through repeated experiments and evaluations.

[0106] Exemplarily, statistically analyze the distribution of the grid risk values in the scenario to obtain its numerical range and variation law, and then set the threshold to a certain percentile, such as the 20th percentile and the 80th percentile, so as to be neither too conservative nor too radical. The threshold can also be adjusted in real time according to the change of the task progress or the risk situation. For example, in the initial stage of the palletizing operation, due to the relatively neat goods and the low scene risk, the threshold can be appropriately relaxed to allow the robot to move and operate more freely. In the later stage of the palletizing operation, due to the increasing height of the pallet and the more crowded environment, the threshold can be correspondingly reduced to make the movement of the robot more cautious and restrained. Another example is that when detecting that a person or other important target enters the operation area, the threshold can be immediately increased to limit the activity range of the robot until the target leaves.

[0107] Set a fixed time interval Δt. Whenever Δt time has passed, recalculate the risk values of the grid area, and perform the division of the safe area according to the recalculated grid risk values.

[0108] Based on grid maps and regional risk assessments, the palletizing robot can dynamically divide the work area into safe and dangerous areas, quantify the risk levels of different areas, provide constraints for the robot's motion decisions, and effectively avoid safety hazards such as collisions and interference.

[0109] S4: According to the dynamic safety area division results, a mobile path planning strategy is constructed to control the palletizing robot to plan the mobile path within the palletizing work area.

[0110] Based on the divided safe areas and dangerous areas, an optimized A* algorithm is used to perform hierarchical mobile path planning for the palletizing robot in the palletizing operation area; the hierarchical mobile path planning includes first-layer mobile path planning, second-layer mobile path planning and third-layer mobile path planning.

[0111] The first-level mobile path planning includes searching for the optimal path in the low-risk area using the A* algorithm, setting the starting point and the target point as inputs of the A* algorithm, and using the low-risk area as a traversable area.

[0112] The second-level mobile path planning includes, if no feasible path can be found in the low-risk area, adding the medium-risk area to the passable area and searching for a path again using the A* algorithm;

[0113] The third-level mobile path planning includes, if no feasible path can be found in the low-risk and medium-risk areas, adding the high-risk areas to the passable areas and using the A* algorithm to search for the path again.

[0114] If no feasible path can be found in all risk areas, a path planning failure alarm is triggered.

[0115] The optimized A* algorithm includes locally optimizing the initial path obtained by the A* algorithm and smoothing the path using a Bezier curve. The optimization goal is:

[0116]

[0117] Among them, (x a ,y a ) is the coordinate of the ath sampling point on the path, is the coordinate of the hth control point, m and n are the number of sampling points and control points respectively, and λ is the smoothness weight coefficient.

[0118] Every time after Δt, when the grid risk value of the grid map changes, the moving path of the palletizing robot is replanned through the optimized A* algorithm.

[0119] Using the hierarchical optimization A* algorithm for mobile path planning, the palletizing robot can autonomously search for the optimal feasible path based on the division results of the safe area, taking into account both movement efficiency and safety. At the same time, the Bezier curve smoothing technology makes the robot's movement trajectory more natural and smooth.

[0120] S5: When the palletizing robot obtains the status of the palletized goods in the palletizing operation area, it detects the status of the palletized goods through the AI ​​vision model. When abnormal palletizing goods are detected, the palletizing robot automatically triggers an alarm and takes safety measures.

[0121] The image data is obtained based on the depth camera, and the stacked goods are identified. When the stacked goods are identified, the point cloud data of the laser radar scanning the stacked goods is obtained at the same time. The obtained point cloud data is marked as the cargo point cloud, and the normal vector and curvature of the cargo point cloud are calculated to obtain the normal vector of each point in the cargo point cloud. and curvature c p value.

[0122] According to prior knowledge, set the normal vector of the goods in normal state and the curvature reference value c ref , and the upper limit of the anomalous curvature c max .

[0123] Calculate the abnormal index A(p) of each point in the cargo point cloud:

[0124]

[0125] Among them, w n is the weight coefficient of the normal vector deviation, w n The value is between [0,1], w n The larger the value is, the greater the contribution of the normal vector deviation to the abnormal index is; c is the weight coefficient for high curvature; w c The value is between [0,1], w c The larger it is, the greater the contribution of high curvature to the abnormal index.

[0126] Set the abnormal threshold T a , used to determine whether a single point is abnormal; when the abnormal index A(p is greater than T a When A(p) is greater than T a When the percentage of points exceeds the percentage threshold TPa, the state of the cargo palletizing is determined to be abnormal.

[0127] According to the distribution and clustering of abnormal points in the cargo point cloud, the abnormal type of cargo stacking is determined, and the abnormal type of cargo stacking includes tilting and scattering.

[0128] Downsample the cargo point cloud to obtain evenly distributed core points; calculate the covariance matrix C of the core points:

[0129]

[0130] Among them, h is the number of core points, p z is the coordinate of the zth core point; are the center coordinates of the core point.

[0131] Perform eigenvalue decomposition on C: C = UΛU T ; Get the feature vector where U is the eigenvector matrix, Λ is the eigenvalue matrix, and U T is the transpose of the eigenvector matrix.

[0132] The feature vector As the main direction of the stacked goods, it indicates the position of the stacked goods; calculates the center coordinates of the core point of the stacked goods As the location of the goods.

[0133] According to the position and abnormality type of the goods, the corresponding correction strategy is searched in the predefined strategy library; if a matching correction strategy is found, the automatic correction process is entered; otherwise, an alarm is triggered; the correction strategy includes Directionally translate and rotate goods to adjust the abnormal stacking posture of goods; and target position, calculate the translation vector of the abnormally stacked goods; and convert the correction strategy into motion control instructions for the stacking robot.

[0134] After the correction is completed, the goods are rescanned to obtain point cloud data; and the corrected state and posture of the palletized goods are detected; if the abnormal state of the palletized goods is eliminated, the palletizing robot continues to operate normally in the palletizing work area; otherwise, an alarm is triggered.

[0135] By introducing AI visual detection and abnormal diagnosis technology, the palletizing robot can automatically monitor the status of cargo palletizing, promptly detect quality defects such as tilting and disarray, and automatically correct or alarm according to predefined strategies, thereby maximizing the quality and safety of cargo stacking.

[0136] Example 2

[0137] Reference Figure 2 , which is the second embodiment of the present invention, provides an intelligent palletizing robot safety management system based on AI vision.

[0138] The system includes: a data acquisition module, a target detection module, a region division module, a path planning module and an anomaly correction module.

[0139] The data acquisition module is used to obtain multimodal data of the palletizing operation area, perform three-dimensional fusion perception of the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and alignment.

[0140] The target detection module is used to build a deep learning-based target detection algorithm to perform real-time detection of three-dimensional models and identify multiple types of targets.

[0141] The area division module is used to divide the palletizing operation area into dynamic safety areas based on the target category and position information identified in the palletizing operation area.

[0142] The path planning module is used to build a mobile path planning strategy based on the dynamic safety area division results and control the palletizing robot to plan the mobile path within the palletizing work area.

[0143] The abnormality correction module is used to detect the status of the palletizing goods in the palletizing operation area through the AI ​​visual model when the palletizing robot obtains the status of the palletizing goods in the palletizing operation area. When abnormal palletizing goods are detected, the palletizing robot automatically takes safety measures and triggers an alarm.

[0144] Example 3

[0145] Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, according to another aspect of the present invention, an electronic device 500 is also provided. The electronic device 500 may include one or more processors and one or more memories. The memory stores a computer readable code, and when the computer readable code is executed by one or more processors, the intelligent palletizing robot safety management method based on AI vision as described above can be executed.

[0146] The method or system according to the embodiment of the present invention can also be used by Figure 3 The architecture of the electronic device shown is implemented.

[0147] like Figure 3 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, and the like.

[0148] The storage device in the electronic device 500, such as ROM 503 or hard disk 507, can store the AI ​​vision-based intelligent palletizing robot safety management method provided by the present invention.

[0149] The safety management method of an intelligent palletizing robot based on AI vision includes: obtaining multimodal data of a palletizing operation area, performing three-dimensional fusion perception of the palletizing operation area, and constructing a three-dimensional model of the palletizing operation area through data synchronization and registration; constructing a target detection algorithm based on deep learning to perform real-time detection of the three-dimensional model and identify multiple types of targets; dynamically dividing the palletizing operation area into safe areas based on the target categories and position information identified in the palletizing operation area; constructing a mobile path planning strategy based on the dynamic safety area division result to control the palletizing robot to plan the mobile path in the palletizing work area; when the palletizing robot obtains the status of the palletizing goods in the palletizing operation area, the status of the palletizing goods is detected through the AI ​​vision model, and when abnormal palletizing goods are detected, the palletizing robot automatically triggers an alarm and takes safety measures.

[0150] Furthermore, the electronic device 500 may also include a user interface 508. Figure 3 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 3 One or more components of an electronic device are shown.

[0151] Example 4

[0152] Figure 4 It is a schematic diagram of the storage medium structure provided by an embodiment of the present invention.

[0153] like Figure 4 As shown is a storage medium 600 according to one embodiment of the present invention.

[0154] The storage medium 600 has computer readable instructions stored thereon.

[0155] When the computer-readable instructions are executed by the processor, the AI ​​vision-based intelligent palletizing robot safety management method according to the embodiments of the present invention described with reference to the above drawings may be executed.

[0156] The storage medium 600 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. In addition, according to an embodiment of the present invention, the process described above with reference to the flowchart may be implemented as a computer software program.

[0157] For example, the present invention provides a non-temporary machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided by the present invention, for example: obtaining multimodal data of the palletizing operation area, performing three-dimensional fusion perception of the palletizing operation area, and constructing a three-dimensional model of the palletizing operation area through data synchronization and registration; constructing a target detection algorithm based on deep learning, performing real-time detection of the three-dimensional model and identifying multiple types of targets; based on the target category and position information identified in the palletizing operation area, dynamically dividing the palletizing operation area into safety areas; according to the dynamic safety area division results, constructing a mobile path planning strategy to control the palletizing robot to plan the mobile path in the palletizing work area; when the palletizing robot obtains the status of the palletizing goods in the palletizing operation area, the status of the palletizing goods is detected through the AI ​​visual model, and when abnormal palletizing goods are detected, the palletizing robot automatically triggers an alarm and takes safety measures.

[0158] When the computer program is executed by the central processing unit (CPU), the above functions defined in the method of the present invention are performed. The method, apparatus, and device of the present invention may be implemented in many ways. For example, the method, apparatus, and device of the present invention may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware.

[0159] The above sequence for the steps of the method is for illustration only, and the steps of the method of the present invention are not limited to the sequence specifically described above unless otherwise specifically stated.

[0160] In addition, in some embodiments, the present invention can also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0161] In addition, the parts of the above technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0162] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The safety management method of intelligent palletizing robot based on AI vision is characterized by: include: Acquire multimodal data of the palletizing operation area, perform 3D fusion perception of the palletizing operation area, and build a 3D model of the palletizing operation area through data synchronization and registration; Build a deep learning-based target detection algorithm to detect 3D models in real time and identify multiple types of targets; Based on the target categories and location information identified in the palletizing operation area, the palletizing operation area is dynamically divided into safety areas; According to the dynamic safety area division results, a mobile path planning strategy is constructed to control the palletizing robot to plan the mobile path in the palletizing work area; When the palletizing robot obtains the status of the palletized goods in the palletizing operation area, it detects the status of the palletized goods through the AI ​​vision model. When abnormal palletizing of goods is detected, the palletizing robot automatically triggers an alarm and takes safety measures.

2. The AI ​​vision-based intelligent palletizing robot safety management method according to claim 1 is characterized in that: The multimodal data includes image data of the palletizing operation area taken by a depth camera and laser point cloud data of the palletizing operation area scanned by a laser radar; the depth camera and the laser radar are respectively fixed on the top of the palletizing robot, and the depth camera and the laser radar are rotated on the top of the palletizing robot to obtain image data and laser point cloud data within a 360° range of the palletizing robot; A multi-view point cloud stitching algorithm is used to register and stitch the point cloud data collected by the lidar at different times, and the image data and point cloud data are synchronized in time; Downsample and voxel filter the fused point cloud to remove noise and outliers; The downsampling includes downsampling the spliced ​​point cloud and dividing the point cloud into uniform small voxels using a voxelized grid method; The voxel filtering includes filtering the point cloud using a voxel filtering algorithm, retaining the center of gravity point in each voxel based on a set voxel size, and obtaining uniform point cloud data; The color image obtained by the depth camera is registered with the point cloud, and the corresponding RGB color information is assigned to each point cloud point; The surface of the fused color point cloud is reconstructed using a greedy triangulation algorithm, and a triangular mesh is generated based on the spatial distribution and normal vector information of the point cloud. The reconstructed triangular mesh is simplified and smoothed to reduce the number of mesh faces and obtain a three-dimensional model of the palletizing operation area.

3. The AI ​​vision-based intelligent palletizing robot safety management method according to claim 2 is characterized in that: The YOLOv5 algorithm is used to perform real-time target detection on the 3D model of the palletizing operation area. The target detection results are fused with the LiDAR point cloud, and the 3D model coordinate position of the target is obtained through nearest neighbor matching. The nearest neighbor matching finds the best match by calculating the distance between the detected target and the point cloud set. The Deep SORT algorithm is used to achieve multi-target tracking; the Deep SORT algorithm uses the Hungarian algorithm to solve the target association problem and optimizes the association matrix by maximizing the similarity between the tracked target and the detected target; After identifying multiple target types based on the three-dimensional model, different risk impact coefficients are assigned to each type of target to perform dynamic safety area division.

4. The AI ​​vision-based intelligent palletizing robot safety management method according to claim 3 is characterized in that: The palletizing operation area is divided into uniform grids to form a grid map. In the grid map, the risk value of each grid is determined by the location and influence range of the target around the grid. The grid risk value calculation formula is: Where R(x,y) represents the risk value at the position (x,y), i is the index of the target object; (x i ,y i ) is the position coordinate of the i-th target object; α i is the risk impact coefficient of the ith target, indicating the degree of danger of the ith target; σ i The parameter representing the size of the influence range of the i-th target; In the grid map, the grid map is divided into a safe area and a dangerous area according to a safety threshold. The safe area includes a low-risk area, and the dangerous area includes a medium-risk area and a high-risk area; Set the safety thresholds TR1 and TR2 of the grid map, and TR2 > TR1; when R(x, y) < TR1, divide the grid area into a low-risk area; when TR1 ≤ R(x, y) ≤ TR2, divide the grid area into a medium-risk area; when R(x, y) > TR2, divide the grid area into a high-risk area; Set a fixed time interval Δt. Whenever Δt time has passed, recalculate the risk value of the grid area, and perform a safety area division according to the recalculated grid risk value.

5. The AI ​​vision-based intelligent palletizing robot safety management method according to claim 4 is characterized in that: Based on the divided safe area and dangerous area, use an optimized A* algorithm to perform a hierarchical movement path planning for the palletizing robot in the palletizing operation area; the hierarchical movement path planning includes a first-layer movement path planning, a second-layer movement path planning, and a third-layer movement path planning; The first-layer movement path planning includes using the A* algorithm to search for the optimal path within the low-risk area, setting the starting point and the target point as the input of the A* algorithm, and taking the low-risk area as the passable area; The second-layer movement path planning includes, if a feasible path cannot be found within the low-risk area, adding the medium-risk area to the passable area and using the A* algorithm to search for the path again; The third-layer movement path planning includes, if a feasible path cannot be found within the low- and medium-risk areas, adding the high-risk area to the passable area and using the A* algorithm to search for the path again; If a feasible path cannot be found within all risk areas, trigger a path planning failure alarm; The optimized A* algorithm includes locally optimizing the initial path obtained by the A* algorithm and smoothing the path using a Bezier curve. The optimization objective is: Among them, (x a ,y a ) is the coordinate of the ath sampling point on the path, is the coordinate of the hth control point, m and n are the number of sampling points and control points respectively, and λ is the smoothness weight coefficient; Whenever Δt time has passed, when the grid risk value of the grid map changes, re-plan the movement path of the palletizing robot through the optimized A* algorithm.

6. The AI ​​vision-based intelligent palletizing robot safety management method according to claim 5 is characterized in that: The image data is obtained based on the depth camera, and the stacked goods are identified. When the stacked goods are identified, the point cloud data of the laser radar scanning the stacked goods is obtained at the same time. The obtained point cloud data is marked as the cargo point cloud, and the normal vector and curvature of the cargo point cloud are calculated to obtain the normal vector of each point in the cargo point cloud. and curvature c p value; Set the normal vector of the cargo in normal state and the curvature reference value c ref , and the upper limit of the anomalous curvature c max ; Calculate the anomaly index A(p) of each point in the cargo point cloud: Among them, w n is the weight coefficient of the normal vector deviation, w c is the weight coefficient for high curvature; Set the abnormal threshold T a , used to determine whether a single point is abnormal; When the abnormal index A(p) of the point is greater than T a When A(p) is greater than T a When the percentage of points exceeds the percentage threshold TPa, the state of the cargo palletizing is determined to be abnormal.

7. The AI ​​vision-based intelligent palletizing robot safety management method according to claim 6 is characterized in that: According to the distribution and clustering of the abnormal points in the cargo point cloud, determine the types of cargo palletizing anomalies, and the types of cargo palletizing anomalies include tilt and scatter; Downsample the cargo point cloud to obtain uniformly distributed core points; calculate the covariance matrix C of the core points: Among them, h is the number of core points, p z is the coordinate of the zth core point; are the center coordinates of the core point; Perform eigenvalue decomposition on C: C = UΛU T ; Get the feature vector where U is the eigenvector matrix, Λ is the eigenvalue matrix, and U T is the transpose of the eigenvector matrix; The feature vector As the main direction of the stacked goods, it indicates the position of the stacked goods; calculates the center coordinates of the core point of the stacked goods as the location of the goods; According to the position and abnormality type of the goods, the corresponding correction strategy is searched in the predefined strategy library; if a matching correction strategy is found, the automatic correction process is entered; otherwise, an alarm is triggered; the correction strategy includes Directionally translate and rotate goods to adjust the abnormal stacking posture of goods; and target position, calculate the translation vector of the abnormally stacked goods; convert the correction strategy into motion control instructions for the stacking robot; After the correction is completed, scan the cargo to re-acquire the cargo point cloud data; and detect the state and pose of the palletized cargo after correction; if the abnormal state of the palletized cargo is eliminated, the palletizing robot continues to operate normally in the palletizing work area; otherwise, trigger an alarm.

8. An AI vision-based intelligent palletizing robot safety management system, which is used to implement the AI ​​vision-based intelligent palletizing robot safety management method according to any one of claims 1 to 7, characterized in that: Including: A data acquisition module, a target detection module, a region division module, a path planning module, and an anomaly correction module; The data acquisition module is used to acquire multi-modal data of the palletizing operation area, perform three-dimensional fusion perception on the palletizing operation area, and construct a three-dimensional model of the palletizing operation area through data synchronization and registration; The target detection module is used to construct a target detection algorithm based on deep learning, and perform real-time detection and recognition of multiple types of targets on the three-dimensional model; An area division module is used to divide the palletizing operation area into dynamic safety areas based on the target categories and location information identified in the palletizing operation area; The path planning module is used to build a mobile path planning strategy based on the dynamic safety area division results, and control the palletizing robot to plan the mobile path within the palletizing work area; The abnormality correction module is used to detect the status of the palletizing goods in the palletizing operation area through the AI ​​visual model when the palletizing robot obtains the status of the palletizing goods in the palletizing operation area. When abnormal palletizing goods are detected, the palletizing robot automatically takes safety measures and triggers an alarm.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; the processor executes the AI ​​vision-based intelligent palletizing robot safety management method according to any one of claims 1 to 7 by calling the computer program stored in the memory.

10. A storage medium, characterized in that: Instructions are stored, and when the instructions are run on a computer, the computer executes the AI ​​vision-based intelligent palletizing robot safety management method according to any one of claims 1 to 7.

Images