Manipulator warehousing intelligent unstacking and stacking control system and device based on machine vision

Abstract

The invention discloses a mechanical arm warehousing intelligent unstacking and stacking control system and device based on machine vision, and relates to the technical field of image data processing. The mechanical arm warehousing intelligent unstacking control system based on machine vision comprises a mechanical arm warehousing clamping monitoring module, a mechanical arm clamping effect quantification module and a mechanical arm clamping force control module. According to the method, the influence condition on point cloud data is monitored when a manipulator performs intelligent unstacking on a specified soft packaging box stack during warehousing, whether flexible packaging influence compensation measures are taken or not is selected, and then the clamping effect of the manipulator for grabbing the specified soft packaging box stack is quantified to judge whether to perform clamping point correction adjustment or not; and finally, qualification judgment is conducted on the grabbing success rate to determine whether a clamping force control adjusting strategy is adopted or not, the control accuracy of the mechanical arm unstacking and stacking is improved, and the problem that in the prior art, soft packaging box materials interfere with the machine vision scanning result, and consequently the control accuracy of the mechanical arm unstacking and stacking is low is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, in particular to a mechanical hand warehouse-in intelligent unstacking control system and device based on machine vision. BACKGROUND

[0002] The existing intelligent stacking / unstacking system based on machine vision is a system integrating multiple technologies, including 3D (Three-Dimensional) visual perception (point cloud combined with depth map), AI (Artificial Intelligence) intelligent recognition and pose estimation, grasp point planning combined with path optimization, stacking / unstacking strategy and task management system, force control combined with fault tolerance mechanism, and system control integration combined with industrial communication. Through 3D visual perception, point cloud and image of the stacking area are collected to reconstruct the surface of the goods, accurately locate the center, pose and grasping surface of the object, and identify the type of goods (box, package, bag). Through AI intelligent recognition and pose estimation, the optimal (anti-shielding, high stability) is selected from multiple possible grasping surfaces, and the object with irregular shape, soft packaging and reflective material can also accurately identify the clamping point, and automatically adjust the pose of the end effector of the mechanical hand. Through grasp point planning combined with path optimization, the path of the mechanical arm is dynamically planned according to the grasp point and target point, obstacles (stacked goods, conveyor belt, shelf, etc.) are avoided, and the compliant motion of the mechanical hand is controlled to achieve accurate grasping and placing. The stacking / unstacking strategy and task management system generates the optimal unstacking sequence (such as from top to bottom, from outside to inside), and manages the FIFO (First In, First Out) / LIFO (Last In, First Out) sequence of warehouse-in / out, and also tracks the state of the goods: position, whether it has been moved, and which shelf position it is placed in. The force control combined with the fault tolerance mechanism not only avoids deformation or falling of soft packaging or irregular objects during grasping, but also implements the "just contact and stop" mechanism to avoid collision, real-time perception of clamping failure / slip, and triggering of the re-grasping logic. The system control integration combined with industrial communication is used for interfacing with the conveying line, vertical warehouse, warehouse WMS (Warehouse Management System) / WCS (Warehouse Control System) system, and multi-robot cooperative scheduling.

[0003] For example, the Chinese invention patent with the publication number CN110322457B discloses a 2D and 3D vision combined unstacking method, which comprises a stereo vision device and an intermediate camera. The unstacking method comprises the following steps: step 1, using the stereo vision device to obtain 3D point cloud data of the stack, and determining the height distribution map of the stack according to the 3D point cloud data; step 2, using the intermediate camera to obtain a 2D image of the upper surface of the stack, and using an edge detection algorithm to determine the edge line of the object in the 2D image; step 3, determining the center coordinates of the object according to the positional relationship of the edge line in the 2D image, and generating the grabbing information of the mechanical hand grabbing the object according to the center coordinates, the longitudinal height of the center coordinates in the height distribution map, and the 3D point cloud data.

[0004] For example, the Chinese invention patent with the publication number CN111524184B discloses an intelligent unstacking method and system based on 3D vision, which comprises the following steps: S1, collecting image information of the box stack and preprocessing; S2, obtaining the top layer region of the stacked box according to the preprocessed image information and setting the unstacking sequence; S3, obtaining a candidate box containing at least one box according to the set unstacking sequence, and respectively using a traditional image processing method and a deep learning method to judge the placement posture of the to-be-grabbed box; S4, determining the placement posture of the to-be-grabbed box in the candidate box by comprehensively considering the judgment results of the two methods; S5, calculating the rotation angle of the grabbing manipulator, and controlling the manipulator to grab the box and unstack.

[0005] The above-mentioned technology at least has the following technical problems: In the prior art, 3D vision can provide shape and posture information, but for soft packaging, easily deformable or uneven surface objects, the gripper grabbing is still prone to failure. Since the surface of the soft packaging object may have irregular shapes such as wrinkles, bulges and deformation, the generated point cloud data is not smooth and continuous, the point cloud "jumps" or is sparse, and the given surface normal and grabbing position are unreliable. In addition, after the soft object is placed, it will deform due to force, and the "posture" seen by the vision system may be a transient state. The center position, normal vector and edge information identified cannot truly reflect its grabbability, resulting in that the holding force cannot be accurately controlled or the grabbing point is not suitable, and the soft packaging box material interferes with the machine vision scanning result, causing low control accuracy of the manipulator. SUMMARY

[0006] In order to solve the technical problem of low control accuracy of the manipulator caused by the interference of the soft packaging box material with the machine vision scanning result in the prior art, the embodiments of the present application provide a manipulator warehouse entry intelligent unstacking control system and device based on machine vision. The technical scheme is as follows: In one aspect, a machine vision-based robot warehouse entry intelligent de-palletizing control system is provided, which comprises: a robot warehouse entry clamping monitoring module, a robot clamping effect quantification module, and a robot clamping force control module; wherein the robot warehouse entry clamping monitoring module is used to monitor the influence of a specified soft packaging box stack on point cloud data in real time when the robot warehouse entry intelligently de-palletizes the specified soft packaging box stack, to select whether to take soft packaging influence compensation measures; the robot clamping effect quantification module is used to quantify the clamping effect of the robot in grabbing the specified soft packaging box stack, to determine whether to perform corresponding clamping point correction adjustment, which is used to improve the accuracy of the robot in selecting clamping points for the specified soft packaging box stack; and the robot clamping force control module is used to determine whether to take clamping force control adjustment strategies to improve the accuracy of the robot in applying clamping force to the specified soft packaging box stack, by judging the success rate of grabbing within a preset grabbing period.

[0007] In another aspect, a machine vision-based robot warehouse entry intelligent de-palletizing control device is provided, which applies the machine vision-based robot warehouse entry intelligent de-palletizing control system, and comprises: a visual perception structure, a robot and clamping structure, and a control decision module; the visual perception structure mainly comprises an industrial camera installed at the end of the robot, a light source system for assisting visual recognition, and a visual computing unit for image processing and point cloud analysis; the robot and clamping structure is a multi-degree-of-freedom industrial robot, which is used to support de-palletizing, rotating, and moving, is equipped with a grabbing detection sensor to determine whether grabbing is successful, and is equipped with various clamps, specifically including a vacuum chuck clamp, an electric clamping jaw, and a pneumatic clamping jaw; and the control decision module is used to run a visual recognition and grabbing path planning algorithm, and is matched with the robot body to control the action of the robot arm.

[0008] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects: 1. When the robot warehouse entry intelligently de-palletizes a specified soft packaging box stack, the influence of the specified soft packaging box stack on point cloud data is monitored in real time to select whether to take soft packaging influence compensation measures, which reduces the interference degree of soft packaging characteristics on mechanical vision, then the clamping effect of the robot in grabbing the specified soft packaging box stack is quantified to determine whether to perform corresponding clamping point correction adjustment, which ensures that the robot can more accurately select clamping points, and finally the success rate of grabbing within a preset grabbing period is judged for eligibility to determine whether to take clamping force control adjustment strategies, so as to apply more accurate clamping force to the specified soft packaging box stack, thereby improving the accuracy of robot de-palletizing control.

[0009] 2、Firstly, the soft packaging point cloud influence proportion and the preset depth value are obtained from the preset database, which provides a data basis for subsequent data analysis, and the corresponding depth deviation influence quantity is obtained by quantifying the deviation degree of the point cloud Z direction depth value from the preset depth value, which initially quantifies the deviation degree of the point cloud Z direction depth value, then the depth deviation influence quantity, point cloud curvature, point cloud jumping frequency and point cloud overlap rate are coupled after weighting operation with the corresponding soft packaging point cloud influence proportion, and finally the point cloud data-pinch influence quantity is obtained, which quantifies the interference degree of the soft packaging characteristics on the machine vision recognition, which is helpful to take corresponding measures in time to improve the accuracy of machine vision recognition.

[0010] 3、Firstly, the pinch effect analysis data is obtained in real time and data normalization processing is performed to ensure the data dimension is uniform, then the pinch effect analysis proportion is extracted from the preset database to accurately analyze the influence degree of each pinch effect analysis data on the pinch effect quantization value, finally the pinch effect quantization value is obtained by coupling the pinch effect analysis data and the corresponding pinch effect analysis proportion after weighting operation, thereby accurately quantifying the accuracy degree of the pinch point of the specified soft packaging box stack selected by the manipulator, and then corresponding measures are taken in time to ensure the accuracy of the selected pinch point. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 is a structural schematic diagram of the manipulator warehouse-in intelligent unstacking control system based on machine vision provided by the embodiments of the present application; Figure 2 is a process schematic diagram of the embodiments of the present application for selecting whether to take soft packaging influence compensation measures; Figure 3 is a process schematic diagram of the embodiments of the present application for judging whether to take corresponding pinch point correction adjustment; Figure 4 is an interface schematic diagram of the grasping parameter control of the manipulator monitoring in the intelligent unstacking platform provided by the embodiments of the present application; Figure 5 is an interface schematic diagram of the gripping force adjustment and limitation setting of the manipulator monitoring in the intelligent unstacking platform provided by the embodiments of the present application. DETAILED DESCRIPTION

[0013] The technical solutions in the present application will be described below in conjunction with the drawings.

[0014] In the embodiments of the present application, the words such as "example", "for example" are used to represent an example, illustration, or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0015] In the embodiments of the present application, sometimes the subscript such as W1 may be written in the form of non-subscript such as W1, and the meanings expressed thereby are consistent when the difference is not emphasized.

[0016] In order to make the technical problems, technical schemes and advantages to be solved by the present application more clear, the following will be described in detail in combination with the drawings and specific embodiments.

[0017] The embodiments of the present application provide a manipulator warehousing intelligent unstacking control system based on machine vision. Figure 1 The embodiments of the present application provide a structure schematic diagram of the manipulator warehousing intelligent unstacking control system based on machine vision, and the processing flow of the system includes the following modules: a manipulator warehousing clamping monitoring module, a manipulator clamping effect quantification module and a manipulator clamping force control module.

[0018] The manipulator warehousing clamping monitoring module is used to monitor the influence of the specified soft packaging box stack on the point cloud data in real time when the manipulator warehousing intelligently unstacks the specified soft packaging box stack, so as to select whether to take soft packaging influence compensation measures, and the soft packaging influence compensation measures are used to improve the accuracy of the machine vision in identifying the specified soft packaging box stack.

[0019] The manipulator clamping effect quantification module is used to quantify the clamping effect of the manipulator in grabbing the specified soft packaging box stack, so as to determine whether to perform corresponding clamping point correction adjustment, which realizes the improvement of the accuracy of the grabbing action from the source to reduce the clamping failure rate, and the clamping point correction adjustment is used to improve the accuracy of the selection of the clamping point of the manipulator for the specified soft packaging box stack.

[0020] The manipulator clamping force control module is used to make a qualification judgment on the success rate of grabbing within a preset grabbing period to determine whether to take a clamping force control adjustment strategy, which realizes intelligent matching of the optimal clamping force, prevents insufficient or excessive clamping force from causing grabbing failure or damage to the goods, and the clamping force control adjustment strategy is used to improve the accuracy of the clamping force applied by the manipulator to the specified soft packaging box stack.

[0021] In this embodiment, from the effect evaluation of the clamping monitoring to the clamping force adjustment, the system connects the closed-loop control process of the robot grasping decision and execution, effectively improving the efficiency and automation level of the de-palletizing operation; and for the characteristics of soft packaging box stacks that are easy to deform and deviate, through point cloud data monitoring and influence compensation mechanism, real-time response to environmental disturbance and physical changes is realized, ensuring the stability of the robot clamping. Through the cooperative work of each module, the operation behavior is continuously optimized in the two key dimensions of clamping point and clamping force, which is beneficial to improve the success rate of robot grasping within the preset grasping period, improve the operation continuity and system stability, and ensure the accuracy of robot de-palletizing control.

[0022] It should be noted that before designing the robot warehouse entry intelligent de-palletizing control system based on machine vision, professional and technical personnel usually prefer to build a preset database of preset parameters for supporting the execution of various control strategies. The database collects a plurality of key control parameters, including: point cloud-clamping decision quantity, soft packaging point cloud influence ratio, preset depth value, camera exposure time adjustment amount, clamping effect decision value, clamping effect analysis proportion, clamping point correction amount, preset success rate, preset clamping force adjustment amount, and clamping force adjustment influence rate, etc. These parameters are pre-set by engineers with professional technical background according to the algorithm model used and the actual hardware configuration conditions. This database provides key data support for subsequent data uploading, storage optimization, screening and decision-making automation processes.

[0023] As shown in Figure 2 The flowchart of selecting whether to take soft packaging influence compensation measures provided by the embodiments of the present application is shown in the figure. The specific logic is as follows: first, the soft packaging point cloud influence data of the specified soft packaging box stack on the point cloud data is obtained in real time; then the point cloud data-clamping influence quantity is obtained based on the soft packaging point cloud influence data, and the point cloud data-clamping influence quantity is compared with the point cloud-clamping decision quantity. If the point cloud data-clamping influence quantity is less than the point cloud-clamping decision quantity, the specified soft packaging box stack is continued to be grabbed, otherwise the difference between the point cloud data-clamping influence quantity and the point cloud-clamping decision quantity is recorded as the clamping influence difference quantity, the clamping influence difference quantity is mapped in the preset database to obtain the camera exposure time adjustment amount, and the initial camera exposure time of the machine vision is multiplied based on the camera exposure time adjustment amount to obtain the optimized camera exposure time. The specified soft packaging box stack is rescanned through the optimized camera exposure time until the point cloud data-clamping influence quantity is less than the point cloud-clamping decision quantity. Finally, the rescan times are obtained in real time and compared with the preset times. If the rescan times are lower than the preset times, the specified soft packaging box stack is continued to be grabbed, otherwise the robot continues to grab the specified soft packaging box stack and issues a robot grasping abnormality warning. Through the above process, the accuracy of robot clamping point identification is improved, and the efficiency of robot operation is improved.

[0024] Further, the influence of the specified soft packaging box stack on the point cloud data is monitored in real time to determine whether to take soft packaging influence compensation measures, and the specific steps are as follows: Step one, real-time acquisition of soft packaging point cloud influence data of the specified soft packaging box stack on the point cloud data, including point cloud curvature, point cloud Z direction depth value, point cloud jumping frequency and point cloud overlap rate.

[0025] Specifically, the point cloud curvature is obtained by point cloud normal estimation, such as NormalEstimation of PCL (Programmable Logic Controller System) library, combined with local curvature calculation (such as eigenvalue method); the 3D camera directly outputs point cloud coordinates (x, y, z), where z is the depth value of the point cloud Z direction depth value; the point cloud jumping frequency is the ratio of the number of abnormal points to the total number of points; the point cloud overlap rate represents the overlap degree of objects in space (especially projection), which is obtained by PCL voxel grid (3D).

[0026] Step two, based on the soft packaging point cloud influence data, the point cloud data-pickup influence quantity is obtained, and the point cloud data-pickup influence quantity is compared with the point cloud-pickup judgment quantity obtained from the preset database. If the point cloud data-pickup influence quantity is less than the point cloud-pickup judgment quantity, the specified soft packaging box stack is continued to be grabbed, otherwise the soft packaging influence compensation measures are taken, and the point cloud data-pickup influence quantity is used to quantify the interference degree of soft packaging characteristics on machine vision recognition.

[0027] Specifically, the point cloud-pickup judgment quantity is obtained from the preset database, and the soft packaging point cloud influence data of the historical data with qualified pickup effect is input into the specific limit expression of the point cloud data-pickup influence quantity by the preset staff, to obtain a data set about the point cloud data-pickup influence quantity, and the average value of the data set is recorded as the point cloud-pickup judgment quantity.

[0028] In this embodiment, based on the multi-dimensional data such as point cloud curvature, point cloud Z direction depth value, point cloud jumping frequency and point cloud overlap rate, the influence of soft packaging characteristics on point cloud data quality is more comprehensively evaluated, which helps to improve the stability and accuracy of the pickup point recognition; and by real-time calculation of the point cloud data-pickup influence quantity and comparison with the corresponding point cloud-pickup judgment quantity, intelligent response under different soft packaging situations is realized, and the adaptability of the system in complex material environment is enhanced; and when the influence of soft packaging characteristics on machine vision exceeds the point cloud-pickup judgment quantity, the compensation mechanism is automatically triggered, which is helpful to avoid mis-grabbing or pickup failure caused by point cloud interference, and improves the efficiency and reliability of the whole operation of the mechanical hand.

[0029] As a further embodiment, the specific acquisition method of the point cloud data-pinch influence quantity is as follows: First, the soft package point cloud influence proportion and the preset depth value are obtained from the preset database. The soft package point cloud influence proportion includes point cloud curvature influence proportion, point cloud Z direction depth value influence proportion, point cloud jumping frequency influence proportion, and point cloud overlap rate influence proportion.

[0030] It should be noted that the system will input the collected soft package point cloud influence data into the trained soft package point cloud influence mapping model to output the corresponding soft package point cloud influence proportion, that is, the influence degree of this kind of data on the point cloud data-pinch influence quantity. The mapping model is constructed based on the logistic regression algorithm and relies on the least square method criterion to complete the training process through the statsmodels framework. The training data used by the model comes from the soft package point cloud influence features collected in the historical period and the reference soft package point cloud influence proportion set by professional technical personnel according to the experience rule, which is used to fit the mapping relationship between the two, so as to accurately reflect the influence degree of the soft package point cloud characteristics on the actual pinch task; in addition, the preset depth value is set by the preset staff based on the change rule of historical data and the operation standard and stored in the preset database.

[0031] Then, the point cloud Z direction depth value and the preset depth value are quantified to obtain the corresponding depth deviation influence quantity, wherein the deviation degree quantification represents the ratio operation of the point cloud Z direction depth value and the preset depth value.

[0032] Finally, the depth deviation influence quantity, point cloud curvature, point cloud jumping frequency, and point cloud overlap rate are coupled after the weighting operation of the corresponding soft package point cloud influence proportion, that is, the corresponding data and the corresponding influence proportion are multiplied and summed to obtain the point cloud data-pinch influence quantity.

[0033] In the embodiment, the algorithm combines the depth deviation influence quantity, the point cloud curvature, the point cloud jumping frequency and the point cloud overlap rate with the corresponding soft packaging point cloud influence proportion to analyze the point cloud data-pinch influence quantity, wherein, as the point cloud curvature, the point cloud jumping frequency and the point cloud overlap rate increase, the higher the influence degree of the soft packaging characteristics on the point cloud data, resulting in the decrease of the accuracy of the specified soft packaging box pile grasped by the manipulator, and the greater the depth deviation influence quantity, the higher the deviation degree of the point cloud Z direction depth value from the preset depth value, and as the depth deviation influence quantity increases, the corresponding point cloud data-pinch influence quantity is greater, indicating that the higher the influence degree of the soft packaging characteristics on the point cloud data, the higher the pinch influence degree on the manipulator; the accurate quantification of the influence degree of the soft packaging characteristics on the point cloud data by the point cloud data-pinch influence quantity helps to determine in time based on the point cloud data-pinch influence quantity and take corresponding optimization measures, thereby improving the accuracy of the control of the specified soft packaging box pile grasped by the manipulator.

[0034] For example, assuming that the point cloud curvature influence proportion is 0.4, the point cloud Z direction depth value influence proportion is 0.3, the point cloud jumping frequency influence proportion is 0.2, and the point cloud overlap rate influence proportion is 0.1, and the preset depth value is 1.2 m, when the point cloud curvature is 0.01, the point cloud Z direction depth value is 1.2 m, the point cloud jumping frequency is 0.5 Hz, and the point cloud overlap rate decreases from 0.85 to 0.3, indicating that the coincidence degree of two frames or two perspective point clouds decreases, which may be caused by scanning angle deviation, increased shielding or sensor movement, target “deformation” or incomplete point cloud, thereby affecting the grasping of the manipulator; when the point cloud curvature is 0.02, the point cloud Z direction depth value increases from 1.2 m to 3 m, the point cloud jumping frequency is 1 Hz, and the point cloud overlap rate is 0.9, indicating that the target is away from the sensor, the point cloud density decreases, the spatial resolution deteriorates, the edge information is not clear or the details are lost, resulting in inaccurate analysis of the point cloud data, thereby affecting the grasping control of the manipulator; when the point cloud curvature increases from 0.01 to 0.12, the point cloud Z direction depth value is 1.5 m, the point cloud jumping frequency is 0.3 Hz, and the point cloud overlap rate is fixed at 0.85, indicating that the target surface becomes more curved or more sharp, which may identify features such as edges, notches and holes, which is helpful for geometric segmentation or edge extraction, thereby affecting the grasping effect of the manipulator; when the point cloud jumping frequency increases from 0.2 Hz to 3 Hz, the point cloud curvature is 0.02, the point cloud Z direction depth value is 1.2 m, and the point cloud overlap rate is fixed at 0.9, indicating that the point cloud frame changes dramatically, which may be caused by sensor jitter, target rapid motion or software registration error, affecting the dynamic tracking stability of the target, thereby affecting the grasping control of the manipulator.

[0035] It should be noted that the specific process of taking the soft packaging influence compensation measures is as follows: A1, the difference between the clamping influence quantity and the point cloud-clamping determination quantity is taken as a clamping influence difference quantity, and the clamping influence difference quantity is mapped in a preset database to obtain a camera exposure time adjustment quantity.

[0036] Specifically, the clamping influence difference quantity is input into a camera exposure time mapping table that has been constructed in the preset database, and a corresponding camera exposure time adjustment quantity is output. The camera exposure time mapping table is used to fit the mapping relationship between the clamping influence difference quantity and the camera exposure time adjustment quantity, and the value range of the camera exposure time adjustment quantity is between 0 and 1.

[0037] A2, the initial camera exposure time of machine vision is multiplied by the camera exposure time adjustment quantity to obtain an optimized camera exposure time.

[0038] A3, the specified soft packaging box stack is rescanned by using the obtained optimized camera exposure time until the point cloud data-clamping influence quantity is less than the point cloud-clamping determination quantity.

[0039] A4, the number of rescanning times is obtained in real time and compared with a preset number of times to prevent the system from falling into an invalid recognition cycle and to ensure the overall operation rhythm. If the number of rescanning times is less than the preset number of times, the specified soft packaging box stack is continuously grabbed, otherwise the specified soft packaging box stack is stopped from being continuously grabbed by the manipulator, and a manipulator grabbing abnormality early warning is issued, which is beneficial to prevent system misoperation or equipment damage, and enhances the safety and stability of system operation; wherein the preset number of times is set by a preset worker based on historical data, work experience and operation standards, and is stored in the preset database in advance for direct extraction when used.

[0040] In the embodiment, by calculating the clamping influence difference quantity and mapping it into the camera exposure time adjustment quantity, the camera exposure parameter is accurately controlled, so that high-quality point cloud data can be obtained under the influence of light interference or soft packaging material, and the visual recognition accuracy is improved. And by constructing the closed-loop process of "recognition->judgment->adjustment exposure->re-identification", it is ensured that the system can automatically optimize the perception condition according to real-time feedback, and improve the adaptability to different soft packaging scenes. Moreover, by optimizing the visual perception, it is ensured that the clamping decision is based on reliable data, which helps to reduce the abnormal situations such as misjudgment, misgrabbing and missed grabbing caused by soft packaging, and improves the operation success rate of the manipulator.

[0041] As Figure 3As shown, it is the flowchart of judging whether to carry out corresponding correction adjustment of the clamping point provided by the embodiment of the application, and the specific logic is: first, the clamping effect analysis data of the specified soft packaging box stack is analyzed in real time to obtain the clamping effect quantitative value, then the clamping effect quantitative value is judged with the clamping effect judgment value, if the clamping effect quantitative value is greater than the clamping effect judgment value, the specified soft packaging box stack is continuously grabbed, otherwise the difference value of the clamping effect quantitative value and the clamping effect judgment value is obtained by difference operation, then the clamping effect difference value is mapped in the preset database to obtain the clamping point correction amount, finally the mechanical hand deviation data is compensated through the clamping point correction amount to improve the accuracy of the mechanical hand in selecting the clamping point of the specified soft packaging box stack. Through the above process, not only the effectiveness of the clamping point selected by the mechanical hand is more accurately determined, but also the self-adaptive ability of the mechanical hand in the soft packaging application scenario is improved.

[0042] As shown in Figure 4 It is the interface diagram of the grabbing parameter control of the mechanical hand control in the intelligent unstacking platform provided by the embodiment of the application, wherein the left side of the interface is the navigation bar of the platform, specifically including the main control panel, the mechanical hand monitoring, the mechanical arm monitoring, the tray monitoring, the task configuration, the system configuration and the alarm and log, there is action control display at the top of the interface, including grabbing test, placing test, single step execution, cycle execution and emergency stop, and the central part of the interface contains the dimension display interface of the grabbing parameter control, specifically the jaw setting, the clamping point setting, the clamping point correction and the candidate grabbing point option. Through the interface, not only the running condition of the mechanical hand is known by the mechanical hand user in real time, but also the control parameters of the mechanical hand are adjusted by the mechanical hand user to improve the accuracy of the unstacking control of the mechanical hand.

[0043] Further, the clamping effect of the mechanical hand grabbing the specified soft packaging box stack is quantified to judge whether to carry out corresponding clamping point correction adjustment, and the specific steps are as follows: S1, the clamping effect analysis data of the specified soft packaging box stack is analyzed in real time to obtain the clamping effect quantitative value to quantify the accuracy of the clamping point of the specified soft packaging box stack selected by the mechanical hand.

[0044] S2, the clamping effect quantitative value is judged with the clamping effect judgment value obtained from the preset database, if the clamping effect quantitative value is greater than the clamping effect judgment value, the specified soft packaging box stack is continuously grabbed, otherwise corresponding clamping point correction adjustment is carried out.

[0045] Specifically, the clamping effect analysis data corresponding to the clamping effect qualified condition in the historical data is input into the specific expression of the clamping effect quantitative value to obtain the data set about the clamping effect quantitative value, and the mean value operation is performed on the data set to obtain the clamping effect judgment value, and the value is stored in the preset database in advance.

[0046] In this embodiment, by real-time quantitative evaluation of the clamping effect, the effectiveness of the manipulator selecting the clamping point is more accurately determined, and deviation clamping caused by soft packaging material deformation or visual error is avoided. At the same time, the system automatically determines whether clamping point correction is needed according to the clamping result, forming a self-loop optimization mechanism of "clamping → evaluation → correction → re-clamping", effectively improving the overall clamping reliability and intelligent level. At the same time, when the clamping effect is not up to standard, the system can quickly respond and actively adjust the clamping strategy to adapt to the uncertainty of soft packaging materials, improve the fault tolerance and stability of the manipulator in complex environments, and avoid repeated grabbing or misoperation caused by improper selection of clamping points. While ensuring the success rate of clamping, it reduces invalid operation and improves the overall line operation efficiency.

[0047] As a further solution, the specific way to obtain the clamping effect quantitative value is as follows: First, real-time clamping effect analysis data is obtained and data normalization processing is performed. The clamping effect analysis data includes clamping point-point cloud density, point cloud noise point number and point cloud void ratio. The data normalization processing is specifically a minimum-maximum normalization method.

[0048] It should be noted that the clamping point-point cloud density is the ratio of the number of point clouds in a fixed size neighborhood range centered on the clamping point to the corresponding object volume; the point cloud noise point number is obtained by statistical filtering; the target region point cloud is projected onto a two-dimensional grid plane (such as XY or XZ plane), then the space is divided into fixed size grids and marked with point cloud and non-point cloud, and the ratio of the number of non-point grids to the total number of grids is the point cloud void ratio.

[0049] Second, the clamping effect analysis proportion is extracted from the preset database. The clamping effect analysis proportion includes clamping point-point cloud density analysis proportion, point cloud noise point number analysis proportion and point cloud void ratio analysis proportion.

[0050] It should be noted that the system inputs the clamping effect analysis data into the trained clamping effect analysis mapping model to output the corresponding clamping effect analysis proportion, i.e. the influence degree of each clamping effect analysis data on the clamping effect quantitative value. The clamping effect analysis mapping model is constructed based on the logistic regression algorithm, uses cross-entropy loss function as the optimization criterion, and is trained through the scikit-learn framework. The training data comes from the clamping effect analysis data collected in the historical stage and the clamping effect analysis proportion set by professionals according to experience, aiming to fit the corresponding relationship between the clamping effect analysis data and the clamping effect quantitative value, so as to realize accurate modeling and evaluation of the influence degree.

[0051] In the third step, the clamping effect analysis data and the corresponding clamping effect analysis proportion are coupled after weighting operation, that is, the product of the clamping point-point cloud density and the clamping point-point cloud density analysis proportion, the product of the reciprocal of the point cloud noise point number and the point cloud noise point number analysis proportion, and the product of the reciprocal of the point cloud hollow proportion and the point cloud hollow proportion analysis proportion are obtained respectively, and then the three are added to obtain the clamping effect quantitative value. In order to avoid the point cloud noise point number and the point cloud hollow proportion being 0, the reciprocal processing is performed by adding 1 to the denominator, that is, when the corresponding data approaches 0, the entire reciprocal approaches 1, and such processing ensures the effectiveness of the fraction.

[0052] In the present embodiment, the algorithm combines the clamping effect analysis data and the corresponding clamping effect analysis proportion to obtain the clamping effect quantitative value. In the formula, when the clamping point-point cloud density increases, it indicates that the deviation degree of the clamping point selected by the manipulator is lower, the clamping point is more accurate, the corresponding clamping effect is better, and the clamping effect quantitative value increases accordingly. When the point cloud noise point number and the point cloud hollow proportion increase, it indicates that the selection of the clamping point deviates more, the corresponding clamping effect is worse, and the clamping effect quantitative value decreases accordingly. Through analysis of the clamping effect quantitative value, the accuracy of the clamping point selected by the manipulator for the specified soft packaging box stack is more accurately quantified, and the clamping effect of the manipulator is also reflected, which helps to take corresponding optimization measures for the clamping point in time, thereby improving the accuracy of the clamping point selected by the manipulator for the specified soft packaging box stack, and further improving the accuracy of the grasping control of the manipulator for the specified soft packaging box stack, and ensuring the grasping effect of the manipulator.

[0053] For example, when the clamping point-point cloud density decreases from 850 points per cubic centimeter to 200 points per cubic centimeter, the point cloud noise point number remains 200, and the point cloud hollow proportion remains 5%, it indicates that the clamping point cloud is sparse, the clamping point may be incomplete, the normal is inaccurate, the clamping position is unstable or fails, and the corresponding clamping effect may be worse. When the clamping point-point cloud density is fixed at 800 points per cubic centimeter, the point cloud noise point number increases from 100 to 1000, and the point cloud hollow proportion remains 3%, it indicates that the grasping point may be misjudged as noise, the clamp is biased or empty, the algorithm misdetects "false target" or surface fluctuation, and the clamping effect is poor. When the clamping point-point cloud density is stable at 900 points per cubic centimeter, the point cloud noise point number is stable at 150, and the point cloud hollow proportion increases from 3% to 18%, it indicates that the available area decreases, the clamping point may be in the missing area, the clamping fails or is unstable, the system may not identify the grasping surface, and the grasping effect is affected.

[0054] It should be noted that the specific content of the corresponding clamping point correction adjustment is: the clamping effect difference value is obtained by difference operation of the clamping effect quantitative value and the clamping effect determination value, the clamping point correction amount is obtained by mapping the clamping effect difference value in the preset database, and the mechanical hand deviation data is compensated by the clamping point correction amount to improve the accuracy of the mechanical hand in selecting the clamping point of the specified soft packaging box stack. The mechanical hand deviation data includes displacement error and torque error.

[0055] Wherein, the system inputs the clamping effect difference value into the clamping point correction mapping model constructed in advance in the preset database to output the corresponding clamping point correction amount, which is used to quantify the adjustment amplitude of the current clamping point. The clamping point correction mapping model is constructed based on linear regression algorithm, adopts least square method as optimization criterion, and is trained relying on scikit-learn framework. The model training data comes from the clamping effect difference value recorded in the history stage and the clamping point correction amount set by professionals according to experience, aiming to fit the mapping relationship between the clamping effect difference value and the clamping point correction amount, so as to realize accurate modeling and evaluation of the clamping point adjustment degree; in addition, the mechanical hand deviation data is compensated by multiplying the clamping point correction amount by the mechanical hand deviation data.

[0056] In this embodiment, by calculating the clamping effect difference value and mapping it into the clamping point correction amount, the displacement error and torque error of the mechanical hand in actual operation are more accurately compensated, which fundamentally improves the accuracy of clamping point selection; and the clamping effect quantitative value is compared with the preset determination value and difference value is calculated, so that the error compensation forms a closed-loop quantitative mechanism based on actual effect, which enhances the adaptive adjustment ability of the system; at the same time, by considering the displacement error and torque error, the clamping path and clamping posture are comprehensively optimized, which helps to deal with the complex grasping challenges of soft packaging articles caused by gravity center offset, irregular surface and other factors; and by compensating the deviation data to improve the end execution action, the slipping, deviation or clamping failure in the clamping process is reduced, and the stability and success rate of the mechanical hand grasping are improved.

[0057] As Figure 5Fig. 1 is an interface diagram of the gripper force adjustment and limit setting of the manipulator control in the intelligent unstacking platform provided by the embodiments of the present application, wherein the left side of the interface is the navigation bar of the platform, specifically including the main control panel, manipulator monitoring, mechanical arm monitoring, tray monitoring, task configuration, system configuration, and alarm and log, and the central part of the interface includes gripper force setting, stage force threshold setting, and soft package gripper force self-adaptive setting. Specifically, the gripper force setting includes maximum gripper force limit and minimum gripper force limit, the stage force threshold setting includes contact stage, lifting stage, moving stage, and placing stage, and the soft package gripper force self-adaptive setting includes soft package-gripper force adjustment amount, gripper point-gripper force adjustment amount, and gripper success rate-gripper force adjustment amount. The control situation of the gripper force is more intuitively displayed through the interface, so that the corresponding parameters are adjusted based on the corresponding control situation, and the accuracy of the gripper force control is improved.

[0058] Further, the success rate of the gripping in the preset gripping period is judged for eligibility to determine whether to adopt the gripper force control adjustment strategy, and the specific process is as follows: obtaining the gripping success rate of the manipulator on the specified soft package box pile after the gripper point correction adjustment in the preset gripping period; comparing the gripping success rate with the preset success rate set in the preset database, if the gripping success rate is less than the preset success rate, adopting the gripper force control adjustment strategy through the preset gripper force adjustment amount, otherwise, continuing to grip the specified soft package box pile, realizing the intelligent triggering of the gripper force control strategy, and effectively guaranteeing the stability and reliability of the gripping action of the manipulator; the preset success rate is set by the preset worker based on the gripping requirements and stored in the preset database.

[0059] The preset gripper force adjustment amount includes a first gripper force adjustment amount, a second gripper force adjustment amount, and a third gripper force adjustment amount; the first gripper force adjustment amount is a result obtained by mapping based on the point cloud data-gripper influence amount in the preset database; the second gripper force adjustment amount is a result obtained by mapping based on the gripper effect quantitative value in the preset database; and the third gripper force adjustment amount is a result obtained by mapping based on the gripping success rate difference amount in the preset database, and the gripping success rate difference amount is the difference between the preset success rate and the gripping success rate.

[0060] Specifically, the preset clamping force adjustment amount is obtained based on a clamping force adjustment mapping model in a preset database, and the clamping force adjustment mapping model includes a first clamping force adjustment mapping model, a second clamping force adjustment mapping model, and a third clamping force adjustment mapping model; the clamping force adjustment mapping model is constructed based on a linear regression algorithm, adopts a least square method as an optimization criterion, and is trained relying on a scikit-learn framework. The model training data are respectively from point cloud data-clamping influence amount, clamping effect quantitative value, and difference amount of success rate of grabbing obtained based on historical data, and the first clamping force adjustment amount, the second clamping force adjustment amount, and the third clamping force adjustment amount set by a professional according to experience, aiming to fit the mapping relationship between the point cloud data-clamping influence amount, the clamping effect quantitative value, the difference amount of success rate of grabbing, and the corresponding preset clamping force adjustment amount, so as to realize accurate modeling and evaluation of the clamping force adjustment degree.

[0061] It needs to be explained that the specific content of the clamping force control adjustment strategy is as follows: B1, obtaining a clamping force adjustment influence rate corresponding to the preset clamping force adjustment amount in the preset database, the clamping force adjustment influence rate including a first clamping force influence rate, a second clamping force influence rate, and a third clamping force influence rate.

[0062] It needs to be pointed out that the preset clamping force adjustment amount is input into a clamping force adjustment mapping set set in the preset database, and the corresponding clamping force adjustment influence rate is output, the clamping force adjustment mapping set being a set for fitting the mapping relationship between the preset clamping force adjustment amount and the clamping force adjustment influence rate.

[0063] B2, the preset clamping force adjustment amount and the corresponding clamping force adjustment influence rate are coupled after being weighted and operated respectively to obtain a clamping force adjustment ratio, avoiding a single adjustment path and improving the adaptability of the system under different physical characteristics and clamping states, the clamping force adjustment ratio being data for regulating the clamping force of the manipulator on the specified soft packaging box stack; through the weighted fusion of the above three, a more scientific and reasonable clamping force adjustment ratio is formed, which helps to improve the pertinence and accuracy of the regulation effect.

[0064] B3, after a preset grabbing period, the clamping force adjustment ratio is transmitted to a preset staff terminal, and the preset staff is prompted to adjust the preset clamping force according to the clamping force adjustment ratio, that is, the product operation of the preset clamping force and the clamping force adjustment ratio is obtained to obtain an optimized clamping force, and the specified soft packaging box stack is clamped through the optimized clamping force. If the success rate of grabbing the specified soft packaging box stack in the next preset grabbing period is still lower than the preset success rate after adjustment, the grabbing is stopped, and a fault repair warning is issued.

[0065] In the embodiment, by more accurately controlling the clamping force, it is helpful to avoid falling off due to insufficient clamping force, or deformation or even damage of the soft package caused by excessive clamping force, thereby improving the overall grasping qualified rate of the manipulator and the operation safety; at the same time, by the terminal prompt mode, the clamping force adjustment ratio is fed back to the operator, so that the system adjustment and manual intervention form a synergistic mechanism, which not only retains the controllability of manual operation, but also reduces the misoperation rate; if multiple adjustments still cannot meet the success rate requirement of grasping, the system automatically terminates the operation and issues a failure maintenance warning, thereby avoiding equipment damage or operation interruption caused by repeated failure operation; through the above method, the quantification, traceability and controllability of the clamping force adjustment process are realized, the demand for high reliability and high adaptability of the manipulator in various industrial scenes is met, and the application value of the system in complex operation environments such as flexible manufacturing and logistics automation is enhanced.

[0066] Meanwhile, the embodiment of the application also provides a manipulator warehousing intelligent unstacking control device based on machine vision, which is used for realizing the foregoing manipulator warehousing intelligent unstacking control system based on machine vision, and specifically includes a visual perception structure, a manipulator and a clamping structure, and a control decision module; the visual perception structure mainly includes an industrial camera installed at the end of the manipulator, a light source system for assisting visual identification, and a visual calculation unit for image processing and point cloud analysis; the manipulator and the clamping structure are a multi-degree-of-freedom industrial robot, which is used for supporting unstacking, rotation and moving operation, is equipped with a grasping detection sensor to judge whether grasping is successful, and is equipped with various clamps, specifically including a vacuum suction cup clamp, an electric clamping jaw and a pneumatic clamping jaw; the control decision module is used for running a visual identification and grasping path planning algorithm, is matched with the robot body, and controls the manipulator action, wherein a control cabinet includes an industrial PLC, a relay, an IO (Input / Output) module and a power management, etc.

[0067] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions according to the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0068] It should be understood that the term "and / or" herein merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it, but it can also represent an "and / or" relationship, which can be understood in the context before and after it.

[0069] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0070] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0071] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0072] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0073] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0074] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0075] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0076] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0077] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A machine vision-based intelligent depalletizing and palletizing control system for robotic arms in warehousing, characterized in that, include: The robotic arm's inbound and gripping monitoring module, the robotic arm's gripping effect quantification module, and the robotic arm's gripping force control module; The robotic arm entry and gripping monitoring module is used to monitor the impact of the designated soft packaging box stack on the point cloud data in real time when the robotic arm enters the warehouse to intelligently unpack and stack the designated soft packaging box stack, so as to select whether to take soft packaging impact compensation measures. The robotic arm gripping effect quantification module is used to quantify the gripping effect of the robotic arm in grasping a specified soft packaging box stack in order to determine whether to perform corresponding gripping point correction adjustment. The gripping point correction adjustment is used to improve the accuracy of the robotic arm in selecting the gripping point of the specified soft packaging box stack. The robotic arm gripping force control module is used to determine the success rate of gripping within a preset gripping cycle to determine whether to adopt a gripping force control adjustment strategy. The gripping force control adjustment strategy is used to improve the accuracy of the gripping force applied by the robotic arm to a specified soft packaging box stack.

2. The intelligent depalletizing and palletizing control system for robotic arms in warehousing based on machine vision according to claim 1, characterized in that, The real-time monitoring of the impact of designated flexible packaging pallets on point cloud data to select whether to take flexible packaging impact compensation measures involves the following steps: Step 1: Real-time acquisition of the impact data of the soft packaging box stack on the point cloud data of the soft packaging point cloud data. The soft packaging point cloud impact data includes point cloud curvature, point cloud depth value in the Z direction, point cloud jump frequency, and point cloud overlap rate. Step 2: Obtain the point cloud data-grip influence based on the point cloud influence data of flexible packaging, and compare the point cloud data-grip influence with the point cloud-grip judgment value obtained from the preset database. If the point cloud data-grip influence is less than the point cloud-grip judgment value, continue to grab the specified flexible packaging box stack; otherwise, take flexible packaging influence compensation measures. The point cloud data-grip influence is used to quantify the degree of interference of flexible packaging characteristics on machine vision recognition.

3. The intelligent depalletizing and palletizing control system for robotic arms in warehousing based on machine vision according to claim 2, characterized in that, The specific method for obtaining the point cloud data-squeezing influence is as follows: The influence ratio and preset depth value of the soft packaging point cloud are obtained from the preset database. The influence ratio of the soft packaging point cloud includes the influence ratio of point cloud curvature, the influence ratio of point cloud depth value in the Z direction, the influence ratio of point cloud jump frequency, and the influence ratio of point cloud overlap rate. The degree of deviation between the depth value in the Z direction of the point cloud and the preset depth value is quantified to obtain the corresponding depth deviation influence. After weighting and coupling the depth deviation influence, point cloud curvature, point cloud jump frequency, and point cloud overlap rate with the corresponding soft packaging point cloud influence ratio, we obtain the point cloud data-pinch influence.

4. The intelligent depalletizing and palletizing control system for robotic arms in warehousing based on machine vision according to claim 2, characterized in that, The specific process for implementing compensation measures for the impact of flexible packaging is as follows: The difference between point cloud data - gripping influence and point cloud - gripping determination is called gripping influence difference. The gripping influence difference is mapped in the preset database to obtain the camera exposure time adjustment amount. The optimized camera exposure time is obtained by multiplying the initial camera exposure time of the machine vision system with the camera exposure time adjustment. The specified soft packaging box stacks are rescanned by optimizing the camera exposure time until the point cloud data-grip influence is less than the point cloud-grip judgment amount. The number of rescans is obtained in real time and compared with the preset number. If the number of rescans is lower than the preset number, the robot continues to grab the specified soft packaging box stack. Otherwise, the robot stops grabbing the specified soft packaging box stack and issues an abnormal grabbing warning.

5. The intelligent depalletizing and palletizing control system for robotic arms in warehousing based on machine vision according to claim 1, characterized in that, The specific steps for quantifying the gripping effect of the robotic arm in grasping a specified stack of soft packaging boxes to determine whether corresponding gripping point correction adjustments are necessary are as follows: S1, Analyze the real-time acquired gripping effect data of the specified soft packaging box stack to obtain a quantitative value of the gripping effect in order to quantify the accuracy of the gripping point of the specified soft packaging box stack selected by the robot arm. S2, compare the quantized value of the gripping effect with the judgment value of the gripping effect obtained from the preset database. If the quantized value of the gripping effect is greater than the judgment value of the gripping effect, continue to grab the specified soft packaging box stack; otherwise, perform the corresponding gripping point correction adjustment.

6. The intelligent depalletizing and palletizing control system for robotic arms in warehousing based on machine vision according to claim 5, characterized in that, The specific method for obtaining the quantification value of the clamping effect is as follows: Real-time acquisition of gripping effect analysis data and data normalization processing are performed. The gripping effect analysis data includes gripping point-point cloud density, number of point cloud noise points, and point cloud hole ratio. Extract the gripping effect analysis weight from the preset database. The gripping effect analysis weight includes the gripping point-point cloud density analysis weight, the point cloud noise point number analysis weight, and the point cloud hole ratio analysis weight. The clamping effect analysis data is coupled with the corresponding clamping effect analysis weights after weighting operations to obtain the clamping effect quantification value.

7. The intelligent depalletizing and palletizing control system for robotic arms in warehousing based on machine vision according to claim 5, characterized in that, The specific details of performing the corresponding grip point correction adjustment are as follows: The difference between the quantified value of the gripping effect and the judgment value of the gripping effect is calculated to obtain the gripping effect difference. The gripping effect difference is mapped in a preset database to obtain the gripping point correction amount. The gripping point correction amount is used to compensate for the deviation data of the robot arm to improve the accuracy of the robot arm in selecting the gripping point of the specified soft packaging box stack. The deviation data of the robot arm includes displacement error and torque error.

8. The intelligent depalletizing and palletizing control system for robotic arms in warehousing based on machine vision according to claim 1, characterized in that, The process of determining whether to adopt a gripping force control adjustment strategy by judging the success rate of gripping within a preset gripping cycle is as follows: After obtaining the gripping point correction adjustment within the preset gripping cycle, the success rate of the robot arm in gripping the specified soft packaging box stack; The grabbing success rate is compared with the preset success rate set in the preset database. If the grabbing success rate is less than the preset success rate, the clamping force control adjustment strategy is adopted through the preset clamping force adjustment amount. Otherwise, the specified soft packaging box stack is continued to be grabbed. The preset clamping force adjustment amount includes a first clamping force adjustment amount, a second clamping force adjustment amount, and a third clamping force adjustment amount; The first clamping force adjustment amount is obtained by mapping point cloud data - clamping influence amount in a preset database; The second clamping force adjustment amount is obtained by mapping the clamping effect quantification value into a preset database; The third clamping force adjustment amount is obtained by mapping the grasping success rate difference amount in a preset database, where the grasping success rate difference amount is the difference between the preset success rate and the grasping success rate.

9. The intelligent depalletizing and palletizing control system for robotic arms in warehousing based on machine vision according to claim 8, characterized in that, The specific details of the clamping force control adjustment strategy are as follows: Obtain the clamping force adjustment influence rate corresponding to the preset clamping force adjustment amount from the preset database. The clamping force adjustment influence rate includes a first clamping force influence rate, a second clamping force influence rate, and a third clamping force influence rate. After weighting the preset clamping force adjustment amount and the corresponding clamping force adjustment influence rate, the clamping force adjustment ratio is obtained by coupling. The clamping force adjustment ratio is the data for controlling the clamping force of the robot arm on the specified soft packaging box stack. After the preset gripping cycle, the clamping force adjustment ratio is transmitted to the preset operator terminal, and the preset operator is prompted to adjust the preset clamping force according to the clamping force adjustment ratio. If, after adjustment, the gripping success rate of the specified soft packaging box stack in the next preset gripping cycle is still lower than the preset success rate, gripping is stopped and a fault repair warning is issued.

10. A machine vision-based intelligent depalletizing and palletizing control device for robotic arm warehousing, used to implement the machine vision-based intelligent depalletizing and palletizing control system for robotic arm warehousing as described in any one of claims 1-9, characterized in that, include: Visual perception structure, robotic arm and gripping structure, and control decision module; The visual perception structure includes an industrial camera mounted on the end effector of a robotic arm, a light source system to assist visual recognition, and a visual computing unit for image processing and point cloud analysis. The robotic arm and gripping structure is a multi-degree-of-freedom industrial robot used to support destacking, rotation and handling actions. It is equipped with gripping detection sensors to determine whether the gripping is successful, and various grippers, including vacuum suction cup grippers, electric grippers and pneumatic grippers. The control decision module is used to run visual recognition and grasping path planning algorithms, which work in conjunction with the robot body to control the movements of the robotic arm.

Images