Stacking method and system based on image processing

Abstract

The invention discloses a stacking method and system based on image processing, and belongs to the field of automation industrials.The stacking method comprises the following specific steps that S1, images of objects to be stacked are captured, and three-dimensional models of the objects are reconstructed; s2, a stacking strategy is formulated, and the stacking path and position are optimized; according to the method, the decision-making path in the stacking process can be dynamically optimized, stable stacking of objects and maximization of the operation efficiency are ensured, the requirement for manual intervention is reduced, the operation risk and cost are reduced, the reliability and the long-term operation capacity of the system are improved, the stacking efficiency and stability are improved, the credibility of the system and the trust feeling of a user are enhanced, and the method is suitable for popularization and application. According to the method, the high-precision virtual image can be generated, the positioning precision can be improved, the robot can still accurately identify and position the object in a changeable actual operation environment, the system can stably operate in various complex scenes, the requirements of physical experiments are reduced, and the development cost and the resource consumption are reduced.

Description

Technical Field

[0001] The present invention relates to the field of automation industry, and in particular to a palletizing method and system based on image processing. Background Art

[0002] In modern industry, palletizing refers to the process of stacking objects together in an orderly manner. It is a vital link in logistics and production. As a key link, palletizing directly affects the efficiency of logistics and production lines. Traditional manual palletizing methods are labor-intensive, inefficient, and prone to errors. With the development of intelligent manufacturing and automation technology, traditional manual palletizing methods are gradually being replaced by automatic palletizing systems. However, achieving efficient and accurate automatic palletizing still faces many challenges. However, existing automatic palletizing systems often face many challenges when facing complex and diverse objects and environments, such as object identification and positioning, path planning optimization, and the stability and robustness of the palletizing process. Therefore, how to improve the intelligence and accuracy of automatic palletizing systems has become a research topic that needs to be solved urgently.

[0003] After searching, Chinese patent number CN202010867912.4 discloses a 3D vision palletizing system and method based on artificial intelligence technology. Although this invention achieves rapid grasping, improves overall palletizing efficiency, further improves palletizing accuracy and speed, and improves stability and safety, it cannot dynamically optimize the decision path during the palletizing process, and the stable stacking of objects and operational efficiency cannot be maximized. The need for manual intervention increases, which increases operational risks and costs. In addition, existing palletizing methods and systems cannot generate high-precision virtual images, and there are deviations in positioning accuracy, which makes it impossible for robots to accurately identify and locate objects in a changing actual operating environment. The demand for physical experiments increases, and development costs and resource consumption increase. To this end, we propose a palletizing method and system based on image processing. Summary of the Invention

[0004] The purpose of the present invention is to solve the defects in the prior art and to propose a palletizing method and system based on image processing.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A palletizing method based on image processing, the specific steps of the palletizing method are as follows:

[0007] S1. Capture images of objects to be palletized and reconstruct a three-dimensional model of the objects: Use a high-resolution camera to capture images of the objects to be palletized from different angles, and pre-process the captured images. Based on the processed images, identify and locate the outline and boundaries of the objects, and reconstruct a three-dimensional model of the objects to be palletized based on the multi-angle images.

[0008] As a further solution of the present invention, the specific steps of identifying and positioning the objects to be palletized in S1 are as follows:

[0009] Step 1: Resize the input image to a fixed size and normalize the image pixel values ​​to construct an image dataset. Use the LabelImg image annotation tool to manually draw annotation boxes to mark the location and category of the object. Record the width w and height h of each annotation box and process the width w and height h of the annotation box as a two-dimensional data point (w, h) to form a dataset {(w i ,h i )}, where i is the index of the annotation box;

[0010] Step 2: Select the number of anchor boxes k to be generated according to the YOLO model type, and randomly select k initial cluster centers μ j =(w j ,h j ), where j = 1, 2, 3, K, k, calculate each data point (w i ,h i ) to each cluster center and assign it to the cluster to which the nearest cluster center belongs. Then, the average value of all data points in each cluster is calculated to update the cluster center.

[0011] Step 3: Repeat the above steps to update the cluster center until the cluster center change value reaches the preset termination threshold, and the final cluster center μ j As the anchor box size, the input image is then divided into multiple grids, and multiple sets of candidate boxes are generated based on the anchor box size, with each grid predicting a fixed number of candidate boxes;

[0012] Step 4: Set the initial candidate box as the initial solution X0, define the objective function E(X) based on the matching degree between the candidate box and the real object boundary and the classification confidence to evaluate the quality of the candidate box, and generate a neighborhood solution X in the neighborhood of the current solution by randomly perturbing the position and size of the candidate box. new , and calculate the acceptance probability P of the new solution according to the change of the objective function;

[0013] Step 5: Choose whether to accept the new candidate box X based on the acceptance probability P new If accepted, update the current candidate box to X new , and update the set control coefficient, repeat the candidate frame search until the predetermined number of iterations is reached or the preset control coefficient drops to the set minimum value;

[0014] Step 6: Perform non-maximum suppression on the multiple candidate boxes searched out, remove redundant boxes with high overlap rates, retain only the boxes with the highest confidence, and output the optimal candidate box parameters and category probability as the final detection result of the object.

[0015] As a further solution of the present invention, the specific calculation formula for updating the cluster center in step 2 is as follows:

[0016]

[0017] Where C j represents the jth cluster; |C j | represents the number of data points in the jth cluster, where i is the index of the annotation box;

[0018] The specific calculation formula of the objective function in step 4 is as follows:

[0019]

[0020] Where E(X) represents the objective function value of evaluating the candidate box; Confidence represents the confidence of the candidate box; Represents the intersection-over-union ratio of the predicted box and the true box;

[0021] The specific calculation formula for the acceptance probability described in step 4 is as follows:

[0022]

[0023] Where P represents the probability of accepting a new solution; E(X old ) represents the objective function value of the current solution, where the initial objective function value is E(X0); E(X new ) represents the objective function value of the new solution; T represents the current control coefficient;

[0024] The specific calculation formula for updating the control coefficient in step 5 is as follows:

[0025] T new =αT old

[0026] Where, T new represents the updated control coefficient; α represents the attenuation coefficient; T old Represents the current control coefficient.

[0027] S2. Develop a palletizing strategy and optimize the palletizing path and position: Based on the various attribute information of the objects to be palletized and the actual palletizing requirements, develop the optimal palletizing strategy. Based on this palletizing strategy, simulate the movement and cooperation of the particle group to optimize the palletizing path and position.

[0028] As a further solution of the present invention, the specific steps of formulating the palletizing strategy in S2 are as follows:

[0029] Step 1: Determine the information of various criteria that affect the stability, space utilization, processing time and ease of operation of the palletizing strategy, and use the decision criterion set C = {c1, c2, K, cm}, where m is the number of criteria, and the range of each set of criteria collected is normalized to unify the dimensions of each set of criteria;

[0030] Step 2: Construct a pairwise comparison matrix based on the collected criteria, and calculate the eigenvalues ​​corresponding to the eigenvectors of the comparison matrix to obtain the corresponding weight vector w = {w1, w2, K, w m}, where the number of weight vectors is consistent with the number of criteria m, and all possible states in the palletizing process and the possible actions that can be performed in each state are collected to determine the state space, action space and state transition probability;

[0031] Step 3: Based on the specific palletizing requirements and constraints, multiple alternative plans are generated. Each plan represents a palletizing strategy. Each alternative plan is then evaluated based on the determined criteria. The standardized criteria values ​​and weights are used to calculate the weighted comprehensive score of each plan. The comprehensive score is then used as the immediate reward, and the reward function is constructed by combining the state and action.

[0032] Step 4: Initialize a set of policies π0 by random selection and calculate the current policy π k The expected return of each state s, where the initial state is π k =π0, and through the iterative method, continuously update the expected return, terminate and reach the preset convergence condition, improve the strategy according to the current expected return, for each state s, calculate the expected return of all possible actions, and select the action that maximizes the expected return as the new strategy π k+1 ;

[0033] Step 5: Compare the current policy π k and the new strategy π k+1 , if π k =π k+1 , then the algorithm converges and the optimal strategy π is obtained * , if π k ≠π k+1 , then let π k =π k+1 , and re-update the expected report of the strategy through iteration, and select a new strategy until the convergence condition is reached;

[0034] Step 6: Select the optimal palletizing strategy π * , and simulate or verify the selected optimal solution through actual operation. If the simulation or verification results do not meet the preset requirements, the criterion weights are adjusted or new alternative solutions are generated based on the feedback.

[0035] As a further solution of the present invention, the specific calculation formula of the reward function in step 3 is as follows:

[0036]

[0037] Where R(s,a) represents the immediate reward of executing action a in state s; S i Representative Plan A i The comprehensive score of c′ ij Represents the standardized i-th solution under criterion c j The score on w j Representative criteria c j The weight of

[0038] The specific calculation formula for the expected return in step 4 is as follows:

[0039]

[0040] Where, Represents the strategy π k Under the state s, the value function; π k (a|s) represents the strategy π k The probability of selecting action a in state s; P(s′|s,a) represents the probability of transitioning to state s′ after executing action a in state s; γ represents the discount factor, which is between 0 and 1.

[0041] S3. Simulate and rehearse the entire palletizing process: Use virtual simulation technology to simulate the entire palletizing process and detect potential problems. Simultaneously, perform visual simulations to generate realistic light and shadow effects to perform curve fitting and optimization on the palletizing path.

[0042] S4. Real-time monitoring of the palletizing process and dynamic adjustment of the palletizing strategy: During the actual palletizing process, the machine's operating status is monitored through cameras and sensors, and the data is fed back to the control system. At the same time, the palletizing strategy is adjusted in real time by combining visual processing and sensor data;

[0043] S5. Record palletizing data to optimize palletizing methods in real time: Record and analyze the data generated during the palletizing process, summarize experience, and learn from each palletizing operation to gradually optimize your own decision-making and operations.

[0044] A palletizing system based on image processing, comprising an acquisition and processing module, a target detection module, a positioning optimization module, a contour extraction module, a 3D reconstruction module, an object recognition module, a posture estimation module, a decision analysis module, a virtual simulation module, a control optimization module, a ray tracing module, and a feedback recording module;

[0045] The acquisition and processing module is used to obtain image data of the objects to be stacked from multiple angles and pre-process the acquired images;

[0046] The object detection module is used to perform object detection on the preprocessed image, identify and locate the bounding box of the object;

[0047] The positioning optimization module is used to eliminate redundant and erroneously detected bounding boxes;

[0048] The contour extraction module is used to extract the contour information of the object to obtain boundary data;

[0049] The 3D reconstruction module is used to reconstruct a 3D model of an object based on multi-angle images and depth information;

[0050] The object recognition module classifies the reconstructed three-dimensional model and identifies the category and features of the object;

[0051] The pose estimation module is used to determine the spatial position and pose of the object;

[0052] The decision analysis module comprehensively considers multiple factors to formulate the optimal palletizing strategy;

[0053] The virtual simulation module is used to simulate the palletizing process in a virtual environment and optimize the palletizing path and position.

[0054] As a further solution of the present invention, the specific steps of optimizing the palletizing path and position of the virtual simulation module are as follows:

[0055] Step 1: Collect the path and placement information in the palletizing strategy, and randomly generate a particle swarm of candidate solutions for the palletizing path and position by random perturbation. where x r Represents the rth palletizing path and position, n represents the position component of each group of candidate solutions in each dimension, and then initializes the speed and position of each particle in the particle swarm;

[0056] Step 2: Based on the evaluation results of the object's shape, size, weight, and stability criteria, calculate the score of each particle under each evaluation criterion. Based on the score and corresponding weight of each criterion, calculate the total fitness value of the particle. Then compare the current fitness value of each group of particles with its individual historical best fitness value.

[0057] Step ③: If the fitness value of the current particle position is better than the fitness value of its historical best position, the historical best position is updated to the position and speed of the current particle. Otherwise, the historical best position remains unchanged, and the fitness value of each particle position in the particle swarm is repeatedly calculated until the preset maximum number of iterations is reached. The optimal palletizing path and position obtained after convergence are output to replace the original palletizing path and position.

[0058] As a further solution of the present invention, the specific calculation formula for initializing the speed and position in step ① is as follows:

[0059]

[0060] Where, represents the initial position of the rth particle in the qth dimension; Represents the minimum position boundary in the qth dimension; Represents the maximum position boundary on the qth dimension; rand() represents generating a random number in the range [0, 1]; represents the initial velocity of the rth particle in the qth dimension; Represents the maximum position boundary in the qth dimension; Represents the minimum position boundary in the qth dimension;

[0061] The specific calculation formula for position and speed update in step ③ is as follows:

[0062]

[0063] Where, represents the velocity of the rth particle in the qth dimension at time t+1; w represents the inertia weight; represents the velocity of the rth particle in the qth dimension at time t; c1 and c2 represent the acceleration coefficient of the particle approaching its own optimal position and the global optimal position, respectively; r1 and r2 represent random numbers in the range [0, 1]; represents the individual optimal position of the rth particle in the qth dimension; represents the position of the rth particle in the qth dimension at time t; g q Represents the component of the global best position in the qth dimension.

[0064] The control optimization module is used to control the robot to perform actual palletizing operations;

[0065] The ray tracing module is used to optimize the robot's vision system.

[0066] As a further solution of the present invention, the specific steps of optimizing the visual system of the ray tracing module are as follows:

[0067] Step I: Collect initial visual system data, including the robot's object recognition rate, positioning accuracy, and ray tracing results, in a virtual environment. Initialize the simulated scene, the maximum number of light reflections, and the refractive index. Based on the camera parameters, project light from the camera viewpoint into the scene to generate the initial light.

[0068] Step II: Calculate the intersection of each ray with an object in the scene, determine whether the ray intersects the object, trace the ray's path in the scene, and record each intersection point and its corresponding normal vector. Calculate the reflection and refraction paths of the ray on the object's surface, and determine the splitting path of the ray based on the object's material properties.

[0069] Step III: Based on the ray tracing path and the intersection information between the ray and the object, the color and brightness of each pixel in the scene are calculated. The colors and brightness of all pixels are combined to generate the final virtual image, and the proxy model is constructed through Gaussian process regression.

[0070] Step IV: Perform initial simulations under different ray tracing parameter settings, collect corresponding positioning error data, calculate the expected improvement for each candidate point, evaluate the potential benefit of each candidate point as the next test point, and select the candidate point with the largest expected improvement value as the parameter for the next ray tracing simulation based on the expected improvement criterion;

[0071] Step VI: Use the selected test points to perform ray tracing simulation, generate a virtual image and perform positioning, calculate the positioning error of the newly simulated image, and record the parameters of the test point and the corresponding error value. Repeat the expected improvement criterion calculation, test point selection, ray tracing simulation and model update until the preset convergence conditions are reached. Verify the optimized visual system in a virtual environment, make adjustments and improvements, and then apply the optimized visual system to actual operations to verify its performance in a real environment.

[0072] As a further solution of the present invention, the specific calculation formula for the light reflection and refraction path in step II is as follows:

[0073] Q reflect =Q-2(Q·n)n

[0074]

[0075] Where Q reflect represents reflected light; Q refract represents refracted light; η represents the refractive index; Q represents the incident light; n represents the normal vector of the object surface;

[0076] The specific calculation formula for the expected improvement in step IV is as follows:

[0077] EI(o)=Ε[max(0,f min -f(o))]

[0078] Where, EI(o) represents the expected improvement value of candidate point o; E represents the expected value; f min represents the current optimal positioning error value; f(o) represents the positioning error function.

[0079] The feedback recording module is used to record the palletizing situation in real time and feed it back to the on-site staff.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] 1. The image processing-based palletizing method determines the various criteria information that affect the palletizing strategy, generates multiple alternatives according to specific palletizing requirements and constraints, and each option represents a palletizing strategy. Each alternative is then evaluated according to the determined criteria, and the weighted comprehensive score of each option is calculated. A set of strategies is randomly selected and initialized, and the expected return of each state under the current strategy is calculated. The expected return is continuously updated through the iterative method, and the action that maximizes the expected return is selected as the new strategy. The path and placement position information in the palletizing strategy are collected, and a particle swarm of candidate solutions for the palletizing path and position is randomly generated through random perturbations. The speed and position of each particle in the particle swarm are then initialized, and the score of each particle under each evaluation criterion is calculated respectively. According to the score of each criterion and the corresponding weight, the candidate solution is randomly generated. , calculate the total fitness value of the particles, and then compare the current fitness value of each group of particles with their individual historical best fitness value. If the fitness value of the current particle position is better than the fitness value of its historical best position, then update the historical best position to the position and speed of the current particle. Otherwise, keep the historical best position unchanged. Then repeatedly calculate the fitness value of each particle position in the particle swarm until the preset maximum number of iterations is reached, and output the optimal stacking path and position obtained after convergence to replace the original stacking path and position. It can dynamically optimize the decision path in the stacking process, ensure the stable stacking of objects and maximize the operation efficiency, reduce the need for manual intervention, reduce operation risks and costs, improve the reliability and long-term operation capability of the system, improve stacking efficiency and stability, and enhance the credibility of the system and the trust of users.

[0082] 2. The image processing-based palletizing system is equipped with a ray tracing module. It collects the initial visual system data of the robot in a virtual environment, initializes the simulated scene, the maximum number of light reflections, and the refractive index, projects light from the camera viewpoint into the scene, generates initial light, and then calculates the reflection and refraction paths of light on the surface of the object. Based on the ray tracing path and the intersection information of the light and the object, it calculates the color and brightness of each pixel in the scene, combines the colors and brightness of all pixels to generate the final virtual image, constructs an agent model through Gaussian process regression, performs initial simulations under different ray tracing parameter settings, collects corresponding positioning error data, calculates the expected improvement of each candidate point, evaluates the potential benefit of each candidate point as the next test point, and selects the expected improvement value based on the expected improvement criterion. The largest candidate point is used as the parameter for the next ray tracing simulation. The selected test points are used for ray tracing simulation to generate a virtual image and perform positioning. The positioning error of the newly simulated image is calculated, and the parameters of the test point and the corresponding error value are recorded. The expected improvement criterion calculation, test point selection, ray tracing simulation and model update are repeated until the preset convergence conditions are reached. The optimized visual system is verified in a virtual environment, and adjustments and improvements are made. The optimized visual system is then applied to actual operations to verify its performance in a real environment. It can generate high-precision virtual images and improve positioning accuracy, so that the robot can still accurately identify and locate objects in a changing actual operating environment, ensuring that the system can operate stably in various complex scenarios, reducing the need for physical experiments, and reducing development costs and resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0084] Figure 1 This is a flowchart of a palletizing method based on image processing proposed by the present invention;

[0085] Figure 2 This is a system block diagram of a palletizing system based on image processing proposed by the present invention. DETAILED DESCRIPTION

[0086] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0087] Example 1

[0088] Reference Figure 1 This embodiment discloses a palletizing method based on image processing. The specific steps of the palletizing method are as follows:

[0089] Capture images of objects to be palletized and reconstruct a 3D model of the objects: Use a high-resolution camera to capture images of the objects to be palletized from different angles, and pre-process the captured images. Based on the processed images, identify and locate the outlines and boundaries of the objects, and reconstruct a 3D model of the objects to be palletized based on the multi-angle images.

[0090] Specifically, the input image is resized to a fixed size and the image pixel values ​​are normalized to construct an image dataset. The LabelImg image annotation tool is used to manually draw annotation boxes to mark the location and category of the object. The width w and height h of each annotation box are recorded and processed as two-dimensional data points (w, h) to form a dataset {(w i ,h i )}, where i is the index of the annotation box, the number of anchor boxes k to be generated is selected according to the YOLO model type, and k initial cluster centers μ are randomly selected. j =(w j ,h j ), where j = 1, 2, 3, K, k, calculate each data point (w i ,h i ) to each cluster center and assign it to the cluster to which the nearest cluster center belongs. Then, the average value of all data points in each cluster is calculated to update the cluster center. Through the above steps, the cluster center is updated repeatedly until the cluster center change value reaches the preset termination threshold, and the final cluster center μ is set. j As the anchor box size, the input image is then divided into multiple grids, and multiple groups of candidate boxes are generated based on the anchor box size. A fixed number of candidate boxes are predicted for each grid, and the initial candidate box is set as the initial solution X0. The objective function E(X) is defined based on the matching degree between the candidate box and the real object boundary and the classification confidence to evaluate the quality of the candidate box. The neighborhood solution X is generated in the neighborhood of the current solution by randomly perturbing the position and size of the candidate box. new , and calculate the acceptance probability P of the new solution according to the change of the objective function, and choose whether to accept the new candidate box X according to the acceptance probability P new If accepted, update the current candidate box to X new , and update the set control coefficient, repeat the candidate box search until the predetermined number of iterations is reached or the preset control coefficient drops to the set minimum value, perform non-maximum suppression on the multiple candidate boxes searched out, remove redundant boxes with high overlap rates, retain only the boxes with the highest confidence, and output the optimal candidate box parameters and category probability as the final detection result of the object.

[0091] In this embodiment, the specific calculation formula for cluster center update is as follows:

[0092]

[0093] Where C j represents the jth cluster; |C j | represents the number of data points in the jth cluster, where i is the index of the annotation box;

[0094] The specific calculation formula of the objective function is as follows:

[0095]

[0096] Where E(X) represents the objective function value of evaluating the candidate box; Confidence represents the confidence of the candidate box; Represents the intersection-over-union ratio of the predicted box and the true box;

[0097] The specific calculation formula for acceptance probability is as follows:

[0098]

[0099] Where P represents the probability of accepting a new solution; E(X old ) represents the objective function value of the current solution, where the initial objective function value is E(X0); E(X new ) represents the objective function value of the new solution; T represents the current control coefficient;

[0100] The specific calculation formula for controlling coefficient update is as follows:

[0101] T new =αT old

[0102] Where, T new represents the updated control coefficient; α represents the attenuation coefficient; T old Represents the current control coefficient.

[0103] Develop a palletizing strategy and optimize the palletizing path and position: Based on the various attribute information of the objects to be palletized and the actual palletizing requirements, develop the optimal palletizing strategy. Based on this palletizing strategy, simulate the movement and cooperation of the particle group to optimize the palletizing path and position.

[0104] Specifically, determine the information of various criteria that affect the stability, space utilization, processing time and operability of the palletizing strategy, and use the decision criterion set C = {c1, c2, K, c m}, where m is the number of criteria, and the range of each set of criteria collected is normalized, the dimensions of each set of criteria are unified, a pairwise comparison matrix is ​​constructed based on each set of criteria collected, and the eigenvalues ​​corresponding to the eigenvectors of the comparison matrix are calculated to obtain the corresponding weight vector w = {w1, w2, K, w m}, where the number of weight vectors is consistent with the number of criteria m. All possible states in the palletizing process and the possible actions that can be performed in each state are collected to determine the state space, action space and state transition probability. According to the specific palletizing requirements and constraints, multiple alternatives are generated. Each solution represents a palletizing strategy. Each alternative is evaluated according to the determined criteria, and the weighted comprehensive score of each solution is calculated using the standardized criterion value and weight. The comprehensive score is then used as the instant reward. The reward function is constructed by combining the state and action. A set of strategies π0 is initialized by random selection, and the current strategy π is calculated. k The expected return of each state s, where the initial state is π k =π0, and through the iterative method, continuously update the expected return, terminate and reach the preset convergence condition, improve the strategy according to the current expected return, for each state s, calculate the expected return of all possible actions, and select the action that maximizes the expected return as the new strategy π k+1 , compare the current policy π k and the new strategy π k+1 , if π k =π k+1 , then the algorithm converges and the optimal strategy π is obtained * , if π k ≠π k+1 , then let π k =π k+1 , and re-update the expected report of the strategy through iteration, and select a new strategy until the convergence condition is reached and stop, and select the optimal stacking strategy π * , and simulate or verify the selected optimal solution through actual operation. If the simulation or verification results do not meet the preset requirements, the criterion weights are adjusted or new alternative solutions are generated based on the feedback.

[0105] It should be further explained that the specific calculation formula of the reward function is as follows:

[0106]

[0107] Where R(s,a) represents the immediate reward of executing action a in state s; S i Representative Plan A i The comprehensive score of c′ ij Represents the standardized i-th solution under criterion c j The score on w j Representative criteria c j The weight of

[0108] The specific calculation formula for expected return is as follows:

[0109]

[0110] Where, Represents the strategy π k Under the state s, the value function; π k (a|s) represents the strategy π k The probability of selecting action a in state s; P(s′|s,a) represents the probability of transitioning to state s′ after executing action a in state s; γ represents the discount factor, which is between 0 and 1.

[0111] Simulate and rehearse the entire palletizing process: Use virtual simulation technology to simulate the entire palletizing process and detect potential problems. Simultaneously, perform visual effect simulation to generate realistic light and shadow effects to perform curve fitting and optimization of the palletizing path.

[0112] Real-time monitoring of the palletizing process and dynamic adjustment of the palletizing strategy: During the actual palletizing process, the machine's operating status is monitored through cameras and sensors, and the data is fed back to the control system. At the same time, the palletizing strategy is adjusted in real time by combining visual processing and sensor data.

[0113] Record palletizing data to optimize palletizing methods in real time: Record and analyze the data generated during the palletizing process, summarize experience, and learn from each palletizing operation to gradually optimize your own decisions and operations.

[0114] Example 2

[0115] Reference Figure 2 This embodiment discloses a palletizing system based on image processing, including an acquisition and processing module, a target detection module, a positioning optimization module, a contour extraction module, a three-dimensional reconstruction module, an object recognition module, a posture estimation module, a decision analysis module, a virtual simulation module, a control optimization module, a ray tracing module, and a feedback recording module.

[0116] The acquisition and processing module is used to obtain multi-angle image data of the objects to be palletized and preprocess the acquired images; the target detection module is used to detect objects in the preprocessed images, identify and locate the bounding boxes of the objects; the positioning optimization module is used to eliminate redundant and erroneously detected bounding boxes; and the contour extraction module is used to extract the contour information of the objects to obtain boundary data.

[0117] The 3D reconstruction module is used to reconstruct the 3D model of the object based on multi-angle images and depth information; the object recognition module classifies the reconstructed 3D model and identifies the category and characteristics of the object; the pose estimation module is used to determine the spatial position and posture of the object; and the decision analysis module comprehensively considers multiple factors to formulate the optimal stacking strategy.

[0118] The virtual simulation module is used to simulate the palletizing process in a virtual environment and optimize the palletizing path and position.

[0119] Specifically, the path and placement information in the palletizing strategy are collected, and a particle swarm of candidate solutions for the palletizing path and position is randomly generated by random perturbation. where x r Represents the rth palletizing path and position, n represents the position component of each group of candidate solutions in each dimension, then initializes the speed and position of each particle in the particle swarm, and calculates the score of each particle under each evaluation criterion according to the evaluation results of the object shape, size, weight and stability criteria. According to the score of each criterion and the corresponding weight, calculate the total fitness value of the particle, then compare the current fitness value of each group of particles with its individual historical best fitness value. If the fitness value of the current particle position is better than the fitness value of its historical best position, then update the historical best position to the position and speed of the current particle. Otherwise, keep the historical best position unchanged, and then repeatedly calculate the fitness value of each particle position in the particle swarm until the preset maximum number of iterations is reached, and output the optimal palletizing path and position obtained after convergence to replace the original palletizing path and position.

[0120] In this embodiment, the specific calculation formulas for initializing speed and position are as follows:

[0121]

[0122] Where, represents the initial position of the rth particle in the qth dimension; Represents the minimum position boundary in the qth dimension; Represents the maximum position boundary on the qth dimension; rand() represents generating a random number in the range [0, 1]; represents the initial velocity of the rth particle in the qth dimension; Represents the maximum position boundary in the qth dimension; Represents the minimum position boundary in the qth dimension;

[0123] The specific calculation formulas for position and speed updates are as follows:

[0124]

[0125] Where, represents the velocity of the rth particle in the qth dimension at time t+1; w represents the inertia weight; represents the velocity of the rth particle in the qth dimension at time t; c1 and c2 represent the acceleration coefficient of the particle approaching its own optimal position and the global optimal position, respectively; r1 and r2 represent random numbers in the range [0, 1]; represents the individual optimal position of the rth particle in the qth dimension; represents the position of the rth particle in the qth dimension at time t; g q Represents the component of the global best position in the qth dimension.

[0126] The control optimization module is used to control the robot to perform actual palletizing operations; the ray tracing module is used to optimize the robot's visual system; and the feedback recording module is used to record the palletizing situation in real time and provide feedback to on-site staff.

[0127] Specifically, the robot's object recognition rate, positioning accuracy, and initial visual system data of ray tracing results are collected in a virtual environment. The simulated scene, the maximum number of light reflections, and the refractive index are initialized. According to the camera parameters, light is projected from the camera viewpoint to the scene to generate initial light. The intersection of each ray and the object in the scene is calculated. It is determined whether the ray intersects with the object, the path of the ray in the scene is traced, and each intersection of the ray and the object and its corresponding normal vector are recorded. The reflection and refraction path of the light on the surface of the object is calculated, and the splitting path of the light is determined according to the material properties of the object. According to the ray tracing path and the intersection information of the light and the object, the color and brightness of each pixel in the scene are calculated. The colors and brightness of all pixels are combined to generate the final virtual image, which is then constructed through Gaussian process regression. Build a proxy model, perform initial simulations under different ray tracing parameter settings, collect corresponding positioning error data, calculate the expected improvement of each candidate point, evaluate the potential benefit of each candidate point as the next test point, and select the candidate point with the largest expected improvement value as the parameter for the next ray tracing simulation based on the expected improvement criterion. Use the selected test point to perform ray tracing simulation, generate a virtual image and perform positioning, calculate the positioning error of the newly simulated image, and record the parameters and corresponding error values ​​of the test point. Repeat the expected improvement criterion calculation, test point selection, ray tracing simulation and model update until the preset convergence conditions are reached. Verify the optimized visual system in a virtual environment, make adjustments and improvements, and then apply the optimized visual system to actual operations to verify its performance in a real environment.

[0128] It should be further explained that the specific calculation formula for the light reflection and refraction path is as follows:

[0129] Q reflect =Q-2(Q·n)n

[0130]

[0131] Where Q reflect represents reflected light; Q refract represents refracted light; η represents the refractive index; Q represents the incident light; n represents the normal vector of the object surface;

[0132] The specific calculation formula for expected improvement is as follows:

[0133] EI(o)=Ε[max(0,f min -f(o))]

[0134] Where, EI(o) represents the expected improvement value of candidate point o; E represents the expected value; f min represents the current optimal positioning error value; f(o) represents the positioning error function.

Claims

1. A palletizing method based on image processing, characterized in that: The specific steps of the palletizing method are as follows: S1. Capturing images of objects to be palletized and reconstructing a three-dimensional model of the objects: Using a high-resolution camera to capture images of the objects to be palletized from different angles, and pre-processing the captured images, identifying and locating the outlines and boundaries of the objects based on the processed images, and reconstructing a three-dimensional model of the objects to be palletized based on the multi-angle images; S2. Develop a palletizing strategy and optimize the palletizing path and position: Based on the attributes of the objects to be palletized and the actual palletizing requirements, develop an optimal palletizing strategy. Based on this strategy, simulate the movement and cooperation of the particle group to optimize the palletizing path and position. S3. Simulate and rehearse the entire palletizing process: Use virtual simulation technology to simulate the entire palletizing process and detect potential problems. Simultaneously, perform visual simulations to generate realistic light and shadow effects to perform curve fitting and optimization on the palletizing path. S4. Real-time monitoring of the palletizing process and dynamic adjustment of the palletizing strategy: During the actual palletizing process, the machine's operating status is monitored through cameras and sensors, and the data is fed back to the control system. At the same time, the palletizing strategy is adjusted in real time by combining visual processing and sensor data; S5. Record palletizing data to optimize palletizing methods in real time: Record and analyze the data generated during the palletizing process, summarize experience, and learn from each palletizing operation to gradually optimize your own decision-making and operations.

2. A palletizing method based on image processing according to claim 1, characterized in that: The specific steps for identifying and positioning the objects to be palletized in S1 are as follows: Step 1: Resize the input image to a fixed size and normalize the image pixel values ​​to construct an image dataset. Use the LabelImg image annotation tool to manually draw annotation boxes to mark the location and category of the object. Record the width w and height h of each annotation box and process the width w and height h of the annotation box as a two-dimensional data point (w, h) to form a dataset {(w i ,h i )}, where i is the index of the annotation box; Step 2: Select the number of anchor boxes k to be generated according to the YOLO model type, and randomly select k initial cluster centers μ j =(w j ,h j ), where j = 1, 2, 3, K, k, calculate each data point (w i ,h i ) to each cluster center and assign it to the cluster to which the nearest cluster center belongs. Then, the average value of all data points in each cluster is calculated to update the cluster center. Step 3: Repeat the above steps to update the cluster center until the cluster center change value reaches the preset termination threshold, and the final cluster center μ j As the anchor box size, the input image is then divided into multiple grids, and multiple sets of candidate boxes are generated based on the anchor box size, with each grid predicting a fixed number of candidate boxes; Step 4: Set the initial candidate box as the initial solution X0, define the objective function E(X) based on the matching degree between the candidate box and the real object boundary and the classification confidence to evaluate the quality of the candidate box, and generate a neighborhood solution X in the neighborhood of the current solution by randomly perturbing the position and size of the candidate box. new , and calculate the acceptance probability P of the new solution according to the change of the objective function; Step 5: Choose whether to accept the new candidate box X based on the acceptance probability P new If accepted, update the current candidate box to X new , and update the set control coefficient, repeat the candidate frame search until the predetermined number of iterations is reached or the preset control coefficient drops to the set minimum value; Step 6: Perform non-maximum suppression on the multiple candidate boxes searched out, remove redundant boxes with high overlap rates, retain only the boxes with the highest confidence, and output the optimal candidate box parameters and category probability as the final detection result of the object.

3. A palletizing method based on image processing according to claim 2, characterized in that: The specific calculation formula for cluster center update in step 2 is as follows: Where C j represents the jth cluster; |C j | represents the number of data points in the jth cluster, where i is the index of the annotation box; The specific calculation formula of the objective function in step 4 is as follows: Where E(X) represents the objective function value of evaluating the candidate box; Confidence represents the confidence of the candidate box; Represents the intersection-over-union ratio of the predicted box and the true box; The specific calculation formula for the acceptance probability described in step 4 is as follows: Where P represents the probability of accepting a new solution; E(X old ) represents the objective function value of the current solution, where the initial objective function value is E(X0); E(X new ) represents the objective function value of the new solution; T represents the current control coefficient; The specific calculation formula for updating the control coefficient in step 5 is as follows: T new =αT old Where, T new represents the updated control coefficient; α represents the attenuation coefficient; T old Represents the current control coefficient.

4. The palletizing method based on image processing according to claim 2, characterized in that: The specific steps for formulating the palletizing strategy described in S2 are as follows: Step 1: Determine the information of various criteria that affect the stability, space utilization, processing time and ease of operation of the palletizing strategy, and use the decision criterion set C = {c1, c2, K, c m }, where m is the number of criteria, and the range of each set of criteria collected is normalized to unify the dimensions of each set of criteria; Step 2: Construct a pairwise comparison matrix based on the collected criteria, and calculate the eigenvalues ​​corresponding to the eigenvectors of the comparison matrix to obtain the corresponding weight vector w = {w1, w2, K, w m }, where the number of weight vectors is consistent with the number of criteria m, and all possible states in the palletizing process and the possible actions that can be performed in each state are collected to determine the state space, action space and state transition probability; Step 3: Based on the specific palletizing requirements and constraints, multiple alternative plans are generated. Each plan represents a palletizing strategy. Each alternative plan is then evaluated based on the determined criteria. The standardized criteria values ​​and weights are used to calculate the weighted comprehensive score of each plan. The comprehensive score is then used as the immediate reward, and the reward function is constructed by combining the state and action. Step 4: Initialize a set of policies π0 by random selection and calculate the current policy π k The expected return of each state s, where the initial state is π k =π0, and through the iterative method, continuously update the expected return, terminate and reach the preset convergence condition, improve the strategy according to the current expected return, for each state s, calculate the expected return of all possible actions, and select the action that maximizes the expected return as the new strategy π k+1 ; Step 5: Compare the current policy π k and the new strategy π k+1 , if π k =π k+1 , then the algorithm converges and the optimal strategy π is obtained * , if π k ≠π k+1 , then let π k =π k+1 , and re-update the expected report of the strategy through iteration, and select a new strategy until the convergence condition is reached; Step 6: Select the optimal palletizing strategy π * , and simulate or verify the selected optimal solution through actual operation. If the simulation or verification results do not meet the preset requirements, the criterion weights are adjusted or new alternative solutions are generated based on the feedback.

5. The palletizing method based on image processing according to claim 4, characterized in that: The specific calculation formula of the reward function in step 3 is as follows: Where R(s,a) represents the immediate reward of executing action a in state s; S i Representative Plan A i The comprehensive score of c′ ij Represents the standardized i-th solution under criterion c j The score on w j Representative criteria c j The weight of The specific calculation formula for the expected return in step 4 is as follows: Where, Represents the strategy π k Under the state s, the value function; π k (a|s) represents the strategy π k The probability of selecting action a in state s; P(s′|s,a) represents the probability of transitioning to state s′ after executing action a in state s; γ represents the discount factor, which is between 0 and 1.

6. A palletizing system based on image processing, used to implement the palletizing method according to any one of claims 1 to 5, characterized in that: It includes acquisition and processing module, target detection module, positioning optimization module, contour extraction module, 3D reconstruction module, object recognition module, posture estimation module, decision analysis module, virtual simulation module, control optimization module, ray tracing module and feedback recording module; The acquisition and processing module is used to obtain image data of the objects to be stacked from multiple angles and pre-process the acquired images; The object detection module is used to perform object detection on the preprocessed image, identify and locate the bounding box of the object; The positioning optimization module is used to eliminate redundant and erroneously detected bounding boxes; The contour extraction module is used to extract the contour information of the object to obtain boundary data; The 3D reconstruction module is used to reconstruct a 3D model of an object based on multi-angle images and depth information; The object recognition module classifies the reconstructed three-dimensional model and identifies the category and features of the object; The pose estimation module is used to determine the spatial position and pose of the object; The decision analysis module comprehensively considers multiple factors to formulate the optimal palletizing strategy; The virtual simulation module is used to simulate the palletizing process in a virtual environment and optimize the palletizing path and position; The control optimization module is used to control the robot to perform actual palletizing operations; The ray tracing module is used to optimize the robot's visual system; The feedback recording module is used to record the palletizing situation in real time and feed it back to the on-site staff.

7. The palletizing system based on image processing according to claim 6, characterized in that: The specific steps of the virtual simulation module palletizing path and position optimization are as follows: Step 1: Collect the path and placement information in the palletizing strategy, and randomly generate a particle swarm of candidate solutions for the palletizing path and position by random perturbation. where x r Represents the rth palletizing path and position, n represents the position component of each group of candidate solutions in each dimension, and then initializes the speed and position of each particle in the particle swarm; Step 2: Based on the evaluation results of the object's shape, size, weight, and stability criteria, calculate the score of each particle under each evaluation criterion. Based on the score and corresponding weight of each criterion, calculate the total fitness value of the particle. Then compare the current fitness value of each group of particles with its individual historical best fitness value. Step ③: If the fitness value of the current particle position is better than the fitness value of its historical best position, the historical best position is updated to the position and speed of the current particle. Otherwise, the historical best position remains unchanged, and the fitness value of each particle position in the particle swarm is repeatedly calculated until the preset maximum number of iterations is reached. The optimal palletizing path and position obtained after convergence are output to replace the original palletizing path and position.

8. The palletizing system based on image processing according to claim 7, characterized in that: The specific calculation formula for initializing the speed and position in step ① is as follows: Where, represents the initial position of the rth particle in the qth dimension; Represents the minimum position boundary in the qth dimension; Represents the maximum position boundary on the qth dimension; rand() represents generating a random number in the range [0, 1]; represents the initial velocity of the rth particle in the qth dimension; Represents the maximum position boundary in the qth dimension; Represents the minimum position boundary in the qth dimension; The specific calculation formula for position and speed update in step ③ is as follows: Where, represents the velocity of the rth particle in the qth dimension at time t+1; w represents the inertia weight; represents the velocity of the rth particle in the qth dimension at time t; c1 and c2 represent the acceleration coefficient of the particle approaching its own optimal position and the global optimal position, respectively; r1 and r2 represent random numbers in the range [0, 1]; represents the individual optimal position of the rth particle in the qth dimension; represents the position of the rth particle in the qth dimension at time t; g q Represents the component of the global best position in the qth dimension.

9. The palletizing system based on image processing according to claim 6, characterized in that: The specific steps for optimizing the ray tracing module visual system are as follows: Step I: Collect initial visual system data, including the robot's object recognition rate, positioning accuracy, and ray tracing results, in a virtual environment. Initialize the simulated scene, the maximum number of light reflections, and the refractive index. Based on the camera parameters, project light from the camera viewpoint into the scene to generate the initial light. Step II: Calculate the intersection of each ray with an object in the scene, determine whether the ray intersects the object, trace the ray's path in the scene, and record each intersection point and its corresponding normal vector. Calculate the reflection and refraction paths of the ray on the object's surface, and determine the splitting path of the ray based on the object's material properties. Step III: Based on the ray tracing path and the intersection information between the ray and the object, the color and brightness of each pixel in the scene are calculated. The colors and brightness of all pixels are combined to generate the final virtual image, and the proxy model is constructed through Gaussian process regression. Step IV: Perform initial simulations under different ray tracing parameter settings, collect corresponding positioning error data, calculate the expected improvement for each candidate point, evaluate the potential benefit of each candidate point as the next test point, and select the candidate point with the largest expected improvement value as the parameter for the next ray tracing simulation based on the expected improvement criterion; Step VI: Use the selected test points to perform ray tracing simulation, generate a virtual image and perform positioning, calculate the positioning error of the newly simulated image, and record the parameters of the test point and the corresponding error value. Repeat the expected improvement criterion calculation, test point selection, ray tracing simulation and model update until the preset convergence conditions are reached. Verify the optimized visual system in a virtual environment, make adjustments and improvements, and then apply the optimized visual system to actual operations to verify its performance in a real environment.

10. The palletizing system based on image processing according to claim 9, characterized in that: The specific calculation formula for the light reflection and refraction path in step II is as follows: Q reflect =Q-2(Q·n)n Where Q reflect represents reflected light; Q refract represents refracted light; η represents the refractive index; Q represents the incident light; n represents the normal vector of the object surface; The specific calculation formula for the expected improvement in step IV is as follows: EI(o)=Ε[max(0,f min -f(o))] Where, EI(o) represents the expected improvement value of candidate point o; E represents the expected value; f min represents the current optimal positioning error value; f(o) represents the positioning error function.

Images