A fruit can sorting path planning method and system based on two-dimensional code recognition

Abstract

The present application relates to the technical field of image processing, more particularly, the present application relates to a fruit canister sorting path planning method and system based on two-dimensional code recognition, comprising: collecting a two-dimensional code image of a fruit canister arranged on a conveying belt, preprocessing the two-dimensional code image to obtain the length of four edges of the two-dimensional code; for a plurality of path nodes in the mechanical arm workspace, a bar code recognition confidence and a canister fluid shaking constraint factor are constructed; the bar code recognition confidence is determined according to the ratio of the difference value and the average value of the length of the four edges, and is used to represent the image distortion degree of the two-dimensional code. The present application makes the mechanical arm not only able to actively bypass the high light area leading to recognition deviation, but also able to automatically slow down the turning radius in high-speed handling, thereby effectively reducing the problems of sorting collision, fruit flesh crushing and seal damage, etc.

Description

Technical Field

[0001] This invention relates to the field of image processing technology. More specifically, this invention relates to a method and system for sorting canned fruit based on QR code recognition. Background Technology

[0002] With the rapid development of automation technology and industrial robots, the technology of machine vision and robotic arm collaborative operation is widely used in intelligent manufacturing and logistics packaging. In such systems, vision sensors are typically used to collect image information of target objects for identification and positioning, and then the motion path of the robotic arm is planned to achieve automated grasping and handling of the target objects.

[0003] In the food processing and packaging industry, the mass production and automated sorting of canned fruit is a typical application of the aforementioned technology. On actual production lines, large quantities of canned fruit are randomly arranged and conveyed forward on high-speed conveyor belts, with QR codes printed on the surface of the cans for classification or traceability. To achieve efficient sorting, robotic arms equipped with vision cameras are used to track and identify the QR codes on the surface of the cans in real time on the dynamic production line, and plan a grasping path to accurately grasp and place specific categories of canned fruit into designated areas.

[0004] However, applying existing robotic arm path planning technology directly to the specific sorting scenario of canned fruit presents serious practical production problems. Traditional path planning algorithms typically only aim to find the shortest geometric distance or the fastest running time when searching for a trajectory, completely ignoring the physical characteristics of the object being grasped. On the one hand, canned fruit is a typical product with both solid and liquid components (containing syrup and fruit pulp). If the robotic arm simply aims to cover the shortest distance and its movement and turning are abrupt with sharp acceleration and deceleration, the inertial effect will cause significant shaking of the liquid inside the can. This violent shaking can easily crush the intact fruit pulp or impact the can lid, causing a leak and directly damaging the product quality. On the other hand, canned goods (such as glass or tinplate) are cylindrical and highly reflective. If the robotic arm only follows a rigid shortest path to grab the can, the camera's viewing angle will often fall within the direct bright light of the factory lighting or the strong distortion area of ​​the cylinder's edge. This will cause severe drift in the spatial coordinates of the QR code recognition, leading to the robotic arm failing to grab the can, or even directly knocking over or breaking the can on the conveyor belt. Therefore, using the traditional path planning method that only pursues the shortest distance is extremely prone to causing serious production accidents such as fruit pulp breakage, product damage, and robotic arm collisions during the sorting process. Summary of the Invention

[0005] This invention provides a method and system for sorting canned fruit based on QR code recognition, aiming to solve the problem that the traditional path planning method that only pursues the shortest distance in related technologies is prone to serious production accidents such as fruit pulp breakage, product damage, and robotic arm collisions during the sorting process.

[0006] In a first aspect, the present invention provides a method for sorting canned fruit based on QR code recognition, comprising: acquiring a QR code image of a can of fruit placed on a conveyor belt; preprocessing the QR code image to obtain the lengths of the four edges of the QR code; constructing a barcode recognition reliability and a can fluid sloshing constraint factor for multiple path nodes in the workspace of a robotic arm; the barcode recognition reliability is determined based on the ratio of the difference to the average of the lengths of the four edges, used to characterize the degree of image distortion of the QR code; the can fluid sloshing constraint factor is determined based on the preset operating speed of the robotic arm, the radius of curvature of the planned trajectory, and the preset filling ratio of the can, used to characterize the sloshing risk of the liquid inside the can; and employing... The algorithm searches the multiple path nodes to plan the sorting trajectory, wherein, The algorithm's cost function includes a geometric distance cost and a cost function composed of the barcode recognition reliability and the canned food sloshing constraint factor, controlling the robotic arm to perform the grasping operation on the canned fruit according to the sorting trajectory. This method simultaneously incorporates the distortion degree of the QR code image caused by reflection and cylindrical projection, as well as the sloshing risk of the solid-liquid mixture inside the can during movement, into the path search stage. In actual high-speed assembly line sorting scenarios, this allows the robotic arm to actively find the clearest angle for visual grasping and automatically avoid abrupt trajectories caused by sharp turns, completely avoiding grasping failures and collisions caused by strong light drift, as well as fruit pulp breakage or seal damage caused by violent inertial shaking, greatly ensuring the accuracy of sorting operations and the quality of the finished product.

[0007] Furthermore, the method for calculating the barcode recognition reliability includes: dividing the difference between the maximum and minimum lengths of the four edges by their arithmetic mean to obtain the distortion ratio; applying an exponential function to the distortion ratio; and wherein the barcode recognition reliability is negatively correlated with the distortion ratio. In workshops with complex and variable lighting environments, the highly sensitive exponential function amplifies the impact of image edge differences, accurately and quantitatively reflecting the severity of visual deviations caused by changes in viewing angle at different positions on the production line. This provides the system with a reliable error-proofing evaluation standard, effectively preventing the robotic arm from blindly issuing grasping commands in high-risk areas with visual uncertainty.

[0008] Furthermore, the calculation method for the fluid sloshing constraint factor of the canned goods includes: the fluid sloshing constraint factor is negatively correlated with the product of the square of the instantaneous linear velocity of the robotic arm end at each node and the filling ratio of the can, and negatively correlated with the radius of curvature of the path at each node. By directly mapping the kinematic parameters to the dynamic feedback of the liquid inside the can, the system can proactively predict the destructive force that centripetal acceleration may cause when customizing sorting strategies for each type of can with different sugar-water ratios and liquid levels, thus fundamentally limiting the violent actions of the robotic arm when handling fragile items with both solid and liquid components.

[0009] Furthermore, the method for constructing the cost function includes: weighted summation of the reciprocal of the barcode recognition reliability and weighted summation of the canned fluid sloshing constraint factor to obtain the cost function. This method endows the automated system with a comprehensive trade-off capability under complex operating conditions, enabling it to flexibly allocate the priority of visual clarity and the priority of motion stability according to the pain points of the actual factory, ensuring that the robotic arm can always find a golden compromise route that satisfies both visual anti-missing and ensures the integrity of the goods when dynamically tracking the target.

[0010] Furthermore, A The formula for calculating the cost function of the algorithm is: ;in, The cost of path node j; From the starting point of the path node to the current path node The actual physical distance is the Euclidean distance between the centers of the two path nodes. From path node The estimated physical distance to the target path node; The visual confidence weight constant; The fluid stability weighting constant; The reliability of barcode recognition at path node j; Let be the fluid sloshing constraint factor at path node j. For the traditional A... The algorithm kernel has been deeply modified to adapt to special industrial scenarios. While retaining its advantage of shortest distance search, it uses a rigorous mathematical model to transform anti-reflective misjudgment and anti-shaking damage into explicit path penalty terms, enabling the underlying algorithm to have the intelligent risk avoidance capabilities of actively avoiding high-reflective areas and actively reducing the curvature of sharp turns.

[0011] Furthermore, the method for calculating physical distance includes: for any two path nodes, determining the center points of the two path nodes, and calculating the Euclidean distance between the center points of the two path nodes as the physical distance. This provides an efficient and standard method for calculating grid spacing in the 3D grid map defined in the factory robotic arm's workspace, ensuring the absolute accuracy of macroscopic path search progressing at the physical spatial scale.

[0012] Furthermore, the formula for calculating the fluid sloshing constraint factor is as follows: ;in, is the fluid sloshing constraint factor at path node j; Let be the instantaneous linear velocity of the robotic arm's end effector at path node j; Let be the radius of curvature of the path at path node j; This refers to the filling ratio of the canned food.

[0013] Furthermore, the preprocessing of the QR code image includes: performing distortion correction on the QR code image using pre-calibrated camera parameters, and enhancing the contrast of the QR code image using an adaptive local threshold binarization algorithm.

[0014] Furthermore, the method also includes: before controlling the robotic arm to perform the grasping operation, using a B-spline curve fitting algorithm to smooth the sorting trajectory to generate continuous and differentiable motion commands. This method eliminates the abruptness associated with the polyline trajectory generated by the discrete grid map, seamlessly transforming the theoretically optimal path into continuous servo commands that the robotic arm motor can directly and smoothly execute. This avoids instantaneous mechanical shocks during joint startup and reversal, achieving ultimate smoothness and safety throughout the entire grasping process.

[0015] In a second aspect, a fruit canning path planning system based on QR code recognition includes a processor and a memory, characterized in that the memory stores a computer program, and the processor executes the computer program to implement the fruit canning path planning method based on QR code recognition as described above.

[0016] Beneficial effects: For automated sorting of canned fruit, a system was constructed to quantify the reliability of visual barcode recognition in the reflective blind zone of a cylindrical surface, and a fluid sloshing physical factor constraining the kinetic energy of a solid-liquid mixture was developed. These two factors were then deeply integrated into the A algorithm as dynamic penalty terms. The algorithm's cost function enables the robotic arm not only to actively avoid the bright areas that cause recognition errors, but also to automatically slow down the turning radius during high-speed handling, thereby effectively reducing problems such as sorting collisions, fruit pulp breakage, and seal damage. Attached Figure Description

[0017] Figure 1 This is a schematic flowchart illustrating a sorting path node planning method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the decrease in reliability as the degree of distortion varies according to an embodiment of the present invention; Figure 3This is a schematic diagram illustrating the relationship between the fluid sloshing constraint factor (Φ) and path parameters according to an embodiment of the present invention. Detailed Implementation

[0018] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] like Figure 1 As shown, S101: Data acquisition and preprocessing.

[0020] In this embodiment, an industrial camera deployed above the high-speed conveyor belt first captures the QR code image of the canned fruit in real time. For example, a high frame rate global shutter can be used. Cameras are used to reduce motion blur. Because the cans are made of cylindrical materials such as tinplate or glass, their surfaces are prone to reflection, and the QR code images will suffer non-linear distortion due to curved projection.

[0021] Therefore, after obtaining the original image, a series of preprocessing operations need to be performed. Specifically, firstly, distortion correction is performed on the image using pre-calibrated camera intrinsic and extrinsic parameters. Then, to address uneven lighting in the production workshop and specular reflections caused by water droplets on the bottle, an adaptive local threshold binarization algorithm can be used to enhance the contrast of the QR code area. Next, noise points are removed through image morphological operations, such as opening and closing operations, and the four corner points of the QR code are initially located using methods such as Hough transform or contour detection, providing basic data for subsequent index calculations. Finally, the workspace of the robotic arm is divided into a grid map, where each cell can be considered a path node.

[0022] S102: Constructing barcode recognition credibility.

[0023] In fruit canning scenarios, the position and spin angle of the cans on the conveyor belt are random. The robotic arm needs to perform grasping operations at different locations on the conveyor belt (upstream, midstream, downstream). There are usually lights above the conveyor belt, and at certain fixed locations (such as where the light shines directly), the QR code on the can surface will experience severe glare. Combined with the projection distortion of the cylindrical surface, this will cause fluctuations in the spatial coordinates of the QR code recognition. To quantify the accuracy of QR code recognition, this embodiment constructs a barcode recognition reliability (e.g., Figure 2 The diagram shows the decrease in barcode recognition reliability as the degree of distortion increases.

[0024] This indicator is based on the fact that when a standard square QR code undergoes geometric distortion due to projection, the projected lengths of its four sides in the image will differ. The magnitude of this difference can effectively characterize the severity of image distortion and indirectly reflect the reliability of the recognition result. Its calculation formula is as follows: ;In the formula, This indicates that the tag for the canned food that needs to be retrieved is located in the path node. The recognition credibility index at the location has a value range of 100%. ; The ambient light suppression coefficient is used to adjust the sensitivity of reliability to the influence of optical reflection caused by changes in humidity or light intensity. As a preferred option, its value range can be set to [value range missing]. ; For path nodes The maximum value among the four edge lengths of the preprocessed QR code image; For path nodes The value at this point corresponds to the minimum of the four edge lengths. For path nodes The arithmetic mean of the lengths of the four edges.

[0025] As can be seen from the above formula, when the can's position is at a path node of image distortion, causing the QR code image distortion to worsen, and The difference will increase significantly, causing the fractional term to... The value increases. Since the exponential function is monotonically decreasing, the final calculated recognition reliability index... This will decrease exponentially. Therefore, identifying confidence metrics can accurately quantify the degree of image distortion caused by cylindrical projection and viewpoint changes in a physical scene, providing a crucial confidence reference for subsequent path node planning.

[0026] S103: Construct fluid sloshing constraint factor for canned goods.

[0027] Canned fruit, as a special sorting object containing both solid and liquid components, contains a large amount of syrup and fruit pulp. If the path planning of the sorting robot arm during the grasping and movement process is too rigid, i.e., the acceleration changes drastically, the liquid inside the can will shake significantly due to inertia, potentially causing the fruit pulp to break or the can to not seal properly, thus affecting product quality. To mitigate this risk during the path planning stage, this embodiment constructs a fluid sloshing constraint factor for canned fruit.

[0028] The rationale behind this index is that the core physical factor causing liquid sloshing is the centripetal acceleration of the robotic arm's end effector during its movement. Centripetal acceleration is directly proportional to the square of the instantaneous linear velocity and inversely proportional to the radius of curvature of the path nodes. Furthermore, the can's filling ratio also determines the size of the free space for liquid sloshing. Based on this, a fluid sloshing constraint factor index is constructed. The calculation formula is as follows: ;In the formula, Path node The fluid sloshing constraint factor, the larger its value, the stronger the path node. The greater the degree of damage to the internal stability of the can; For the end effector of the sorting robot at this path node The instantaneous linear velocity can be obtained through the preset speed of the robotic arm; To plan the path at the path nodes The radius of curvature can be obtained by taking the second derivative of the analytical geometric equation of the path node; The filling ratio of the can is the ratio of the liquid level to the total height of the can. In this embodiment, The preferred value range is... ; The value is 0.1 to avoid the denominator being 0.

[0029] From this formula, we can see that when the operating speed of the robotic arm... When the radius of curvature increases, or when the path node turns too sharply (i.e., the radius of curvature decreases), The value will increase dramatically. Specifically, by introducing the filling ratio... This formula can also reflect the influence of the physical state inside the tank: when Smaller, meaning more air inside the can, the easier it is for the inside of the can to shake, i.e., the molecules within... It will get bigger, causing The value increases significantly, which perfectly matches the physical phenomenon that a larger sloshing space makes the liquid more prone to sloshing. This index provides a clear dynamic constraint for the path node planning algorithm.

[0030] S104: Improvement A Cost function and path node search.

[0031] standard When calculating the cost function of path nodes, algorithms typically only consider geometric distance costs, which cannot meet the dual constraints of resisting visual bias and preventing swaying required in this invention. This embodiment integrates the physical indices constructed in the first two steps into... The algorithm reshapes its cost function, thereby proactively avoiding areas with high vibration and low recognition accuracy during the path node search stage.

[0032] Improved adaptive cost function The construction is based on adding a penalty term related to the physical state to the traditional distance cost. This penalty term consists of two parts, corresponding to the uncertainty of visual recognition and the instability of fluid dynamics, and the fluid sloshing constraint factor needs to be normalized before calculation. Its calculation formula is as follows: ;In the formula, The improved total value; From the starting point of the path node to the current path node The actual physical distance is the Euclidean distance between the centers of the two path nodes. From path node The estimated physical distance to the target path node; The visual confidence weight constant has a value of [value]. ; The fluid stationary weighting constant has a value of [value missing]. ; and Path nodes The identification credibility index and fluid sloshing constraint factor.

[0033] The logical relationship of this formula is as follows: In A When the algorithm searches for path nodes, if the recognition confidence index is low on a straight path, the algorithm imposes a high penalty on the straight path, forcing the robotic arm's end effector to make a slight arc, cutting in from an angle above or avoiding highlights. This ensures the clearest QR code recognition at the last moment before grasping. For example, calculations show that if the robotic arm grasps at position C, the recognition confidence index is extremely low due to reflection, resulting in large coordinate errors. However, if the robotic arm waits an extra 0.5 seconds, allowing the can to move with the conveyor belt to position D before grasping, the change in light angle results in a higher recognition confidence index, allowing for accurate grasping of the can. This enables the automatically generated grasping trajectory to guide the camera away from light convergence points and physical shadow areas, achieving accurate QR code locking in dynamic environments and effectively reducing the risk of the robotic arm colliding with the can due to inaccurate recognition. In other words, when a path node has strong reflection or causes violent shaking, the passage weight of that path node increases significantly, prompting A... When searching for the minimum cost path, the algorithm naturally avoids this high-risk area, thereby achieving optimal visual stability and motion smoothness while ensuring path continuity.

[0034] However, when a certain segment of the path node has excessive speed or a sharp turn, it causes a swaying factor. When it increases, The number of parameters also increases, forcing the algorithm to choose smoother trajectories with smaller acceleration changes. Understandably, this leads to fluid stability weights. It needs to be greater than the visual confidence weight. This indicates that in this application scenario, ensuring product quality takes precedence over optimizing the recognition angle. Through comprehensive penalties on path nodes, the algorithm can automatically search for a path with a total cost. Minimum, which is the trajectory that best combines distance, clarity of recognition, and stability of the liquid surface.

[0035] S105: Generate the optimal sorting trajectory and execute it.

[0036] During the path node search phase, a cost function with the aforementioned improvements is used. of The algorithm searches within a grid map consisting of the conveyor belt and the robotic arm's workspace. Upon convergence, the algorithm outputs a series of optimal path nodes. To ensure a smooth trajectory for the robotic arm, this series of discrete path nodes can be processed... Spline curve fitting generates a continuous and high-order differentiable motion trajectory. Finally, this trajectory is transformed into a sequence of control commands for each joint of the robotic arm, driving the sorting system to perform grasping and placement actions. In this way, the present invention achieves closed-loop optimization from visual perception and physical constraint modeling to motion control, ensuring high efficiency, accuracy, and high quality in the sorting process.

[0037] like Figure 3 As shown, this 3D graph visually illustrates how the motion parameters of the robotic arm affect the stability of the fluid inside the tank. The 3D surface in the graph plots the fluid sloshing constraint factor (Z-axis) based on the robotic arm's movement speed (X-axis) and the radius of curvature of the path nodes (Y-axis, representing the sharpness of turns). The surface shows that the faster the speed or the sharper the turn (smaller the radius of curvature), the higher the fluid sloshing constraint factor and the more severe the sloshing. This provides theoretical support for suppressing rapid acceleration and sharp turns in subsequent path node planning.

[0038] This invention also provides a fruit canning sorting path planning system based on QR code recognition. The system includes a processor and a memory, the memory storing computer program instructions. When the processor executes the computer program instructions, it implements the fruit canning sorting path planning method based on QR code recognition according to the first aspect of this invention.

[0039] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and therefore will not be described in detail here.

[0040] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions stored or otherwise maintained on such a computer-readable medium.

[0041] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for sorting canned fruit based on QR code recognition, characterized in that, include: The QR code image of the fruit cans set on the conveyor belt is acquired, and the QR code image is preprocessed to obtain the lengths of the four edges of the QR code; For multiple path nodes in the workspace of the robotic arm, a barcode recognition confidence level and a can fluid sloshing constraint factor are constructed. The barcode recognition confidence level is determined based on the ratio of the difference and average of the lengths of the four edges, and is used to characterize the degree of image distortion of the QR code. The can fluid sloshing constraint factor is determined based on the preset operating speed of the robotic arm, the radius of curvature of the planned trajectory, and the preset filling ratio of the can, and is used to characterize the sloshing risk of the liquid inside the can. The method for calculating the reliability of barcode recognition includes: The distortion ratio is obtained by dividing the difference between the maximum and minimum lengths of the four edges by their arithmetic mean. An exponential function is applied to the distortion ratio, wherein the barcode recognition reliability is negatively correlated with the distortion ratio. The current requirement is to retrieve the tags for canned goods from the path nodes. Reliability index of identification for: ； This is the ambient light suppression coefficient, used to adjust the sensitivity of credibility to the impact of optical reflections caused by changes in humidity or light intensity. For path nodes The maximum value among the four edge lengths of the preprocessed QR code image; For path nodes The value at this point corresponds to the minimum of the four edge lengths. For path nodes The arithmetic mean of the lengths of the four edges; use The algorithm searches the multiple path nodes to plan the sorting trajectory, wherein, The algorithm's cost function includes a geometric distance cost and a cost function composed of the barcode recognition confidence and the canned food fluid sloshing constraint factor, controlling the robotic arm to perform the grasping operation on the canned fruit according to the sorting trajectory; A The formula for calculating the cost function of the algorithm is: ； in, The cost of path node j; From the starting point of the path node to the current path node The actual physical distance is the Euclidean distance between the centers of the two path nodes; From path node The estimated physical distance to the target path node; The visual confidence weight constant; The fluid stability weighting constant; The reliability of barcode recognition at path node j; is the constraint factor for fluid sloshing in the can at path node j.

2. The fruit canning path planning method based on QR code recognition according to claim 1, characterized in that, The calculation method for the fluid sloshing constraint factor in the canned food includes: The fluid sloshing constraint factor of the can is negatively correlated with the product of the square of the instantaneous linear velocity of the robotic arm end at each node and the preset filling ratio of the can, and negatively correlated with the radius of curvature of the path at each node.

3. The fruit canning path planning method based on QR code recognition according to claim 1, characterized in that, The method for constructing the cost function includes: The cost function is obtained by weighted summation of the reciprocal of the barcode recognition reliability of the canned goods and the weighted summation of the fluid sloshing constraint factor of the canned goods.

4. The fruit canning path planning method based on QR code recognition according to claim 1, characterized in that, Methods for calculating physical distance include: For any two path nodes, determine the center point of the two path nodes, and calculate the Euclidean distance between the center points of the two path nodes as the physical distance.

5. The fruit canning path planning method based on QR code recognition according to claim 1 or 2, characterized in that, The formula for calculating the fluid sloshing constraint factor in the canned food is as follows: ； in, The constraint factor for the sloshing of the canned fluid at path node j; Let be the instantaneous linear velocity of the robotic arm's end effector at path node j; Let be the radius of curvature of the path at path node j; The preset filling ratio for canned goods.

6. The fruit canning path planning method based on QR code recognition according to claim 1, characterized in that, Preprocessing the QR code image further includes: The QR code image is distorted using pre-calibrated camera parameters, and the contrast of the QR code image is enhanced using an adaptive local threshold binarization algorithm.

7. The fruit canning path planning method based on QR code recognition according to claim 1, characterized in that, The method further includes: Before controlling the robotic arm to perform the grasping operation, the sorting trajectory is smoothed using a B-spline curve fitting algorithm to generate continuous and differentiable motion commands.

8. A fruit canning sorting path planning system based on QR code recognition, comprising a processor and a memory, characterized in that, The memory stores a computer program, and the processor executes the computer program to implement the fruit canning path planning method based on QR code recognition as described in any one of claims 1-7.

Images