Dried fruit and vegetable product identification and sorting system and control method

Abstract

This application belongs to the field of intelligent material sorting technology, specifically to a dried fruit and vegetable product identification and sorting system and control method. The system includes a vibrating loading mechanism, a conveying device, a camera, a horizontal articulated robotic arm, a weighing and counting mechanism, and a control device. Based on the recognition results of the control device, the horizontal articulated robotic arm is controlled to pick up each identified dried fruit and vegetable product. This invention can improve the production efficiency of dried fruit and vegetable products.

Description

Technical Field

[0001] The present application belongs to the technical field of intelligent material sorting, and specifically relates to a dried fruit and vegetable product identification and sorting system and control method. Background Art

[0002] Dried fruit and vegetable products are dried foods made from fruits and vegetables through cooking, freezing and other techniques. During the sorting and bagging process of dried fruit and vegetable products, taking pumpkin slices as an example, it is required that when they are packaged into bags, the number of pumpkin slices in each bag is the same and the total weight has a fixed requirement, and no adhesion occurs. However, after slicing, cooking and refrigeration and other processing techniques, multiple pumpkin slices will stick together, which affects the difficulty and efficiency of subsequent sorting and bagging; and in order to avoid the frozen pumpkin slices from melting and deforming, the sorting process cannot consume too much time. Therefore, the automatic breaking up and identification of pumpkin slices, as well as the rapid sorting and packaging, have become technical difficulties that limit the automated production of pumpkin slices. Currently, there is no product on the market that can accurately sort out individual dried fruits and vegetables from a pile and combine them into a set of fixed weight products. The technology proposed in the present invention can make up for this deficiency. Summary of the Invention

[0003] In order to address the deficiencies of the prior art, the present invention aims to provide a dried fruit and vegetable product identification and sorting device and a control method thereof, so as to achieve automated breaking up, identification, and rapid sorting of dried fruit and vegetable products, thereby improving the production efficiency of dried fruit and vegetable products. The technical solution is as follows:

[0004] A dried fruit and vegetable product identification and sorting system includes a vibration feeding mechanism, a conveying device, a camera device, a horizontal joint mechanical arm, a weighing and counting mechanism, and a control device;

[0005] The vibration feeding mechanism is used to output the material to the conveying device;

[0006] The horizontal joint robot arm adjusts its direction and cooperates with the end mechanism to pick up materials and place them in the weighing and counting mechanism;

[0007] The camera device is used to obtain images of materials on the conveyor device;

[0008] The weighing and numbering mechanism weighs the obtained materials, and when the weight of the materials reaches a set threshold, the control device pushes the materials out;

[0009] The control device is used to control the vibration feeding mechanism, the conveying device, the horizontal joint robot arm and the weighing and making up the number mechanism movement as well as image processing.

[0010] Preferably, the vibrating feeding mechanism includes a feeding tray, a vibrator and a primary conveyor belt; a vibrator is provided under the feeding tray, and the vibrator is fixed to the conveyor belt base through a feeding base; a breaking wheel is provided on the primary conveyor belt, and a plurality of groups of rubber soft rods with uniform gaps are provided on the breaking wheel.

[0011] Preferably, the conveying device includes a secondary conveyor belt, a leveling wheel and a leveling wheel motor are arranged above the secondary conveyor belt, and a leveling soft plate is installed on the leveling wheel; the leveling wheel is fixed on the conveyor belt base two, and is fixed to the bottom plate through the conveyor belt base two.

[0012] Preferably, the horizontal joint robotic arm is located on one side of the secondary conveyor belt of the conveying device, and the end of the horizontal joint robotic arm has at least four degrees of freedom: translation along the X, Y, and Z axes and rotation around the Z axis; the end of the horizontal joint robotic arm is also provided with an end mechanism, which includes a suction cup, an elastic buffer rod, a suction cup connecting plate and an air pipe; the suction cup is connected to the end of the elastic buffer rod, and the elastic buffer rod is fixed on the suction cup connecting plate; a spring is provided on the elastic buffer rod, and the suction cup connecting plate includes an L-shaped plate and a straight plate, and a through hole is formed between the two for clamping and fixing the elastic buffer rod and the end sleeve of the robotic arm together.

[0013] Preferably, the weighing and making up mechanism includes a plurality of identical weighing units, each weighing unit including, from top to bottom, a discharge hopper, a buffer hopper, a weighing tray, a pushing slider, and a weighing sensor; the discharge hopper and the buffer hopper are both provided with a fan-shaped switch plate, which is opened and closed by a rotating cylinder, and the material falls from the discharge hopper to the buffer hopper below, and then falls to the weighing tray below. The weighing sensor below the weighing tray obtains the weight value of the material and feeds it back to the control device. The control device solves the optimal weight combination and controls the pushing slider to push out the group of materials.

[0014] Preferably, the control device includes a host computer and a slave computer, which are communicatively connected. The slave computer controls the movement of the vibration feeding mechanism, the conveying device, the horizontal joint robot arm, the end mechanism and the weighing and making-up mechanism; the host computer divides the area according to the image processing structure and controls the movement trajectory of the horizontal joint robot arm.

[0015] A method for identifying and sorting dried fruit and vegetable products comprises the following steps:

[0016] S100, acquiring an image of a working area on a conveying device;

[0017] S200, when it is recognized that there is a material to be identified in the working area, the conveying device is controlled to advance a set distance and then stop;

[0018] S300, after the vibrating feeding mechanism and the conveying device stop, an image of the working area on the conveying device is acquired again to identify the material to be sucked in the image;

[0019] S400: Control the horizontal joint robot arm to sequentially absorb the identified materials and place them in several weighing and numbering mechanisms. The weighing and numbering mechanisms feed back the weight values ​​to the host computer, which solves the optimal weight combination and controls the pusher slider to push the materials out.

[0020] S500: When the amount of dried fruit and vegetable products in the working area is lower than the set threshold, the vibration feeding mechanism and the conveying device are controlled to move forward a set distance again and then stop, and steps S300-S500 are repeated.

[0021] Preferably, in step S200, the secondary conveyor belt of the conveying device carries the scattered materials and advances at a constant speed, and is preliminarily broken up and flattened by the scattering wheel, and the camera continuously acquires the image of the working area below. When the front material enters the edge of the working area, the camera captures the image with the first material target. At this time, the upper computer recognizes the material target at the front side of the working area in the image, and controls the conveying device to advance a set distance and then stop. The set distance here is the length of the working area in the advancing direction of the secondary conveyor belt of the conveying device. When the secondary conveyor belt runs at a constant speed, the advancing distance of the secondary conveyor belt can be controlled by controlling the advancing time of the secondary conveyor belt. When the front material on the secondary conveyor belt just covers the working area, the secondary conveyor belt stops advancing.

[0022] In step S300, the Yolov5 model is used as the basis, and the labeled enhanced image of the material on the conveyor belt is input to perform supervised training on the model. At the same time, the confidence threshold of the target frame is adjusted during training, the anchor frame selection mechanism is improved, and the probability of overlapping frames and containing multiple target frames is reduced. A trained recognition model is obtained and deployed to the host computer; the camera captures the current frame image in real time and uploads it to the host computer. The host computer recognizes the materials in the collected image of the working area, marks each target with a rectangular frame, and plans the coordinate sequence of all materials. The planned coordinate sequence forms the picking path of the robotic arm;

[0023] In step S400, the upper computer plans the movement trajectory of the horizontal joint manipulator according to the weighing units at different positions of the weighing and making-up mechanism, and numbers the rotary cylinders corresponding to the discharge hopper and the buffer hopper; when the horizontal joint manipulator places the material into the set discharge hopper of the weighing and making-up mechanism, the lower computer controls the rotary cylinder of the discharge hopper to open the fan-shaped switch plate at the bottom of the discharge hopper, and the material falls into the buffer hopper on the lower layer. According to the set time, the upper discharge hopper closes the fan-shaped switch plate, and the lower buffer hopper opens its fan-shaped switch plate, and the material falls into the material bag on the tray again;

[0024] The horizontal joint robot arm uses a "gate"-shaped trajectory when picking and placing. A fixed-ratio Lam'e curve is used to optimize the vertical height and lateral movement corners. Then, the horizontal straight line segments are eliminated and directly replaced by Lam'e curves.

[0025] Preferably, the communication mechanism adopts a mutual feedback mechanism of the "hook principle", specifically:

[0026] During the first hook, the camera acquires an image, and the host computer recognizes the presence of material. The first and second conveyor belts advance a set distance and then stop. The host computer receives information that the motors of the first and second conveyor belts have stopped. At this time, the camera takes a picture of a still and stable image, and obtains accurate information about all targets in the image. When all targets in the image are picked up by the horizontal joint robotic arm, the camera takes a picture and recognizes that there are no targets in the image, and feedback is sent to the host computer, and the conveyor belt continues to advance to the next working distance. During the second hook, a single material is picked up by the robotic arm and placed in the hopper of each weighing unit. In the initial state of weight addition, each weighing unit contains one piece of material, and materials are continuously put into the weighing unit. When the material weight combination of a certain weighing unit is calculated to meet the requirements, the lower computer receives a signal that the material on the weighing tray is pushed out, and controls the action of the push slider; the buffer hoppers of other units open to refill the weighing tray below.

[0027] The third time the hand is hooked, during the discharge process, the discharge hopper and the buffer hopper have opening and closing signals, and the suction cup exhales when discharging. The upper computer knows that the horizontal joint robotic arm has put down the picked up material, and the upper computer can control the horizontal joint robotic arm to absorb the next piece of material; when all the discharge hoppers do not generate the opening and closing signal of the material dropping, the horizontal joint robotic arm grabs the material and waits in the air just above the weighing and making up the number mechanism.

[0028] Preferably, in the identification of dried fruits and vegetables, the modified Yolov5 Model recognition, the steps are as follows: Improve the original IoU Expression:

[0029] ;

[0030] ;

[0031] in:

[0032] : Dynamic weight, adjusted according to confidence IoU The geometric contribution weight of

[0033] B and B gt The predicted box and the real box in the detection respectively; b and b gt Defined asB and B gt the center of C diag is the diagonal length of the smallest rectangle that covers both boxes;

[0034] d and d gt yes B and B gt Vector expression for diagonal; | d | and | d gt | indicates d and d gt length;

[0035] b sc_max and b i Represent the maximum confidence score box and other redundant boxes respectively;

[0036] : The Euclidean distance between the center of the predicted box and the true box, indicating the center offset;

[0037] : Diagonal vector difference, measuring the shape difference of the box;

[0038] : The directional angle difference of the diagonal vector is used to capture the directional change of the frame;

[0039] Improved loss function for:

[0040] ;

[0041] ;

[0042] in:

[0043] IoU Item: represents the intersection-over-union ratio of the predicted box and the true box:

[0044] IoU= ;

[0045] Center shift penalty: ;

[0046] is the offset distance of the frame center;

[0047] is the minimum bounding box diagonal length, used to normalize the offset distance;

[0048] Diagonal Difference Term: ;

[0049] , represents the diagonal vector difference, The product of the diagonal lengths is used to represent the normalized weight and measure the similarity of the box size;

[0050] Direction angle difference term: ;

[0051] Represents the square of the angle difference, normalizing the orientation difference of the weighted box;

[0052] Dynamic weight adjustment:

[0053] : Weight factor, dynamically adjusted to adapt to different data sets and detection tasks;

[0054] ;

[0055] Improved dense area suppression function DR_NMS : Taking into account the overlapping area ratio of the box, the diagonal direction vector of the box, and the confidence, a direction-aware IoU Suppression strategy, design similarity score , to improve the accuracy of dense target detection:

[0056] ;

[0057] ;

[0058] in:

[0059] Area ratio, which measures the similarity of the areas of two boxes:

[0060] ;

[0061] Diagonal ratio, which measures the similarity of the diagonal lengths of boxes:

[0062] ;

[0063] The direction angle ratio reflects the consistency of the frame direction through the ratio of the cosine values ​​of the direction angles:

[0064] ;

[0065] 、 、 are the weights of these three ratios, the sum of which is 1;

[0066] Direction-aware similarity S dir and improvements DR_IoU Combined with the final suppression for dynamic adjustment of the frame, the final DR_NMS After adjustment for:

[0067] .

[0068] Compared with the prior art, this application has the following beneficial effects:

[0069] The device's overall structure is primarily based on a horizontally articulated robotic arm, suitable for high-speed sorting of materials (such as pumpkin slices and dried fruit and vegetable products) on a conveyor belt. The arm's end features an elastic buffer shaft and a flexible suction cup to prevent collisions during high-speed picking, ensuring the slices are not damaged. A scattering wheel on the conveyor belt automatically scatters the slices to facilitate subsequent pickup by the suction cup. When the suction cups pick up the slices, a pause-and-grab system is employed. When the conveyor belt, laden with slices, reaches the camera's working area, it pauses. The camera takes a single image, identifying all objects in the current working area. This system then plans a complete path, guiding the robotic arm to sequentially pick and place all objects. Once all objects have been picked up, the conveyor belt moves on to the next working area, ensuring the quality of the picked slices. The robotic arm's path planning incorporates an arc-optimized "gate"-shaped trajectory, significantly improving both picking speed and smoothness. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A perspective view of the present invention;

[0071] Figure 2 It is a front view of the present invention;

[0072] Figure 3 It is a left side view of the present invention;

[0073] Figure 4 A top view of the present invention;

[0074] Figure 5 It is a partial schematic diagram of the terminal mechanism;

[0075] Figure 6 It is a schematic diagram of the scattering wheel and the leveling wheel;

[0076] Figure 7 A schematic diagram of the local effect of identifying dried fruit and vegetable products in the work area;

[0077] Figure 8Schematic diagram of the movement trajectory of the end of the robotic arm.

[0078] Among them, 1. Vibration feeding mechanism; 2. Feeding tray; 3. Vibrator; 4. Feeding base; 5. Control cabinet; 6. Conveyor belt base 1; 7. Bottom plate; 8. Industrial camera; 9. End sleeve of robot arm; 10. End mechanism; 11. Horizontal joint robot arm; 12. Camera bracket; 13. Weighing and numbering mechanism; 14. Robot arm base; 15. Base fixing plate; 16. Scattering wheel; 17. Discharge hopper; 18. Buffer hopper; 1 9. Weighing tray; 20. Pushing slider; 21. Weighing sensor; 22. Primary conveyor belt; 23. Breaking wheel motor; 24. Leveling wheel; 25. Secondary conveyor belt; 26. Leveling wheel motor; 27. Robot arm working range trajectory; 28. Suction cup connecting plate; 29. ​​Elastic buffer rod; 30. Suction cup; 31. Rubber soft rod; 32. Leveling soft plate; 33. Fan-shaped switch plate; 34. Conveyor belt base II; 35. Weighing fixed plate. DETAILED DESCRIPTION

[0079] In order to facilitate the understanding of the present invention, the present invention is further described below through examples in conjunction with the accompanying drawings, but the present invention is not limited thereto and is not intended to be limited thereto.

[0080] The embodiment of the present invention, as Figure 1-Figure 7 As shown, a device for identifying and sorting dried fruit and vegetable products is first provided, comprising:

[0081] The vibrating feeding mechanism 1 includes a feeding tray 2 and a vibrator 3, which is used to drop the piled dried fruits and vegetables more evenly onto the primary conveyor belt 22. The vibrating feeding mechanism 1 is fixed to the conveyor belt base 1 6 through the feeding base 4 and is located above the primary conveyor belt 22. The conveyor belt base 1 6 is fixed to the bottom plate 7.

[0082] The primary conveyor belt 22 is provided with a scattering wheel 16 , which is designed with multiple sets of rubber rods 31 with uniform gaps, so as to scatter the accumulated dried fruits and vegetables without damaging them.

[0083] A leveling wheel 24 and a leveling wheel motor 26 are provided above the secondary conveyor belt 25. Two leveling soft plates 32 are installed on the leveling wheel 24, which can level the dried fruits and vegetables that still have individual vertical spirals after being scattered into a layer; the leveling wheel 24 is fixed on the conveyor belt base 34 and is fixed to the bottom plate 7 through the conveyor belt base 34.

[0084] The primary conveyor belt 22 is located above the secondary conveyor belt 25 , and the ends of the primary conveyor belt 22 and the secondary conveyor belt 25 are arranged to overlap each other.

[0085] The camera mechanism includes an industrial camera 8 and a camera bracket 12. The industrial camera 8 is located above the horizontal joint mechanical arm 11 and is fixedly arranged toward the conveyor belt to obtain images of the scattered dried fruits and vegetables on the conveyor belt.

[0086] The horizontal joint robot arm 11 is located on one side of the secondary conveyor belt 25 and is fixed to the robot arm base 14. The robot arm base 14 is fixed to the bottom plate 7 through the base fixing plate 15. The end of the horizontal joint robot arm 11 has at least four degrees of freedom: translation along the X, Y, and Z axes and rotation around the Z axis; the end of the horizontal joint robot arm 11 is also provided with an end mechanism 10, which includes a suction cup 30 for sucking dried fruits and vegetables on the conveyor belt. The suction cup 30 is connected to the end of the joint robot arm through an elastic buffer rod 29. The elastic buffer rod 29 is fixed to the end sleeve 9 of the robot arm through an end connecting plate. The end connecting plate is divided into a straight plate and an L-shaped plate.

[0087] The weighing and counting mechanism 13, located on the same side of the horizontally articulated robotic arm 11 and secured to the robotic arm base 14 via a weighing fixing plate 35, comprises four identical weighing units, each of which includes a discharge hopper 17, a buffer hopper 18, a weighing tray 19, a pusher slide 20, and a weighing sensor 21. The discharge hopper 17 and buffer hopper 18 are equipped with fan-shaped switch plates 33, which are opened and closed by a rotating cylinder, allowing dried fruits and vegetables to fall onto the buffer hopper below. Similarly, dried fruits and vegetables in the buffer hopper fall onto the weighing tray below. The weighing sensor below the weighing tray quickly determines the weight of the dried fruits and vegetables and feeds it back to the host computer. The host computer determines the optimal weight combination and controls the pusher slide via the lower computer to push the dried fruits and vegetables out.

[0088] The control device includes a host computer and a slave computer. The slave computer is located in the control cabinet 5. The host computer is a computer. The computer is used to identify each dried fruit and vegetable product in the image captured by the industrial camera and to perform weight combination calculations on the four hoppers to determine the tray number from which the material is pushed out. The slave computer is used to control the horizontal joint robot arm to absorb the identified dried fruit and vegetable products based on the computer's recognition results and to control the opening and closing of the cylinders of the hoppers and the pusher slide of the counting and weighing mechanism.

[0089] During the production and processing of dried fruit and vegetable products such as pumpkin slices, after slicing, cooking, and refrigeration, the randomly stacked pumpkin slices of varying shapes and sizes are transported via conveyor belts to sorting and packaging stations. Traditionally, pumpkin slice sorting and packaging involves manually breaking apart stuck-together pumpkin slices, counting and removing the required number of individual pieces, and placing them into bags. This approach suffers from low efficiency and poor hygiene. To address this issue, this embodiment provides a dried fruit and vegetable product identification and sorting device. By installing a breaking wheel on the conveyor belt, it automatically breaks up stuck pumpkin slices during transport. Suction cups on a horizontally articulated robotic arm then pick up the broken-up pumpkin slices and package them separately.

[0090] Specifically, the horizontal articulated robotic arm 11 in this embodiment is a four-degree-of-freedom robotic arm. Its distal end has four degrees of freedom: translation along the X, Y, and Z axes, and rotation about the Z axis. This facilitates grasping pumpkin slices at various locations within the workspace. It can also rapidly ascend and descend along the Z axis, meeting high-speed pick-and-place requirements. The articulated robotic arm is bolted to the robotic arm base 14. The robotic arm has guide holes for connecting to the air path. The distal end of the robotic arm is provided with a hollow robotic arm sleeve 9 for mounting a suction cup and air tube. The distal end mechanism 10 includes a suction cup 30, an elastic buffer rod 29, a suction cup connecting plate 28, and an air tube. The suction cup 30 is a food-grade silicone film suction cup with a skirt. It adheres tightly to irregular pumpkin slice surfaces, ensuring stable suction without dropping. The suction cup 30 is threadedly connected to the distal end of the elastic buffer rod 29, which is bolted to the suction cup connecting plate 28. The elastic buffer rod is equipped with a spring, which can provide a certain buffer floating range to prevent excessive pressure on the pumpkin stem from the end. The suction cup connecting plate 28 includes an L-shaped plate and a straight plate. The two plates are combined to form a through hole in the middle, which is used to clamp the elastic buffer shaft and the sleeve at the end of the robot arm together.

[0091] The suction cup 30 is connected to the pneumatic component, and the pumpkin stem is sucked under the control of the pneumatic component. The pneumatic component includes an air filter, a vacuum generator, and an air pump, which are sequentially connected to the suction cup and the rear of the elastic buffer shaft with an air pipe as a connector. The air pipe on the terminal mechanism is connected to the upper end of the elastic buffer shaft, and is connected to the air filter after passing through the end sleeve of the robotic arm. The air filter is connected to the vacuum generator through the air pipe, and then connected to the air pump through the air pipe. The air pump serves as an air source to supply air to the entire device. The vacuum generator can generate positive and negative pressure, and is controlled by a solenoid valve to achieve rapid switching between positive and negative pressure. When sucking pumpkin slices, negative pressure is generated to suck the target, and positive pressure is generated when placing the pumpkin slices to make the pumpkin slices fall quickly.

[0092] The primary conveyor belt 22 is equipped with a scattering wheel 16 and a scattering wheel motor 23. Driven by the scattering wheel motor 23, the scattering wheel 16 continuously moves the pumpkin slices on the primary conveyor belt, thereby initially flattening the pile of pumpkin slices. The algorithm identifies and removes any stuck pumpkin slices. At a later location, personnel will separate and return them to the beginning of the production line for further sorting by the horizontal articulated robotic arm 11.

[0093] The surfaces of the secondary conveyor belt 25 and the primary conveyor belt 22 are green PVC materials, which are used to transport the scattered and disordered pumpkin slices after processing techniques such as slicing, cooking and refrigeration.

[0094] The camera mechanism includes an industrial camera 8 and a camera bracket 12. The industrial camera 8 faces the center of the conveyor belt below and is used to obtain real-time images of the pumpkin slices on the conveyor belt. The bottom of the camera bracket 12 is fixed to the floor, and the industrial camera 8 is fixedly mounted on the top. In some embodiments, the height of the industrial camera is 700 mm higher than the workbench plane.

[0095] The robotic arm base 14 is also equipped with multiple weighing units, four in this embodiment. These units are mounted side by side on the side of the horizontally articulated robotic arm's base and fixed to the arm base 14. Each time the horizontally articulated robotic arm's end places a pumpkin slice onto a weighing unit, the control device receives the coordinates of the robotic arm at that moment and a signal from the solenoid valve controlling the suction cup's exhalation. Based on these two signals, the control device marks the number of slices in each weighing unit. The slices, when gathered to the appropriate weight, fall onto the gathering production line.

[0096] The hardware of the control device includes a control cabinet 5 and a computer. The control cabinet includes a robotic arm motion control module, a conveyor belt motion control module and a suction cup control module. The computer is a GPU computer.

[0097] The GPU computer is used to process images taken by industrial cameras, run recognition algorithms, calculate coordinate transformation relationships, and send the coordinate information in the processing results to the horizontal joint robot arm motion control module through TCP communication.

[0098] The horizontal articulated robotic arm motion control module is used to control the start and stop, speed regulation, and path planning of the articulated robotic arm. In this embodiment, the horizontal articulated robotic arm 11 is a four-axis robotic arm. Therefore, the robotic arm motion control module can independently control the rotation angles J1, J2, J3, and J4 of the robotic arm's four axes, or jointly control the linear motion of the horizontal articulated robotic arm's end along the X, Y, and Z axes and the rotational motion around the Z axis.

[0099] The primary conveyor belt and secondary conveyor belt motion control module are used to receive signals from the lower computer and control the rapid start and stop and uniform speed advance of the primary conveyor belt and secondary conveyor belt.

[0100] The suction cup control module is used to quickly control the suction and blowing of the end suction cup by controlling the on and off of the solenoid valve of the vacuum generator, corresponding to picking up and placing the target respectively.

[0101] When the pumpkin slices move via the conveyor belt to the working area below the industrial camera, the industrial camera captures the current frame image and sends it to the host computer, namely the GPU computer. The host computer extracts the pumpkin slice target information in the current frame based on the recognition algorithm. Then, through mathematical derivation and inverse solution, it obtains the position information of the target required by the robotic arm. Finally, it sends this information to the control cabinet via socket communication, controlling the robotic arm to sequentially capture the pumpkin slice target according to the planned path and point information. The camera captures each frame image in real time and processes it according to the above process. The horizontal joint robotic arm 11 will quickly pick up and place the pumpkin slices in the working area on the conveyor belt. Due to the large number and high density of pumpkin slices on the conveyor belt in the required industrial scenario, the dynamic grabbing method of grabbing as it goes cannot pick up all the pumpkin slices in the current working area. The remaining pumpkin slices will flow forward along the faster secondary conveyor belt. Because the pumpkin slices are frozen, they will thaw and melt by the next cycle, reducing their quality. Therefore, the present invention adopts a pause-grabbing method. When the secondary conveyor belt carrying a large number of pumpkin slices reaches the work area, it pauses. The camera only needs to take a single photo, identifying all objects in the current area. The complete coordinate sequence is sent to the control cabinet lower computer. After the lower computer plans the entire path, the horizontal joint controls the robotic arm to sequentially pick and place all objects. After all objects are captured, the upper computer recognizes that the number of objects is zero and sends a signal, the conveyor belt continues one work area and begins work in the next area.

[0102] The lower computer controls the start and stop of the conveyor belt through the conveyor belt start and stop control device. Figure 6 As shown, in this embodiment, the conveyor motion control module within the lower computer is a programmable controller (PLC). The PLC output acts on the coil of relay KA4 to control the opening and closing of relay KA4's normally open terminal, thereby controlling the start and stop of the conveyor motor. When the PLC controls the coil of relay KA4, the normally open terminal of relay KA4 closes, the motor rotates forward, and the conveyor belt advances at a constant speed. When the coil of relay KA4 is de-energized, the normally open terminal of relay KA4 opens, the motor stops, and the conveyor belt stops.

[0103] After receiving images of the work area from the industrial camera, the host computer identifies and labels the dried fruits and vegetables in the images. A dried fruit and vegetable product recognition model is built based on a deep learning algorithm. This model identifies and selects each dried fruit product in the image.

[0104] Taking pumpkin slices as an example, a dataset of 5,000 images of pumpkin slices on a conveyor belt was collected. These images were taken from different angles, lighting conditions, and distances. Data augmentation methods such as inversion, cropping, splicing, and color changes were used to expand the final dataset to 20,000 images. This makes the learned images richer and more complex than in reality, further improving recognition accuracy. Subsequently, after manually annotating image samples representing various conditions, a deep learning algorithm, based on the Yolov5 model, was trained on the image dataset to learn the semantic information of edge contact cases. Because the maximum suppression (NMS) algorithm used in this model is not suitable for the densely packed small dried fruit and vegetable object detection method used in this scheme, the NMS mechanism sorts the anchor boxes in the score set S by their scores, iterates over the overlap area ratio between each anchor box and the highest-scoring candidate box, and removes the anchor box if the overlap ratio exceeds a threshold a. This approach may result in failed detection of objects in overlapping areas. Furthermore, if a is too small, adjacent objects may be missed, while if a is too large, multiple duplicate detection boxes may appear for the same object. Based on this, we improved the anchor selection mechanism to take advantage of the dense and closely spaced distribution of dried fruits and vegetables on the conveyor belt. We used the SoftNMS algorithm to select anchor frames, reducing the probability of overlapping frames and frames containing multiple objects. When the overlap ratio between two anchor frames exceeds a, the score for that anchor frame is not reset to zero. Instead, it is lowered, and the higher the overlap ratio, the greater the score reduction. The resulting trained recognition model can intelligently identify and select a single object from a cluster of closely packed pumpkin slices.

[0105] According to the characteristics of the device's action of sucking up pumpkin slices from a flat position and placing them in the weighing unit, this implementation uses a "door"-shaped trajectory ( Figure 8 ), and add arc interpolation to optimize the corners to improve smoothness, reduce vibration and travel time, such as Figure 5 As shown in the figure, we optimized the vertical height and lateral movement corners using a fixed-ratio Lam'e curve. We then eliminated the horizontal straight lines and replaced them directly with Lam'e curves. This design reduces acceleration and deceleration during lift-translation-landing, significantly shortening the time spent on the reciprocating pick-and-place process and minimizing jitter. In actual industrial production, the robot can perform up to 40 pick-and-place operations per minute, meeting the demands of industrial scenarios.

[0106] The embodiment of the present invention, based on the above-mentioned dried fruit and vegetable product identification and sorting device, further provides a control method of the identification and sorting device, including:

[0107] S100, acquiring an image of the working area on the secondary conveyor belt;

[0108] S200: When dried fruit and vegetable products to be identified are detected in the working area, the secondary conveyor belt is controlled to advance a set distance and then stop;

[0109] S300: After the secondary conveyor belt stops, an image of the working area is acquired again to identify the dried fruit and vegetable products to be sucked out of the image;

[0110] S400: Controlling the joint robotic arm to sequentially absorb the identified dried fruit and vegetable products;

[0111] S500: When the amount of dried fruit and vegetable products in the working area is lower than the set threshold, the secondary conveyor belt is controlled to advance the set distance again and then stop, and steps S300-S500 are repeated.

[0112] In step S100, the working area is the field of view of the industrial camera on the conveyor belt. In this embodiment, the industrial camera is set 700 mm above the conveyor belt, and the working range is a semicircular area with a diameter of 550 mm.

[0113] The working area of ​​the horizontal joint robot is a circle, but the conveyor belt is on the right side of the robot arm. The picking working area is the overlap of the semicircular area on the conveyor belt and the rectangular area of ​​the camera, and the placement area is the semicircular area on the left side of the conveyor belt. Figure 4 shown.

[0114] In step S200, the secondary conveyor belt carries the scattered pumpkin slices forward at a constant speed and is initially broken up and flattened by the scattering wheel. The industrial camera continuously captures images of the working area below. When the front pumpkin slice enters the edge of the working area, the camera captures an image with the first pumpkin slice target. At this point, the host computer recognizes the pumpkin slice target at the front of the working area in the image and controls the secondary conveyor belt to advance a set distance before stopping. The set distance here is the length of the working area in the direction of the secondary conveyor belt's advance. When the secondary conveyor belt runs at a constant speed, the distance it advances can be controlled by controlling the secondary conveyor belt's advance time. In this embodiment, after the secondary conveyor belt is controlled to advance for another 0.5 seconds, the front pumpkin slice, that is, the pumpkin slice previously identified, moves to the back of the working area. At this point, the front pumpkin slice on the secondary conveyor belt just covers the working area, and the secondary conveyor belt stops.

[0115] In step S300, a dried fruit and vegetable product recognition model is constructed based on a deep learning algorithm. Each dried fruit and vegetable product in the image is identified and framed using this model. For example, using pumpkin slices as the dried fruit and vegetable product, the dried fruit and vegetable product recognition model is a pumpkin slice recognition model. This model is built based on the Yolov5 model. The model is fed with an image of annotated, enhanced pumpkin slices on a conveyor belt. Supervised training is performed on the model, and the confidence threshold for the target frame is adjusted during training. The NMS algorithm selects the optimal frame, reducing the probability of overlapping frames and frames containing multiple targets. The trained pumpkin slice recognition model is then deployed to a host computer. An industrial camera captures the current frame image in real time and uploads it to the host computer. The host computer identifies each pumpkin slice in the captured image of the work area, marks each target with a rectangular frame, and plans a coordinate sequence for all pumpkin slices. This planned coordinate sequence forms the robotic arm's picking path.

[0116] In step S400, the horizontal joint robot arm motion control module picks up each pumpkin slice in sequence according to the robot arm picking path planned by the host computer. During the picking process, the motion trajectory of the robot arm is a gate-shaped trajectory, and circular arc interpolation is used at the top bend of the gate-shaped trajectory to optimize the trajectory to improve the smoothness of the robot arm's movement. The robot arm places the pumpkin slices into the four weighing units in order. After the picking is completed, the industrial camera acquires the current frame image again to identify whether there are any remaining pumpkin slices. If the remaining number is less than the set threshold, it is qualified. Otherwise, the above algorithm is repeated and the picking is continued until all pumpkin slices in the working area are picked up.

[0117] In step S500, after all pumpkin slices in the current working area have been picked up, the host computer issues a command to control the conveyor belt to continue moving forward at a constant speed for 0.5 seconds to obtain a new working area range, so that the pumpkin slices in the next working area can be sorted and picked up. Steps S300-S500 are repeated to continuously pick up pumpkin slices on the conveyor belt.

[0118] After step S500 is completed, the process further includes: S600: If no dried fruit or vegetable products are identified within the working area within a set time, a completion signal is generated, prompting a user to replenish materials or shut down the device. If the host computer does not identify any pumpkin slices within the conveyor belt image for three consecutive seconds, i.e., within the six working areas, the process deems that all pumpkin slices on the conveyor belt have been sorted, and a completion signal is generated.

[0119] In the identification of dried fruits and vegetables, since dried fruits and vegetables are usually arranged in a highly dense manner, and their shapes have certain directionality and size differences, Yolov5 The original model IoU and NMS The algorithm is prone to overlap and confusion, which may lead to a decrease in detection accuracy. For the application scenario of dense dried fruit and vegetable recognition, we proposed Dense Region IoU (DR_ IoU), Dense Region NMS (DR_NMS) and the corresponding loss function Loss DR_IoU To better handle the detection of densely packed targets, especially in dried fruits and vegetables with complex arrangements and directional differences, the final algorithm model can accurately identify and select individual targets from closely packed materials. The design mainly includes the following four aspects:

[0120] 1. Dense target separation: Reduces false detection and missed detection of overlapping targets, suitable for situations where dried fruits and vegetables are closely arranged.

[0121] 2. Shape and orientation differences: Consider that dried fruits and vegetables may have orientation or shape differences (such as irregular slice shapes).

[0122] 3. Dynamic weight adjustment: Dynamically adjust the weight of the detection algorithm according to the confidence difference to make high-confidence targets more prominent.

[0123] 4. More precise loss optimization: Optimize box position, shape, orientation, and size simultaneously during the regression stage.

[0124] The formula is as follows:

[0125] (one) DR_IoU : Through dynamic adjustment IoU The contribution weight of the candidate box is considered, considering the geometric similarity characteristics (area, diagonal length ratio and angle difference) of the candidate box and the confidence of the box to improve the original IoU form of expression.

[0126] ;

[0127] ;

[0128] in:

[0129] : Dynamic weight, adjusted according to confidence IoU The geometric contribution weight of .

[0130] B and B gt The predicted boxes and the true boxes in the detection are respectively. b and b gt Defined as B and B gt center. C diag is the diagonal length of the smallest rectangle that covers both boxes.

[0131] d and dgt yes B and B gt Vector expression for the diagonal. | d | and | d gt | indicates d and d gt length.

[0132] b sc_max and b i Represent the maximum confidence score box and other redundant boxes respectively.

[0133] : The Euclidean distance between the center of the predicted box and the true box, indicating the center offset.

[0134] : Diagonal vector difference, measuring the shape difference of the box.

[0135] : Diagonal vector direction angle difference, used to capture the direction change of the frame.

[0136] (2) Proposing an improved loss function It can minimize the geometric differences between boxes and maximize the overlapping area, while penalizing center position offset, box size difference, and orientation difference. It can adapt to highly densely arranged objects, capture differences in position, shape, and orientation, and optimize the performance of the detection model in complex scenes.

[0137] ;

[0138] ;

[0139] in:

[0140] 1. IoU Item: represents the intersection-over-union ratio between the predicted box and the true box, which is used to measure the degree of regional overlap.

[0141] ;

[0142] 2. Center offset penalty:

[0143] ;

[0144] is the offset distance of the frame center.

[0145] is the minimum bounding box diagonal length, used to normalize the offset distance.

[0146] 3. Diagonal difference terms:

[0147]

[0148] , represents the diagonal vector difference, The product of the diagonal lengths is used to represent the normalized weight and measure the similarity of the box sizes.

[0149] 4. Direction angle difference:

[0150]

[0151] Represents the square of the angle difference, which normalizes the orientation difference of the weighted box.

[0152] 5. Dynamic weight adjustment:

[0153] : Weight factor, dynamically adjusted to adapt to different data sets and detection tasks.

[0154]

[0155] (III) Improved dense area suppression function DR_NMS : Taking into account the overlapping area ratio of the box, the diagonal direction vector of the box, and the confidence, a direction-aware IoU Suppression strategy, design similarity score , to improve the accuracy of dense target detection.

[0156] ;

[0157] ;

[0158] in:

[0159] 1. Area ratio, which measures the similarity of the areas of two boxes.

[0160] ;

[0161] 2. Diagonal ratio, which measures the similarity of the diagonal lengths of the boxes.

[0162] ;

[0163] 3. Direction angle ratio, which reflects the consistency of the frame direction through the ratio of the cosine values ​​of the direction angles.

[0164] ;

[0165] 4. 、 、 are the weights of these three ratios, and their sum is 1.

[0166] Direction-aware similarity S dir and improvements DR_IoU Combined with the final suppression for dynamic adjustment of the frame, the final DR_NMS After adjustment for:

[0167]

[0168] DR_NMS advantage:

[0169] 1. More accurate overlapping box suppression: Introducing direction-aware similarity to reduce false suppression between densely packed objects.

[0170] 2. Stronger adaptability: Combined with dynamic weight adjustment, it can adapt to different dense target scenarios.

[0171] 3. Improve recall rate: IoU Moderate retention of the frame to improve the integrity of the detection frame

[0172] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for identifying and sorting dried fruit and vegetable products, characterized in that: The following steps are involved: S100, acquiring an image of a working area on a conveying device; S200, when it is recognized that there is a material to be identified in the working area, the conveying device is controlled to advance a set distance and then stop; S300, after the vibrating feeding mechanism and the conveying device stop, an image of the working area on the conveying device is acquired again to identify the material to be sucked in the image; Based on the Yolov5 model, we input annotated, enhanced image of materials on a conveyor belt and conduct supervised training on the model. During training, we adjust the confidence threshold of the target box and improve the anchor box selection mechanism. We use the SoftNMS algorithm to select the optimal box and reduce the probability of overlapping boxes and boxes containing multiple targets. Finally, we obtain a trained recognition model and deploy it to the host computer. S400: Control the horizontal joint robot arm to sequentially absorb the identified materials and place them in several weighing and numbering mechanisms. The weighing and numbering mechanisms feed back the weight values ​​to the host computer, which solves the optimal weight combination and controls the pusher slider to push the materials out. The horizontal joint robot arm uses a "gate"-shaped trajectory for pick-and-place operations. A fixed-ratio Lam'e curve is used to optimize the vertical height and lateral movement corners. The horizontal straight line segments are then eliminated and replaced directly with Lam'e curves. S500: When the amount of dried fruit and vegetable products in the working area is lower than the set threshold, the vibration feeding mechanism and the conveying device are controlled to move forward a set distance again and then stop, and steps S300-S500 are repeated.

2. A method for identifying and sorting dried fruit and vegetable products according to claim 1, characterized in that: In step S200, the secondary conveyor belt of the conveying device carries the scattered materials and advances at a constant speed. The materials are initially scattered and flattened by the scattering wheel. The camera continuously acquires images of the working area below. When the front material enters the edge of the working area, the camera captures an image with the first material target. At this time, the host computer recognizes the material target at the front of the working area in the image and controls the conveying device to advance a set distance before stopping. The set distance here is the length of the working area in the advancing direction of the secondary conveyor belt of the conveying device. When the secondary conveyor belt runs at a constant speed, the advancing distance of the secondary conveyor belt can be controlled by controlling the advancing time of the secondary conveyor belt. When the front material on the secondary conveyor belt just covers the working area, the secondary conveyor belt stops advancing. In step S300, the camera captures the current frame image in real time and uploads it to the host computer. The host computer identifies the materials in the captured image of the working area, marks each target with a rectangular box, and plans the coordinate sequence of all materials. The planned coordinate sequence forms the picking path of the robot arm; In step S400, the upper computer plans the movement trajectory of the horizontal joint robot arm according to the weighing units at different positions of the weighing and making-up mechanism, and numbers the rotary cylinders corresponding to the discharge hopper and the buffer hopper; when the horizontal joint robot arm places the material into the set discharge hopper of the weighing and making-up mechanism, the lower computer controls the rotary cylinder of the discharge hopper to open the fan-shaped switch plate at the bottom of the discharge hopper, and the material falls into the buffer hopper on the lower layer. According to the set time, the upper discharge hopper closes the fan-shaped switch plate, and the lower buffer hopper opens its fan-shaped switch plate, and the material falls into the material bag on the tray again.

3. The method for identifying and sorting dried fruit and vegetable products according to claim 2, characterized in that: The communication mechanism adopts the mutual feedback mechanism of the "hook principle", specifically: During the first hook, the camera acquires an image, and the host computer recognizes the presence of material. The first and second conveyor belts advance a set distance and then stop. The host computer receives information that the motors of the first and second conveyor belts have stopped. At this time, the camera takes a picture of a still and stable image, and obtains accurate information about all targets in the image. When all targets in the image are picked up by the horizontal joint robotic arm, the camera takes a picture and recognizes that there are no targets in the image, and feedback is sent to the host computer, and the conveyor belt continues to advance to the next working distance. During the second hook, a single material is picked up by the robotic arm and placed in the hopper of each weighing unit. In the initial state of weight addition, each weighing unit contains one piece of material, and materials are continuously put into the weighing unit. When the material weight combination of a certain weighing unit is calculated to meet the requirements, the lower computer receives a signal that the material on the weighing tray is pushed out, and controls the action of the push slider; the buffer hoppers of other units open to refill the weighing tray below. The third time the hand is hooked, during the discharge process, the discharge hopper and the buffer hopper have opening and closing signals, and the suction cup exhales when discharging. The upper computer knows that the horizontal joint robotic arm has put down the picked up material, and the upper computer can control the horizontal joint robotic arm to absorb the next piece of material; when all the discharge hoppers do not generate the opening and closing signal of the material dropping, the horizontal joint robotic arm grabs the material and waits in the air just above the weighing and making up the number mechanism.

4. The method for identifying and sorting dried fruit and vegetable products according to claim 3, characterized in that: In the identification of dried fruits and vegetables, the modified Yolov5 The model is identified as follows: Improve the original IoU Expression: ； ； in: : Dynamic weight, adjusted according to confidence IoU The geometric contribution weight of B and B gt The predicted box and the real box in the detection respectively; b and b gt Defined as B and B gt the center of C diag is the diagonal length of the smallest rectangle that covers both boxes; d and d gt yes B and B gt Vector expression for diagonal; | d | and | d gt | indicates d and d gt length; b sc_max and b i Represent the maximum confidence score box and other redundant boxes respectively; : The Euclidean distance between the center of the predicted box and the true box, indicating the center offset; : Diagonal vector difference, measuring the shape difference of the box; : The directional angle difference of the diagonal vector is used to capture the directional change of the frame; Improved loss function for: ； ； in: IoU Item: represents the intersection-over-union ratio of the predicted box and the true box: IoU= ； Center shift penalty: ; is the offset distance of the frame center; is the minimum bounding box diagonal length, used to normalize the offset distance; Diagonal Difference Term: ; , represents the diagonal vector difference, The product of the diagonal lengths is used to represent the normalized weight and measure the similarity of the box size; Direction angle difference term: ; Represents the square of the angle difference, normalizing the orientation difference of the weighted box; Dynamic weight adjustment: : Weight factor, dynamically adjusted to adapt to different data sets and detection tasks; ； Improved dense area suppression function DR_NMS : Taking into account the overlapping area ratio of the box, the diagonal direction vector of the box, and the confidence, a direction-aware IoU Suppression strategy, design similarity score , to improve the accuracy of dense target detection: ； ； in: Area ratio, which measures the similarity of the areas of two boxes: ； Diagonal ratio, which measures the similarity of the diagonal lengths of boxes: ； The direction angle ratio reflects the consistency of the frame direction through the ratio of the cosine values ​​of the direction angles: ； 、 、 are the weights of these three ratios, the sum of which is 1; Direction-aware similarity S dir and improvements DR_IoU Combined with the final suppression for dynamic adjustment of the frame, the final DR_NMS After adjustment for: 。 5. A dried fruit and vegetable product identification and sorting system, using the method according to any one of claims 1 to 4, characterized in that: It includes a vibrating feeding mechanism, a conveying device, a photographing device, a horizontal joint mechanical arm, a weighing and counting mechanism and a control device; The vibration feeding mechanism is used to output the material to the conveying device; The horizontal joint robot arm adjusts its direction and cooperates with the end mechanism to pick up materials and place them in the weighing and counting mechanism; The camera device is used to obtain images of materials on the conveyor device; The weighing and numbering mechanism weighs the obtained materials, and when the weight of the materials reaches a set threshold, the control device pushes the materials out; The control device is used to control the vibration feeding mechanism, the conveying device, the horizontal joint robot arm and the weighing and making up the number mechanism movement as well as image processing.

6. The dried fruit and vegetable product identification and sorting system according to claim 5, characterized in that: The vibrating feeding mechanism includes a feeding tray, a vibrator and a primary conveyor belt; a vibrator is provided under the feeding tray, and the vibrator is fixed to the conveyor belt base through a feeding base; a scattering wheel is provided on the primary conveyor belt, and a plurality of groups of rubber soft rods with uniform gaps are provided on the scattering wheel.

7. The dried fruit and vegetable product identification and sorting system according to claim 5, characterized in that: The conveying device includes a secondary conveyor belt, a leveling wheel and a leveling wheel motor are arranged above the secondary conveyor belt, and a leveling soft board is installed on the leveling wheel; the leveling wheel is fixed on the conveyor belt base 2 and fixed to the bottom plate through the conveyor belt base 2.

8. The dried fruit and vegetable product identification and sorting system according to claim 5, characterized in that: The horizontal joint robot arm is located on one side of the secondary conveyor belt of the conveying device. The end of the horizontal joint robot arm has at least four degrees of freedom: translation along the X, Y, and Z axes and rotation around the Z axis; the end of the horizontal joint robot arm is also provided with an end mechanism, which includes a suction cup, an elastic buffer rod, a suction cup connecting plate and an air pipe; the suction cup is connected to the end of the elastic buffer rod, and the elastic buffer rod is fixed to the suction cup connecting plate; a spring is provided on the elastic buffer rod, and the suction cup connecting plate includes an L-shaped plate and a straight plate, and a through hole is formed between the two for clamping and fixing the elastic buffer rod and the end sleeve of the robot arm together.

9. The dried fruit and vegetable product identification and sorting system according to claim 5, characterized in that: The weighing and making up mechanism includes a plurality of identical weighing units, each weighing unit including, from top to bottom, a discharge hopper, a buffer hopper, a weighing tray, a push slider, and a weighing sensor; the discharge hopper and the buffer hopper are both provided with a fan-shaped switch plate, which is opened and closed by a rotating cylinder, and the material falls from the discharge hopper to the buffer hopper below, and then falls to the weighing tray below. The weighing sensor below the weighing tray obtains the weight value of the material and feeds it back to the control device. The control device solves the optimal weight combination and controls the push slider to push out the group of materials.

10. The dried fruit and vegetable product identification and sorting system according to claim 5, characterized in that: The control device includes an upper computer and a lower computer, which are communicatively connected. The lower computer controls the movement of the vibration feeding mechanism, the conveying device, the horizontal joint robotic arm, the end mechanism, and the weighing and making-up mechanism; the upper computer divides the area according to the image processing structure and controls the movement trajectory of the horizontal joint robotic arm.

Images