Intelligent feed detection and sorting device based on binocular camera and yolo model and SAM2 model
The intelligent detection and sorting device, which combines a binocular camera and a YOLO model with a SAM2 model, solves the problems of color feature interference and detection frame size differences in feed purity detection, and achieves accurate product sorting and purity detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAOSHU YOUTAI (BEIJING) BIOTECHNOLOGY CO LTD
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-09
AI Technical Summary
Existing feed purity detection methods rely on color features, which are easily affected by background objects, leading to a decrease in detection accuracy. Furthermore, the difference in the size of the detection boxes on the left and right sides of the YoloV11 object detection binocular camera affects sorting accuracy.
An intelligent detection and sorting device based on binocular cameras, YOLO models, and SAM2 models is adopted. The detection module obtains the position of the target box, the target box matching module performs precise matching, the target segmentation and positioning module determines the area and position, and the robotic arm and transmission mechanism are used to realize sorting.
It achieves more accurate acquisition of product color purity information, precise matching of target positions, calculation of target distance by combining binocular imaging model, and control of robotic arm for precise sorting, thereby improving the accuracy of detection and sorting.
Smart Images
Figure CN122175930A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of feed purity detection technology, specifically to an intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model. Background Technology
[0002] Feed purity testing refers to the process of detecting the purity and impurity content of target components in feed (including raw materials, compound feed, additives, etc.) through sensory evaluation, physical analysis, chemical determination, and microbiological testing. It is a key link to ensure the quality of feed raw materials, protect animal health, and improve breeding efficiency. The core of feed purity testing is to determine whether the proportion of effective components in feed meets the standards, or whether impurities (such as foreign matter, harmful components, and unlabeled components) are within the allowable range, so as to ensure feed quality and safety. Feed purity directly affects the nutritional value of feed. When feed impurities exceed the standard, it will lead to a decrease in animal digestibility, increase the risk of intestinal diseases, and impurities will also lead to an increase in feed clumping rate and abnormal fluctuations in fecal moisture content.
[0003] Current detection and sorting methods rely on color features when detecting feed purity, but background objects can also be mixed in, affecting the judgment of detection accuracy. While YoloV11 object detection can detect the approximate area where the target is located, when using a binocular camera, the target detection boxes on the left and right sides often differ in size, so they cannot be accurately matched, affecting the accuracy of feed detection and sorting. Summary of the Invention
[0004] This invention provides an intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model. It can effectively solve the problem mentioned in the background art that current detection and sorting methods rely on color features when detecting feed purity, but background objects can also be mixed in, affecting the judgment of detection accuracy. While YOLO V11 target detection can detect the approximate area where the target is located, when using a binocular camera, the target detection boxes on the left and right sides often have different sizes, so they cannot be accurately matched, affecting the accuracy of feed detection and sorting.
[0005] To achieve the above objectives, the present invention provides the following technical solution: an intelligent feed detection and sorting device based on binocular cameras, YOLO models, and SAM2 models, including a detection module, a target box matching module, a target segmentation and positioning module, and a sorting and conveying module. It utilizes the advantage of YOLOV11 in detecting the position of target boxes and the advantage of SAM2 in accurate segmentation of large models to obtain product color purity information. Based on the position information, a robotic arm and a transmission mechanism are used to achieve sorting.
[0006] The detection module obtains the position of the target in the bounding boxes in the left and right images, and the target bounding box matching module matches the target detection box results on the left and right sides;
[0007] The target segmentation and positioning module determines the region and position of the target in the left and right images, and the sorting and conveying module moves the product to the designated classification area to achieve sorting.
[0008] According to the above technical solution, the detection module uses a binocular camera to take pictures of the calibration plate at different positions and angles, calibrate it, and calculate the intrinsic parameter matrices Ml, Mr, distortion coefficients K1, Kr, and the camera transformation matrix R, t of the left and right cameras.
[0009] The camera is fixed to the end of the robotic arm. Hand-eye calibration is performed on the camera and the robotic arm. The calibration plate is fixed and the camera position is changed so that the calibration plate appears in different positions in the image. The images are collected and saved. 15 sets of posture data of the end of the robotic arm and the corresponding calibration plate images are collected. The position transformation matrix T of the camera and the end of the robotic arm is calculated.
[0010] A binocular camera is deployed above the device, and the image acquisition results are transmitted to the intelligent detection module to perform distortion correction on the images obtained by the left and right cameras.
[0011] According to the above technical solution, the detection module improves Yolov11 by replacing the attention layer in the C2PSA module of Yolov11 with MLCA (Hybrid Local Channel Attention), and replacing the Bottleneck module of Yolov11 with DEAB (Detail Enhancement Attention Module), thereby improving the network's target detection capability.
[0012] The YOLOv11 model is trained, and the intelligent detection module uses the trained YOLOv11 model to detect the left and right images, obtaining the position of the target in the bounding box in the image.
[0013] According to the above technical solution, the target box matching module calculates the normalized length and width vector vector_width_height_left for the target detected in the left eye;
[0014] Based on the y-coordinate value yleft, targets with y-coordinates close to yleft are selected from the right image, that is, targets with y-coordinates in the range of [yleft-offset, yleft+offest], as the initial matching result. Here, offset is the error range, which can be set to 5 pixels. For the selected targets, the normalized length and width vector vector_width_height_right is calculated.
[0015] The similarity S1 is obtained by calculating the inner product of the two vectors. If S1 exceeds the threshold, the two targets are considered to be the same target. The areas of the initially matched targets in the left and right images are compared. If the similarity S2 = smallArea / bigArea is greater than the threshold, the two images are considered to be the same target.
[0016] According to the above technical solution, the target box matching module aligns the upper left corner of the box region where the initially matched target is located in the left and right images, and aligns the center position of the box region where the target is located in the left and right images, calculates the overlapping area, and calculates the cross-correlation coefficient S3 for the overlapping area.
[0017] If the cross-correlation coefficient S3 is greater than the threshold, the two images are considered to be the same target. If the same target matches multiple candidate targets, they are further excluded in the target segmentation and localization module.
[0018] According to the above technical solution, the target segmentation and localization module segments the detected target using the SAM2 model based on the detection bounding box of YOLOv11;
[0019] Feature points are detected in the left and right images, then matched, and feature points in the segmentation region are selected and retained.
[0020] If the number of feature points is greater than 5, calculate the disparity of each feature point, use ransac to find the disparity of the inlier with the most disparity as the disparity of the target in the left and right images, and use it to calculate the distance of the target and then obtain the coordinate value P_cam(x,y,z) in the left camera coordinate system.
[0021] If the number of feature points is less than 5, calculate the centroid xleft in the left image and the centroid xright in the right image based on the segmentation region of the detected target in the left image, and obtain the centroid dist_center.
[0022] Simultaneously, the disparity of the four boundary points—leftmost, rightmost, topmost, and bottommost—is calculated to obtain dist_left, dist_right, dist_up, and dist_down. Based on the coordinates of the matching point in the left and right images, the average disparity dist_mean and the coordinate values P_cam(x,y,z) in the left camera coordinate system can be obtained.
[0023] According to the above technical solution, the target segmentation and localization module aligns the center of the segmented regions on both sides based on the obtained disparity, calculates the overlap ratio S4 and the correlation coefficient S5 of the segmented regions on both sides. When the overlap ratio S4 is greater than the threshold and the correlation coefficient S5 is greater than the threshold, the current target is considered to be correctly matched. If there are multiple candidate target matching results for the same target, the target with the largest S4*S5 is selected as the correct matching result.
[0024] Based on the transformation relationship T between the robot arm coordinate system and the camera coordinate system, the coordinates of the left camera coordinate system are transformed to the world coordinate system P_world(x,y,z) where the robot arm is located, where P_world(x,y,z)=T*P_cam(x,y,z).
[0025] According to the above technical solution, the target segmentation and localization module converts the pixels of corresponding regions within the target segmentation areas of the left and right images to HSV space, calculates the color distribution histogram of each channel, and obtains the histogram feature vector vector_hist for each channel.
[0026] Normalize the vector_hist of all channels, then merge and normalize it to obtain vector_normal. Calculate the inner product of vector_normal and vector_std, which is the normalized vector of the color histogram of the standard sample, to obtain an estimate R of the target purity. Based on the estimate, the target purity information can be obtained, and the target can be classified according to purity to obtain the classification result Class.
[0027] According to the above technical solution, the sorting and conveying module controls the mechanical structure to move the product to the designated sorting area based on the position P_world(x,y,z) and the category result Class, thereby completing the sorting.
[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0029] This invention, through a detection module, a target bounding box matching module, a target segmentation and positioning module, and a sorting and conveying module, can leverage the advantage of YOLOv11 in detecting the position of the target bounding box to accurately detect the location of the target. It also utilizes the advantage of SAM2's large model for precise segmentation to easily determine the precise region and position of the target in the left and right images. Based on this, more accurate product color purity information can be obtained, while erroneous matches can be eliminated. Furthermore, by combining the principle of binocular imaging, the precise 3D position information of the target can be obtained, enabling more accurate sorting based on the position information.
[0030] The SAM2 model is used to accurately segment the target and combine it with the detection box of the YoloV11 to accurately match the left and right cameras, thereby obtaining the precise position of the target. Based on the precise position of the target in the left and right images, the distance information of the target can be calculated using the binocular camera imaging model. At the same time, more accurate color information can be used to obtain more accurate product precision. Based on the position information, the robotic arm and transmission mechanism are controlled to sort and transport the product to the accurate classification area, thus completing accurate detection and sorting. Attached Figure Description
[0031] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0032] In the attached diagram:
[0033] Figure 1 This is a flowchart illustrating the detection and sorting process of the present invention;
[0034] Figure 2 This is a schematic diagram of the composition of the detection and sorting device of the present invention.
[0035] Figure 3 This is a schematic diagram of the camera imaging model of the present invention;
[0036] Figure 4 This is a schematic diagram of the target to be detected by the present invention;
[0037] Figure 5 This is a schematic diagram of the segmentation of the target region according to the present invention;
[0038] Figure 6 This is a schematic diagram of feature point matching and filtering of segmented region points according to the present invention;
[0039] Figure 7 This is a schematic diagram of the positioning results output by the present invention. Detailed Implementation
[0040] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0041] Example: Figure 1 As shown, the present invention provides a technical solution, an intelligent feed detection and sorting device based on binocular cameras, YOLO models and SAM2 models, including a detection module, a target box matching module, a target segmentation and positioning module and a sorting and conveying module. It utilizes the advantage of YOLOV11 in detecting the position of target boxes and the advantage of SAM2 large model in accurate segmentation to obtain more accurate product color purity information. Based on the position information, a robotic arm and transmission mechanism are used to achieve more accurate sorting.
[0042] The detection module obtains the position of the target in the bounding boxes in the left and right images, and the target bounding box matching module accurately matches the target detection box results on the left and right sides;
[0043] The target segmentation and positioning module determines the precise area and position of the target in the left and right images, and the sorting and conveying module moves the product to the designated sorting area to achieve more accurate sorting.
[0044] Based on the above technical solution, the detection module uses a binocular camera to take pictures of the calibration plate at different positions and angles, calibrate it, and calculate the intrinsic parameter matrices Ml, Mr, distortion coefficients K1, Kr, and the transformation matrix R, t between the left and right cameras.
[0045] The camera is fixed to the end of the robotic arm. Hand-eye calibration is performed on the camera and the robotic arm. The calibration plate is fixed and the camera position is changed so that the calibration plate appears in different positions in the image. The images are collected and saved. 15 sets of posture data of the end of the robotic arm and the corresponding calibration plate images are collected. The position transformation matrix T of the camera and the end of the robotic arm is calculated.
[0046] A binocular camera is deployed above the device. The image acquisition results are transmitted to the intelligent detection module, which performs distortion correction on the images obtained from the left and right cameras. Figure 2 As shown.
[0047] Based on the above technical solutions, the detection module improves Yolov11 by replacing the attention layer in the C2PSA module of Yolov11 with MLCA (Hybrid Local Channel Attention), and replacing the Bottleneck module of Yolov11 with DEAB (Detail Enhancement Attention Module), thereby improving the network's target detection performance and enhancing the network's target detection capability.
[0048] The YOLOv11 model is trained, and the intelligent detection module uses the trained YOLOv11 model to detect objects in the left and right images, obtaining the location of the target's bounding box in the image, such as... Figure 4 As shown.
[0049] Based on the above technical solution, the target bounding box matching module calculates a normalized width and height vector `vector_width_height_left` for targets detected in the left eye, where `width` and `height` are the width and height of the target bounding box, specifically calculated using the following formula:
[0050] ;
[0051] Based on the y-coordinate value yleft, targets with y-coordinates close to yleft are selected from the right image, that is, targets with y-coordinates in the range of [yleft-offset, yleft+offest], as the initial matching result. Here, offset is the error range, which can be set to 5 pixels. For the selected targets, the normalized length and width vector vector_width_height_right is calculated.
[0052] The similarity S1 is obtained by calculating the inner product of the two vectors. If S1 exceeds the threshold of 0.9, the two targets are considered to be the same target. The areas of the initially matched targets in the left and right images are compared. If the similarity S2 = smallArea / bigArea is greater than the threshold of 0.9, the two images are considered to be the same target.
[0053] Based on the above technical solution, the target bounding box matching module aligns the top-left corner of the bounding box region containing the initially matched target in the left and right images, and simultaneously aligns the center position of the bounding box region containing the target in the left and right images, calculates the overlapping area, and calculates the cross-correlation coefficient S3 for the overlapping area, specifically calculated by the following formula:
[0054] ;
[0055] If the cross-correlation coefficient S3 is greater than the threshold, and the threshold is 0.9, then the two images are considered to be the same target. If the same target matches multiple candidate targets, then further elimination is performed in the target segmentation and localization module.
[0056] Based on the above technical solution, the target segmentation and localization module uses the SAM2 model to segment the detected targets according to the detection bounding boxes of YOLOv11, such as... Figure 5 As shown;
[0057] Feature point detection is performed on the left and right images, followed by matching to filter out feature points located within the segmentation region. Feature points within the segmentation region are then retained. Figure 6 As shown;
[0058] If the number of feature points is greater than 5, calculate the disparity of each feature point, use `ransac` to find the disparity with the most inliers as the disparity of the target in the left and right images, and use this disparity to calculate the target distance, thus obtaining the coordinate values P_cam(x,y,z) in the left camera coordinate system. Figure 3 As shown, f is the focal length, b is the baseline distance between the left and right cameras, the pixel value of the left camera is (xl, yl), and the pixel value of the right camera is (xr, yr), which is calculated using the following formula:
[0059] ;
[0060] ;
[0061] or ;
[0062] If the number of feature points is less than 5, calculate the centroid xleft in the left image and the centroid xright in the right image based on the segmentation region of the detected target in the left image, and obtain the centroid dist_center.
[0063] Simultaneously, the disparity of the four boundary points—leftmost, rightmost, topmost, and bottommost—is calculated to obtain dist_left, dist_right, dist_up, and dist_down. Based on the coordinates of the matching point in the left and right images, the average disparity dist_mean and the coordinate values P_cam(x,y,z) in the left camera coordinate system can be obtained.
[0064] Based on the above technical solution, the target segmentation and localization module aligns the centers of the segmented regions on both sides according to the obtained disparity, calculates the overlap ratio S4 and the correlation coefficient S5 of the segmented regions on both sides. If the overlap ratio S4 is greater than the threshold and the threshold is 0.95, and the correlation coefficient S5 is greater than the threshold and the threshold is 0.95, then the current target is considered to be matched correctly. If there are multiple candidate target matching results for the same target, the target with the largest S4*S5 is selected as the correct matching result.
[0065] Based on the transformation relationship T between the robotic arm coordinate system and the camera coordinate system, the coordinates of the left camera coordinate system are transformed to the world coordinate system P_world(x,y,z) where the robotic arm is located, where P_world(x,y,z)=T*P_cam(x,y,z), as follows. Figure 7 As shown.
[0066] Based on the above technical solution, the target segmentation and localization module converts the pixels of corresponding regions within the target segmentation areas of the left and right images to HSV space, calculates the color distribution histogram of each channel, and obtains the histogram feature vector vector_hist for each channel.
[0067] Normalize the vector_hist of all channels, then merge and normalize it to obtain vector_normal. Calculate the inner product of vector_normal and vector_std, which is the normalized vector of the color histogram of the standard sample, to obtain an estimate R of the target purity. Based on the estimate, the target purity information can be obtained, and the target can be classified according to purity to obtain the classification result Class.
[0068] Based on the above technical solution, the sorting and conveying module controls the mechanical structure to move the product to the designated sorting area according to the position P_world(x,y,z) and the category result Class, thereby completing the sorting. The mechanical structure specifically consists of a robotic arm and a transmission device.
[0069] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An intelligent feed detection and sorting device based on binocular cameras, YOLO models, and SAM2 models, characterized in that: It includes a detection module, a target box matching module, a target segmentation and positioning module, and a sorting and conveying module. It utilizes the advantages of YOLOv11 in detecting the position of the target box and the advantages of SAM2 in accurate segmentation of the large model to obtain the product color purity information. Based on the position information, a robotic arm and a transmission mechanism are used to achieve sorting. The detection module obtains the position of the target in the bounding boxes in the left and right images, and the target bounding box matching module matches the target detection box results on the left and right sides; The target segmentation and positioning module determines the region and position of the target in the left and right images, and the sorting and conveying module moves the product to the designated classification area to achieve sorting.
2. The intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model as described in claim 1, characterized in that: The detection module uses a binocular camera to take pictures of the calibration plate at different positions and angles, calibrate it, and calculate the intrinsic parameter matrices Ml, Mr, distortion coefficients K1, Kr, and the transformation matrix R, t between the left and right cameras. The camera is fixed to the end of the robotic arm. Hand-eye calibration is performed on the camera and the robotic arm. The calibration plate is fixed and the camera position is changed so that the calibration plate appears in different positions in the image. The images are collected and saved. 15 sets of posture data of the end of the robotic arm and the corresponding calibration plate images are collected. The position transformation matrix T of the camera and the end of the robotic arm is calculated. A binocular camera is deployed above the device, and the image acquisition results are transmitted to the intelligent detection module to perform distortion correction on the images obtained by the left and right cameras.
3. The intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model according to claim 2, characterized in that: The detection module improves upon Yolov11 by replacing the attention layer in the C2PSA module with MLCA (Hybrid Local Channel Attention), and replacing the Bottleneck module with DEAB (Detail Enhancement Attention), thereby enhancing the network's target detection capability. The YOLOv11 model is trained, and the intelligent detection module uses the trained YOLOv11 model to detect the left and right images, obtaining the position of the target in the bounding box in the image.
4. The intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model according to claim 2, characterized in that: The target box matching module calculates a normalized width and height vector vector_width_height_left for the target detected in the left eye; Based on the y-coordinate value yleft, targets with y-coordinates close to yleft are selected from the right image, that is, targets with y-coordinates in the range of [yleft-offset, yleft+offest], as the initial matching result. Here, offset is the error range, which can be set to 5 pixels. For the selected targets, the normalized length and width vector vector_width_height_right is calculated. The similarity S1 is obtained by calculating the inner product of the two vectors. If S1 exceeds the threshold, the two targets are considered to be the same target. The areas of the initially matched targets in the left and right images are compared. If the similarity S2 = smallArea / bigArea is greater than the threshold, the two images are considered to be the same target.
5. The intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model according to claim 4, characterized in that: The target box matching module aligns the top left corner of the box region where the initially matched target is located in the left and right images, and aligns the center position of the box region where the target is located in the left and right images, calculates the overlapping area, and calculates the cross-correlation coefficient S3 for the overlapping area. If the cross-correlation coefficient S3 is greater than the threshold, the two images are considered to be the same target. If the same target matches multiple candidate targets, they are further excluded in the target segmentation and localization module.
6. The intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model according to claim 5, characterized in that: The target segmentation and localization module uses the SAM2 model to segment the detected targets based on the detection bounding boxes of YOLOv11. Feature points are detected in the left and right images, then matched, and feature points in the segmentation region are selected and retained. If the number of feature points is greater than 5, calculate the disparity of each feature point, use ransac to find the disparity of the inlier with the most disparity as the disparity of the target in the left and right images, and use it to calculate the distance of the target and then obtain the coordinate value P_cam(x,y,z) in the left camera coordinate system. If the number of feature points is less than 5, calculate the centroid xleft in the left image and the centroid xright in the right image based on the segmentation region of the detected target in the left image, and obtain the centroid dist_center. Simultaneously, the disparity of the four boundary points—leftmost, rightmost, topmost, and bottommost—is calculated to obtain dist_left, dist_right, dist_up, and dist_down. Based on the coordinates of the matching point in the left and right images, the average disparity dist_mean and the coordinate values P_cam(x,y,z) in the left camera coordinate system can be obtained.
7. The intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model according to claim 6, characterized in that: The target segmentation and localization module aligns the centers of the segmented regions on both sides based on the obtained disparity, calculates the overlap ratio S4 and the correlation coefficient S5 of the segmented regions on both sides. When the overlap ratio S4 is greater than the threshold and the correlation coefficient S5 is greater than the threshold, the current target is considered to be matched correctly. If there are multiple candidate target matching results for the same target, the target with the largest S4*S5 is selected as the correct matching result. Based on the transformation relationship T between the robot arm coordinate system and the camera coordinate system, the coordinates of the left camera coordinate system are transformed to the world coordinate system P_world(x,y,z) where the robot arm is located, where P_world(x,y,z)=T*P_cam(x,y,z).
8. The intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model according to claim 7, characterized in that: The target segmentation and localization module converts the pixels of corresponding regions within the target segmentation areas of the left and right images to HSV space, calculates the color distribution histogram of each channel, and obtains the histogram feature vector vector_hist for each channel. Normalize the vector_hist of all channels, then merge and normalize it to obtain vector_normal. Calculate the inner product of vector_normal and vector_std, which is the normalized vector of the color histogram of the standard sample, to obtain an estimate R of the target purity. Based on the estimate, the target purity information can be obtained, and the target can be classified according to purity to obtain the classification result Class.
9. The intelligent feed detection and sorting device based on a binocular camera, a YOLO model, and a SAM2 model according to claim 8, characterized in that: The sorting and conveying module controls the mechanical structure to move the product to the designated sorting area based on the position P_world(x,y,z) and the category result Class, thus completing the sorting.