Optimal target recognition method for automatic individual feeding of bamboo products

Abstract

This invention relates to the field of bamboo product processing, specifically to the optimization of a method for the bamboo product feeding process. The purpose of this invention is to provide an optimal target recognition method suitable for the automatic, one-by-one feeding of bamboo products. Based on binocular stereo vision, it simultaneously acquires two-dimensional images and three-dimensional depth information of the scene, autonomously determining the optimal object to be grasped. This provides a recognition and instruction basis for the subsequent automatic and reliable identification and separation of individual bamboo strips from a chaotic pile of bamboo strips. It facilitates the full automation of the front-end feeding process, seamlessly connecting with downstream processing equipment, and significantly improving the overall automation level and operational efficiency of the production line.

Description

Technical Field

[0001] This invention relates to the field of bamboo product processing, and specifically to the optimization of a method for feeding bamboo products. Background Technology

[0002] Bamboo product processing refers to the process of using bamboo, especially pre-processed bamboo strips, as the main raw material and processing it into products with specific shapes, sizes, surface characteristics, and functions through a series of procedures. The starting point of bamboo product processing is the bamboo strip feeding process.

[0003] In existing technologies, the feeding of bamboo strips is a high-intensity, repetitive manual operation. Specifically, unsorted bamboo strips are piled haphazardly on the ground or in material bins, forming a random accumulation. Operators need to remove individual bamboo strips from this chaotic pile one by one and manually place or deliver them to the feed inlet or workbench of the processing equipment for the next processing step. This process relies entirely on manual labor for material identification, separation, and feeding.

[0004] The existing design process has significant flaws. Manual feeding is inefficient and costly, heavily reliant on continuous manual labor, resulting in high work intensity and limiting production scale. A more fundamental technical deficiency lies in the haphazard stacking of non-standard bamboo strips, which poses a major obstacle to automation. Traditional automated equipment, such as programmed robotic arms or simple conveyor mechanisms, can transport bamboo strips in batches and piles, but cannot reliably identify and separate individual bamboo strips. Therefore, although the processing equipment itself, such as CNC machine tools, possesses automated processing capabilities, the automation bottleneck in the front-end feeding stage prevents the entire process from achieving a fully automated closed loop, hindering the effective improvement of production efficiency and automation levels. Summary of the Invention

[0005] The purpose of this invention is to provide an optimal target recognition method suitable for the automatic, one-by-one feeding of bamboo products. Based on binocular stereo vision, it simultaneously acquires two-dimensional images and three-dimensional depth information of the scene, autonomously determining the optimal object to be grasped. This provides a recognition and instruction basis for the subsequent automatic and reliable identification and separation of individual bamboo strips from a chaotic pile of bamboo strips. It facilitates the full automation of the front-end feeding process, seamlessly connecting with downstream processing equipment, and significantly improving the overall automation level and operational efficiency of the production line.

[0006] This invention provides an optimal target recognition method suitable for automatic individual feeding of bamboo products, comprising the following steps:

[0007] Use a binocular recognition device to acquire image one from a first-view perspective and image two from a second-view perspective;

[0008] A depth map is obtained based on Image 1 and Image 2, and the depth map contains the depth data of each point in the image.

[0009] The images obtained using the binocular recognition device are input into the bamboo strip recognition model to obtain multiple candidate targets and the supplementary information of each candidate target;

[0010] Based on the depth map and the associated information of the multiple candidate targets, the optimal target is selected from the multiple candidate targets according to a preset target selection strategy.

[0011] As a preferred embodiment of the present invention, the ancillary information includes category, confidence level, bounding box, instance segmentation mask and / or contour point set.

[0012] As a preferred embodiment of the present invention, the step of selecting the optimal target from the multiple candidate targets based on the depth map and the auxiliary information of the multiple candidate targets, according to a preset target selection strategy, specifically involves:

[0013] Candidate targets that are determined to be independent and non-overlapping are selected through a first-level screening. The candidates that pass the first-level screening are sorted according to their depth data, and the candidate farthest from the ground is selected as the optimal target.

[0014] As a preferred embodiment of the present invention, when there are multiple candidate targets with consistent depth data, the candidate target with the highest confidence score is selected as the optimal target.

[0015] As a preferred embodiment of the present invention, the step of selecting the optimal target from the multiple candidate targets based on the depth map and the auxiliary information of the multiple candidate targets, according to a preset target selection strategy, specifically involves:

[0016] The candidate targets are sorted according to their depth data, and the candidate target farthest from the ground is selected as the optimal target. When there are multiple candidate targets with the same depth data, the independence is further determined, and the independent and non-overlapping candidate targets are selected as the optimal target.

[0017] As a preferred embodiment of the present invention, the step of selecting the optimal target from the multiple candidate targets based on the depth map and the auxiliary information of the multiple candidate targets, according to a preset target selection strategy, specifically involves:

[0018] For each candidate target, a confidence score, a high value score, and an independence score are calculated. A comprehensive score for each candidate target is obtained based on the scores of the three. The candidate target with the highest comprehensive score is determined as the optimal target.

[0019] As a preferred embodiment of the present invention, before selecting the optimal target from the multiple candidate targets based on the depth map and the auxiliary information of the multiple candidate targets, and according to a preset target selection strategy, a confidence screening step is further performed:

[0020] The confidence level of each candidate target is compared with a preset confidence threshold, and candidate targets with a confidence level lower than the confidence threshold are directly deleted from the list of candidate targets with filtering.

[0021] As a preferred embodiment of the present invention, the independence determination of the candidate target specifically involves comparing the instance segmentation mask of a candidate target with the instance segmentation masks of other candidate targets. If the comparison ratio is zero, the candidate target is determined to be an independent target.

[0022] As a preferred embodiment of the present invention, in the process of inputting the image obtained by the binocular recognition device into the bamboo strip recognition model to obtain multiple candidate targets and the auxiliary information of each candidate target, only the image one or the image two obtained by the binocular recognition device is input into the bamboo strip recognition model.

[0023] As a preferred embodiment of the present invention, before obtaining the depth map based on the first image and the second image, wherein the depth map includes depth data for each point in the image, an image preprocessing step is further included:

[0024] After performing image rotation correction and image stereo correction on Image 1 and Image 2, we obtain Corrected Image 1 and Corrected Image 2, and the depth map is derived from the corrected images.

[0025] In summary, the present invention has the following beneficial effects:

[0026] 1. By acquiring depth information through binocular vision and using a deep learning model to identify bamboo strips, multiple candidate targets can be automatically and accurately identified from a messy pile of bamboo strips. Combining depth information with target attributes, the optimal grasping target is intelligently selected based on a preset strategy, realizing reliable machine vision recognition and decision-making for non-standard and disordered bamboo strips, laying the core perception and decision-making foundation for fully automated feeding.

[0027] 2. By acquiring multi-dimensional supplementary information, including instance segmentation masks, it is possible not only to locate bamboo strips but also to accurately obtain their contours and spatial occupancy information. This provides detailed data support for subsequent filtering based on target independence and precise geometric location, significantly improving the accuracy of target filtering and the reliability of decision-making.

[0028] 3. One screening strategy prioritizes independent and non-overlapping targets, effectively avoiding the risk of accidentally touching or colliding with other bamboo strips during the grasping process, thus ensuring operational stability. Furthermore, selecting targets furthest from the ground optimizes the grasping path and improves the efficiency and safety of continuous operations.

[0029] 4. When multiple candidate targets are at the same height, confidence level is introduced as the final decision criterion. Prioritizing the target with the highest confidence level, the crawling model can maximize the success rate of a single crawl when depth information cannot distinguish between them, and reduce crawling failures or erroneous operations caused by target misidentification.

[0030] 5. One filtering strategy first identifies the topmost target based on depth information, prioritizing grabbing efficiency. When multiple targets exist at the same level, the most isolated target is selected based on independence criteria. This balances efficiency with stability in the grabbing action and minimal disruption to the stacking state, achieving a balance between efficiency and stability.

[0031] 6. One screening strategy, by constructing a quantitative scoring model that integrates confidence, height, and independence, enables multi-dimensional and quantifiable comprehensive evaluation of candidate targets. The weights can be adjusted according to actual production needs, achieving flexible and optimal target selection and enhancing the system's ability to adapt to different process requirements.

[0032] 7. Setting a confidence threshold filtering step before the core screening process can preemptively eliminate targets with low identification reliability. This reduces the processing burden and interference on downstream decision-making modules, improves overall processing efficiency, reduces the risk of performing crawling actions based on erroneous identification results, and enhances the robustness of the system.

[0033] 8. By calculating the intersection-union ratio (IUU) between instance segmentation masks, the independence of targets is quantitatively determined, providing an objective and accurate mathematical criterion. This method can effectively distinguish between bamboo strips that are in contact or overlapping, ensuring the accuracy of the independence screening process and providing a reliable basis for judging spatial relationships for stable grasping.

[0034] 9. By inputting only monocular images into the recognition model, the requirements for the complexity of the recognition model and computing resources are reduced, thereby improving the real-time performance of image processing and target recognition.

[0035] 10. Rotation and stereo corrections are performed on the original image to eliminate geometric errors caused by camera mounting pose and lens distortion. This step ensures the accuracy of subsequent stereo matching for depth map generation and provides a standardized input image for the target recognition model. Attached Figure Description

[0036] Figure 1A flowchart illustrating several embodiments of the optimal target recognition method applicable to the automatic individual feeding of bamboo products is shown.

[0037] Figure 2 The flowcharts illustrating several embodiments of this specification demonstrate optimal target selection methods based on a progressive filtering approach of completeness, proximity, and confidence.

[0038] Figure 3 A schematic diagram of the structure of an electronic device according to some embodiments of this specification is shown. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings.

[0040] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings.

[0041] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0042] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this specification. Various processes or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0043] Figure 1 This specification illustrates several embodiments of an optimal target recognition method suitable for the automatic individual feeding of bamboo products. The specific operation steps of this method 100 are as follows.

[0044] In box 101, method 100 is used for image acquisition and performs a certain degree of preprocessing.

[0045] Box 101 is the perception entry point of the entire automated grasping system. Its core function is to acquire and prepare high-quality image data to provide accurate and reliable input for subsequent 3D reconstruction and object recognition.

[0046] In this step, the system's main control unit, typically an industrial computer or embedded controller, sends a trigger signal to the hardware-synchronized binocular cameras via a specific communication protocol, such as GigE Vision or USB3 Vision. This signal ensures that the shutters of the left and right cameras inside the camera operate within the same microsecond interval, exposing the same physical scene. The image sensor inside the camera converts the captured light signals into electrical signals, which are then converted from analog to digital to form two digital images.

[0047] The camera transmits the two raw image data streams to the main control unit's memory via the data interface. Strictly speaking, what is obtained at this point are two raw left-eye images and a raw right-eye image that are completely synchronized in time but have horizontal parallax in space.

[0048] Then, image rotation correction is performed.

[0049] Due to mechanical structure or field-of-view optimization requirements, binocular cameras are often mounted in a non-horizontal orientation. This results in the orientation of the acquired raw images in memory not conforming to the conventional image coordinate system. The system's image preprocessing program applies a rigid geometric transformation matrix to each of the two raw images in memory, based on the camera's fixed physical mounting angle parameters, for example, a 90-degree counterclockwise rotation. This matrix operation recalculates the coordinates and color values ​​of each pixel in the image under the new orientation, generating a new pixel array through an image resampling algorithm (such as bilinear interpolation). The output is a rotation-corrected left-eye image and a rotation-corrected right-eye image, where the orientation of the image content is consistent with the standard orientation expected by the algorithm.

[0050] The core purpose of image rotation correction is to eliminate orientation interference caused by physical installation, so that objects in the image present a positive orientation that conforms to the preset values ​​of subsequent algorithms.

[0051] Then, a stereo calibration operation is performed.

[0052] The system calls a stereo correction algorithm, such as the Bouguet algorithm. This algorithm requires a set of stereo camera parameters obtained beforehand through precise calibration, including:

[0053] Intrinsic parameter matrix: describes the imaging characteristics of each camera, such as focal length (fx, fy) and principal point (cx, cy).

[0054] Distortion coefficient: describes the radial and tangential distortion caused by the lens.

[0055] The extrinsic parameter matrix describes the relative position and orientation relationship between the left and right cameras, including a rotation matrix R and a translation vector T.

[0056] The algorithm first corrects the distortion of the two rotated images based on the distortion coefficients, eliminating image distortion caused by the lens. Then, based on the extrinsic parameter matrix, the algorithm calculates a pair of projection transformation matrices. Applying these matrices, the algorithm reprojects the two images pixel-level, with the core mathematical goal of reprojecting the imaging planes of the two cameras onto a completely coplanar and row-aligned virtual plane. This process is called epipolar correction. After correction, the algorithm outputs the corrected left and right images. In these two new images, the image point of the same physical point in space will be strictly located on the same horizontal scan line in both images, meaning they will have the same ordinate.

[0057] Thus, in box 101, method 100 has obtained two two-dimensional images based on the binocular camera device. In some embodiments, after two preprocessing corrections, namely rotation correction and stereo correction, the corrected left-eye image and right-eye image are obtained. This is also the data basis for subsequent steps.

[0058] In box 102, method 100 obtains a depth map based on the left and right eye images. The depth map contains depth data for each point, which refers to the spatial distance of that point from the camera in the real three-dimensional world.

[0059] Specifically, a stereo matching algorithm is invoked and begins execution, for example, using the existing semi-global block matching algorithm SGBM. This algorithm takes the corrected left-eye and right-eye images as input. Its core task is to perform pixel-level correspondence search. Since the images have undergone epipolar correction, the search is simplified to be performed within the same image rows. The algorithm finds a matching point in the right image for each pixel in the left image by comparing the texture features of corresponding pixel blocks in the left and right images. The difference in the horizontal coordinates of two matching points on the same row is calculated; this difference is called disparity. The stereo matching algorithm aggregates all these disparity values ​​to generate a new image of the same size as the input image, representing the disparity magnitude with grayscale values—this is the disparity map.

[0060] Subsequently, the 3D reconstruction module immediately intervenes. Based on the pre-calibrated intrinsic parameters of the binocular cameras and the physical distance between the optical centers of the two camera lenses (i.e., the baseline distance), and applying the principle of triangulation, it converts each pixel value in the disparity map into the actual physical distance value corresponding to that pixel. This calculation process is performed pixel by pixel, ultimately outputting a depth map.

[0061] Each pixel value in this depth map directly represents the spatial distance of that point in the scene relative to the camera lens, thus expanding two-dimensional image information into depth data containing three-dimensional geometric information.

[0062] In box 103, a pre-trained deep learning model, such as YOLOv8, is introduced and used as the bamboo strip recognition model. The image is input into this model for contour recognition, identifying the comprehensive information of the bamboo strips in the image. This comprehensive information can be presented as a list containing multiple candidate targets and supplementary information for each target.

[0063] The image of the bamboo strip recognition model is input here. It is neither the depth map in box 102 nor the two two-dimensional views in box 101. Instead, you only need to input one of the two-dimensional images obtained in box 101, such as the left eye image.

[0064] This is because in a binocular vision system, the core function of the left and right images is to calculate depth through parallax. However, in box 103, the goal is to identify the outline information of the bamboo strips, and a single high-resolution color image usually provides sufficient information, such as texture, color, and shape. Therefore, object recognition is performed using a single image, while depth calculation is performed using a pair of images. This avoids the increased computational load and model complexity that would result from simultaneously inputting two images into the recognition model.

[0065] When the corrected left image is input into the bamboo strip recognition model, feature extraction begins first: the model analyzes the input image from shallow to deep using a multi-layer convolutional network. For example, the primary network layer identifies basic features such as lines and corners, while the deeper network combines these basic features into more advanced abstract features, such as the elongated structure of bamboo strips, bamboo joint texture, or the color features of the green / yellow side.

[0066] Subsequently, based on the rich features extracted above, the model makes predictions in different regions of the image and generates structured outputs, namely, bamboo strip comprehensive information. This bamboo strip comprehensive information typically includes category, confidence score, bounding box, instance segmentation mask, and / or contour point set.

[0067] The category refers to the type of object to which it is identified. In this application, the category is "bamboo strip". The model is able to distinguish the bamboo strip from other objects or noise in the background.

[0068] Confidence level is a numerical value between 0 and 1, quantifying how confident the model is about its prediction. For example, a confidence level of 0.95 means the model is 95% confident that the area is a bamboo strip. This provides a crucial quantitative basis for determining the best target to grab.

[0069] A bounding box is a rectangular box used to define the approximate location and extent of a target object in an image. It is defined by the coordinates of the box's center point, width, and height, providing a rough spatial location of the target's existence.

[0070] Instance segmentation masks or contour point sets are more refined outputs than bounding boxes. An instance segmentation mask is a binary image of the same size as the input image, where each pixel containing the target bamboo strip is precisely labeled as foreground (e.g., a value of 1), while background pixels are labeled with other values ​​(e.g., a value of 0). Contour point sets are collections of consecutive pixels that delineate the outer boundary of the bamboo strip. Both forms provide precise shape and contour information of the bamboo strip.

[0071] It should be noted that boxes 102 and 103 are independent steps and do not have a necessary order. Box 102 can be executed first, followed by box 103, or vice versa, or they can be processed simultaneously in parallel.

[0072] Box 104 is for selecting the best grasping target. Based on the results obtained from boxes 103 and 102, namely the comprehensive information of the bamboo strips obtained in box 103 and the depth map obtained in box 102, the current best grasping target, i.e. the bamboo strip that needs to be grasped, is selected according to the preset selection strategy.

[0073] The selection strategy is preset by engineers, and different selection strategies can be used in different scenarios. In some embodiments, confidence filtering is included.

[0074] The confidence level is represented by a value between 0 and 1. The higher the value, the more confident the model is that the area is a real bamboo strip target; the lower the value, the more the model believes that it may be background, noise, or an unidentifiable object.

[0075] The system obtains an initial list of recognition results from an object detection model, such as YOLOv8. This list contains all objects in the current image that the model initially identifies as bamboo strips, with each object accompanied by a confidence score.

[0076] Engineers preset a confidence threshold. This threshold is an empirical value, usually determined through extensive testing during the model development and debugging phase, for example, set to 0.7. This threshold represents a balance between reliability and sensitivity.

[0077] The system automatically iterates through each candidate target in the identification list, comparing its confidence score with a preset threshold of 0.7. If a candidate target has a confidence score greater than or equal to 0.7, it is considered a "reliable identification," likely a clear and complete bamboo strip, and is retained. If a candidate target has a confidence score less than 0.7, it is considered an "unreliable identification," as its confidence score is below the threshold. This could be a false alarm, such as a stick in the background that resembles the color of bamboo, or a blurry bamboo strip that is severely obscured and only a small portion is visible. The system will directly remove it from the current processing list.

[0078] By eliminating low-confidence targets, the system effectively avoids the risk of performing subsequent operations based on erroneous identification results. For example, it prevents the robotic arm from attempting to grab a bamboo strip that does not actually exist or a background object, thereby avoiding empty grabs, collisions, or other unexpected actions.

[0079] Different implementations employ different selection strategies, which involve choosing the target to be grasped from multiple candidate targets. In other words, from multiple bamboo strips identified through image recognition, the bamboo strip to be grasped is selected. Examples of selection strategies are provided below.

[0080] like Figure 2 As shown, in one embodiment, an optimal target selection method for grasping is based on a progressive filtering of completeness, proximity, and confidence. Its core principle is to prioritize grasping complete and independent targets, and under the same conditions, prioritize grasping targets closer to the robotic arm, ultimately using the highest recognition confidence as the deterministic criterion. This method aims to maximize the success rate and operational efficiency of a single grasp while minimizing disturbances to the overall stability of the bamboo strip stack.

[0081] The screening process and specific steps are as follows: Figure 2 As shown in box 201, spatial independence filtering based on integrity is used. Bamboo strips that have an interaction ratio of zero after being compared with other bamboo strips are judged as independent bamboo strips and added to the candidate set.

[0082] Specifically, the system acquires all candidate targets (let's say bamboo strips A, B, C, D, E, F, and G) that pass a basic confidence threshold (e.g., >0.7). For each candidate target, the system calculates the intersection-union ratio (IUR) between its instance segmentation mask and the masks of all other candidate targets. If the IUR between a target's mask and the masks of any other target is zero, it indicates that the target has no overlap, and the target is determined to be an independent bamboo strip. The system will retain only this set.

[0083] This step is to avoid a domino effect. By excluding targets that are physically in contact with or obstructed by other bamboo strips (such as A, B, and C), the system ensures that the selected target is physically isolated. When grasping such targets, the robotic arm's movements are less likely to involve or knock down other bamboo strips, thus greatly ensuring the safety of the grasping action and the stability of the entire working state, laying the foundation for continuous automated operation.

[0084] For example, after this level of screening, targets D, E, F, and G are determined to be complete and independent, and proceed to the next level of screening. Targets A, B, and C are eliminated because they overlap.

[0085] In box 202, there is the second level of filtering: depth-first selection based on spatial proximity.

[0086] In this step, it is necessary to obtain the depth information data for each bamboo strip.

[0087] For example, the process of acquiring specific bamboo strip depth data in box 104 essentially involves associating and fusing the recognition result of box 103 with the depth map of box 102. Specifically, firstly, the depth map output by box 102 and the input image used for recognition by box 103 correspond one-to-one in the pixel coordinate system. That is, the pixel value at coordinate (u, v) in the depth map represents the depth of the object point captured by the pixel at the same coordinate (u, v) in the left image.

[0088] Subsequently, depth data is extracted using a mask. The bamboo strip recognition model in box 103 not only outputs bounding boxes but also a precise instance segmentation mask for each bamboo strip target. This mask is a binary image in which all pixel positions occupied by the target bamboo strip are marked as foreground, for example, with a value of 1. When box 104 needs to obtain the depth information of "bamboo strip A", the system uses the instance segmentation mask of "bamboo strip A" as an index.

[0089] Finally, the system extracts and statistically analyzes the regional depth values. Based on the mask of "Bamboo Strip A," the system locates all corresponding pixel positions in the depth map, thereby extracting a batch of discrete depth values ​​belonging to the surface of Bamboo Strip A. Since the bamboo strip surface may contain reflections, missing textures, or measurement noise, the directly extracted raw depth values ​​may contain outliers. Therefore, the system typically performs robust filtering, such as the interquartile range (ICM), to remove outliers and calculates statistics on the remaining valid depth values, such as the mean or median. This statistic is then used as a robust depth value representing the overall depth or position of "Bamboo Strip A."

[0090] Therefore, in this step, the depth information of the corresponding region is queried and statistically analyzed from the pre-calculated global depth map by using the precise pixel-level positioning provided by the recognition results.

[0091] In this embodiment, the system reads the depth information associated with each remaining target (D, E, F, G). Specifically, in the depth map, the system calculates a robust average of the depth values ​​within the pixel region defined by the instance segmentation mask for each target, such as the median, as an approximate distance of the bamboo strip relative to the camera or robotic arm base. Subsequently, the system sorts the distance values ​​of all targets in ascending order and selects one or more targets with the smallest distance values ​​as output.

[0092] This step optimizes the system's operational efficiency and the robotic arm's energy consumption. Selecting the nearest target means that the robotic arm's end effector typically needs to move a shorter trajectory, thereby shortening the cycle time for a single grasp and improving the overall cycle time. Simultaneously, the shorter travel distance also helps reduce positioning errors caused by inertia and decreases energy consumption, aligning with the design principles of efficient automated production.

[0093] For example, the calculated distances to targets D, E, F, and G are 0.4 meters, 0.3 meters, 0.3 meters, and 0.5 meters, respectively. The system selects the targets with the smallest distances, namely E and F, both at 0.3 meters, for the final round of selection. Targets D (0.4 meters) and G (0.5 meters) are eliminated.

[0094] In some embodiments, the target to be captured has already been selected based on the previous two levels of filtering. However, in some cases, multiple targets may coexist, such as those with an EF of 0.3 meters as mentioned above. In this case, the process proceeds to the next level of filtering.

[0095] Box 203 represents the third level of screening, which is a confidence-based decision based on the reliability of identification.

[0096] After the first two levels of screening, among the remaining targets that have equal priority in terms of completeness and proximity, the system directly compares their confidence scores output by the target detection model. The system identifies the target with the higher confidence score as the final best candidate for capture.

[0097] Selecting a target with higher confidence means that the system chooses a more accurately recognized target for capture. This directly reduces the risk of capture failure due to visual misjudgment.

[0098] For example, target E has a confidence level of 0.94, and target F has a confidence level of 0.89. After comparison, the system selects target E as the best candidate for crawling in this round of iteration. Thus, the system completes the decision-making process of locking onto a single target E from seven initial targets, and outputs all associated data of target E, including its mask, bounding box, and 3D coordinates, to the subsequent crawling point calculation and execution module.

[0099] In another embodiment, unlike the previous embodiment where the three-level screening followed the order of independence judgment—height judgment—confidence judgment, this embodiment follows the order of height judgment—independence judgment—confidence judgment. This is because in this specific scenario, the bamboo strips to be processed are small in size and light in weight. Therefore, the screening strategy prioritizes height. For example, compared to bamboo strip B, bamboo strip A has a smaller average or median depth, meaning it is closer to the camera and has a higher height. In this case, the better choice for the gripper is to grab bamboo strip A. Even if bamboo strip A is partially obscured or overlapped by other bamboo strips, the light weight of the bamboo strips means that any overlap will be easily shaken off by the gripper, preventing significant jamming or obstruction during the gripper's handling of bamboo strip A.

[0100] In another embodiment, instead of selecting the optimal target according to the three-level screening and sorting, or the two-level and four-level screening and sorting described above, it is achieved by sorting the comprehensive scores of multiple factors.

[0101] Let's take confidence level, height value, and independence as an example again. In this case, the ranking is based on the comprehensive scores of each candidate target. Confidence level, height value, and independence are scored on a 100-point scale, multiplied by their corresponding weight values, and finally added together to get the total score of the candidate target. The candidate target with the highest total score is the one to be captured.

[0102] Specifically, the confidence score (S_conf) is calculated by multiplying the confidence score output by the object detection model by 100 and converting it to a percentage. This score directly reflects the reliability of the model's recognition results.

[0103] The height score (S_height) is calculated based on depth map data, representing the median depth of the target area and reflecting the relative distance between the target and the grasping device. The system presets an optimal grasping distance range; the closer the target's depth value is to the center of this range, the higher the score. For example, the score can be mapped to a negative correlation function with the deviation from the optimal distance, ultimately normalized to 0-100 points. This score aims to prioritize grasping targets at the most suitable operating distance.

[0104] Independence Score (S_iso): Calculates the maximum cross-union ratio (CURRR) of the target instance segmentation mask with all other target masks. A lower CURRR indicates less overlap or occlusion between the target and other objects, resulting in better independence. The independence score can be designed to be negatively correlated with the maximum CURRR, with completely independent targets (CURRR of 0) receiving 100 points and completely overlapping targets receiving 0 points. This score is used to assess the risk of interfering with stacking stability when grasping the target.

[0105] Based on a prior assessment of the importance of each factor to successful capture, weighted coefficients are assigned to the three scores, as pre-defined by the engineer. For example, confidence weight (W_conf = 0.5), height weight (W_height = 0.3), and independence weight (W_iso = 0.2). The sum of the weights is 1.0.

[0106] For any candidate target i, its total score (Total_Score_i) is calculated as follows:

[0107] Total_Score_i = (S_conf_i × W_conf) + (S_height_i × W_height) + (S_iso_i × W_iso). After calculating the comprehensive score of all candidate targets, the system sorts them in descending order. The target with the highest score is determined as the "best target to grab" for the current loop.

[0108] For example, given three bamboo sticks A, B, and C, the system calculates the following scores for stick A: confidence 88, height 70, and independence 60, for a total score of 77.0. Stick B scores: confidence 95, height 50, and independence 90, for a total score of 80.5. Stick C scores: 82, 90, and 40, for a total score of 76.0. Comparing these results, stick B ranks first with a total score of 80.5. Therefore, the system identifies stick B as the optimal target for this round of crawling.

[0109] This method achieves an optimal balance between reliability, operational efficiency, and operational safety by incorporating multidimensional decision-making factors into a unified mathematical framework. Furthermore, the weighting coefficients can be flexibly adjusted according to specific process requirements, demonstrating high adaptability and decision-making transparency.

[0110] At this point, box 104 is complete. It has identified and calculated the optimal target for the next grasping step, such as bamboo strip B.

[0111] In block 105, method 100 is used to instruct the gripping device to grasp the corresponding bamboo strip according to the instructions in block 104. The gripping device can be a conventional mechanical claw capable of movement along the XYZ axes. Block 105 specifically performs the positioning, coordinate calculation, and final physical grasping of the target. This process begins with the precise calculation of the target grasping point and ends with the closed-loop motion control of the robotic arm, exemplarily as follows:

[0112] First, perform gripping point localization and 3D reconstruction.

[0113] The system performs geometric calculations based on the instance segmentation mask of the target bamboo strip B. The system obtains user-preset grab reference lines, such as the image center line, and calculates the central axis of the bamboo strip B's contour. Then, it calculates the intersection point of this central axis with the grab reference line and determines this point as the optimal grab point on the two-dimensional image plane; if there is no intersection, the end of the contour closest to the reference line is selected as the grab point.

[0114] Subsequently, region depth estimation is performed to obtain robust distance information.

[0115] The system extracts all depth values ​​corresponding to the depth map within the masked region of bamboo strip B. First, invalid values, such as 0, are removed. Then, outlier noise is filtered using outlier removal algorithms such as the interquartile range (ICM). Finally, the average of the remaining valid depth values ​​is calculated to obtain the robust depth value representing bamboo strip B in that region. Finally, combining the 2D pixel coordinates of this optimal grasping point, the robust depth value, and camera intrinsic parameters, the 3D spatial coordinates of this grasping point in the camera coordinate system are calculated using the back-projection formula of the pinhole camera model.

[0116] Subsequently, coordinate transformation and compensation are performed.

[0117] The system reads the hand-eye matrix, which describes the spatial relationship between the camera and the robot base and is obtained through pre-calibration. This matrix includes the rotation matrix R and the translation vector T. The system then transforms the three-dimensional coordinates of the grasping point in the camera coordinate system to the coordinates in the robot base coordinate system using the formula P_robot = R* P_camera + T.

[0118] To further compensate for end effector installation deviations or meet specific process requirements, the system superimposes the preset grasping offset and the fine-tuning values ​​in the X, Y, and Z directions onto the coordinates to obtain the final coordinates of the robot's grasping target point and its posture command.

[0119] Finally, the automatic grasping strategy is executed. For example, the control system drives the grasping device to perform physical operations according to a standardized process: The first step is initial reset, controlling the robotic arm to move to a preset safe waiting position. The second step is pre-grabbing action, controlling the end effector gripper to fully open and hold for a moment to ensure the action is in place. The third step is precise movement, the robotic arm maintains the end effector posture and precisely moves the center point of the end effector tool to the calculated final grasping target point using linear interpolation motion, and waits briefly after reaching the target to eliminate aftershocks. The fourth step is closing grip, controlling the gripper to close reliably clamp the bamboo strip B, and confirming the grasping status through sensor feedback. The fifth step is transport reset, controlling the robotic arm to smoothly return to the safe waiting position carrying the grasped bamboo strip B.

[0120] At this point, the entire process from visual positioning to physical grasping described in box 105 has been completed. This method ensures high accuracy, reliability, and automation in grasping a specific target, namely bamboo strip B, in a jumbled pile through precise grasping point geometry and 3D reconstruction, rigorous coordinate space transformation, and standardized, robust motion control strategies, thus fully realizing the closed loop of "perception-decision-execution".

[0121] Figure 3 A block diagram of an electronic device 300 that can implement various embodiments of the present disclosure is shown. For example... Figure 3 As shown, device 300 includes a processor 301, which can perform various appropriate actions and processes based on computer program instructions loaded into random access memory (RAM) 303 according to computer program instructions stored in read-only memory (ROM) 302. RAM 303 may also store various programs and data required for the operation of device 300. The processor 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0122] The various processes and procedures described above, such as method 100, can be executed by processor 301. For example, in some embodiments, method 100 may be implemented as a software program tangibly contained in a machine-readable medium. In some embodiments, part or all of the software program may be loaded and / or installed on device 300 via ROM 302. When the software program is loaded into RAM 303 and executed by processor 301, one or more actions of method 300 described above may be performed.

[0123] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0125] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An optimal target recognition method applicable to the automatic individual feeding of bamboo products, characterized in that, It includes the following steps: Use a binocular recognition device to acquire image one from a first-view perspective and image two from a second-view perspective; A depth map is obtained based on Image 1 and Image 2, and the depth map contains the depth data of each point in the image. The images obtained using the binocular recognition device are input into the bamboo strip recognition model to obtain multiple candidate targets and the supplementary information of each candidate target; Based on the depth map and the associated information of the multiple candidate targets, the optimal target is selected from the multiple candidate targets according to a preset target selection strategy.

2. The optimal target recognition method for automatic individual feeding of bamboo products according to claim 1, characterized in that: The additional information includes category, confidence level, bounding box, instance segmentation mask, and / or contour point set.

3. The optimal target recognition method for automatic individual feeding of bamboo products according to claim 2, characterized in that: The step of selecting the optimal target from the multiple candidate targets based on the depth map and the auxiliary information of the multiple candidate targets, according to a preset target selection strategy, specifically involves: Candidate targets that are determined to be independent and non-overlapping are selected through a first-level screening. The candidates that pass the first-level screening are sorted according to their depth data, and the candidate farthest from the ground is selected as the optimal target.

4. The optimal target recognition method for automatic individual feeding of bamboo products according to claim 3, characterized in that: When there are multiple candidate targets with consistent depth data, the candidate target with the highest confidence score is selected as the optimal target.

5. The optimal target recognition method for automatic individual feeding of bamboo products according to claim 2, characterized in that: The step of selecting the optimal target from the multiple candidate targets based on the depth map and the auxiliary information of the multiple candidate targets, according to a preset target selection strategy, specifically involves: The candidate targets are sorted according to their depth data, and the candidate target farthest from the ground is selected as the optimal target. When there are multiple candidate targets with the same depth data, the independence is further determined, and the independent and non-overlapping candidate targets are selected as the optimal target.

6. The optimal target recognition method for automatic individual feeding of bamboo products according to claim 2, characterized in that: The step of selecting the optimal target from the multiple candidate targets based on the depth map and the auxiliary information of the multiple candidate targets, according to a preset target selection strategy, specifically involves: For each candidate target, a confidence score, a high value score, and an independence score are calculated. A comprehensive score for each candidate target is obtained based on the scores of the three. The candidate target with the highest comprehensive score is determined as the optimal target.

7. The optimal target recognition method for automatic individual feeding of bamboo products according to any one of claims 2-6, characterized in that: Before selecting the optimal target from the multiple candidate targets based on the depth map and the associated information of the multiple candidate targets, according to a preset target selection strategy, there is a confidence screening step: The confidence level of each candidate target is compared with a preset confidence threshold, and candidate targets with a confidence level lower than the confidence threshold are directly deleted from the list of candidate targets with filtering.

8. The optimal target recognition method for automatic individual feeding of bamboo products according to any one of claims 2-6, characterized in that: The independence determination of candidate targets is as follows: for a candidate target, its instance segmentation mask is compared with the instance segmentation masks of other candidate targets. If the interaction ratio is zero, the candidate target is determined to be an independent target.

9. The optimal target recognition method for automatic individual feeding of bamboo products according to claim 1, characterized in that: In the process of inputting the image obtained by the binocular recognition device into the bamboo strip recognition model to obtain multiple candidate targets and the auxiliary information of each candidate target, only the first image or the second image obtained by the binocular recognition device is input into the bamboo strip recognition model.

10. The optimal target recognition method for automatic individual feeding of bamboo products according to claim 1, characterized in that: Before obtaining the depth map based on Image 1 and Image 2, wherein the depth map contains depth data for each point in the image, an image preprocessing step is also included: After performing image rotation correction and image stereo correction on Image 1 and Image 2, we obtain Corrected Image 1 and Corrected Image 2, and the depth map is derived from the corrected images.

Images