Methods, apparatus, and storage media for detecting surface defects in small fruits of the Rosaceae (Prunus) subfamily.

By combining depth information of fruit trees and small fruits with a multi-round benefit update method and utilizing the structural information of fruit stalks and sutures, the problem of accuracy in detecting surface defects on small fruit trees of the Rosaceae subfamily was solved, achieving efficient and accurate detection results.

CN123118455APending Publication Date: 2026-11-17SHAANXI UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611080296.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies cannot effectively detect surface defects on small fruit trees of the Rosaceae subfamily, leading to inaccurate detection and increased harvesting costs. Furthermore, inconsistent human judgment standards result in missed detections and misjudgments.

Method used

By acquiring the initial image to be inspected, a multi-round revenue update method is used, combined with the depth information of the fruit tree and small fruit, and the revenue parameter is used to constrain the candidate observation pose to obtain comprehensive revenue parameters to improve detection accuracy. In particular, the structural information of the fruit stalk and suture line is used to guide the selection of observation pose.

Benefits of technology

This method improves the accuracy and efficiency of surface defect detection in small fruits of the Rosaceae subfamily, reduces the waste of computing resources, and ensures the precision and completeness of the detection results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118455A_ABST
    Figure CN123118455A_ABST
Patent Text Reader

Abstract

This invention relates to the field of defect detection technology, and discloses a method, device, and storage medium for detecting surface defects in small fruits belonging to the Rosaceae subfamily. The method employs multiple rounds of payoff updates, using payoff parameters to constrain the selection of candidate observation poses based on an initial image to be inspected. Specifically, the first and second payoff parameters, through a hierarchical perspective of the "tree-fruit" relationship, encourage candidate observation poses to prioritize acquiring visually uncovered areas as supplementary information. For small fruits, the stem and suture line are recessed areas, considered difficult to detect, and these areas contain structural information, further guiding the construction of surface information during payoff updates, thereby improving the accuracy of candidate observation poses. Finally, by applying a first threshold, the updated set of images that meet the requirements is output for surface defect detection, thus saving computational resources while improving the accuracy of surface defect detection in small fruits.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of defect detection technology, and in particular to a method, apparatus, and storage medium for detecting surface defects in small fruits of the Rosaceae family (Prunus subfamily). Background Technology

[0002] Small fruits of the Prunoideae subfamily of the Rosaceae family, including peaches, apricots, plums, and cherries, are widely cultivated globally and have extremely high economic value. However, due to their thin skin and soft flesh, these fruits are prone to various surface defects during growth. Current defect detection processes mainly target surface defects in harvested Prunoideae small fruits. However, during the harvesting process, different farmers have different standards for judging defects manually, and the fatigue caused by repetitive labor leads to missed detections and misjudgments, increasing ineffective harvesting costs and creating a hidden danger of diseased fruits being mixed with contaminated healthy fruits.

[0003] Existing technologies for tree-based inspection employ robots and drones that move along preset paths and viewpoints to collect visual information and perform inspections. However, these methods are not well-suited for small fruits of the Rosaceae subfamily (Prunus spp.). This is because relying on fixed collection trajectories or passive sensing strategies often results in incomplete fruit information. Furthermore, the collection process wastes computational resources by repeatedly collecting data from already observed areas. Areas requiring precise observation may remain inaccurate if the pedicel, ventral suture, and calyx end are not adequately covered. Summary of the Invention

[0004] This invention provides a method, apparatus, and storage medium for detecting surface defects in small fruits of the Rosaceae family, Prunoideae subfamily. It can improve the accuracy of surface defect detection results by utilizing the unique characteristics (pedicel, suture line) of small fruits of the Rosaceae family, Prunoideae subfamily, and combining information gain methods, while saving observation and computing resources.

[0005] The first aspect of this invention discloses a method for detecting surface defects in small fruits of the Rosaceae subfamily Prunoideae, the method comprising: Acquire an initial image to be inspected, the initial image to be inspected including a fruit tree and at least two small fruits on the fruit tree and the depth information corresponding to each small fruit, the small fruits being of the Rosaceae family, Prunoideae subfamily, the small fruits including suture lines and fruit stalks; Based on the initial image to be inspected, multiple rounds of revenue update operations are performed, wherein each round of revenue update operation includes: Multiple candidate observation poses are obtained. For each candidate observation pose: based on all images to be inspected before the current round, a first benefit parameter, a second benefit parameter, and a third benefit parameter are obtained. All benefit parameters are weighted and fused to obtain a comprehensive benefit parameter. The first benefit parameter is used to measure the completeness of surface information corresponding to the fruit tree, the second benefit parameter is used to measure the completeness of surface information corresponding to all small fruits, and the third benefit parameter is used to measure the confidence of the predicted pose. The predicted pose is obtained based on the predicted orientation of the first representation of each small fruit, which includes the suture line and the fruit stalk. When it is determined that the comprehensive benefit parameter is greater than or equal to a preset first threshold, a preferred image to be inspected is obtained based on the candidate observation pose with the largest comprehensive benefit parameter. The preferred image to be inspected is used for the next round of benefit update. When it is determined that the comprehensive benefit parameter is less than the first threshold, all images to be inspected are identified as the image set to be inspected, and the surface defect detection result corresponding to each small fruit is obtained based on the image set to be inspected.

[0006] As an optional implementation, in the first aspect of the present invention, in each round of revenue update operation, for each candidate observation pose, the method for obtaining the second revenue parameter includes: Construct a first prediction surface, which is obtained by fitting the unknown region surface of the fruit tree corresponding to all images to be inspected before the current round; Construct a candidate imaging plane, which is used to represent the simulated imaging plane of the candidate observation pose; The first predicted surface is projected onto the candidate imaging plane, and the second revenue parameter is obtained based on the projected area corresponding to the first predicted surface.

[0007] As an optional implementation, in the first aspect of the invention, in each round of revenue update operation, for each candidate observation pose, obtaining the third revenue parameter based on all images to be inspected includes: Based on the first prediction surface, the total predicted length of the first representation of each small fruit is obtained; Based on the projection result of the first predicted surface onto the candidate imaging plane, the ideal observation length of the first characterization of each small fruit and the extrapolated suture line are obtained. The extrapolated suture line is obtained by extrapolation and fitting based on the suture line extrapolated from the observable part of the projection result. The first benefit sub-parameter is obtained based on the ratio of the ideal observation length to the predicted total length. The first benefit sub-parameter is used to measure the visibility of the first representation that the candidate observation pose is expected to observe. The suture line fitted to the unknown region in the first prediction surface is three-dimensionally stitched together with the suture lines in all the images to be inspected to obtain the prior suture line. A second benefit sub-parameter is obtained based on the extrapolated suture line and the prior suture line. The second benefit sub-parameter is used to measure the difference between the extrapolated suture line and the prior suture line. The third revenue parameter is obtained based on the first revenue sub-parameter and the second revenue sub-parameter.

[0008] As an optional implementation, in the first aspect of the present invention, in each round of revenue update operation, for each candidate observation pose, the method for obtaining the third revenue parameter includes: An occlusion model is constructed for the fruit tree, the occlusion model including the fitted surface for the leaves and the fitted surface for the branches; based on the occlusion model, the first prediction surface is occluded to obtain an occlusion projection surface, the occlusion projection surface is used to represent the projection result of the first prediction surface onto the candidate imaging plane when it is occluded; Based on all the images to be inspected and the first predicted surface, the corresponding position of the fruit center point of each small fruit in the occlusion projection surface is determined as a reference point; Based on the first continuous line and the reference point, a third benefit sub-parameter is obtained. The first continuous line is the longest continuous line segment corresponding to the suture line in the occlusion projection surface. The third benefit sub-parameter is obtained based on the angular span of the start and end points of the first continuous line relative to the reference point. The third benefit sub-parameter is used to measure the continuity of the line segment corresponding to the suture line in the occlusion projection surface. The third revenue parameter is obtained based on the first revenue sub-parameter, the second revenue sub-parameter, and the third revenue sub-parameter.

[0009] As an optional implementation, in the first aspect of the present invention, in each round of revenue update operation, for each candidate observation pose, the method for obtaining the second revenue parameter includes: Based on all the images to be inspected and the first predicted surface, construct the UV unfolded map corresponding to each small fruit; Based on the projection result of the first predicted surface onto the candidate imaging plane, the coverage area of ​​the projection result in the UV unfolding diagram is obtained; The fourth revenue sub-parameter is obtained based on the ratio of the coverage area to the total area of ​​the UV unfolded map; The region corresponding to the first representation in the UV unfolded map is determined as the representation region; the region in the UV unfolded map whose distance from the representation region is less than a preset second threshold is determined as the first optimal detection region; the region in the UV unfolded map whose distance from the representation region is greater than or equal to the second threshold and less than a preset third threshold is determined as the second optimal detection region; a fifth benefit sub-parameter is obtained based on the first overlapping area ratio and the second overlapping area ratio, wherein the first overlapping area ratio is the ratio of the overlapping area of ​​the representation region in the first optimal detection region to the total area, and the second overlapping area ratio is the ratio of the overlapping area of ​​the representation region in the second optimal detection region to the total area; The fourth and fifth revenue sub-parameters are weighted and fused to obtain the second revenue parameter.

[0010] As an optional implementation, in the first aspect of the invention, each round of revenue update operation further includes: When it is determined that all of the first revenue parameters are less than the preset fourth threshold, the weight of the first revenue parameter in the weighted fusion process in the next round is reduced. When it is determined that all of the third revenue parameters are less than the preset fifth threshold, the weight of the third revenue parameters in the weighted fusion process in the next round is reduced.

[0011] As an optional implementation, in the first aspect of the present invention, in each round of revenue update operation, determining all images to be inspected as the set of images to be inspected when it is determined that the comprehensive revenue parameter is less than the first threshold includes: When it is determined that the comprehensive benefit parameter is less than the first threshold, the candidate observation pose whose orientation angle meets the preset condition is determined as the candidate observation pose, and the orientation angle is the angle between the orientation of the candidate observation pose and the orientation corresponding to the fruit stalk. Based on multiple randomly selected candidate observation poses, multiple candidate images to be inspected are obtained; based on all images to be inspected before the current round, the preferred images to be inspected, and all candidate images to be inspected, a set of images to be inspected is obtained.

[0012] A second aspect of this invention discloses a device for detecting surface defects in small fruits belonging to the Rosaceae subfamily Prunoideae, the device comprising: The image acquisition module is used to acquire an initial image to be inspected, which includes a fruit tree, at least two small fruits on the fruit tree, and depth information corresponding to each small fruit. The small fruits are from the Rosaceae family, Prunoideae subfamily, and include suture lines and fruit stalks. The revenue update module is used to perform multiple rounds of revenue update operations based on the initial image to be inspected, wherein each round of revenue update operation includes: Multiple candidate observation poses are obtained. For each candidate observation pose: based on all images to be inspected before the current round, a first benefit parameter, a second benefit parameter, and a third benefit parameter are obtained. All benefit parameters are weighted and fused to obtain a comprehensive benefit parameter. The first benefit parameter is used to measure the completeness of surface information corresponding to the fruit tree, the second benefit parameter is used to measure the completeness of surface information corresponding to all small fruits, and the third benefit parameter is used to measure the confidence of the predicted pose. The predicted pose is obtained based on the predicted orientation of the first representation of each small fruit, which includes the suture line and the fruit stalk. When it is determined that the comprehensive benefit parameter is greater than or equal to a preset first threshold, a preferred image to be inspected is obtained based on the candidate observation pose with the largest comprehensive benefit parameter. The preferred image to be inspected is used for the next round of benefit update. When it is determined that the comprehensive benefit parameter is less than the first threshold, all images to be inspected are identified as the image set to be inspected, and the surface defect detection result corresponding to each small fruit is obtained based on the image set to be inspected.

[0013] A third aspect of this invention discloses another device for detecting surface defects in small fruits of the Rosaceae subfamily Prunoideae, the device comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the method for detecting surface defects of small fruits of the Rosaceae subfamily as described in any of the first aspects of the present invention.

[0014] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the method for detecting surface defects of small fruits of the Rosaceae subfamily as described in any of the first aspects of the present invention.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention acquires an initial image to be inspected and, based on this initial image, uses multiple rounds of reward updates, employing reward parameters to constrain the selection of candidate observation poses. Specifically, the first and second reward parameters, through a hierarchical perspective of the "tree-fruit" relationship, encourage candidate observation poses to prioritize acquiring information from visually uncovered areas. For small fruits, the stem and suture line are recessed areas, making them difficult to inspect. Furthermore, the stem and suture line are unique structures of small fruits, containing structured information that further guides the construction of surface information during the reward update operation, thereby improving the accuracy of candidate observation poses. Finally, a first threshold is used to determine update conditions, and the updated image set that meets the requirements is output for surface defect detection. This improves acquisition accuracy and observation efficiency at the image level, thus saving computational resources while increasing the accuracy of surface defect detection results. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic flowchart of a method for detecting surface defects in small fruits of the Rosaceae subfamily, as disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the suture line and pedicel of a small fruit of the Rosaceae subfamily Prunoideae disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a surface defect detection device for small fruits of the Rosaceae subfamily disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a device including a memory and a processor disclosed in an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Example 1 Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a method for detecting surface defects in small fruits of the Rosaceae (Prunus) subfamily, as disclosed in an embodiment of the present invention. Figure 1 The described method for detecting surface defects in small fruits of the Rosaceae (Prunus) subfamily can be applied to surface defect detection devices for small fruits of the Rosaceae (Prunus) subfamily. For example... Figure 1 As shown, the Methods for detecting surface defects in small fruits of the Rosaceae (Prunoideae) subfamily include the following steps: Step 101: Obtain the initial image to be inspected.

[0022] In this embodiment of the invention, the initial image to be inspected may include a fruit tree, at least two small fruits on the tree, and depth information corresponding to each small fruit. The initial image to be inspected may be a complete view of the fruit tree or a partial view. The purpose of including the fruit tree in the initial image to be inspected is to ensure that the visual information subsequently acquired can completely cover every fruit on the entire tree, avoiding omissions. The purpose of including at least two small fruits in the initial image to be inspected is to emphasize the validity constraint of the initial image to be inspected, because this embodiment of the invention can simultaneously acquire visual information of multiple small fruits in a single image and perform integrated analysis. If only an initial image of a single small fruit is acquired, it will not only affect the efficiency of surface defect detection but also affect the accuracy of the integrated analysis of the visual information in the benefit update stage.

[0023] Small fruits, belonging to the subfamily Prunoideae of the Rosaceae family (drupes), include the individual fruits and hybrid varieties of peaches, plums, apricots, cherries, sweet cherries, and prunes. Their common characteristics are the presence of a pedicel (at the end of the pedicel) and a suture line (ventral suture). The suture line is a marker formed by the natural fusion of the carpels during the growth of these fruits, located at the other end or side of the pedicel. For example... Figure 2 As shown, Figure 2 This is a schematic diagram of the suture line and pedicel of a small fruit belonging to the Rosaceae subfamily of the present invention, as disclosed in an embodiment of the invention. The pedicel is located obliquely to the upper right of the small fruit (within the image shown). Figure 2 A small columnar protrusion (defined by the perspective direction), the suture line is an arc extending obliquely to the lower left from the base of the fruit stalk on the fruit body.

[0024] In this embodiment of the invention, the initial image to be inspected can be obtained by any device with shooting function, such as a robot (or drone, unmanned vehicle) with shooting function for automated shooting, or by human shooting. This embodiment of the invention does not limit the method.

[0025] It should be noted that since subsequent multiple rounds of update operations require the acquisition of candidate observation poses, and the analysis process of candidate observation poses based on benefit parameters involves the measurement of surface information integrity, and the acquisition of such surface information is actually based on the construction of three-dimensional information, in this embodiment of the invention, while acquiring the initial image to be inspected, it is necessary to acquire the camera pose (initial observation pose) in the initial observation state, so as to construct a relative coordinate system or an absolute coordinate system on this basis, and input the acquired visual information into the information processing device or system according to the coordinate system adapted to the pose, so as to facilitate the implementation of various subsequent processing and analysis processes.

[0026] Specifically, the initial image to be inspected may include depth information, which can be directly acquired by the imaging device. For example, an RGB-D depth camera can record distance information for each pixel while capturing a color image, or a binocular / multi-view stereo vision camera can estimate the distance to an object through the parallax in the images from multiple cameras. This depth information can also be obtained through computer vision and deep learning algorithms, such as Magma (mobile app), Depth-Anything-V2, PaddleDepth, etc., or LiDAR can be used to assist in acquiring depth information. This embodiment of the invention does not limit the scope of this invention.

[0027] It's important to note that the imaging equipment needs to be high-resolution to ensure that the surface details and defects of small fruits can be captured clearly with high pixel counts. Furthermore, the surface curvature of small fruits is greater than that of normal fruits; therefore, in subsequent revenue updates, the surface information of small fruits needs to be described using high-resolution voxels or Gaussian ellipsoids. Additionally, if robots, drones, or unmanned vehicles are used for image acquisition, high-precision guide rails or robotic arms are required to achieve precise movement at minute angles across multiple observation poses.

[0028] Furthermore, acquiring the initial image to be inspected describes the initial acquisition process of the image. All images acquired in the initial stage before the profit update stage for analysis and processing can be considered initial images to be inspected. This embodiment of the invention does not limit the number of initial images to be inspected. Moreover, if multiple initial images to be inspected are acquired in the early stages (before multiple rounds of profit updates), the verification between these images can improve the accuracy of visual information, making subsequent prior surface information more complete. This is more conducive to selecting candidate observation poses, thereby indirectly improving the efficiency and accuracy of acquiring the preferred image to be inspected.

[0029] Step 102: Based on the initial image to be inspected, perform multiple rounds of revenue update operations.

[0030] Each round of profit update operations includes: Multiple candidate observation poses are obtained. For each candidate observation pose: based on all images to be inspected before the current round, the first benefit parameter, the second benefit parameter and the third benefit parameter are obtained. All benefit parameters are weighted and fused to obtain the comprehensive benefit parameter. When it is determined that the comprehensive benefit parameter is greater than or equal to the preset first threshold, the preferred image to be inspected is obtained based on the candidate observation pose with the largest comprehensive benefit parameter. When the comprehensive benefit parameter is determined to be less than the first threshold, all images to be inspected are identified as the image set to be inspected. Based on the image set to be inspected, the surface defect detection results corresponding to each small fruit are obtained.

[0031] In this embodiment of the invention, the benefit update operation essentially analyzes the benefit of the observation pose used to acquire a new round of images based on the surface information corresponding to all images to be inspected before the current round, and then selects the pose with the highest benefit to save time in acquiring new images, avoid repeatedly acquiring images with the same content and generating information redundancy, thereby improving the modeling efficiency of small fruits (the efficiency of acquiring image information completeness), so as to efficiently pick up complete surface information of small fruits and use this surface information for detection, making the surface defect detection process more accurate from the source.

[0032] In this embodiment of the invention, the pose includes data information in six independent degrees of freedom, including three dimensions for representing position and three dimensions for representing orientation (rotation). The observation pose represents the coordinates of the capturing device (camera) and the orientation of the camera when acquiring the image. The candidate observation pose is the shooting pose when acquiring the image to be inspected. This shooting pose and the captured image have a one-to-one correspondence. Furthermore, the image acquired in each round of profit update should have depth information, just like the initial image to be inspected, so as to determine the corresponding surface coverage of each small fruit based on the depth information. Acquiring multiple candidate observation poses actually means acquiring all possible observation poses within the observation space allowed by the observation conditions. Since there may be many candidate poses, the following methods can be used in actual operation to further improve computational efficiency: First, the orientation of the candidate observation pose is limited to a central bounding box within the area where the fruit tree is located, thereby restricting the rotational degrees of freedom of the candidate observation pose. Furthermore, a sampling shell is defined based on camera resolution or other performance characteristics. This sampling shell is used to represent the restricted range of the observation pose's position.

[0033] Second, a sparse sampling method is adopted. First, several hundred candidate points are generated in the observation space, and the scores are roughly calculated. Then, local dense sampling is performed near the top few points with the highest scores.

[0034] In this embodiment of the invention, the first benefit parameter is used to measure the completeness of the surface information corresponding to the fruit tree. The surface information includes image information and three-dimensional information. Specifically, the image information is the visual information obtained corresponding to the pose, and the three-dimensional information can be obtained from the depth information corresponding to the obtained image, used to represent the spatial position of each pixel in the image. The surface information of the fruit tree can be only the surface information of the trunk, or it can include the surface information of the fruit, leaves, and branches.

[0035] Surface information completeness is essentially a prediction value based on known information. Specifically, it's the ratio of known information to the total predicted information. For example, mapping surface information to a point cloud allows us to fit the distribution of points in unknown areas to the point cloud. Alternatively, we can avoid fitting and instead count the possible circumscribed cuboids of the point cloud; the completeness is then the ratio of known to unknown areas within that cuboid. The purpose of obtaining the first revenue parameter is to observe the fruit tree from a holistic perspective to ensure there are any missing information points, guaranteeing that the image content fully corresponds to the actual number of fruits on the tree, and preventing the measurement of the subsequent second revenue parameter from being affected by the omission of some fruit information.

[0036] In this embodiment of the invention, the second revenue parameter is used to measure the completeness of surface information corresponding to all small fruits. This surface information includes both image information and 3D information. Specifically, the measurement method for the completeness of surface information can adopt the method for measuring the completeness of the fruit tree described in the previous embodiment. Alternatively, it can project the 3D surface onto a 3D point cloud based on image and depth information, mesh the 3D point cloud, and connect adjacent pixels to form triangular patches. An algorithm is set to traverse every edge in the entire mesh. A normal edge is shared by two triangles. If an edge is found to connect only one triangle, and the other side is a hole, then that edge is determined to be an edge on the boundary. The area obtained by unfolding (projecting) the known 3D surface onto a 2D plane and the area of ​​the surface enclosed by the boundary are calculated. The completeness can be obtained based on these two areas. At this point, the second revenue parameter and the first revenue parameter form a hierarchical / step-like completeness evaluation method, where the hierarchy transitions from the overall fruit tree to the details of the fruit. Since this surface defect detection method targets small fruits, obtaining the completeness of the fruit tree surface information is to assist in obtaining the surface information of the small fruits. Therefore, a higher weight can be set for the first benefit parameter in the early stage to capture the information points of all small fruits on the fruit tree. Then, the weight of the first benefit parameter can be reduced and the weight of the second benefit parameter can be increased so that the obtained preferred image to be inspected focuses on improving the details of the small fruits themselves.

[0037] In this embodiment of the invention, the third benefit parameter is used to measure the confidence level of the predicted pose. The predicted pose is obtained based on the predicted orientation of the first representation of each small fruit. The first representation includes the suture line and the pedicel. Specifically, for calculation, the orientation of the pedicel can be defined as the z-axis, and the normal of the spatial plane containing the suture line can be defined as the x-axis (the pedicel is located in the spatial plane containing the suture line). This defines the predicted pose of the small fruit, which is only used to describe the orientation of the small fruit and not its inherent attribute. Therefore, the key to this confidence level lies in the confidence level of the pedicel orientation and the confidence level of the plane containing the suture line. This confidence level is also affected by the completeness of the surface information of the pedicel and suture line. The more complete the surface information, the more accurate the predicted pose, and the higher the confidence level. At the same time, for the suture line, due to the regularity of its curvature change, the importance of some nodes also affects the confidence level. The acquisition of surface information of some key nodes can have a significant impact on the change of this confidence level, as will be reflected in subsequent embodiments. Essentially, the fruit body of the Liyaco small fruit is a rotationally symmetric body, but the suture line breaks this symmetry. It is a rigid curve with a specific spatial curvature, which can serve as the anchor or skeleton of the entire fruit. Therefore, measuring the confidence of the predicted pose can further improve the accuracy of structural properties on the basis of measuring the integrity of surface information, so as to ensure that the obtained surface is not limited to the two-dimensional level.

[0038] It should be noted that, in addition to the three parameters mentioned above, more benefit parameters can be set. For example, the fourth benefit parameter is used to measure the expected number of small fruits observed from the candidate observation pose. In principle, the observation pose corresponding to the acquisition of more small fruit images at the highest resolution viewpoint is more worthy of priority selection. The fifth benefit parameter is used to measure the confidence of the known information. Specifically, this confidence refers to the error caused by the difference between images from different rounds in the known information. Measuring the confidence helps determine which information to use as a reference, etc., and this embodiment of the invention does not limit this. Furthermore, the above benefit parameters are used to guide the selection of candidate observation poses, while the model's loss function is used to constrain the training direction of the model. Therefore, the measurement logic of the benefit parameters can be applied to the corresponding loss function to constrain the training of the model, prompting the model to actively seek the optimal observation pose during actual operation. This embodiment of the invention also does not limit this.

[0039] When the first and second benefit parameters are guided by the completeness of surface information to find the viewpoint (candidate observation pose) with the highest benefit in the unknown area, the third benefit parameter is constrained by the confidence of the predicted pose. Considering that the purpose of the predicted pose is that during the switching of observation poses, the same small fruit may sway due to wind or the equipment contacting the branch, relying solely on surface information, i.e., image information and relative position information, may cause deviations in the surface information of the small fruit. The predicted pose, however, is the unique structural information of each small fruit, and using it to assist in the acquisition of surface information can further improve the acquisition accuracy. At the same time, using the first representation as a skeleton makes the embodiments of the present invention focus more on spatial structure rather than texture speculation. Since the benefit update process relying solely on surface information may result in forced distortion of features to align with the surface curvature gradient of the small fruit due to missing field of view or occlusion, a large amount of noise will appear in the process of integrating surface information in the image set to be inspected. Therefore, the acquisition of this spatial structure information is more for improving the surface information accuracy used for defect detection.

[0040] Furthermore, based on the actual structure of small fruits, the depressions at the stem and suture line make the surface of that part difficult to detect. The third benefit parameter can also enable the embodiments of the present invention to achieve more refined and comprehensive sampling at that location, and at the same time improve the accuracy of subsequent surface detection results.

[0041] Optionally, for areas close to the suture line, the gain parameters can be further set so that the observation pose orientation (e.g., the camera optical axis) is at an angle of 30 to 60 degrees to the normal of the suture line plane, resulting in higher gains. Alternatively, the observation pose orientation can be at an angle of 45 to 60 degrees to the fruit stalk axis, as the fruit stalk indentation is more clearly observed at this angle, and setting this range of angles yields higher gains. This embodiment of the invention does not limit this.

[0042] In this embodiment of the invention, the preferred image to be inspected is used for the next round of revenue update operation. A comprehensive revenue parameter is used as a metric, which is obtained by weighted fusion of three revenue parameters. The specific weights can be adjusted according to actual task needs. For example, initially, the weight of the first revenue parameter used in the above embodiment is greater than the weight of the second revenue parameter to ensure that all fruits on the tree can be initially acquired. Then, the weight of the first revenue parameter is reduced. This is also reflected in subsequent embodiments. Furthermore, to improve task efficiency, when acquiring surface information of small fruits, acquiring structural information is prioritized, and a higher weight is assigned to the third revenue parameter. This embodiment describes the case where the observation pose is determined by the highest comprehensive revenue parameter. Optionally, several observation poses with the highest comprehensive revenue parameters and significant differences in angle and distance between candidate observation poses can simultaneously acquire preferred images to supplement 3D information for the next round of revenue update. This embodiment does not limit this approach.

[0043] In summary, the embodiments of the present invention have the following functions and effects: I. Efficient Static Observation Applicable to Small Fruits. In this embodiment of the invention, the process of acquiring various observation poses and obtaining images based on the observation poses is discussed from a static perspective. Static here can include ultra-low speed (speed that does not affect high-resolution observation). This is because small fruits are relatively small, and the traditional fruit industry mainly uses dynamic observation methods, which saves time and facilitates multiple rounds of observation. However, this embodiment of the invention focuses on ensuring that the high-resolution effect is not lost under a static perspective, and that information from a small number of perspectives is integrated without overfitting due to redundant visual information.

[0044] Second, surface inspection during the on-tree stage can reduce losses at the source. Small fruits can suffer significant damage on the tree, and manual harvesting can easily overlook some of this damage, leading to contamination of healthy fruit by diseased fruit and causing even greater losses. Damage during the on-tree stage includes overcrowding of fruit, friction between fruit and branches caused by wind, damage from extreme weather, pests, fungal and bacterial infections, human-caused abrasions, or irritation to the fruit surface due to improper spraying.

[0045] Third, the unique morphological characteristics of this type of small fruit are fully utilized. This is used as structural information to improve the accuracy of surface information acquisition, thereby enhancing the accuracy of surface defect detection. Because small fruits (drupes) of the Rosaceae family (Prunus subfamily) have unique pedicels and a suture extending from the pedicel, this combination constitutes a structural feature of the small fruit. A third benefit parameter designed based on this further improves the accuracy of surface information acquisition from a structural perspective. Furthermore, the suture and pedicel are difficult-to-measure points on the surface of small fruits; the third benefit parameter makes the selection of observation poses more inclined towards observation of these difficult-to-measure points, thus improving the accuracy of acquiring difficult-to-measure information.

[0046] Fourth, a hierarchical and efficient image acquisition method based on information gain is adopted. The comprehensive benefit parameter and the three weight parameters are used to determine the acquisition path for the next round of observation poses based on the benefit of the candidate observation poses. This maximizes the conservation of observation resources and avoids redundant information from repeated images. Secondly, the three benefit parameters form a hierarchical structure of "fruit tree (all fruits) - fruit structure (third information parameter) - fruit surface," which can meet the task requirements at different stages and realize the process from acquiring general information to refining detailed information.

[0047] Fifth, multiple small fruits can be acquired simultaneously as much as possible within the resolution support, so as to obtain comprehensive benefit parameters through comprehensive judgment, select the optimal observation pose of multiple fruits to obtain the optimal image to be inspected, and achieve efficient detection in the tree stage.

[0048] As can be seen, the embodiments of the present invention can acquire an initial image to be inspected, and then use multiple rounds of revenue updates to constrain the selection of candidate observation poses based on the initial image. Specifically, the first and second revenue parameters, through a hierarchical perspective of "fruit tree – fruit structure – fruit," encourage candidate observation poses to prioritize acquiring visually uncovered areas as supplementary information. For small fruits, the stem and suture line have indentations, which are difficult-to-detect areas. Furthermore, the stem and suture line are unique structures of small fruits, containing structured information, which can further guide the construction of surface information in the revenue update operation, thereby improving the accuracy of candidate observation poses. Finally, by using a first threshold to determine the update conditions, the updated image set that meets the requirements is output for surface defect detection. This improves the acquisition accuracy and observation efficiency at the image level, thereby saving observation computing resources while improving the accuracy of surface defect detection results.

[0049] In an optional embodiment, the method for measuring the completeness of the surface information using the first and second revenue parameters is further refined. In each round of revenue update operation, for each candidate observation pose, the first, second, and third revenue parameters obtained based on all images to be inspected before the current round may include: Construct a first prediction surface, which is obtained by fitting the unknown region surface of the fruit tree corresponding to all images to be inspected before the current round; Construct candidate imaging planes, which are used to represent simulated imaging planes of candidate observation poses; The first predicted surface is projected onto the selected imaging plane, and the second revenue parameter is obtained based on the projected area corresponding to the first predicted surface.

[0050] In this optional embodiment, the surface fitting of the unknown region can follow the construction method of the 3D point cloud described in the above embodiments, and the fitting operation is performed according to the algorithm. Optionally, the first predicted surface is based on the least squares method to find a set of ellipsoidal parameters that minimizes the sum of the squares of the algebraic or geometric distances from all points to the ellipsoidal surface. The surface of the unknown region corresponding to this ellipsoid is then used as the predicted surface. Alternatively, the fitting problem can be regarded as a probabilistic inference process, using prior knowledge (such as the shape of a small fruit) and the data itself to infer the most likely ellipsoidal parameters.

[0051] Specifically, the aforementioned 3D point cloud is first preprocessed (denoising, downsampling, etc.). Using the preprocessed point cloud, a standard ellipsoid is fitted using the least squares method, yielding the ellipsoid's center point, the radii of its three axes, and its spatial orientation. For each 3D point, the signed distance from that point to the surface of the standard ellipsoid is calculated, with positive values ​​for points outside the ellipsoid, negative values ​​for points inside, and zero values ​​for points on the surface. This distance is defined as the residual. A radial basis function network is trained, taking the coordinates of the 3D points as input and outputting the predicted residuals. The final predicted surface is defined by an implicit equation: signed distance + predicted residual = 0. The solution to this equation is the first predicted surface.

[0052] In this optional embodiment, the candidate imaging plane is the spatial plane where the simulated imaging result under the viewpoint is located, determined according to the imaging depth. The candidate imaging plane and the candidate observation pose can correspond one-to-one. Accordingly, the imaging window and imaging depth of the candidate observation pose should be kept fixed to ensure the fairness of the comparison of candidate observation poses.

[0053] It should be noted that the first benefit parameter refers to the completeness of the surface information of the fruit tree. Since this optional embodiment focuses on the acquisition effect of surface information of small fruits, different construction methods can be used to calculate this completeness for the fruit tree. Optionally, a voxel prediction method can be used, placing the known point cloud (leaves, branches) into a voxel mesh to determine the effective space corresponding to the canopy of a fruit tree. This effective space is continuously updated as known information is input, approximating the real structure (convex hull) of the canopy. Using known voxels as occluders, virtual rays are emitted from the candidate observation pose, and the volume of the unknown voxels that the ray passes through under the occlusion is calculated. The ratio of this volume to the effective space volume is the first benefit parameter. Here, a voxel, or volumetric pixel, is the smallest unit in 3D space. The shape and size of the voxel can be adjusted according to the actual task requirements; this optional embodiment does not limit this.

[0054] In this optional embodiment, the second benefit parameter is specifically described as follows: the second benefit parameter can be used to measure the change in the integrity of the surface information of the small fruit corresponding to the candidate observation pose, that is, the proportion of information in the unknown area that can be obtained, which directly reflects the information benefit brought by the candidate observation pose. Furthermore, the second benefit parameter is obtained based on the projected area corresponding to the first predicted surface. This projected area can be directly used for benefit comparison, or it can be the ratio of the projected area to the total predicted area. Since the denominator is consistent, the comparison of benefit parameters in selecting the observation pose also focuses only on the size of the projected area. Optionally, the second benefit parameter can also be any parameter further calculated based on the projected area to represent the benefit of obtaining unknown information, in order to adjust the sensitivity of the comprehensive benefit parameter to the influence of the second benefit parameter. This optional embodiment does not limit this aspect.

[0055] As can be seen, this optional embodiment obtains a first predicted surface through surface fitting, and then projects the first predicted surface onto the imaging plane to obtain the projection area ratio, simulating the expected observation results of the candidate observation pose. Based on the projection area, a second benefit parameter is obtained. The larger the projection area of ​​the predicted surface projection method, the more unknown areas can be observed from the perspective of that observation pose. This method can reduce the complex 3D calculations to 2D pixel area calculations, thereby exponentially reducing the computational load, further improving the efficiency of selecting the next observation pose and saving computational resources.

[0056] In another optional embodiment, in each round of revenue update operation, for each candidate observation pose, the third revenue parameter obtained based on all images to be inspected may include: Based on the first prediction surface, the total predicted length of the first representation of each small fruit is obtained; Based on the projection results of the first predicted surface onto the candidate imaging plane, the ideal observation length of the first characterization of each small fruit and the extrapolated suture line are obtained. The extrapolated suture line is obtained by extrapolating and fitting the suture line of the observable part in the projection results. The first benefit sub-parameter is obtained based on the ratio of the ideal observation length to the predicted total length. The first benefit sub-parameter is used to measure the visibility of the first representation that the candidate observation pose is expected to observe. The suture line fitted to the unknown region in the first prediction surface is three-dimensionally stitched together with the suture lines in all the images to be inspected to obtain the prior suture line. Based on the extrapolated suture line and the prior suture line, a second benefit sub-parameter is obtained, which is used to measure the difference between the extrapolated suture line and the prior suture line. Based on the first and second revenue sub-parameters, the third revenue parameter is obtained.

[0057] In this optional embodiment, the predicted total length of the first characterization can be the total length of the fruit stalk and the total length of the suture line predicted based on known information. The ideal observation length is the length corresponding to the observable suture line portion in the projection result, and the extrapolated suture line is obtained by extrapolation fitting based on the observable portion.

[0058] The specific operation methods of extrapolation fitting can include: For each small fruit: Construct a prior model based on all images to be inspected prior to the current round, and the prior model includes the first prediction surface; Based on the projection result of the prior model onto the selected imaging plane, the ideal suture line is obtained. Based on the prior model, the corresponding tangent of the ideal suture line is obtained, and the curve equation of the ideal suture line on the tangent plane is determined (it can be solved by the tangent direction of the starting and ending points of the ideal suture line and the integration method). In the process of solving the equation, the first constraint is applied to obtain the extrapolated suture line. The first constraint is that the derivative of the curvature amplitude with respect to the arc length is greater than zero.

[0059] The first constraint is applied because the actual shape of the small fruit is not spherical, but rather oblate or ellipsoidal. At the top (stalk), the surface is almost flat or has a large radius of curvature, while the bottom surface contracts sharply, creating a compact curve. Therefore, the rate of change of curvature is monotonic. This perfectly matches the growth pattern of the small fruit.

[0060] Specifically, the difference between the extrapolated suture line and the prior suture line can be measured using geometric similarity or by comparing the angle between the tangent vectors at the ends of the two lines. The purpose of this second benefit sub-parameter is to measure the observable structured benefit at this observation point. If the extrapolated suture line differs too little from the prior suture line, it means the confidence level is high, indicating that the region can provide little new structural information and there is no need to continue observation from this pose. On the other hand, if the difference is too large, it means that the direction of the suture line has unexpectedly and sharply deviated from this candidate viewpoint, that is, it has just captured the 3D spatial distortion at the occlusion boundary (at the limit of the side contour of the small fruit in this viewpoint, the Jacobian matrix of the perspective projection degenerates the instant the suture line disappears from the field of vision after turning away). In this case, the observation pose should be selected first.

[0061] In this optional embodiment, the purpose of the first benefit sub-parameter is to measure the amount of information expected to be observed from the first representation at the candidate observation pose, and the purpose of the second benefit sub-parameter is to measure the amount of new structural information expected to be provided by the observable content. The more information that can be obtained corresponding to the two benefit sub-parameters, the lower the confidence level, and the more benefit parameters need to be allocated to select the observation pose corresponding to this viewpoint. It should be noted that the benefit sub-parameters are only to emphasize that these two parameters have the same role as the third benefit parameter. Sub-parameters that are not decomposed from the third benefit parameter do not have any order or subordinate relationship to constrain the corresponding calculation process in actual calculation. They can be regarded as the comprehensive benefit parameter obtained by weighted fusion of the first benefit parameter, the second benefit parameter, the first benefit sub-parameter, and the second benefit sub-parameter in this optional embodiment.

[0062] As can be seen, this optional embodiment refines the confidence measurement criteria of the third benefit parameter through the first benefit sub-parameter and the second benefit sub-parameter. Furthermore, it refines the calculation process of the third benefit parameter from two perspectives: the visibility of the first representation, i.e., the expected total amount of information that can be obtained, and the difference between the extrapolated suture line and the prior suture line, i.e., the expected benefit of obtaining new structural information. This enriches the selection criteria for candidate observation poses, making the observation poses focus more on key nodes that have a significant impact on structural information. It can further avoid visual information redundancy, improve the efficiency and accuracy of surface information acquisition, and thus improve the accuracy of surface defect detection results while saving observation computing resources.

[0063] In another optional embodiment, in each round of revenue update operation, the method for obtaining the third revenue parameter for each candidate observation pose may include: An occlusion model is constructed for the fruit tree, which includes a fitted surface for the leaves and a fitted surface for the branches. Based on the occlusion model, the first prediction surface is occluded to obtain an occlusion projection surface. The occlusion projection surface is used to represent the projection result of the first prediction surface onto the candidate imaging plane when it is occluded. Based on all the images to be inspected and the first predicted surface, the corresponding position of the center point of each small fruit in the occluded projection surface is determined as the reference point. Based on the first continuous line and the reference point, the third benefit sub-parameter is obtained. The first continuous line is the longest continuous line segment corresponding to the suture line in the occluded projection plane. The third benefit sub-parameter is obtained based on the angular span of the start and end points of the first continuous line relative to the reference point. The third benefit sub-parameter is used to measure the continuity of the line segment corresponding to the suture line in the occluded projection plane. The third revenue parameter is obtained based on the first revenue sub-parameter, the second revenue sub-parameter, and the third revenue sub-parameter.

[0064] In this optional embodiment, the occlusion model can be constructed by surface fitting or by voxel construction as described in the above embodiments. The occlusion model mainly simulates branches, leaves and other parts of the tree canopy that are not small fruits, so as to obtain the first prediction surface projected onto the imaging plane under occlusion.

[0065] For small fruits, images acquired at the tree stage are often obstructed by leaves and branches. These small fruits are generally characterized by dense canopies and foliage. Even if farmers prune the branches and leaves, obstruction is still common. This alternative embodiment focuses on how to reasonably and efficiently introduce viewpoint occlusion in the revenue update.

[0066] The reason for not using the traditional surface occlusion rate is that the surface position deviates when the wind blows or branches and leaves sway, and small fruits are mostly isotropic, so the difference in the projected surface obtained under different observation poses may be very small. Furthermore, the surface occlusion rate is a static regional area statistic. Considering the occlusion situation is not helpful when the second benefit parameter is set based on the unknown region as a guide for selecting the benefit of the observation pose. The continuity of the suture line is a dynamic topological index. In the above embodiments, if the continuity is too low in the process of surface fitting and curve extrapolation, the simulation results will be full of multiple solutions. At the same time, the continuity provides a strong directional signal. For example, if it is known that the surface information only covers half of the small fruit and the continuous suture line ends at a certain breakpoint, the benefit sub-parameter will reach a maximum value on the other side of the breakpoint to guide the selection of the observation pose to supplement the directional information for that breakpoint. The surface occlusion rate is difficult to achieve this strong directional guidance and will only continue to advance with the goal of supplementing the surface information.

[0067] Furthermore, instead of using the three-dimensional curves obtainable in the above embodiments to compare continuity, the angular span measurement method is adopted. The key lies in the measurement accuracy and measurement standard. The expected observable suture line described in the above embodiments, as well as the overall suture line corresponding to the prior model, have errors in the actual curvature change and length of the suture line due to the use of surface fitting method. The angular span, on the other hand, precisely addresses the visual continuity. Regardless of the actual length or curvature change in space, the geometric angle corresponding to the first continuous line on the imaging plane remains constant. This makes the continuity and measurement standard absolutely objective.

[0068] In this optional embodiment, the reference point can be the projection point of the center of the small fruit onto the imaging plane (occluding projection plane). Since the overall shape of the small fruit is close to an ellipsoid, the geometric center can be directly used as the center point. The start and end points of the first continuous line form a complete arc on this projection plane. The angular span is the rotation angle (polar angle) from the start point to the end point of the arc with the center point as the center. The method for obtaining the third benefit sub-parameter based on this angular span can be the ratio of this angular span to the angle range corresponding to the ideal suture line. This ratio can be exponentialized based on the number of breakpoints to amplify the influence of discontinuous suture lines observed from the candidate observation pose. It should be noted that the third benefit sub-parameter plays the same role as the first and second benefit sub-parameters in the process of weighted fusion to obtain the comprehensive benefit parameter. Optionally, low continuity can also be used as a penalty to reduce the comprehensive benefit; this optional embodiment does not limit this.

[0069] As can be seen, this optional embodiment, through the third benefit sub-parameter, makes the selection process of the candidate observation pose based on the continuity of the expected suture line, fully considers the situation of visual occlusion, further refines the composition structure of the comprehensive benefit parameter, and further improves the selection criteria of the candidate observation pose, making the process more efficient and accurate.

[0070] In another optional embodiment, the composition of the second revenue parameter is refined. In each round of revenue update operation, for each candidate observation pose, the method of obtaining the second revenue parameter may include: Based on all the images to be inspected and the first predicted surface, construct the UV unfolded map corresponding to each small fruit; Based on the projection result of the first predicted surface onto the selected imaging plane, the coverage area of ​​the projection result in the UV unfolding diagram is obtained; The fourth revenue sub-parameter is obtained based on the ratio of the coverage area to the total area of ​​the UV unfolded map; The region corresponding to the first representation in the UV unfolded map is determined as the representation region; the region in the UV unfolded map whose distance from the representation region is less than a preset second threshold is determined as the first optimal detection region; the region in the UV unfolded map whose distance from the representation region is greater than or equal to the second threshold and less than a preset third threshold is determined as the second optimal detection region; based on the first overlapping area ratio and the second overlapping area ratio, the fifth benefit sub-parameter is obtained, where the first overlapping area ratio is the ratio of the overlapping area of ​​the representation region in the first optimal detection region to the total area, and the second overlapping area ratio is the ratio of the overlapping area of ​​the representation region in the second optimal detection region to the total area; The second revenue parameter is obtained by weighted fusion of the fourth and fifth revenue sub-parameters.

[0071] In this optional embodiment, the UV unfolding map is a three-dimensional model formed by combining the three-dimensional information corresponding to the image to be inspected with the fitting of the unknown surface and mapping it onto a two-dimensional plane. This mapping process records each vertex of the three-dimensional model surface and maps it to the specific pixel position on the two-dimensional image. This is a common method in 3D computer graphics, and this optional embodiment will not be described in detail.

[0072] In this optional embodiment, the coverage area is the surface information that can be obtained under the observation pose, and the fourth benefit sub-parameter focuses on the acquisition rate of unknown information under this viewpoint.

[0073] In this optional embodiment, based on the above embodiment, the measurement of detection benefit for the difficult-to-measure area around the first characterization is refined. The characterization area is the area corresponding to the first characterization in the UV unfolding diagram. It can be the line corresponding to the suture line and the short curved surface corresponding to the fruit stalk, or it can be the curved surface adjacent to the suture line. The second threshold and the third threshold are preset values, which are usually as small as possible, and are limited by the distance before the depression extends to the flat curved surface, so as to control the measurement range within the area where the depression is located.

[0074] The purpose of setting the second and third thresholds is to form a gradient, that is, the closer to the periphery (representation region) of the first representation, the higher the detection benefit at that location. The first and second overlapping area ratios are actually used to measure the amount of information on the surface of the hard-to-detect region that can be obtained from this perspective. The first overlapping area ratio is more important than the second overlapping area ratio because it is closer to the representation region. This importance is reflected in its influence on the fifth benefit sub-parameter. The process of obtaining the fifth benefit sub-parameter based on the two area ratios can be done by weighted fusion.

[0075] Optionally, the fifth benefit sub-parameter can be calculated directly on the UV unfolded map based on the distance to the location of the characterization region. This optional embodiment is not limited to this.

[0076] The second benefit parameter is obtained by weighted fusion of the fourth and fifth benefit sub-parameters. The fourth benefit sub-parameter measures the amount of information about the surface of unknown areas that can be obtained from this perspective, while the fifth benefit sub-parameter measures the amount of information about the surface of unknown and difficult-to-detect areas that can be obtained from this perspective. The fourth and fifth benefit sub-parameters are the same as the three benefit sub-parameters in the above embodiments; the second benefit parameter only limits the function of the benefit sub-parameters and will not be elaborated here. Specifically, the weights of the fourth and fifth benefit sub-parameters can be adjusted according to the needs of the actual task. For tasks requiring the acquisition of complete surface information of small fruits, the weight corresponding to the fourth benefit sub-parameter should be set as high as possible. For tasks requiring precise details of small fruits, the weight corresponding to the fifth benefit sub-parameter should be set as high as possible. This optional embodiment does not limit this.

[0077] In yet another optional embodiment, each round of revenue update operations may further include: When it is determined that all first-return parameters are less than the preset fourth threshold, the weight of the first-return parameters in the weighted fusion process in the next round is reduced. When it is determined that all third-revenue parameters are less than the preset fifth threshold, the weight of the third-revenue parameters in the weighted fusion process in the next round is reduced.

[0078] In this optional embodiment, the purpose of setting the fourth threshold is that when all the first benefit parameters are too small, it means that there is too little unknown information for the fruit tree, making it difficult to meet the detection benefit requirements of the first benefit parameters. At this time, the overall information of the fruit tree tends to be complete, and ideally, some information of all small fruits has been obtained. The next step is to increase the weight of the second and third benefit parameters, changing the task objective from finding all small fruits to obtaining and refining the surface information of small fruits.

[0079] In this optional embodiment, the purpose of setting the fifth threshold is that when all the third benefit parameters are too small, it means that the confidence of the predicted pose of the small fruit is high enough, that is, enough information related to the first representation of the small fruit has been obtained, so that the candidate observation pose no longer focuses on the morphological representation, so as to avoid overfitting caused by redundant information and thus affecting the observation accuracy.

[0080] In another optional embodiment, in each round of revenue update operation, when it is determined that the comprehensive revenue parameter is less than a first threshold, all images to be inspected are identified as the image set to be inspected, including: When the comprehensive benefit parameter is determined to be less than the first threshold, the candidate observation pose whose orientation angle meets the preset condition is determined to be the candidate observation pose, and the orientation angle is the angle between the orientation of the candidate observation pose and the orientation corresponding to the fruit stalk. Based on multiple randomly selected candidate observation poses, multiple candidate images to be inspected are obtained; based on all images to be inspected before the current round, the preferred images to be inspected, and all candidate images to be inspected, a set of images to be inspected is obtained.

[0081] In this optional embodiment, the orientation angle is the angle between the orientation of the selected observation pose and the orientation corresponding to the fruit stalk. The preset condition has been stated in the above embodiment, that is, the angle between the camera optical axis and the fruit stalk can be in the range of 45 degrees to 60 degrees. Under this angle, the surface details at the fruit stalk can be observed better.

[0082] The purpose of randomly selecting multiple images to be inspected from candidate observation poses is twofold: first, to further supplement the surface information of difficult-to-inspect areas with a better specified observation angle; and second, to use this randomness to test the accuracy of the above detection process, so as to adjust the weights and thresholds, so as to continue the observation pose selection method of comprehensive benefit parameters and continue to update the surface information of small fruits.

[0083] Example 2 Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a surface defect detection device for small fruits of the Rosaceae subfamily (Prunoideae) disclosed in an embodiment of the present invention. Figure 3 As shown, the device may include: Image acquisition module 201 is used to acquire an initial image to be inspected. The initial image to be inspected includes a fruit tree, at least two small fruits on the fruit tree, and depth information corresponding to each small fruit. The small fruits are of the Rosaceae family and the Prunoideae subfamily. The small fruits include suture lines and fruit stalks. Revenue update module 202 is used to perform multiple rounds of revenue update operations based on the initial image to be inspected. Each round of revenue update operations includes: Multiple candidate observation poses are obtained. For each candidate observation pose: based on all images to be inspected before the current round, a first benefit parameter, a second benefit parameter, and a third benefit parameter are obtained. All benefit parameters are weighted and fused to obtain a comprehensive benefit parameter. The first benefit parameter is used to measure the completeness of surface information corresponding to the fruit tree, the second benefit parameter is used to measure the completeness of surface information corresponding to all small fruits, and the third benefit parameter is used to measure the confidence of the predicted pose. The predicted pose is obtained based on the predicted orientation of the first representation of each small fruit. The first representation includes the suture line and the fruit stalk. When it is determined that the comprehensive benefit parameter is greater than or equal to the preset first threshold, the preferred inspection image is obtained based on the candidate observation pose with the largest comprehensive benefit parameter, and the preferred inspection image is used for the next round of benefit update. When the comprehensive benefit parameter is determined to be less than the first threshold, all images to be inspected are identified as the image set to be inspected. Based on the image set to be inspected, the surface defect detection results corresponding to each small fruit are obtained.

[0084] Example 3 Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a device including a memory and a processor, as disclosed in an embodiment of the present invention. Figure 4 As shown, the device may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute some or all of the steps in any of the surface defect detection methods for small fruits of the Rosaceae subfamily in Embodiment 1 of the present invention.

[0085] Example 4 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in any of the surface defect detection methods for small fruits of the Rosaceae subfamily disclosed in Embodiment 1 of this invention.

[0086] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0087] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0088] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting surface defects in small fruits of the Rosaceae (Prunus) subfamily, characterized in that, The method includes: Acquire an initial image to be inspected, the initial image to be inspected including a fruit tree and at least two small fruits on the fruit tree and the depth information corresponding to each small fruit, the small fruits being of the Rosaceae family, Prunoideae subfamily, the small fruits including suture lines and fruit stalks; Based on the initial image to be inspected, multiple rounds of revenue update operations are performed, wherein each round of revenue update operation includes: Multiple candidate observation poses are obtained. For each candidate observation pose: based on all images to be inspected before the current round, a first benefit parameter, a second benefit parameter, and a third benefit parameter are obtained. All benefit parameters are weighted and fused to obtain a comprehensive benefit parameter. The first benefit parameter is used to measure the completeness of surface information corresponding to the fruit tree, the second benefit parameter is used to measure the completeness of surface information corresponding to all small fruits, and the third benefit parameter is used to measure the confidence of the predicted pose. The predicted pose is obtained based on the predicted orientation of the first representation of each small fruit, which includes the suture line and the fruit stalk. When it is determined that the comprehensive benefit parameter is greater than or equal to a preset first threshold, a preferred image to be inspected is obtained based on the candidate observation pose with the largest comprehensive benefit parameter. The preferred image to be inspected is used for the next round of benefit update. When it is determined that the comprehensive benefit parameter is less than the first threshold, all images to be inspected are identified as the image set to be inspected, and the surface defect detection result corresponding to each small fruit is obtained based on the image set to be inspected.

2. The method for detecting surface defects in small fruits of the Rosaceae subfamily Prunoideae according to claim 1, characterized in that, In each round of revenue update operations, for each candidate observation pose, the second revenue parameter is obtained in the following ways: Construct a first prediction surface, which is obtained by fitting the unknown region surface of the fruit tree corresponding to all images to be inspected before the current round; Construct a candidate imaging plane, which is used to represent the simulated imaging plane of the candidate observation pose; The first predicted surface is projected onto the candidate imaging plane, and the second revenue parameter is obtained based on the projected area corresponding to the first predicted surface.

3. The method for detecting surface defects in small fruits of the Rosaceae subfamily Prunoideae according to claim 2, characterized in that, In each round of revenue update operation, for each candidate observation pose, the third revenue parameter is obtained based on all images to be inspected, including: Based on the first prediction surface, the total predicted length of the first representation of each small fruit is obtained; Based on the projection result of the first predicted surface onto the candidate imaging plane, the ideal observation length of the first characterization of each small fruit and the extrapolated suture line are obtained. The extrapolated suture line is obtained by extrapolation and fitting based on the suture line extrapolated from the observable part of the projection result. The first benefit sub-parameter is obtained based on the ratio of the ideal observation length to the predicted total length. The first benefit sub-parameter is used to measure the visibility of the first representation that the candidate observation pose is expected to observe. The suture line fitted to the unknown region in the first prediction surface is three-dimensionally stitched together with the suture lines in all the images to be inspected to obtain the prior suture line. A second benefit sub-parameter is obtained based on the extrapolated suture line and the prior suture line. The second benefit sub-parameter is used to measure the difference between the extrapolated suture line and the prior suture line. The third revenue parameter is obtained based on the first revenue sub-parameter and the second revenue sub-parameter.

4. The method for detecting surface defects in small fruits of the Rosaceae subfamily Prunoideae according to claim 3, characterized in that, In each round of revenue update operation, for each candidate observation pose, the method for obtaining the third revenue parameter includes: An occlusion model is constructed for the fruit tree, the occlusion model including the fitted surface for the leaves and the fitted surface for the branches; based on the occlusion model, the first prediction surface is occluded to obtain an occlusion projection surface, the occlusion projection surface is used to represent the projection result of the first prediction surface onto the candidate imaging plane when it is occluded; Based on all the images to be inspected and the first predicted surface, the corresponding position of the fruit center point of each small fruit in the occlusion projection surface is determined as a reference point; Based on the first continuous line and the reference point, a third benefit sub-parameter is obtained. The first continuous line is the longest continuous line segment corresponding to the suture line in the occlusion projection surface. The third benefit sub-parameter is obtained based on the angular span of the start and end points of the first continuous line relative to the reference point. The third benefit sub-parameter is used to measure the continuity of the line segment corresponding to the suture line in the occlusion projection surface. The third revenue parameter is obtained based on the first revenue sub-parameter, the second revenue sub-parameter, and the third revenue sub-parameter.

5. The method for detecting surface defects in small fruits of the Rosaceae subfamily Prunoideae according to claim 4, characterized in that, In each round of revenue update operations, for each candidate observation pose, the second revenue parameter is obtained in the following ways: Based on all the images to be inspected and the first predicted surface, construct the UV unfolded map corresponding to each small fruit; Based on the projection result of the first predicted surface onto the candidate imaging plane, the coverage area of ​​the projection result in the UV unfolding diagram is obtained; The fourth revenue sub-parameter is obtained based on the ratio of the coverage area to the total area of ​​the UV unfolded map; The region corresponding to the first representation in the UV unfolded map is determined as the representation region; the region in the UV unfolded map whose distance from the representation region is less than a preset second threshold is determined as the first optimal detection region; the region in the UV unfolded map whose distance from the representation region is greater than or equal to the second threshold and less than a preset third threshold is determined as the second optimal detection region; a fifth benefit sub-parameter is obtained based on the first overlapping area ratio and the second overlapping area ratio, wherein the first overlapping area ratio is the ratio of the overlapping area of ​​the representation region in the first optimal detection region to the total area, and the second overlapping area ratio is the ratio of the overlapping area of ​​the representation region in the second optimal detection region to the total area; The fourth and fifth revenue sub-parameters are weighted and fused to obtain the second revenue parameter.

6. The method for detecting surface defects in small fruits of the Rosaceae subfamily Prunoideae according to claim 5, characterized in that, Each round of profit update operations also includes: When it is determined that all of the first revenue parameters are less than the preset fourth threshold, the weight of the first revenue parameter in the weighted fusion process in the next round is reduced. When it is determined that all of the third revenue parameters are less than the preset fifth threshold, the weight of the third revenue parameters in the weighted fusion process in the next round is reduced.

7. The method for detecting surface defects in small fruits of the Rosaceae subfamily Prunoideae according to claim 6, characterized in that, In each round of revenue update operation, when it is determined that the comprehensive revenue parameter is less than the first threshold, determining all images to be inspected as the image set to be inspected includes: When it is determined that the comprehensive benefit parameter is less than the first threshold, the candidate observation pose whose orientation angle meets the preset condition is determined as the candidate observation pose, and the orientation angle is the angle between the orientation of the candidate observation pose and the orientation corresponding to the fruit stalk. Based on multiple randomly selected candidate observation poses, multiple candidate images to be inspected are obtained; based on all images to be inspected before the current round, the preferred images to be inspected, and all candidate images to be inspected, a set of images to be inspected is obtained.

8. A device for detecting surface defects in small fruits of the Rosaceae (Prunus) subfamily, characterized in that, The device includes: The image acquisition module is used to acquire an initial image to be inspected, which includes a fruit tree, at least two small fruits on the fruit tree, and depth information corresponding to each small fruit. The small fruits are from the Rosaceae family, Prunoideae subfamily, and include suture lines and fruit stalks. The revenue update module is used to perform multiple rounds of revenue update operations based on the initial image to be inspected, wherein each round of revenue update operation includes: Multiple candidate observation poses are obtained. For each candidate observation pose: based on all images to be inspected before the current round, a first benefit parameter, a second benefit parameter, and a third benefit parameter are obtained. All benefit parameters are weighted and fused to obtain a comprehensive benefit parameter. The first benefit parameter is used to measure the completeness of surface information corresponding to the fruit tree, the second benefit parameter is used to measure the completeness of surface information corresponding to all small fruits, and the third benefit parameter is used to measure the confidence of the predicted pose. The predicted pose is obtained based on the predicted orientation of the first representation of each small fruit, which includes the suture line and the fruit stalk. When it is determined that the comprehensive benefit parameter is greater than or equal to a preset first threshold, a preferred image to be inspected is obtained based on the candidate observation pose with the largest comprehensive benefit parameter. The preferred image to be inspected is used for the next round of benefit update. When it is determined that the comprehensive benefit parameter is less than the first threshold, all images to be inspected are identified as the image set to be inspected, and the surface defect detection result corresponding to each small fruit is obtained based on the image set to be inspected.

9. An apparatus comprising a memory and a processor, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the surface defect detection method for small fruits of the Rosaceae subfamily as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked by a processor, execute the surface defect detection method for small fruits of the Rosaceae subfamily as described in any one of claims 1-7.