Agricultural product fine-grained defect detection method and system based on normal sample RGB-FPFH anomaly positioning and branch normalization multi-modal classification

Abstract

The application discloses a kind of agricultural product fine-grained defect detection method and system based on normal sample RGB-FPFH abnormal positioning and branch normalization multi-modal classification.The method obtains registered RGB image and ordered XYZ point cloud;Only with the same normal sample, local RGB and FPFH features are extracted, aligned and spliced in a unified grid, and a normal sample memory bank is established;Calculate the nearest neighbor distance of the test feature to the memory bank to generate an abnormal score map, and calibrate it with a normal verification pixel threshold, select a RGB product support gate and an over-threshold abnormal quality connected domain, determine the candidate area and map it to the original resolution;Extract the RGB appearance and XYZ geometric statistical features of the candidate area, and in the RGB-only, XYZ-only, direct splicing and branch normalization fusion model, the optimal configuration is selected through nested cross-validation, and the defect category and area coordinates are output.The application reduces the risk of label leakage and modal scale imbalance.

Description

Technical Field

[0001] This invention relates to the fields of machine vision, 3D point cloud processing, and agricultural product quality inspection technology, specifically to a method and system for detecting fine-grained defects in agricultural products based on RGB-FPFH anomaly localization and branch normalization multimodal classification of normal samples. Background Technology

[0002] Agricultural product sorting typically requires answering two core questions simultaneously: "Does an anomaly exist?" and "What type of defect is the anomaly?" Traditional two-dimensional visual inspection methods mainly rely on the texture information of RGB images. However, in actual production environments, when lighting conditions change, the surface color of agricultural products is similar to the color of defects, or the defect area is small, two-dimensional texture features are easily interfered with by the intact surface background, leading to missed or false detections.

[0003] 3D point cloud data can provide morphological and geometric information of agricultural product surfaces, and has good potential for detecting defects with obvious geometric changes such as depressions, bulges, and cracks. However, directly using high-dimensional point-level features for anomaly detection will significantly increase storage, computation, and annotation requirements, and the disordered nature of point cloud data poses challenges to feature alignment and cross-sample comparison.

[0004] In existing 3D anomaly detection technologies, publicly available methods typically only use normal samples to build a memory bank and output continuous anomaly scores to determine "whether an anomaly exists," but they cannot directly provide supervised defect subcategories. Furthermore, the stable conversion of pixel-level or point-level anomaly scores into regions of interest (ROIs) suitable for classification still faces the following technical challenges: how to calibrate the anomaly score threshold to avoid dependence on defect samples; how to accurately separate agricultural product areas from the background in complex environments; how to generate stable, recalculated candidate regions from discrete anomaly responses; and how to accurately map the generated candidate regions back to the original resolution for subsequent feature extraction.

[0005] In multimodal fusion, when simultaneously utilizing high-dimensional RGB appearance features (e.g., 2048-dimensional) and low-dimensional geometric statistical features (e.g., 21-dimensional), direct splicing leads to a severe scale imbalance between modalities. The high-dimensional modality dominates distance calculation and classification decisions, thus drowning out the effective information of the low-dimensional modality. Furthermore, if test fold data is used for hyperparameter tuning during model selection, label leakage will occur, resulting in overly optimistic and unreproducible performance evaluations.

[0006] Therefore, there is a need for a fine-grained defect detection technology for agricultural products that can isolate evidence between unsupervised anomaly localization and supervised defect classification, support original resolution coordinate mapping, have a modal scale balancing mechanism, and adopt a nested model selection strategy. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a method, system, and storage medium for detecting fine-grained defects in agricultural products based on RGB-FPFH anomaly localization and branch normalization multimodal classification of normal samples.

[0008] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0009] A method for detecting fine-grained defects in agricultural products based on RGB-FPFH anomaly localization and branch-normalized multimodal classification of normal samples includes the following steps:

[0010] S1. Acquire RGB images of the same agricultural product to be tested and ordered XYZ point clouds spatially registered with the RGB images;

[0011] S2. Using only normal training samples of the same category as the object to be tested, extract local RGB features and local FPFH features respectively. Align and stitch the local RGB features and local FPFH features on a preset grid according to their positions to form multimodal local features. Establish a normal sample memory bank based on the multimodal local features.

[0012] S3. Calculate the nearest neighbor distance from the multimodal local features of the sample to be tested to the normal sample memory, and generate an anomaly score map;

[0013] S4. Using the anomaly score threshold calibrated only by normal verification pixels, the RGB product support mask, and the connected component over-threshold anomaly quality selection rule, candidate regions are determined from the anomaly score map, and the candidate regions are mapped to the original resolution of the RGB image and the XYZ point cloud.

[0014] S5. Extract RGB appearance features and XYZ geometric statistical features from the mapped candidate regions respectively;

[0015] S6. Select the classification feature family, dimensionality reduction, and classifier hyperparameters from the candidate feature family through inner-layer cross-validation, and output the defect category, candidate region coordinates, and traceable audit records on the outer-layer test layer, wherein the outer-layer test layer does not participate in the selection process.

[0016] Furthermore, both the RGB image and the XYZ point cloud are ordered raster data. The coordinate transformation of the candidate region between the RGB image and the XYZ point cloud is determined by their respective width and height ratios. The coordinates of the upper left corner of the candidate region are rounded down, and the coordinates of the lower right corner are rounded up. The coordinates are then cropped to the valid range of the corresponding data.

[0017] Furthermore, the local RGB features are extracted by a wide ResNet-50 network pre-trained on ImageNet. The feature maps with network output indices 2 and 3 are used, and then subjected to 3×3 average pooling with a stride of 1 and adaptive average pooling to a grid size of 28×28, respectively. The resulting grids are then concatenated according to their positions to obtain a 1536-dimensional RGB feature vector for each grid position.

[0018] Furthermore, the local FPFH feature is a point cloud composed of valid non-zero points, with a coordinate scaling factor of 0.05, a normal estimation radius of 0.10, and an FPFH feature search radius of 0.25. Each valid point yields a 33-dimensional FPFH descriptor, which is then concatenated with the RGB feature on the 28×28 grid according to position to form a 1569-dimensional multimodal local feature.

[0019] Furthermore, the number of local features in the normal sample memory is the number of normal samples multiplied by the number of grids of 28×28. Core set compression is performed using a core set ratio of 0.1 and a sparse random projection parameter epsilon of 0.9. The core set is achieved by successively selecting the feature vector with the largest minimum Euclidean distance to the selected set.

[0020] Furthermore, the generation of the anomaly score map includes:

[0021] For each query local feature Calculate its nearest neighbor distance to the normal sample memory. ;

[0022] make , and order for Nearest neighbor memory features;

[0023] by Of the three nearest neighbors, excluding The reweighting coefficients are calculated for the second and third neighbors outside the user:

[0024] Where D is the local feature dimension, and e2 and e3 are... The Euclidean distance to the two neighbors;

[0025] Image-level anomaly score Pixel-level anomaly score map is composed of The result was obtained by rearranging the grid into a 28×28 grid and then performing bilinear interpolation and Gaussian smoothing.

[0026] Furthermore, the anomaly score threshold is determined by the empirical 99.5% quantile of the anomaly score of the surface support pixels of the normal verification sample, so that the false alarm target for normal pixels is 0.005; the RGB product support mask is obtained by grayscale Otsu threshold segmentation, 3×3 binary opening operation, 9×9 binary closing operation, hole filling and maximum connected component selection.

[0027] Furthermore, the candidate region mask is:

[0028] ;

[0029] Where S(x,y) is the pixel-level anomaly score map, τ is the anomaly score threshold, and V(x,y) is the RGB product support mask;

[0030] Calculate the out-of-threshold quality for each connected component C:

[0031] Select the connected region with the largest E(C), calculate its original resolution bounding rectangle, expand it to the surrounding area by a preset number of pixels, and then crop it to the image boundary to obtain the candidate region.

[0032] Furthermore, the RGB appearance feature is a 2048-dimensional vector obtained by adjusting the RGB region corresponding to the candidate region to a 224×224 input size, normalizing it with ImageNet mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225), and then using global average pooling to obtain the last convolutional block of the frozen ResNet-50 network.

[0033] Furthermore, the XYZ geometric statistical features calculate the mean, standard deviation, 10th percentile, 90th percentile, range, median, and mean absolute deviation for all finite points within the candidate region that are not all zero, respectively, forming a 21-dimensional vector; when the 21-dimensional vector is directly concatenated with the 2048-dimensional RGB appearance features, a 2069-dimensional vector is formed.

[0034] Furthermore, the candidate feature families include RGB-only feature families, XYZ-only feature families, directly concatenated RGB+XYZ feature families, and branch-normalized RGB+XYZ feature families;

[0035] The branch-normalized RGB+XYZ feature family first performs standardization and principal component analysis to reduce the dimensionality of the 2048-dimensional RGB branch and the 21-dimensional XYZ branch respectively, then performs sample-by-sample L2 normalization, and finally splices the two branches together.

[0036] The principal component analysis dimension of the RGB branch is selected from {10, 20, 40}, and the principal component analysis dimension of the XYZ branch is selected from {3, 5, 10}.

[0037] Furthermore, the model selection employs five-fold hierarchical outer-layer cross-validation and three-fold hierarchical inner-layer cross-validation. The inner-layer cross-validation uses the macro-average F1 score as the selection metric. The penalty parameter C of the RBF support vector machine is selected from {0.1, 1, 10, 100}. The RBF kernel parameter adopts the scale rule. All candidate configurations are selected only by sorting within each outer-layer training partition.

[0038] Furthermore, when no valid connected components are formed in the abnormal score graph, the system outputs the invalid ROI status and reason and retains the original abnormal score graph; when it is necessary to continue the controlled classification comparison, the predefined RGB product bounding box is called as an explicit fallback and the fallback is marked in the audit record. The fallback box is not generated by the defect mask.

[0039] Furthermore, the candidate region extraction also includes a secure hybrid selector: generating compact high-scoring seeds from a thresholded and product-supported gated anomaly score map, sorting them according to the density of anomalies exceeding the threshold, the compactness of connected components, and the fill rate, performing region area constraints and recording confidence and rejection states, and setting the original resolution outer boundary to 0 pixels; when the candidate region does not trigger the pathological area or low confidence condition, retaining the selection result of the largest connected component of the anomaly quality exceeding the threshold, the secure hybrid selector does not read the test defect mask or defect category during inference.

[0040] A fine-grained defect detection system for agricultural products includes:

[0041] The synchronous acquisition module is used to acquire RGB images of the same agricultural product to be tested and ordered XYZ point clouds spatially registered with the RGB images;

[0042] The normal sample memory module is used to extract local RGB features and local FPFH features using only normal training samples of the same category as the object to be tested and to build a normal sample memory.

[0043] An anomaly score map generation module is used to calculate the nearest neighbor distance from the local features of the sample to be tested to the memory bank and generate an anomaly score map;

[0044] The candidate region extraction module is used to determine candidate regions using a threshold calibrated only by normal verified pixels, an RGB product support mask, and a connected component over-threshold abnormal quality selection rule, and to map the candidate regions to the RGB image and the XYZ point cloud;

[0045] The multimodal feature extraction module is used to extract RGB appearance features and XYZ geometric statistical features from candidate regions;

[0046] The nested model selection and classification module is used to select the classification feature family, dimensionality reduction, and classifier hyperparameters from the candidate feature families through inner-layer cross-validation, and output the defect category on the outer-layer test fold; and

[0047] The audit output module is used to output the coordinates of the candidate area and traceable audit records;

[0048] Each module is configured to perform the method described in any one of claims 1 to 14.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] First, this invention establishes an RGB+FPFH memory library using only normal samples, strictly separating the unsupervised anomaly localization stage from the supervised defect classification stage, clearly recording the data range that can be used in each stage, and avoiding the leakage of defect sample information in the anomaly localization stage.

[0051] Second, this invention forms a set of candidate region generation rules that are independent of defect sample annotation, recalculated, and auditable by calibrating a normal verification pixel threshold on the original resolution, implementing RGB product support gating, selecting connected components based on abnormal quality exceeding the threshold, and expanding fixed boundaries.

[0052] Third, this invention effectively reduces the scale imbalance problem when directly splicing the high-dimensional RGB appearance branch and the low-dimensional XYZ geometric statistics branch by branch independence standardization, principal component analysis dimensionality reduction and sample-by-sample L2 normalization, so that the information of the two modes can be expressed in a balanced manner after fusion.

[0053] Fourth, this invention uses a nested structure of five-fold outer cross-validation and three-fold inner cross-validation to ensure that the outer test fold data does not participate in the selection process of model family, dimensionality reduction, and classifier hyperparameters, while retaining the inner validation score of each candidate configuration, making the performance evaluation results true and reliable.

[0054] Fifth, this invention improves the traceability and engineering maintainability of the entire processing chain by using explicit status output, rollback flags, input parameter recording, program version recording, and runtime recording, which facilitates recalculation, migration, and production line verification. Attached Figure Description

[0055] Figure 1 The overall flowchart of RGB+3D anomaly localization and fine-grained classification for evidence separation provided in the embodiments of the present invention;

[0056] Figure 2 This is an example of an abnormal score map of a normal sample using RGB+FPFH provided in an embodiment of the present invention;

[0057] Figure 3 This is an example of automatic candidate region extraction based on anomaly score, product support, and connected component quality, provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0059] Example 1: Overall Method Flow

[0060] like Figure 1 As shown, this embodiment provides a fine-grained defect detection method for agricultural products based on RGB-FPFH anomaly localization and branch-normalized multimodal classification of normal samples. This method includes the following core stages:

[0061] Phase 1: Data Acquisition and Registration. Acquire RGB images (I) and ordered XYZ point clouds (P) of the same agricultural product, ensuring that the pixel indices of the two data sources correspond to the same spatial location. For sensor outputs of different sizes, when mapping candidate boxes from RGB coordinates to point cloud coordinates, the right boundary of the mapped box uses the upper bound of an open interval to avoid missing the last column. Before point cloud statistics, remove points containing NaN or infinite values, as well as points with all three coordinates being zero. If there are insufficient valid points within the candidate region, output a geometric feature missing status and enter the traceable anomaly handling branch.

[0062] The coordinate mapping formula is as follows:

[0063] ;

[0064] ;

[0065] Phase Two: Multimodal Local Feature Extraction and Memory Establishment for Normal Samples. A wide ResNet-50 network (wide_resnet50_2) pre-trained on ImageNet is used to extract feature maps at indices 2 and 3 from the RGB images. Each feature map is first subjected to 3×3 average pooling with a stride of 1, and then adjusted to a 28×28 grid using adaptive average pooling. At grid position p, the two feature layers are concatenated along the channel direction:

[0066] For an ordered XYZ point cloud, all zeros are removed. The coordinates are multiplied by a scaling factor of 0.05 before calculating the normal and Fast Point Feature Histogram (FPFH). The normal search radius is 2v = 0.10, and the FPFH search radius is 5v = 0.25, where v = 0.05. Each valid point yields a 33-dimensional FPFH descriptor, which is then restored to an ordered raster and pooled to 28×28. The FPFH calculation formula is:

[0067] ;

[0068] RGB and FPFH features are stitched together at the same grid location:

[0069] ;

[0070] For each category, only normal training samples are used to form a memory bank M. Taking the Carrot category as an example, 286 normal samples each generate 784 local features, resulting in 286 × 784 = 224224 1569-dimensional vectors in the core set before compression; after rounding to the nearest 0.1 (the core set ratio), 22422 vectors are retained. The core set selection first uses sparse random projection (parameter epsilon = 0.9) to reduce dimensionality, and then, starting from the first vector, iteratively selects the candidate vector with the largest minimum Euclidean distance from the selected set.

[0071] ;

[0072] Phase 3: Calculation, reweighting, and generation of local anomaly scores. This involves 784 query local features of the test sample. Calculate its nearest neighbor distance to the normal memory. Let the local location corresponding to the maximum distance be... and order for Nearest neighbor memory features. Using Reweight the three nearest neighbors and exclude Use the 2nd and 3rd nearest neighbors after itself:

[0073] ;

[0074] ;

[0075] ;

[0076] Will The local anomaly score map is rearranged into a 28×28 grid, bilinearly interpolated to the original RGB resolution, and then smoothed using Gaussian blur (radius 4) to obtain the pixel-level anomaly score map S(x,y). The image-level reweighted score and the pixel-level score map are saved separately to avoid mixing them into the same metric.

[0077] Phase Four: Threshold Calibration, Product Support, and Candidate ROI Extraction. The threshold is calibrated solely by the surface-supporting pixels of normal validation samples of the same category. Let the set of normal validation pixels be... The threshold τ is Based on the empirical 99.5th percentile of the outlier score, the false alarm rate for normal target pixels is 0.005.

[0078] The RGB product supports mask V, which is obtained by grayscale Otsu thresholding, 3×3 binary opening, 9×9 binary closing, hole filling, and maximum connected component preservation. The thresholded anomaly map is multiplied by the product support to obtain the candidate mask M.

[0079]

[0080] For each connected component C of M, calculate the out-of-threshold anomaly quality E(C), select the component with the largest E(C), and calculate the bounding rectangle of that connected component. Expand outwards by δ=20 pixels and crop to the image boundary:

[0081] ;

[0082] If no valid connected component is formed, the system outputs the no_roi_component status, reason, and original anomaly score graph; when it is necessary to continue controlled comparison, the predefined RGB product bounding box is called as a fallback and marked as fallback in the record.

[0083] In one alternative implementation, candidate region extraction employs a secure hybrid selector with the original resolution outer boundary set to 0 pixels. This selector only receives the anomaly score map S, the RGB product support mask V, and the normal validation threshold τ. It generates compact high-scoring seeds from the thresholded results, sorts them by over-threshold anomaly density, connected component compactness, and fill rate, applies region area constraints, and outputs confidence and rejection (abstention) states. It selects the candidate with the best quality, recording the abstraction when quality conditions are insufficient and retaining the connected component with the highest over-threshold anomaly quality as the baseline result. This selector is decoupled from the detector and classifier, does not read the test mask and defect labels during inference, and supports parameter optimization by object category.

[0084] Phase 5: Extraction of RGB appearance features and XYZ geometric statistical features. The corresponding RGB regions are cropped and resized to 224×224, normalized to ImageNet mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225), input into the frozen ResNet-50 network, remove the last fully connected layer, and perform global average pooling on the last convolutional block to obtain the appearance feature vector. .

[0085] For the valid set of points mapped to the same region For each coordinate axis a∈{X,Y,Z}, calculate seven types of statistics:

[0086] , , The standard deviation is calculated based on the population standard deviation, and the percentiles are defined using linear interpolation. NaN, infinite values, and all zeros in the point cloud are excluded before statistical analysis.

[0087] Phase Six: Branch Normalization, Multiple Model Candidates, and Nested Selection. Each outer layer training layer individually fits the normalizer, principal component analysis (PCA) model, and classifier. Candidate feature families include:

[0088] RGB-only: Only uses 2048-dimensional RGB appearance features;

[0089] XYZ-only: Uses only 21-dimensional XYZ geometric statistical features;

[0090] Directly concatenating RGB+XYZ: PCA is performed after standardizing the entire 2069-dimensional vector;

[0091] Branch normalization RGB+XYZ: Standardize, perform PCA, and sample-by-sample L2 normalization on the RGB and XYZ branches respectively.

[0092] , The total number of model candidates is 72: RGB-only (3 PCA dimensions × 4 C values ​​= 12); direct RGB+XYZ (12); XYZ-only (3 PCA dimensions × 4 C values ​​= 12); branch normalization fusion (3 RGB dimensions × 3 XYZ dimensions × 4 C values ​​= 36).

[0093] For each outer training fold, perform three-fold inner-layer hierarchical cross-validation, sort by macro-average F1 score, select the unique optimal configuration, and then refit and predict the outer test fold using that outer training partition. The kernel function of the RBF support vector machine is:

[0094] The implementation configuration adopts five-fold hierarchical outer-layer cross-validation and three-fold hierarchical inner-layer cross-validation. The inner layer uses macro-average F1 to select the model family, PCA dimension, and RBF-SVM penalty parameters, and writes the selection results to the running record each time.

[0095] Example 2: Application of the Carrot Category

[0096] The Carrot category includes defect types such as combined, contamination, crack, cut, and hole. The dataset contains 132 defect samples, of which 128 have public pixel masks for paired classification comparison; 286 are normal training samples and 29 are normal validation samples. The automatic ROI path uses a threshold τ = 43.312679290771484 (determined by the 99.5% quantile of the surface supporting pixels of the 29 normal validation samples), with the original resolution expanded by 20 pixels.

[0097] On the representative sample combined / 002 of the Carrot category, the following deterministic calculated values ​​were recorded: image-level reweighted score 36.9537239074707; pixel score minimum 17.3281307220459, maximum 57.8417854309082, mean 25.443071365356445; after applying a threshold of 43.312679290771484, three candidate connected components were formed, with the maximum out-of-threshold abnormal quality being 47590.86510848999. The recorded candidate bounding boxes are as follows: The selected component has 6659 pixels. The above parameters are used to illustrate the deterministic calculation process of this invention. The public mask of this sample is only used for the alignment record after inference.

[0098] Example 3: Independent Reproduction of Bagel Categories

[0099] The defect types for the Bagel category include combined, contamination, crack, and hole. A memory bank of 244 normal samples was independently built, and a threshold τ = 48.722115 was given from 22 normal validation samples. All 88 defect samples had public masks and selected components. Independent calibration demonstrates the adaptability of this invention's process to different object categories; the Carrot and Bagel categories use independent memories, independent thresholds, and independent classifiers.

[0100] Example 4: Classification Performance and Audit Results

[0101] Classification robustness results were recorded under three hierarchical random partitioning schemes. The table below reports the core results for two categories: overall input and mask reference ROI. Automatic ROI, as a configurable engineering path, has its threshold, region constraints, and expansion strategy optimized by object category.

[0102] The performance evaluation of safe hybrid operation focuses on operational efficiency, repeatability, inference-time data isolation, anomaly area control, and rollback traceability. As shown in the table below, the average online time for Carrot / p95 is 0.8931 / 1.0769 seconds / figure, and for Bagel it is 0.9454 / 1.0780 seconds / figure; both passed the repeatability, no test mask leakage, area control, and rollback recording checks. The times mentioned above are the time taken from online detection to ROI under recording conditions, excluding offline memory construction. Parameters can be calibrated according to object category and production line cycle time.

[0103]

[0104] The audit results demonstrate that the present invention can provide a low-latency, recalculated, and traceable automated ROI processing link in a controlled hardware environment.

[0105] Example 5: System Architecture

[0106] This embodiment provides a fine-grained defect detection system for agricultural products, including:

[0107] Synchronous acquisition module: Used to synchronously acquire RGB images and ordered XYZ point clouds of the agricultural products to be tested, and to ensure spatial registration between the two.

[0108] Normal Sample Memory Module: Used to extract and concatenate local RGB features and local FPFH features using only normal training samples of the same type, to build a normal sample memory and compress the core set.

[0109] Anomaly score generation module: Used to calculate the nearest neighbor distance from the local features of the sample to be tested to the memory, and generate image-level anomaly scores and pixel-level anomaly score maps after reweighting.

[0110] Candidate Region Extraction Module: Used to determine candidate regions based on normal verification threshold, RGB product support mask, and connected component over-threshold abnormal quality selection rules, and map them to the original resolution.

[0111] Multimodal feature extraction module: Used to extract RGB appearance features and XYZ geometric statistical features from candidate regions.

[0112] Nested Model Selection and Classification Module: Used to select the optimal configuration from candidate feature families through inner-layer cross-validation and output the defect category on the outer-layer test fold.

[0113] Audit output module: Used to output candidate area coordinates, running status, rollback markers, and traceable audit records.

[0114] Each module works in concert to execute the steps in the aforementioned method embodiments.

[0115] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting fine-grained defects in agricultural products based on RGB-FPFH anomaly localization and branch-normalized multimodal classification of normal samples, characterized in that, Includes the following steps: S1. Acquire RGB images of the same agricultural product to be tested and ordered XYZ point clouds spatially registered with the RGB images; S2. Using only normal training samples of the same category as the object to be tested, extract local RGB features and local FPFH features respectively. Align and stitch the local RGB features and local FPFH features on a preset grid according to their positions to form multimodal local features. Establish a normal sample memory bank based on the multimodal local features. S3. Calculate the nearest neighbor distance from the multimodal local features of the sample to be tested to the normal sample memory, and generate an anomaly score map; S4. Using the anomaly score threshold calibrated only by normal verification pixels, the RGB product support mask, and the connected component over-threshold anomaly quality selection rule, candidate regions are determined from the anomaly score map, and the candidate regions are mapped to the original resolution of the RGB image and the XYZ point cloud. S5. Extract RGB appearance features and XYZ geometric statistical features from the mapped candidate regions respectively; S6. Select the classification feature family, dimensionality reduction, and classifier hyperparameters from the candidate feature family through inner-layer cross-validation, and output the defect category, candidate region coordinates, and traceable audit records on the outer-layer test layer, wherein the outer-layer test layer does not participate in the selection process.

2. The method according to claim 1, characterized in that, Both the RGB image and the XYZ point cloud are ordered raster data. The coordinate transformation of the candidate region between the RGB image and the XYZ point cloud is determined by their respective width and height ratios. The coordinates of the upper left corner of the candidate region are rounded down, and the coordinates of the lower right corner are rounded up. The coordinates are also cropped to the effective range of the corresponding data. The local RGB features are extracted by a wide ResNet-50 network pre-trained on ImageNet. The feature maps with network output indices 2 and 3 are used. After being averaged by 3×3 with a stride of 1 and adaptive averaged by 28×28 grid size, they are concatenated according to their positions to obtain a 1536-dimensional RGB feature vector for each grid position. The local FPFH feature is a point cloud composed of valid non-zero points, with a coordinate scaling factor of 0.05, a normal estimation radius of 0.10, and an FPFH feature search radius of 0.

25. Each valid point yields a 33-dimensional FPFH descriptor, which is then concatenated with the RGB feature on the 28×28 grid according to position to form a 1569-dimensional multimodal local feature. The number of local features in the normal sample memory is the number of normal samples multiplied by the number of grids of 28×28. The core set is compressed using a core set ratio of 0.1 and a sparse random projection parameter epsilon of 0.

9. The core set is achieved by successively selecting the feature vector with the largest minimum Euclidean distance to the selected set.

3. The method according to claim 1, characterized in that, The generation of the anomaly score map includes: For each query local feature Calculate its nearest neighbor distance to the normal sample memory. ; make , and order for Nearest neighbor memory features; by Of the three nearest neighbors, excluding The reweighting coefficients are calculated for the second and third neighbors outside the user: Where D is the local feature dimension, and e2 and e3 are... The Euclidean distance to the two neighbors; Image-level anomaly score Pixel-level anomaly score map is composed of The result was obtained by rearranging the grid into a 28×28 grid and then performing bilinear interpolation and Gaussian smoothing.

4. The method according to claim 1, characterized in that, The anomaly score threshold is determined by the empirical 99.5% quantile of the anomaly score of the surface support pixels of the normal verification samples, so that the false alarm target for normal pixels is 0.005; the RGB product support mask is obtained by grayscale Otsu threshold segmentation, 3×3 binary opening operation, 9×9 binary closing operation, hole filling and maximum connected component selection.

5. The method according to claim 1, characterized in that, The candidate region mask is: ； Where S(x,y) is the pixel-level anomaly score map, τ is the anomaly score threshold, and V(x,y) is the RGB product support mask; Calculate the out-of-threshold quality for each connected component C: Select the connected region with the largest E(C), calculate its original resolution bounding rectangle, expand it to the surrounding area by a preset number of pixels, and then crop it to the image boundary to obtain the candidate region.

6. The method according to claim 1, characterized in that, The XYZ geometric statistical features calculate the mean, standard deviation, 10th percentile, 90th percentile, range, median, and mean absolute deviation for all finite points within the candidate region that are not all zero, with respect to the X, Y, and Z coordinate axes, forming a 21-dimensional vector. When the 21-dimensional vector is directly concatenated with the 2048-dimensional RGB appearance features, a 2069-dimensional vector is formed.

7. The method according to claim 1, characterized in that, The candidate feature families include RGB-only feature families, XYZ-only feature families, directly concatenated RGB+XYZ feature families, and branch-normalized RGB+XYZ feature families; The branch-normalized RGB+XYZ feature family first performs standardization and principal component analysis to reduce the dimensionality of the 2048-dimensional RGB branch and the 21-dimensional XYZ branch respectively, then performs sample-by-sample L2 normalization, and finally splices the two branches together. The principal component analysis dimension of the RGB branch is selected from {10, 20, 40}, and the principal component analysis dimension of the XYZ branch is selected from {3, 5, 10}.

8. The method according to claim 1, characterized in that, The model selection employs a five-fold hierarchical outer-layer cross-validation and a three-fold hierarchical inner-layer cross-validation. The inner-layer cross-validation uses the macro-average F1 score as the selection metric. The penalty parameter C of the RBF support vector machine is selected from {0.1, 1, 10, 100}. The RBF kernel parameters adopt the scaling rule. All candidate configurations are selected only by sorting within each outer-layer training partition.

9. The method according to claim 1, characterized in that, The candidate region extraction also includes a secure hybrid selector: generating compact high-scoring seeds from a thresholded and product-supported gated anomaly score map, sorting them according to the density of anomalies exceeding the threshold, the compactness of connected components, and the fill rate, performing region area constraints and recording confidence and rejection states, and setting the original resolution outer boundary to 0 pixels; when the candidate region does not trigger the pathological area or low confidence condition, retaining the selection result of the largest connected component of the anomaly quality exceeding the threshold, the secure hybrid selector does not read the test defect mask or defect category during inference.

10. A fine-grained defect detection system for agricultural products, characterized in that, include: The synchronous acquisition module is used to acquire RGB images of the same agricultural product to be tested and ordered XYZ point clouds spatially registered with the RGB images; The normal sample memory module is used to extract local RGB features and local FPFH features using only normal training samples of the same category as the object to be tested and to build a normal sample memory. An anomaly score map generation module is used to calculate the nearest neighbor distance from the local features of the sample to be tested to the memory bank and generate an anomaly score map; The candidate region extraction module is used to determine candidate regions using a threshold calibrated only by normal verified pixels, an RGB product support mask, and a connected component over-threshold abnormal quality selection rule, and to map the candidate regions to the RGB image and the XYZ point cloud; The multimodal feature extraction module is used to extract RGB appearance features and XYZ geometric statistical features from candidate regions; The nested model selection and classification module is used to select the classification feature family, dimensionality reduction, and classifier hyperparameters from the candidate feature families through inner-layer cross-validation, and output the defect category on the outer-layer test fold; and The audit output module is used to output the coordinates of the candidate area and traceable audit records; Each module is configured to perform the method described in any one of claims 1 to 9.

Images