A machine vision-based method for detecting defects in agricultural products

Abstract

The present application relates to the field of machine vision and intelligent detection technology, and particularly relates to a kind of agricultural product defect detection method based on machine vision, its method includes: obtaining the multi-view observation data of different polarization state visible light image, different illumination direction visible light image and near infrared image;Determine surface parameterization coordinate based on multi-view observation data, construct normal appearance model on surface parameterization coordinate and generate visible light residual graph, near infrared residual graph, geometric feature map and near infrared scattering anomaly graph;Visible light residual graph, near infrared residual graph and geometric feature map are carried out low rank sparse decomposition to obtain sparse anomaly graph and extract candidate defect area, when candidate defect area does not meet cross-view consistency, obtain supplementary multi-view observation data and update sparse anomaly graph;According to the updated sparse anomaly graph, determine the defect mask, and determine the defect type in combination with the near-infrared scattering anomaly map, thereby improving the stability and accuracy of defect detection and classification.

Description

Technical Field

[0001] This invention relates to the field of machine vision and intelligent inspection technology, specifically to a method for detecting defects in agricultural products based on machine vision. Background Technology

[0002] In the post-harvest grading, storage, distribution, and final sales stages of agricultural products, defects such as rot, insect infestation, and mold can lead to increased spoilage, cross-contamination, and food safety risks, directly impacting grading and pricing, supply chain fulfillment, and brand reputation. The industry has long relied on manual sorting or single optical inspection methods, which suffer from high subjectivity, insufficient stability, and numerous false positives and false negatives due to variations in lighting and fruit surface morphology. Furthermore, defects often exhibit cross-modal variations, making it difficult for a single imaging method to capture surface texture, geometric undulations, and internal scattering changes, thus hindering the formation of unified standards for sorting consistency and traceability quality indicators. Summary of the Invention

[0003] This invention provides a machine vision-based method for detecting defects in agricultural products, which addresses the problem of how to stably and accurately extract defective regions and identify defect types in agricultural products under multiple viewing angles and lighting conditions.

[0004] This invention provides a machine vision-based method for detecting defects in agricultural products, the method comprising: Acquire multi-view observation data of agricultural products, including visible light images with different polarization states, visible light images with different illumination directions, and near-infrared images; The surface parameterized coordinates are determined based on multi-view observation data. A normal appearance model is constructed based on the surface parameterized coordinates, and visible light residual map, near-infrared residual map, geometric feature map and near-infrared scattering anomaly map are generated. Low-rank sparse decomposition is performed on the visible light residual map, near-infrared residual map and geometric feature map to obtain sparse anomaly map and determine candidate defect regions. When the candidate defect regions do not meet the cross-view consistency condition, supplementary multi-view observation data is obtained and the supplementary multi-view observation data is mapped to surface parameterized coordinates to update the sparse anomaly map. The defect mask is determined based on the updated sparse anomaly map, and the defect type is determined based on the defect mask and the near-infrared scattering anomaly map.

[0005] In one possible implementation, visible light images with different polarization states include visible light images acquired under cross-polarization states and visible light images acquired under co-polarization states, and visible light images with different illumination directions include visible light images acquired under illumination from at least three different illumination directions at the same viewing angle.

[0006] In one possible implementation, a diffuse reflection reference plate image is acquired before acquiring multi-view observation data of agricultural products; after acquiring multi-view observation data of agricultural products, gain correction and intensity normalization are performed on the multi-view observation data based on the diffuse reflection reference plate image.

[0007] In one possible implementation, determining the surface parameterized coordinates includes: segmenting agricultural product regions from multi-view observation data to obtain the outlines of agricultural products from each viewpoint; fitting an ellipsoid based on the agricultural product outlines; and establishing a parametric coordinate mapping from pixel coordinates to the ellipsoid to obtain the surface parameterized coordinates.

[0008] In one possible implementation, constructing a normal appearance model includes: resampling multi-view observation data to a surface map based on surface parameterized coordinates; and performing smooth regression on the surface map to obtain a normal appearance model.

[0009] In one possible implementation, generating the visible light residual map includes: selecting a visible light image corresponding to a preset polarization state, and generating the visible light residual map based on the difference between the predicted visible light image output by the normal appearance model and the visible light image corresponding to the preset polarization state; generating the near-infrared residual map includes: generating the near-infrared residual map based on the difference between the predicted near-infrared image output by the normal appearance model and the near-infrared image; generating the geometric feature map includes: calculating the brightness difference and gradient magnitude based on visible light images with different illumination directions and generating the geometric feature map; generating the near-infrared scattering anomaly map includes: normalizing the intensity of the near-infrared image and the visible light image corresponding to the preset polarization state on the surface parameterized coordinates, and generating the near-infrared scattering anomaly map based on the normalized intensity difference.

[0010] In one possible implementation, performing low-rank sparse decomposition includes: concatenating the visible light residual map, near-infrared residual map, and geometric feature map by channel on the surface parameterized coordinates to form a feature matrix; decomposing the feature matrix into a low-rank baseline matrix and a sparse anomaly matrix; and generating a sparse anomaly map based on the sparse anomaly matrix.

[0011] In one possible implementation, determining candidate defect regions involves performing connected component analysis on a sparse anomaly graph to obtain a set of candidate defect regions; the cross-view consistency condition includes that the candidate defect regions have an overlap area ratio greater than an overlap ratio threshold after being mapped to surface parameterized coordinates from at least two different viewpoints.

[0012] In one possible implementation, supplementing the multi-view observation data includes at least one of switching polarization state to acquire visible light images, switching illumination direction to acquire visible light images, and acquiring images from adjacent viewpoints; updating the sparse anomaly map includes: mapping the supplemented multi-view observation data to surface parameterized coordinates; regenerating the visible light residual map, near-infrared residual map, and geometric feature map based on the mapping result; and performing low-rank sparse decomposition again on the regenerated visible light residual map, near-infrared residual map, and geometric feature map to update the sparse anomaly map.

[0013] In one possible implementation, defect types include rot, insect infestation, and mold. Determining the defect type involves: determining rot as the defect type when the average value of the near-infrared scattering anomaly map within the corresponding area of ​​the defect mask is greater than a first threshold; determining insect infestation as the defect type when the average value of the geometric feature map within the corresponding area of ​​the defect mask is greater than a second threshold; determining mold as the defect type when the average value of the visible light residual map within the corresponding area of ​​the defect mask is greater than a third threshold and the average value of the near-infrared scattering anomaly map within the corresponding area of ​​the defect mask is less than a fourth threshold; and simultaneously outputting the defect type, the ratio of the defect mask area to the effective area corresponding to the surface parameterized coordinates is output as a severity quantification result.

[0014] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: By combining visible light imaging with near-infrared imaging under different polarization states and illumination directions, complementary characterization of surface reflection, geometric undulations, and scattering anomalies was achieved. Alignment and fusion of multi-view data using surface parameterized coordinates enabled consistent modeling and localization of the same surface location across different viewpoints. Separation of gradually varying textures and illumination terms from defects and anomalies was achieved through normal appearance models and residual construction. Sparse highlighting and noise suppression of local anomalies were achieved through low-rank sparse decomposition and candidate defect region extraction. Closed-loop correction of false anomalies was achieved through cross-view consistency judgment and supplementary acquisition and update mechanisms. Interpretable differentiation of defect types was achieved through joint discrimination of defect masks and near-infrared scattering anomalies. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the execution flow of the method of the present invention; Figure 2 This is a statistical chart of the calibration stability of the reference board in a specific embodiment of the present invention; Figure 3 This is a statistical chart showing cross-view consistency and supplementary data acquisition triggering in a specific embodiment of the present invention; Figure 4 This is a statistical chart showing the distribution of defect severity quantification results in a specific embodiment of the present invention; Figure 5 This is a statistical chart of the detection effect in a specific embodiment of the present invention. Detailed Implementation

[0016] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation.

[0017] Machine vision typically refers to a technological system that uses imaging sensors to acquire two-dimensional or three-dimensional observation data of a target, and then uses image processing and pattern recognition algorithms to complete target localization, feature representation, state determination, and result output. Its core lies in converting observable optical information into computable digital representations, and achieving stable automated detection under controlled or semi-controlled imaging conditions. Compared to discrimination methods that rely solely on a single image, machine vision can expand the information dimension through multi-view, multi-illumination, multi-spectral, and polarization imaging techniques, allowing complementary features of the same target under different observation conditions to be jointly utilized, thus forming a more robust detection chain. Based on the above technological foundation, this invention addresses the typical appearance and material anomaly problem of agricultural product defects, introducing mechanisms such as multi-view observation data organization, surface parameterized alignment, normal appearance modeling, and anomaly decomposition to achieve a closed-loop processing flow for defect region extraction and defect type discrimination.

[0018] like Figure 1 As shown, a machine vision-based method for detecting defects in agricultural products includes: Acquire multi-view observation data of agricultural products, including visible light images with different polarization states, visible light images with different illumination directions, and near-infrared images; In one embodiment, agricultural products are placed at the center of a stage, which rotates around a vertical axis at a preset angular sequence. The imaging module triggers acquisition at each angular position, forming multi-view observation data. The imaging module includes a visible light imaging channel and a near-infrared imaging channel. The visible light imaging channel acquires visible light images under at least two polarization states and sequentially switches between at least three different illumination directions at the same viewpoint to acquire visible light images under different illumination directions. The near-infrared imaging channel acquires near-infrared images when near-infrared illumination is on. Each frame of image records the viewpoint number, polarization state, illumination direction, exposure time, gain, and timestamp as input for subsequent alignment and fusion.

[0019] Visible light images with different polarization states include visible light images acquired under cross-polarization states and visible light images acquired under the same polarization states. Visible light images with different illumination directions include visible light images acquired under illumination from at least three different illumination directions at the same viewing angle.

[0020] In one embodiment, the present invention provides acquisition rules for visible light images with different polarization states and different illumination directions, enabling multiple frames of images from the same viewpoint to correspond one-to-one. The visible light source employs a constant-current driven array of light-emitting diodes, with a linear polarizer fixedly mounted at the light-emitting end.

[0021] A linear polarization analyzer is mounted at the front of the imaging lens. Driven by a rotating motor, the analyzer can be positioned to a preset angle. A zero-degree angle between the analyzer and the polarizer is defined as a co-polarization state, while a ninety-degree angle is defined as a cross-polarization state. The relative positions of the camera and stage are kept constant during image acquisition. For each viewing angle, a co-polarized visible light image is acquired first, followed by a switch to the cross-polarization state. Both frames are acquired using the same exposure time and gain.

[0022] After polarization switching is complete, a stabilization period is waited before triggering exposure. If a saturation region exists in the same polarization state, the exposure is reduced and two frames of images from that viewpoint are re-acquired. Polarization-paired images refer to visible light images in the same polarization state and visible light images in cross-polarization state from the same viewpoint. Visible light images with different illumination directions are achieved by a partitioned controllable light source array. The light source array is arranged around the field of view and divided into at least three independent partitions, each corresponding to a primary illumination direction. During acquisition, only one independent partition is illuminated sequentially from the same viewpoint, and one frame of visible light image is acquired until acquisition of at least three illumination directions is completed.

[0023] Acquisition of images from different illumination directions employs a fixed polarization state, with cross-polarization being the preferred method to reduce specular interference and enhance texture contrast. Exposure time and gain are kept consistent across all three illumination directions. If brightness is insufficient in a particular illumination direction, the drive current of the corresponding zone is increased, but maintained within a range that does not induce saturation. The acquisition sequence for visible light images from a single viewpoint is fixed as follows: same-polarization image, cross-polarization image, first illumination direction image, second illumination direction image, and third illumination direction image. After each frame is acquired, a sharpness index is calculated, and blurring is detected. If the sharpness index does not reach a threshold, only that frame is reacquired, while the remaining frames are retained.

[0024] After acquisition, all visible light images of that viewpoint are written to the same data unit, along with polarization state and illumination direction markers, to ensure accurate image retrieval in subsequent processing stages. Illumination direction is determined by the physical orientation of the light source partition, which can be used to limit stray light and reduce shadow edge sharpness via light shields and diffusers. Each light source partition number corresponds one-to-one with its physical orientation and is recorded in the metadata for each frame. After each rotation of the stage to its designated position, it stops and waits for stabilization before performing full acquisition of that viewpoint; the waiting time and rotation angle are fixed in the parameter table.

[0025] Before acquiring multi-view observation data of agricultural products, a diffuse reflection reference plate image is obtained; after acquiring multi-view observation data of agricultural products, gain correction and intensity normalization are performed on the multi-view observation data based on the diffuse reflection reference plate image.

[0026] In one embodiment, the present invention introduces a diffuse reflection reference plate image before and after multi-view observation data acquisition. This image is used to perform gain correction and intensity normalization on the multi-view observation data, thereby reducing the impact of light source brightness drift, lens vignetting, and sensor response differences. The diffuse reflection reference plate is a plate with a surface that approximates Lambertian reflection, and its reflectivity is stable in the visible and near-infrared bands. Before acquisition, the diffuse reflection reference plate is placed at the center of the stage, ensuring that the surface of the reference plate is at the same working distance and field of view as the agricultural products.

[0027] Subsequently, diffuse reflection reference images are acquired using the same configuration as those acquired for agricultural products. This configuration covers visible light images with the same polarization state, visible light images with cross-polarization states, visible light images with at least three different illumination directions, and near-infrared images. When the imaging module has a single camera or a single lens channel, diffuse reflection reference images consistent with those acquired for agricultural products are acquired, and corresponding calibration records are saved. When the imaging module contains multiple cameras or multiple lens channels, each camera or lens channel independently acquires its corresponding diffuse reflection reference image and saves its corresponding calibration record. The calibration record includes the reference image, exposure time, gain, polarization state markers, and illumination direction markers. Subsequently, multi-view observation data of agricultural products are acquired, maintaining the same polarization state switching method, illumination direction sequence, and near-infrared illumination state as the calibration records. After acquisition, gain correction and intensity normalization are performed on each frame of agricultural product image. Gain correction includes dark level subtraction and flat field correction.

[0028] Dark level subtraction is achieved using the shading image or sensor black level parameters. Flat field correction generates a pixel-level correction coefficient map from a reference image and performs pixel-by-pixel correction on the agricultural product image according to the corresponding configuration to compensate for lens vignetting and sensor response inhomogeneity. Subsequently, intensity statistics are calculated within the agricultural product area to align the overall brightness of images from different viewing angles, polarization states, and illumination directions to a preset target level, achieving intensity normalization.

[0029] To avoid artifacts introduced by local anomalies in the reference image, anomalous pixels are thresholded and removed when generating the correction coefficient map, and the neighborhood mean is used to fill in the anomalous areas. If overexposure, smudges, or shadows in the reference image cause instability in the correction coefficients, the reference image is reacquired and the calibration record is updated. When changes in ambient temperature or light source aging cause brightness drift exceeding the threshold, the diffuse reference plate image is reacquired and the original calibration record is replaced. The multi-view observation data after gain correction and intensity normalization have a stable intensity baseline, providing consistent input for the subsequent generation of residual and anomaly maps.

[0030] During intensity normalization, an agricultural product region mask is first segmented from the visible light image. Then, the average and median gray levels are calculated within the agricultural product region mask to suppress the influence of a small number of highlight pixels on the statistics. After normalization, a correction quality index is generated and written to the calibration record. The correction quality index is used to identify whether the reference image needs to be updated and whether the corresponding viewpoint data needs to be reacquired.

[0031] The surface parameterized coordinates are determined based on multi-view observation data. A normal appearance model is constructed based on the surface parameterized coordinates, and visible light residual map, near-infrared residual map, geometric feature map and near-infrared scattering anomaly map are generated. In one embodiment, after multi-view observation data enters the processing, a unified surface parameterized coordinate system is first established to map the agricultural product surface from different perspectives to the same coordinate system, facilitating subsequent comparison and fusion. After the surface parameterized coordinate system is established, a normal appearance model is constructed on the surface parameterized coordinate system. This model is used to predict the appearance of agricultural products under defect-free conditions. Subsequently, four types of intermediate results are generated as inputs for subsequent anomaly decomposition and defect determination. These four types of intermediate results include visible light residual maps, near-infrared residual maps, geometric feature maps, and near-infrared scattering anomaly maps. Visible light residual maps and near-infrared residual maps are used to characterize the differences between observed appearance and normal appearance. Geometric feature maps are used to characterize the differences in shadow and morphological response under different illumination directions. Near-infrared scattering anomaly maps are used to characterize the differences in scattering characteristics between near-infrared and visible light. All of the above processing is performed on the same surface parameterized coordinate system, ensuring comparability of the same surface location under different channels and reducing displacement interference caused by changes in viewing angle.

[0032] Determining the surface parameterized coordinates includes: segmenting agricultural product regions from multi-view observation data to obtain the outlines of agricultural products from each viewpoint; fitting an ellipsoid based on the agricultural product outlines; and establishing a mapping from pixel coordinates to parametric coordinates of the ellipsoid to obtain the surface parameterized coordinates.

[0033] In one embodiment, the present invention provides a specific method for determining surface parametric coordinates. A new limitation is that an ellipsoid is fitted to the agricultural product contour, and a mapping from pixel coordinates to ellipsoidal parametric coordinates is established, thereby forming a achievable unified surface coordinate system. First, agricultural product region segmentation is performed on the multi-view observation data from each perspective to obtain an agricultural product region mask.

[0034] Agricultural product region segmentation can be achieved based on the condition of a fixed background, preferably using visible light images with cross-polarization to reduce the influence of highlights. The segmentation method can be a combination of threshold segmentation and connected component screening: first, adaptive thresholding is performed on the brightness channel to obtain candidate regions; then, the connected component with the largest area is selected as the agricultural product region; morphological closing operations are performed on the agricultural product region to fill small holes, and morphological opening operations are performed to remove edge burrs.

[0035] After obtaining the agricultural product region mask, a boundary point set is extracted from the mask. This set represents the agricultural product outline from that viewpoint. To improve outline stability, the boundary point set is sampled at equal intervals, and large cusps with abnormal curvature are removed to avoid fitting bias caused by petioles, fruit stalks, or small attachments. Subsequently, the agricultural product outlines from multiple viewpoints are used to fit an ellipsoid. During fitting, the ellipsoid parameters are used as the variables to be estimated, with the goal of minimizing the distance between the projected outlines from multiple viewpoints and the observed outline.

[0036] The camera projection parameters can be the intrinsic parameters calibrated by the equipment, with the camera extrinsic parameters corresponding to the stage rotation angle as initial values. Fitting can be achieved through iterative optimization: first, the ellipsoidal scale parameters are estimated using the major and minor axes and area of ​​the contour; then, the ellipsoidal center position and attitude parameters are iteratively updated using multi-view contour errors until the error decreases to a preset threshold or reaches the maximum number of iterations. To ensure feasibility, the error can be the average distance from the contour points to the projected contour, with weights added to each viewpoint, the weights being related to the segmentation quality of that viewpoint.

[0037] After fitting, a mapping from pixel coordinates to ellipsoidal parameter coordinates is established. The mapping is established as follows: for any viewpoint pixel, the intersection of the camera ray direction and the ellipsoid is calculated to obtain the intersection coordinates; these intersection coordinates are then converted to ellipsoidal parameter coordinates, which can be in longitude and latitude form or an equal-area unfolded form. To avoid mapping discontinuities, a uniform starting meridian and starting parallel are defined on the ellipsoidal parameter coordinates, and the zero-degree viewpoint of the stage is used as the initial alignment reference.

[0038] Mapped locations in invisible areas are marked as invalid and skipped in subsequent resampling stages. The final output surface parameterized coordinates are a set of indexable two-dimensional parametric coordinate grids, where each location on the grid corresponds to a small region on the agricultural product surface, and the corresponding pixel location from each viewpoint can be looked up. Through this mapping, images from different viewpoints can be aligned on the same surface coordinates, providing consistent input for subsequent normal appearance model and residual generation.

[0039] Constructing a normal appearance model involves: resampling multi-view observation data to a surface mapping map based on surface parameterized coordinates; and performing smooth regression on the surface mapping map to obtain a normal appearance model.

[0040] In one embodiment, the present invention provides a method for constructing a normal appearance model, with the added limitation of first resampling multi-view observation data to a surface mapping map, and then performing smooth regression on the surface mapping map to obtain a achievable prediction of the normal appearance. Based on surface parameterized coordinates, the visible light image and near-infrared image of each viewpoint are mapped onto a surface parameterized coordinate grid to form a surface mapping map. The surface mapping map uses the surface parameterized coordinate grid as an index, and each grid position aggregates the corresponding pixel values ​​from multiple viewpoints.

[0041] Weighted fusion is preferred during aggregation to reduce the impact of occlusion, reflection, and noise on the results. The weights are determined by three factors: smaller angle of incidence, higher image sharpness, and lower pixel saturation. Multiple observations at the same grid location are weighted and averaged to obtain the observed surface mapping value for that location. For missing observations, neighborhood interpolation is used to fill in the gaps, but this is limited to small areas, and the filled locations have a lower confidence weight in subsequent residual calculations. After resampling, visible light and near-infrared surface mappings are obtained. Smoothing regression is then performed on the surface mappings to obtain a normal appearance model. The goal of smoothing regression is to fit the low-frequency variations of the surface mappings, preserving the gradually changing structure caused by shape and illumination, and suppressing local abrupt changes.

[0042] Smoothing regression can be implemented using locally weighted regression or bilateral filtered regression. To avoid treating large-scale defects as part of the normal appearance, a robust loss is introduced into the smoothing regression, reducing the regression weight of observations with large deviations. After regression, the normal appearance model outputs a predicted surface map. The predicted surface map includes a predicted visible light surface map and a predicted near-infrared surface map, which can be back-projected back to the image plane of each viewpoint when needed to obtain predicted visible light and near-infrared images. After the normal appearance model is built, the model parameters and surface parameterized coordinates are saved together, allowing the same modeling framework to be reused and updated quickly in subsequent stages after supplementary data acquisition. Through the path of "resampling to surface map and then smoothing regression," the normal appearance model is both feasible and maintains a consistent prediction benchmark across different viewpoints and channels.

[0043] Generating a visible light residual map includes: selecting a visible light image corresponding to a preset polarization state, and generating a visible light residual map based on the difference between the predicted visible light image output by the normal appearance model and the visible light image corresponding to the preset polarization state; generating a near-infrared residual map includes: generating a near-infrared residual map based on the difference between the predicted near-infrared image output by the normal appearance model and the near-infrared image; generating a geometric feature map includes: calculating the brightness difference and gradient magnitude based on visible light images with different illumination directions and generating a geometric feature map; generating a near-infrared scattering anomaly map includes: normalizing the intensity of the near-infrared image and the visible light image corresponding to the preset polarization state on the surface parameterized coordinates, and generating a near-infrared scattering anomaly map based on the normalized intensity difference.

[0044] In one embodiment, the present invention provides four methods for generating intermediate results. The added limitation is to clearly define the difference relationship and calculation path between each intermediate result and the normal appearance model and multi-view observation data, thereby avoiding unclear definitions of intermediate results. The generation of the visible light residual map first selects a visible light image corresponding to a preset polarization state. The preset polarization state is preferably a cross-polarization state to reduce the interference of specular highlights on the residual.

[0045] The visible light image corresponding to the preset polarization state is mapped to the surface parameterized coordinates to obtain the observed visible light surface mapping map. The predicted visible light surface mapping map is output from the normal appearance model. The visible light residual map is the difference between the observed and predicted visible light surface mapping maps. The difference can be an absolute difference or a signed difference; the signed difference is used to distinguish between changes that are brighter than normal and darker than normal. The generation method of the near-infrared residual map is the same as that of the visible light residual map. Figure 1The near-infrared image is mapped to surface parameterized coordinates to obtain the observed near-infrared surface map. The predicted near-infrared surface map is output from the normal appearance model. The near-infrared residual map is the difference between the observed and predicted near-infrared surface maps. The geometric feature map is used to reflect the differences in illumination response under different illumination directions. First, visible light images from different illumination directions at the same viewpoint are mapped to surface parameterized coordinates to obtain visible light surface maps for multiple illumination directions.

[0046] The brightness difference is calculated pairwise for each illumination direction to obtain a brightness difference map. Then, a gradient magnitude map is calculated for the visible light surface mapping map for each illumination direction; the gradient magnitude reflects local edge and texture changes. The brightness difference map and gradient magnitude map are normalized and then superimposed according to weights to generate a geometric feature map. The weights ensure that the brightness difference has a higher proportion in low-texture areas and the gradient magnitude has a higher proportion in high-texture areas, thus taking into account both shadow and edge variations. The near-infrared scattering anomaly map is used to reflect the difference in scattering characteristics between near-infrared and visible light in the material. First, the intensity of the near-infrared surface mapping map and the visible light surface mapping map corresponding to the preset polarization state are normalized on the parametric coordinates of the surface. Intensity normalization can be scaled by the median and quantile within the effective area of ​​the agricultural product to suppress local extreme values.

[0047] After normalization, the difference between the two is calculated to obtain the near-infrared scattering anomaly map. In rotten or moldy areas, the near-infrared scattering anomaly map typically exhibits anomalous changes asynchronous with visible light, serving as an important basis for subsequent defect type differentiation. All four types of intermediate results are generated on the surface parameterized coordinates and correspond one-to-one with the same surface location, enabling subsequent low-rank sparse decomposition to align inputs by channel and avoid mismatches introduced by viewing angle differences. After generation, the visible light residual map, near-infrared residual map, geometric feature map, and near-infrared scattering anomaly map are written into a data structure with a unified index, and the preset polarization state, illumination direction number, and normalization parameters are recorded to facilitate subsequent experimental reproduction and online parameter tuning.

[0048] Low-rank sparse decomposition is performed on the visible light residual map, near-infrared residual map and geometric feature map to obtain sparse anomaly map and determine candidate defect regions. When the candidate defect regions do not meet the cross-view consistency condition, supplementary multi-view observation data is obtained and the supplementary multi-view observation data is mapped to surface parameterized coordinates to update the sparse anomaly map. In one embodiment, the visible light residual map, near-infrared residual map, and geometric feature map, as multi-channel anomaly representations of the same surface location, are input into a low-rank sparse decomposition process. Low-rank sparse decomposition is used to separate large-scale, slowly varying normal appearance errors from localized, abrupt anomalous responses. The sparse anomaly map in the separation result is used to highlight local anomalous regions of defects such as decay and insect infestation on the surface parameterized coordinates. After the sparse anomaly map is generated, candidate defect regions are extracted based on connectivity, and a cross-view consistency check is performed on these regions to eliminate false anomalies caused by occasional highlights, shadow edges, or single-view occlusion. When a candidate defect region does not meet the cross-view consistency condition, a supplementary multi-view observation data acquisition process is triggered. After the supplementary multi-view observation data acquisition is completed, it is mapped to surface parameterized coordinates and used to regenerate the residual and geometric feature inputs. Low-rank sparse decomposition is then performed again to update the sparse anomaly map, thereby confirming or eliminating candidate defect regions with more observational evidence.

[0049] Performing low-rank sparse decomposition includes: stitching visible light residual map, near-infrared residual map and geometric feature map into a feature matrix by channel on the surface parameterized coordinates; decomposing the feature matrix into a low-rank baseline matrix and a sparse anomaly matrix; and generating a sparse anomaly map based on the sparse anomaly matrix.

[0050] In one embodiment, the present invention provides an input organization and output generation method for low-rank sparse decomposition. A new limitation is that multi-channel intermediate results are concatenated channel-wise to form a feature matrix, and this feature matrix is ​​decomposed into a low-rank baseline matrix and a sparse anomaly matrix. First, the visible light residual map, near-infrared residual map, and geometric feature map are aligned to the same size on a unified grid of surface parameterized coordinates, ensuring a one-to-one correspondence between the three maps at each grid position. Intensity normalization is then performed on each map to make the numerical scales of different channels comparable.

[0051] Intensity standardization calculates the mean and standard deviation within the effective area of ​​agricultural products, followed by zero-mean and unit-variance transformation for each plot. To avoid noise amplification due to excessively small standard deviations, a minimum value is set at the lower limit of the standard deviation. After standardization, three-channel values ​​are taken at each grid location to form a feature vector. The surface parameterized coordinate grid is expanded into a one-dimensional index in a fixed order, and all feature vectors are stacked according to the index to form a feature matrix. The rows of the feature matrix correspond to the surface parameterized coordinate grid positions, and the columns correspond to the channel features. To improve the robustness of the decomposition, local neighborhood statistics, such as the local mean and local variance of the residuals, can be added to the channel dimension, but the number of new features should be kept limited to avoid overcomplication. Then, low-rank sparse decomposition is performed on the feature matrix to obtain a low-rank baseline matrix and a sparse anomaly matrix.

[0052] The low-rank baseline matrix represents the gradual variation components caused by shape, illumination inhomogeneity, and modeling errors. The sparse anomaly matrix represents the anomalous components with local mutations. To ensure the decomposition is feasible, the low-rank sparse decomposition can be solved using an alternating iterative approach: the low-rank baseline matrix is ​​updated while the sparse anomaly matrix is ​​fixed, and the sparse anomaly matrix is ​​updated while the low-rank baseline matrix is ​​fixed, until the iteration converges or the maximum number of iterations is reached. The iteration stopping condition can be determined by the decrease in reconstruction error between two adjacent iterations. The reconstruction error is defined as the mean square value of the difference between the characteristic matrix and the sum of the low-rank baseline matrix and the sparse anomaly matrix.

[0053] To suppress random noise from entering the sparse anomaly matrix, a thresholding process is applied to the sparsity during matrix updates. This thresholding performs soft-thresholding shrinkage on each element of the sparse anomaly matrix, compressing anomalies with amplitudes less than the threshold to zero, thus ensuring a sparse distribution of anomalies. The threshold value can be related to the channel noise level and can be estimated from the standard deviation during the aforementioned intensity normalization. After low-rank sparse decomposition, the sparse anomaly matrix is ​​inversely expanded using the surface-parameterized coordinate grid index to obtain the sparse anomaly map. The anomaly intensity can be taken as the sum of the absolute values ​​of the sparse anomaly matrix in the channel dimension, and the anomaly intensity is normalized for subsequent thresholding and connected component analysis. The generated sparse anomaly map is retained in the surface-parameterized coordinate space, ensuring direct connection with subsequent candidate defect region extraction.

[0054] Determining candidate defect regions involves performing connected component analysis on the sparse anomaly graph to obtain a set of candidate defect regions; the cross-view consistency condition includes that the candidate defect regions have an overlap area ratio greater than the overlap ratio threshold after being mapped to surface parameterized coordinates from at least two different viewpoints.

[0055] In one embodiment, the present invention provides definitions of extraction rules for candidate defect regions and cross-view consistency conditions. The added limitation is that a set of candidate defect regions is obtained using connected component analysis, and the cross-view consistency of the candidate defect regions is tested using the overlap area ratio. Candidate defect region extraction first involves thresholding the sparse anomaly graph.

[0056] Thresholding can employ either a global threshold or an adaptive threshold. To reduce threshold fluctuations caused by individual differences in various agricultural products, the threshold can be determined based on the quantiles of the sparse anomaly map within the effective region of the agricultural product, for example, using the 95th quantile as the initial threshold, and limiting the upper and lower bounds of the threshold within a preset range. Thresholding yields a binary anomaly map. Connectivity analysis is performed on the binary anomaly map to obtain a set of candidate defect regions. For each candidate defect region, the area, perimeter, compactness, and aspect ratio are calculated, and candidate defect regions with areas smaller than the minimum area threshold are removed to eliminate noise points. A cross-view consistency check of candidate defect regions is used to determine whether the candidate defect region is supported from multiple viewpoints.

[0057] The inspection process uses surface parametric coordinates as a bridge. First, the region masks of candidate defect areas on the surface parametric coordinates are numbered to obtain the surface masks of candidate defect areas. Then, at least two original images from different perspectives are selected from the multi-view observation data as verification perspectives. The verification perspectives are preferably chosen from those with a larger visible area corresponding to the surface position of the candidate defect area to reduce occlusion effects. For each verification perspective, the surface mask of the candidate defect area is projected onto the pixel plane of that perspective through the inverse mapping from pixel coordinates to ellipsoidal parametric coordinates, obtaining the pixel mask of the candidate defect area under that perspective. Subsequently, the mean anomalous intensity within the pixel mask is calculated based on the visible light residual map, near-infrared residual map, or geometric feature map under that perspective, and a consistency judgment within the perspective is performed based on the difference between the mean anomalous intensity and the background area. If the mean anomalous intensity within the perspective does not exceed the corresponding threshold, the perspective is considered not to support the candidate defect area. The cross-perspective consistency condition is defined by the overlap area ratio.

[0058] The overlap area ratio is calculated as follows: The pixel masks of candidate defect regions under different viewpoints are mapped back to surface parameterized coordinates to obtain corresponding surface mask sets. The ratio of the intersection area to the union area of ​​these surface mask sets is then calculated. When the ratio of the intersection area to the union area is greater than the overlap ratio threshold, the candidate defect region is deemed to meet the cross-viewpoint consistency condition. The overlap ratio threshold can be set according to the number of viewpoints. The more viewpoints there are, the lower the threshold can be to tolerate slight deviations, but the threshold should not be lower than a preset lower limit to avoid random noise passing the consistency check. Through this consistency check, anomalies caused by single-viewpoint highlights, shadow edges, or local occlusion are more easily eliminated, while true defects are more easily preserved because they have a stable response across multiple viewpoints.

[0059] Supplementing multi-view observation data includes at least one of the following: acquiring visible light images by switching polarization states, acquiring visible light images by switching illumination directions, and acquiring images from adjacent viewpoints. Updating the sparse anomaly map includes: mapping the supplemented multi-view observation data to surface parameterized coordinates; regenerating the visible light residual map, near-infrared residual map, and geometric feature map based on the mapping results; and performing low-rank sparse decomposition again on the regenerated visible light residual map, near-infrared residual map, and geometric feature map to update the sparse anomaly map.

[0060] In one embodiment, the present invention provides a specific process for triggering and updating the sparse anomaly map using supplementary multi-view observation data. The added limitation is the explicit definition of the supplementary acquisition options and the update path of regenerating intermediate results and re-executing low-rank sparse decomposition after supplementary acquisition. When a candidate defect region does not meet the cross-view consistency condition, the current evidence is deemed insufficient, and the supplementary acquisition process begins. The goal of supplementary acquisition is to obtain observational evidence that better distinguishes defects from interference around the candidate defect region. Supplementing multi-view observation data includes at least one of three types of operations.

[0061] The first type of operation involves acquiring visible light images by switching polarization states. If the current preset polarization state used for residual generation is cross-polarization, then visible light images under the same polarization state are acquired to observe whether the candidate defect region exhibits pseudo-anomalies caused by specular highlights. The second type of operation involves acquiring visible light images by switching illumination directions. New illumination directions are added to the same viewing angle, or the original illumination direction is repeatedly acquired to verify stability, determining whether the candidate defect region continues to exhibit abnormal responses under different shadow conditions. The third type of operation involves acquiring images from adjacent viewing angles. Adjacent viewing angles can be selected from two adjacent corner positions before and after the current viewing angle, allowing the candidate defect region to be repeatedly observed under changing viewing angles. The selection of supplementary acquisition viewing angles is determined by the surface mask of the candidate defect region. First, based on the surface mask of the candidate defect region, it is determined which viewing angles have the largest visible area for that region, and then supplementary acquisition is performed preferentially from these viewing angles, thereby reducing invalid acquisitions.

[0062] After supplementary data acquisition, the supplementary multi-view observation data is mapped to surface parameterized coordinates. The mapping method is consistent with the aforementioned resampling, still using pixel coordinates to ellipsoidal parameterized coordinates, and alignment is completed on the surface parameterized coordinate grid. After mapping, the visible light residual map, near-infrared residual map, and geometric feature map are regenerated based on the mapping results. During regeneration, the grid positions corresponding to candidate defect areas are replaced or supplemented first, while other areas use the original data to reduce the overall computational load and maintain the update focus. When regenerating the visible light residual map, the visible light image corresponding to the preset polarization state is reselected according to the polarization state obtained from the supplementary acquisition, and the difference is calculated using the predicted visible light surface mapping map output by the normal appearance model. When regenerating the near-infrared residual map, the difference is calculated using the near-infrared image mapping result obtained from the supplementary acquisition and the predicted near-infrared surface mapping map output by the normal appearance model.

[0063] When regenerating the geometric feature map, the visible light surface map of the newly added illumination direction is incorporated into the brightness difference and gradient magnitude calculations, making the geometric feature map more sensitive to shadow changes. After regeneration, low-rank sparse decomposition is performed again on the regenerated visible light residual map, near-infrared residual map, and geometric feature map to obtain an updated sparse anomaly map. Connected component analysis and cross-view consistency checks are then performed again. If the updated candidate defect region meets the cross-view consistency condition, it proceeds to the next stage to output the defect mask. If the updated region still does not meet the cross-view consistency condition, it is marked as an uncertain region, and further acquisition or direct removal is selected based on a preset strategy. Through this supplementary acquisition and update process, defect judgment does not rely on the randomness of a single acquisition, but rather gradually stabilizes the sparse anomaly map through supplementary evidence, thereby improving the reliability of candidate defect regions.

[0064] The defect mask is determined based on the updated sparse anomaly map, and the defect type is determined based on the defect mask and the near-infrared scattering anomaly map.

[0065] In one embodiment, after the sparse anomaly map is updated, it is used as the primary basis for defect localization. First, a defect mask is generated on the surface parameterized coordinates, and then defect type determination is performed under the constraints of the defect mask. The defect mask is used to identify the area where defects may exist on the surface of agricultural products and serves as the spatial result for subsequent output. Defect type determination uses the area defined by the defect mask as the calculation object, introducing a near-infrared scattering anomaly map as auxiliary information to distinguish different defect mechanisms. The near-infrared scattering anomaly map can reflect the response changes of near-infrared and visible light to differences in tissue structure, water content, and epidermal scattering, and is suitable for distinguishing defects such as rot and mold. The final output includes the defect mask, defect type, and severity quantification result. The severity quantification result uses the ratio of the defect mask area to the effective surface area, facilitating classification and subsequent processing.

[0066] Defect types include rot, insect infestation, and mold. Determination of defect types includes: rot is identified when the average value of the near-infrared scattering anomaly map within the corresponding area of ​​the defect mask exceeds a first threshold; insect infestation is identified when the average value of the geometric feature map within the corresponding area of ​​the defect mask exceeds a second threshold; and mold is identified when the average value of the visible light residual map within the corresponding area of ​​the defect mask exceeds a third threshold and the average value of the near-infrared scattering anomaly map within the corresponding area of ​​the defect mask is less than a fourth threshold. Simultaneously with outputting the defect type, the ratio of the defect mask area to the effective area corresponding to the surface parameterized coordinates is output as a severity quantification result.

[0067] In one embodiment, the present invention provides rules for generating a defect mask and determining the defect type. The added limitation is that the defect type is limited to rot, insect infestation, and mold, and the determination is based on the relationship between statistics and thresholds within the corresponding region of the defect mask. The defect mask generation process is completed on surface parameterized coordinates. First, thresholding processing is performed on the updated sparse anomaly map, resulting in a binary anomaly mask. The threshold can be determined based on the quantiles of the sparse anomaly map within the effective region of the agricultural product, and its upper and lower bounds are limited within a preset range to ensure the stability of data from different batches.

[0068] After generating the binary anomaly mask, morphological closing operations are performed to fill small holes, followed by morphological opening operations to remove isolated small noise points. Then, connected component analysis is performed on the binary anomaly mask, retaining connected components that satisfy the minimum area threshold to obtain the defect mask. The defect mask can consist of multiple connected regions, each corresponding to a defect region. For each defect region, the area, perimeter, aspect ratio of the bounding rectangle, and compactness are calculated to exclude regions that clearly do not conform to the defect morphology. If a defect region is located near an invalid boundary of the surface parametric coordinates, boundary clipping or direct removal is performed on the defect region to avoid false defects caused by mapping errors.

[0069] Defect type determination is performed within the area defined by the defect mask. First, the average values ​​of the near-infrared scattering anomaly map, the average value of the geometric feature map, and the average value of the visible light residual map are calculated within the corresponding area of ​​the defect mask. The average value calculation uses only the effective grid positions within the corresponding area of ​​the defect mask, and extreme values ​​are truncated to avoid individual anomalies affecting the overall statistics. Defect types include decay, insect infestation, and mold. The rule for determining decay is: when the average value of the near-infrared scattering anomaly map within the corresponding area of ​​the defect mask is greater than a first threshold, the defect type is determined to be decay. This rule utilizes the characteristic that decayed areas show more obvious scattering response anomalies in the near-infrared channel. The rule for determining insect infestation is: when the average value of the geometric feature map within the corresponding area of ​​the defect mask is greater than a second threshold, the defect type is determined to be insect infestation. This rule utilizes the fact that insect-infested areas typically form local depressions, holes, or abrupt edge changes, thus producing stable morphological and shadow differences under multiple illumination directions. The rule for determining mold is as follows: when the average value of the visible light residual image within the corresponding area of ​​the defect mask is greater than the third threshold and the average value of the near-infrared scattering anomaly image within the corresponding area of ​​the defect mask is less than the fourth threshold, the defect type is determined to be mold. This rule utilizes the characteristic that mold is obvious in visible light texture and color, but its near-infrared scattering anomaly may not be significant. If the same defect area meets multiple rules simultaneously, the decision is executed according to the preset priority. The priority can be set to first determine insect infestation, then rot, and finally mold, to avoid morphologically abrupt defects being covered by scattering anomalies. If the defect area does not meet any rule, the defect type is marked as other anomalies, and the defect mask output is retained.

[0070] The first, second, third, and fourth thresholds are obtained through offline statistical analysis of labeled samples and are embedded in a parameter table during equipment deployment. This parameter table can be configured separately for each agricultural product variety. To ensure the transferability of the thresholds, the average values ​​can be normalized to the background statistics of the effective area of ​​the agricultural product before being compared with the threshold. Severity quantification results are generated simultaneously with the output defect type. The severity quantification result is the ratio of the defect mask area to the effective area corresponding to the surface parameterized coordinates. The effective area is the total number of effective grid positions in the surface parameterized coordinate grid multiplied by the area of ​​a single grid. The area of ​​a single grid can be approximated by the local area element of the ellipsoidal parameterized mapping, or replaced by the proportion of grid positions in engineering implementation. The severity quantification results allow for the classification of defect severity, for example, dividing the ratio into three intervals: mild, moderate, and severe. The classification results and defect types are output together for subsequent sorting or quality management.

[0071] In one specific embodiment, a Fuji apple is used as the object of study, and the defect types include rot, insect infestation, and mold. The acquisition end includes a visible light camera and a near-infrared camera, installed in the same field of view; the visible light channel is equipped with a linear polarizer and supports switching between co-polarization and cross-polarization states; the illumination end is equipped with three sets of directional visible light sources, with azimuth angles of 0 degrees, 120 degrees, and 240 degrees respectively at the same viewing angle, and each set of sources can be lit individually; the support end is equipped with a controllable turntable to achieve multi-view acquisition. The camera resolution is set to 1280×720, and the exposure time and gain level are fixed. Six viewing angles are acquired for each sample, with turntable angles of 0 degrees, 60 degrees, 120 degrees, 180 degrees, 240 degrees, and 300 degrees respectively; for each viewing angle, one frame of cross-polarized visible light image, one frame of co-polarized visible light image, three frames of visible light images in the three illumination directions, and one frame of near-infrared image are acquired, forming a data packet for that viewing angle. The six viewing angle data packets are merged to form the multi-view observation data for a single sample. To ensure data scale consistency across batches, a diffuse reference image was acquired before each batch began, under the same exposure and light source settings. The average grayscale of the effective area of ​​the reference image was used as the calibration value for that batch.

[0072] The reference plate calibration employs a two-step method: gain correction + intensity normalization. The formula for gain correction is: in, This is the gain coefficient; The target grayscale value is set to 200 in this embodiment; This represents the average grayscale of the effective area of ​​the reference plate. Taking the first batch as an example, the cross-polarized visible light channel... ,get If the original grayscale value of a pixel in this batch is 120, then the grayscale value after gain correction is 120 × 1.0652 = 127.828. Intensity normalization is used to align the effective areas of different viewpoints within the same batch to a unified dynamic range. Percentile truncation is used to suppress the effects of highlights and dark noise: 1% quantile values ​​are calculated within the effective area. and 99th percentile For the grayscale after gain correction Linear normalization is performed to obtain The calculation formula is: in, This is the truncation function. The process is executed independently for the three channels: cross-polarized visible light, co-polarized visible light, and near-infrared light. Figure 2 To obtain the reference plate calibration stability statistics, diffuse reflection reference plate images were collected for each batch, and the average gray level of the effective area of ​​the reference plate was statistically analyzed. Batch stability curves for the three channels were plotted before and after calibration, with the horizontal axis representing the batch number and the vertical axis representing the average gray level of the reference plate. Before calibration, the average gray level of the cross-polarized visible light channel across 12 batches ranged from approximately 187.7523 to 223.6399, with a standard deviation of approximately 10.1004; after calibration, the range was approximately 198.4561 to 203.2486, with a standard deviation of approximately 1.0800. The standard deviation of the same-polarized visible light channel before calibration was approximately 12.5633, and after calibration, it was approximately 1.7822; the standard deviation of the near-infrared channel before calibration was approximately 6.8141, and after calibration, it was approximately 1.8711. This ensures that subsequent residual construction and anomaly decomposition have comparable numerical scales across batches.

[0073] In the multi-view alignment stage, agricultural product region segmentation is first performed on the image from each viewpoint to obtain the agricultural product outline. An ellipsoid is fitted based on the outline of each viewpoint, and the ellipsoid parameters are used as the surface parameterization coordinate system. A mapping from pixel coordinates to ellipsoid parameter coordinates is established, and the data from each viewpoint is resampled to a unified surface mapping grid to obtain cross-view aligned surface mapping data. This alignment result is directly used as input for normal appearance modeling, enabling the aggregation and calculation of observations of the same surface location from different viewpoints.

[0074] In the normal appearance modeling stage, normal appearance models are constructed on the surface parametric coordinate grid for both visible and near-infrared light. To avoid bias in the model caused by defect regions, robust smooth regression is used: using the local neighborhood of the surface grid as a window, weighted regression is performed on cross-view observations within the window, and outliers are reduced in weight, outputting the predicted intensity at that grid location. The predicted visible light image and predicted near-infrared image are output from the normal appearance model, and the difference is calculated with the aligned actual observations to generate visible light residual maps and near-infrared residual maps. The formula for calculating the residuals is: in, The residual value; The normalized observation intensity; This represents the predicted intensity at the same surface location for a normal appearance model. For example, the intensity at a surface location on cross-polarized visible light. Normal appearance model prediction The visible light residual is The geometric feature map is calculated from visible light images of three illumination directions: first, the pairwise differences of the brightness in the three directions at the same surface location are calculated and the average of the absolute values ​​is taken to obtain the brightness difference value; then, the brightness gradient magnitude is calculated on the surface mapping grid and the local average is taken; the two are fused according to a preset weight to obtain the geometric feature value. The near-infrared scattering anomaly map is used to characterize the abnormal scattering deviation of near-infrared light relative to visible light. First, the intensity of near-infrared light and visible light under a preset polarization state are normalized to the same scale, and then the absolute value of the difference between the two is taken to obtain the scattering anomaly intensity, which serves as one of the key bases for subsequent defect type identification.

[0075] In the anomaly extraction stage, the visible light residual image, near-infrared residual image, and geometric feature image are concatenated on the surface parameterized grid to form a feature matrix by channel. The feature matrix is ​​organized as "surface position × feature channel," with each row corresponding to a three-channel feature vector of a surface position. Low-rank sparse decomposition is performed on the feature matrix, with the decomposition target taking the form of "low-rank baseline + sparse anomaly." in, The characteristic matrix; It is a low-rank baseline matrix used to absorb background terms that vary over a wide range; This is a sparse anomaly matrix, used to highlight local defect anomalies; For nuclear norm; It is the sum of the absolute values ​​of the elements; As the weighting coefficient, this embodiment takes... , For the number of surface positions, Let be the number of channels. When using alternating updates, the sparse term update can be implemented using a soft threshold operator: in, This is the current residual matrix; The threshold is used. The obtained sparse anomaly matrix is ​​converged into sparse anomaly intensities by channel and mapped back to the surface mesh to form a sparse anomaly map. Thresholding is performed on the sparse anomaly map and connected component analysis is conducted to obtain a set of candidate defect regions.

[0076] To suppress false anomalies caused by shadow edges and specular residue, a cross-view consistency determination is introduced. For each candidate defect region, the region is back-projected to at least two different viewpoints, then mapped back to the surface parameterized mesh, and the overlap area ratio of the two mapped regions is calculated. in, This represents the percentage of overlapping areas. The intersection area; Let be the area of ​​the union. In this embodiment, the consistency threshold is set to 0.35. Supplementary data collection and updates are triggered periodically. Figure 3 To establish cross-view consistency and supplementary acquisition trigger statistics, the cross-view overlap area ratio of candidate defect regions was calculated for all samples to determine whether supplementary acquisition was triggered. Trigger ratios were statistically analyzed by category and plotted as a bar chart, with the horizontal axis representing sample category and the vertical axis representing the trigger ratio. Based on 240 samples, the trigger ratio was approximately 0.8125 for normal samples, 0.25 for moldy samples, 0.15 for insect-infested samples, and 0.0333 for rotten samples. This trend indicates that consistency determination is primarily used to screen out unstable false anomalies and to introduce uncertain samples into the supplementary observation closed loop. Supplementary multi-view observation data preferentially uses visible light images acquired by switching illumination directions, followed by cross-polarized visible light images acquired from adjacent perspectives. After the supplementary data undergoes the same correction, normalization, and surface parameterization mapping, visible light residuals, near-infrared residuals, and geometric features are regenerated. Low-rank sparse decomposition is then performed again to update the sparse anomaly map, thereby converging candidate defect regions with supplementary information.

[0077] In the output stage, a defect mask is determined based on the updated sparse anomaly map. The defect mask is obtained by thresholding the sparse anomaly intensity and then removed by morphological opening and closing operations to remove isolated noise and fill small holes, ensuring mask coherence. The severity quantification result is expressed as the ratio of the defect mask area to the effective surface area. in, This is a highly quantified result; The area of ​​the defect mask; The effective surface area. Figure 4The distribution chart of defect severity quantification results is generated by calculating the severity quantification result (the ratio of defect mask area to effective area) for all samples, statistically analyzing its cumulative distribution by category, and plotting a cumulative curve. The horizontal axis represents the severity quantification result, and the vertical axis represents the cumulative proportion. Based on 240 samples, the median severity quantification result for rotten samples is approximately 0.0635, for insect-eaten samples approximately 0.0330, for mold approximately 0.0295, and for normal samples approximately 0.0062. The four cumulative curves are clearly separated, reflecting the trend of severity quantification results in distinguishing categories. Defect type discrimination is performed within the defect mask-defined area: the mean near-infrared scattering anomaly intensity, the mean geometric feature, and the mean visible light residual are calculated and compared with preset thresholds to distinguish between rot, insect-eaten, and mold; simultaneously, the severity quantification result is output as a grading basis.

[0078] The evaluation of the testing effectiveness used a test set of 240 samples: 60 rotten, 60 insect-infested, 40 moldy, and 80 normal. The evaluation indicators were... The value is calculated using the following formula: in, For accuracy; Recall rate; To ensure the correct number is detected; This represents the number of false positives. This represents the number of missed detections. Taking the decay category as an example, if 56 were correctly detected, 4 were falsely detected, and 4 were missed, then... , ,get . Figure 5 To generate a statistical chart of the detection results, the defect type is output for each sample in the test set, and the macro-average of each category is calculated. The value, and compare this embodiment with the control method. The results were plotted as a statistical chart, showing the differences between the control method and the method of this embodiment in terms of rot, insect infestation, mold, and macro-average. Value comparison: The control methods yielded values ​​of approximately 0.83, 0.79, 0.74, and 0.79, respectively, while the method in this embodiment yielded values ​​of approximately 0.93, 0.91, 0.89, and 0.91, respectively. Combined with... Figure 2 Input stabilization trend Figure 3 Consistent closed-loop triggering trend, Figure 4 The separation trend in severity distribution verifies that the process has a consistent data support chain in batch drift, pseudo-anomaly suppression, and hierarchical quantization, ultimately reflected in... Figure 5 Overall detection and classification performance improved.

[0079] The consistency threshold controls the trigger boundary for determining whether candidate defect regions require supplementary data collection. The overlap area ratio is calculated by dividing the intersection area by the union area, denoted as... In engineering implementation, the consistency threshold is denoted as... ,when Supplementary data collection is triggered at certain times. Consideration should be given. Figure 3 The category distribution trend shown is intended to avoid over-triggering resampling for defective samples while maintaining a high correction intensity for normal samples. It is recommended to select a value within the range of 0.30 to 0.40; this embodiment uses... When deploying to new production areas or new batches of varieties, a small-sample calibration method can be used to determine the optimal production location. Without changing the algorithm structure, statistically analyze a set of known normal samples. The distribution is used, and the 25th percentile is selected as the candidate. Then, a small number of defective samples are used to verify that the trigger ratio for supplementary sampling does not exceed the preset upper limit. If the trigger ratio for defective samples is too high, the limit will be lowered. If there are too many false positives for normal samples, the adjustment will be increased. This continues until a balance is reached between supplementary sampling triggering and false positives / false negatives. This configuration is consistent with... Figure 3 The data trend is consistent with the statement that "the proportion of normal triggers is significantly higher than that of defect categories".

[0080] The upper limit on the number of supplementary data collections is used to control the resource boundary of closed-loop updates. The upper limit on the number of supplementary data collections for a single candidate defect region is denoted as... The maximum number of total re-collection rounds for a single sample is denoted as... In engineering practice, a "two-level upper limit" is recommended to avoid infinite iterations caused by extreme samples: for the same candidate defect region, a maximum of [number] triggers should be allowed. This is a supplementary sampling update; for the same sample, it can trigger a maximum of [number] times. The system continuously updates the sparse anomaly map. Once the upper limit is exceeded, the defect mask is directly output using the currently updated sparse anomaly map, and the model is then used for type determination. This embodiment recommends using... , The reason is that Figure 3 The supplementary sampling triggers shown are mainly concentrated on samples with insufficient consistency, and the purpose of supplementary sampling is to "supplement visual or illumination evidence" rather than to continuously optimize the model; usually, a single supplementary sampling can significantly change the model. The distribution and convergence of candidate regions are achieved through a second sampling process to handle extreme cases such as strong reflections and local occlusion. The priority of the sampling strategy can be configured as follows: prioritize sampling a set of three-directional visible light images by switching the illumination direction; if... Still below Furthermore, if the candidate region is close to a high-reflectivity zone, the polarization state is switched to acquire a cross-polarized visible light image. If this is still not satisfied, cross-polarized visible light images from adjacent viewpoints are acquired to enhance geometric consistency evidence. By limiting the acquisition path to the above three types, it is ensured that the acquired data and the original multi-view observation data can be directly mapped and updated within the same data structure, avoiding the introduction of additional sensing information incompatible with this process.

[0081] Severity grading intervals are used to translate severity quantification results into executable sorting actions. The severity quantification result is denoted as... ,in, For the defect mask area, This represents the effective surface area. (Combined with...) Figure 4 The cumulative distribution curves of different categories can be used to... The project is divided into three levels to support engineering decisions: Level 1, minor defect range, is... The range of secondary moderate defects is: The range of level three severe defects is: The basis for this classification is the normal sample. The median values ​​for insect infestation and mold are mainly concentrated below 0.015, with the median for rotten samples falling between 0.02 and 0.04. The median and upper quartile of rotten samples are significantly higher than 0.045. In engineering implementation, the three-level intervals can be linked to actions such as "sellable," "downgraded," and "rejected." If the business focuses more on food safety, stricter thresholds can be set for the three-level intervals of rotten samples, for example, lowering the rejection threshold to 0.04 to reduce the risk of missed detections. This classification does not change the algorithm output, only the subsequent use of the output results, and is consistent with... Figure 4 The distribution shows a consistent trend of separation.

[0082] The defect type discrimination threshold is used to map statistics within the defect mask area to three categories: rot, insect infestation, and mold. In engineering implementation, the mean value of near-infrared scattering anomalies is denoted as... Let the mean of the geometric features be denoted as The mean of the visible light residual is denoted as Set four configurable thresholds. , , , And output the type according to the following rules: when Output decay; when and Output of insect infestation; when and Output mold growth in real time. To avoid threshold drift between different batches, you can... , , The calculation is performed on a unified scale after reference plate calibration and intensity normalization, ensuring that the threshold has cross-batch portability. This embodiment recommends an initial value of [value to be filled in]. , , , When deploying to new fruits or new near-infrared bands, the threshold calibration should prioritize the following order: first separate rotten fruit, then insect infestation, and finally mold: begin by determining the threshold using a small number of rotten samples. To achieve a predetermined lower limit for the rotten recall rate; then adjust using insect-infested samples. Suppressing false positives from insect infestation; finally, adjust using moldy samples. and Ensure the boundary between mold and mild decay is stable.

[0083] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for detecting defects in agricultural products based on machine vision, characterized in that, The method includes: Acquire multi-view observation data of agricultural products, including visible light images with different polarization states, visible light images with different illumination directions, and near-infrared images; The surface parameterized coordinates are determined based on the multi-view observation data. A normal appearance model is constructed based on the surface parameterized coordinates, and visible light residual map, near-infrared residual map, geometric feature map and near-infrared scattering anomaly map are generated. The visible light residual map, the near-infrared residual map, and the geometric feature map are subjected to low-rank sparse decomposition to obtain a sparse anomaly map and determine candidate defect regions. When the candidate defect regions do not meet the cross-view consistency condition, supplementary multi-view observation data is obtained and the supplementary multi-view observation data is mapped to the surface parameterized coordinates to update the sparse anomaly map. The defect mask is determined based on the updated sparse anomaly map, and the defect type is determined based on the defect mask and the near-infrared scattering anomaly map.

2. The method according to claim 1, characterized in that, The visible light images with different polarization states include visible light images acquired under cross polarization and visible light images acquired under the same polarization state. The visible light images with different illumination directions include visible light images acquired under illumination from at least three different illumination directions at the same viewing angle.

3. The method according to claim 1, characterized in that, Before acquiring the multi-view observation data of the agricultural products, a diffuse reflection reference plate reference image is obtained; After acquiring the multi-view observation data of the agricultural products, the multi-view observation data are subjected to gain correction and intensity normalization based on the reference image of the diffuse reflection reference plate.

4. The method according to claim 1, characterized in that, Determining the surface parameterized coordinates includes: The agricultural product region segmentation is performed on the multi-view observation data to obtain the outlines of agricultural products from each viewpoint. An ellipsoid is fitted based on the outline of the agricultural product; Establish a parametric coordinate mapping from pixel coordinates to the ellipsoid to obtain the surface parametric coordinates.

5. The method according to claim 4, characterized in that, Constructing the normal appearance model includes: The multi-view observation data is resampled to a surface mapping map based on the surface parameterized coordinates. The surface mapping is subjected to smooth regression to obtain the normal appearance model.

6. The method according to claim 1, characterized in that, Generating the visible light residual map includes: selecting a visible light image corresponding to a preset polarization state, and generating the visible light residual map based on the difference between the predicted visible light image output by the normal appearance model and the visible light image corresponding to the preset polarization state; Generating the near-infrared residual map includes: generating the near-infrared residual map based on the difference between the predicted near-infrared image output by the normal appearance model and the near-infrared image; Generating the geometric feature map includes: calculating the brightness difference and gradient magnitude based on the visible light images with different illumination directions and generating the geometric feature map; Generating the near-infrared scattering anomaly map includes: normalizing the intensity of the near-infrared image and the visible light image corresponding to the preset polarization state on the surface parameterized coordinates, and generating the near-infrared scattering anomaly map based on the normalized intensity difference.

7. The method according to claim 6, characterized in that, Performing the low-rank sparse decomposition includes: The visible light residual map, the near-infrared residual map, and the geometric feature map are spliced ​​together by channel on the surface parameterized coordinates to form a feature matrix; The feature matrix is ​​decomposed into a low-rank baseline matrix and a sparse anomaly matrix; The sparse anomaly map is generated based on the sparse anomaly matrix.

8. The method according to claim 2, characterized in that, Determining the candidate defect regions includes performing connected component analysis on the sparse anomaly graph to obtain a set of candidate defect regions. The cross-view consistency condition includes the candidate defect region having an overlap area ratio greater than an overlap ratio threshold after being mapped to the surface parameterized coordinates from at least two different viewpoints.

9. The method according to claim 6, characterized in that, The supplementary multi-view observation data includes at least one of the following: acquiring visible light images by switching polarization states, acquiring visible light images by switching illumination directions, and acquiring images from adjacent viewpoints. Updating the sparse anomaly graph includes: Map the supplementary multi-view observation data to the surface parameterized coordinates; The visible light residual map, the near-infrared residual map, and the geometric feature map are regenerated based on the mapping results. The low-rank sparse decomposition is performed again on the regenerated visible light residual map, the near-infrared residual map, and the geometric feature map to update the sparse anomaly map.

10. The method according to claim 1, characterized in that, The defect types include rot, insect infestation, and mold; the determination of the defect types includes: When the average value of the near-infrared scattering anomaly map within the corresponding area of ​​the defect mask is greater than a first threshold, the defect type is determined to be rot. When the average value of the geometric feature map within the corresponding area of ​​the defect mask is greater than a second threshold, the defect type is determined to be insect infestation. When the average value of the visible light residual image in the corresponding area of ​​the defect mask is greater than the third threshold and the average value of the near-infrared scattering anomaly image in the corresponding area of ​​the defect mask is less than the fourth threshold, the defect type is determined to be mold. And while outputting the defect type, the ratio of the defect mask area to the effective area corresponding to the surface parameterized coordinates is output as the severity quantification result.

Images