Intelligent appearance defect detection method and system for agricultural products

Abstract

The invention relates to the field of appearance defect detection, in particular to an intelligent appearance defect detection method and system for agricultural products, and the method comprises the steps: enabling a collected agricultural product point cloud image to pass through a point cloud branch constructed by a Point Transform network architecture based on embedded curvature similarity to generate a browning morphology feature map; calculating a browning index and an enhanced texture of the collected agricultural product color image, and performing image branching on the browning index and the enhanced texture to generate a browning texture feature map; and performing cross-modal feature fusion on the browning morphology feature map and the browning texture feature map to generate an early browning region segmentation map of the agricultural product. According to the method, the curvature similarity and the color texture defect features are combined to avoid early browning misinformation of a single feature, accurate recognition and positioning of the early browning defect area of the pixel-level agricultural product are achieved, then transition of early appearance defect detection of the agricultural product from visible to microcosmic is achieved, and support is provided for unmanned sorting.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of appearance defect detection, and in particular to an intelligent appearance defect detection method and system for agricultural products. BACKGROUND

[0002] The appearance quality of fruits, vegetables and other agricultural products is a key indicator of their market value and safety. Appearance defects, especially early browning caused by diseases, mechanical damage or physiological disorders, not only affect product appearance and consumer acceptance, but also are an early sign of spoilage, significantly shortening shelf life and causing significant postharvest losses. Therefore, developing automated detection technology for early browning appearance defects of agricultural products is of great significance for ensuring food safety, improving the commercialization rate of agricultural products and reducing economic losses.

[0003] Currently, automated detection based on machine vision is the mainstream technology for agricultural product sorting. However, traditional methods rely on two-dimensional color image analysis, while early browning areas of agricultural products often exhibit weak color changes or very fine texture changes on two-dimensional images, with low contrast compared to normal areas, and are easily affected by factors such as uneven lighting, background interference, and changes in fruit color and texture, resulting in low recognition accuracy and poor robustness of algorithms based on threshold segmentation, traditional texture features or shallow machine learning, making it difficult to achieve reliable early warning.

[0004] Moreover, traditional methods rely solely on two-dimensional color image analysis and cannot effectively perceive the three-dimensional surface topography defects of slight depressions, wrinkles, and roughness changes on the surface of objects, which are exactly the important manifestations of early browning or damage, such as increased surface roughness caused by cell collapse. Therefore, the use of single modality information limits the upper limit of detection capability.

[0005] Three-dimensional point cloud technology acquires precise three-dimensional coordinate information of the surface of an object through laser scanning, which can reflect the geometric details and spatial structure of the surface, providing a new approach for detecting three-dimensional surface topography defects. However, direct application of point cloud technology to early browning detection of agricultural products faces the challenge of how to effectively extract and quantify three-dimensional geometric features highly related to early browning and establish a strong correlation model between them and the degree of browning.

[0006] In summary, how to detect the appearance defects of early browning of agricultural products by combining three-dimensional point cloud detection of three-dimensional surface topography defects and two-dimensional color image detection of surface color and texture defects to achieve accurate early browning area segmentation is a technical problem to be solved at present. SUMMARY

[0007] To this end, the present application provides an intelligent appearance defect detection method and system for agricultural products, which constructs a point cloud branch through a Point Transformer network embedded with curvature similarity to accurately extract surface geometric roughness features highly related to browning, effectively perceive three-dimensional surface topography defects of early browning of agricultural products, combine uncertainty perception, geometric guidance and cross-modal attention to establish the association between color image texture features and curvature up-sampling to establish color texture defect features and roughness features, avoid false positives of early browning of single features, realize accurate identification and positioning of pixel-level early browning defect areas of agricultural products, and further enable the transition of early appearance defect detection of agricultural products from visible to microscopic, thereby providing support for unmanned sorting.

[0008] To achieve the above-mentioned purpose, the present application provides an intelligent appearance defect detection method for agricultural products, comprising:

[0009] The collected point cloud image of agricultural products is processed through a point cloud branch constructed based on a Point Transformer network embedded with curvature similarity to generate a browning topography feature map, wherein the curvature similarity is adjusted based on the variety of the agricultural products;

[0010] The collected color image of the agricultural products is calculated to obtain a browning index and enhanced texture, and the browning index and enhanced texture are processed through an image branch constructed based on a UNet network architecture to generate a browning texture feature map;

[0011] The browning topography feature map and the browning texture feature map are processed through a gating fusion mechanism, a cross-attention mechanism and a decoder of a cross-modal feature fusion model to generate an early browning region segmentation map of the agricultural products.

[0012] Further, the process of generating the browning topography feature map through the point cloud branch based on the collected point cloud image of the agricultural products comprises:

[0013] A dynamic radius is determined based on the variety of the agricultural products in the point cloud image;

[0014] Gaussian curvature, average curvature and directional curvature are calculated based on local point cloud data within the dynamic radius of the detected point cloud in the point cloud image of the agricultural products;

[0015] The curvature similarity is calculated based on the Gaussian curvature, the average curvature and the directional curvature;

[0016] The curvature similarity and the attention generation value are fused to generate the browning topography feature map.

[0017] Further, the process of calculating the Gaussian curvature, the average curvature and the directional curvature based on the local point cloud data within the judgment radius of the detected point cloud in the point cloud image of the agricultural products comprises:

[0018] computing a covariance matrix based on the local point cloud data;

[0019] performing matrix eigen decomposition based on the covariance matrix to generate a plurality of eigenvalues;

[0020] calculating a surface shape ratio of the agricultural product based on a plurality of the eigenvalues, generating the Gaussian curvature, mean curvature and directional curvature.

[0021] Further, the process of calculating curvature similarity based on the Gaussian curvature, mean curvature and directional curvature comprises:

[0022] calculating a query point curvature feature and a key point curvature feature based on the Gaussian curvature, mean curvature and directional curvature;

[0023] generating a gating weight based on the query point curvature feature through a learnable parameter and a sigmoid activation function;

[0024] calculating a distance based on the query point curvature feature and the key point curvature feature;

[0025] determining the curvature similarity based on a product of the gating weight and an exponential function of the distance.

[0026] In particular, by combining the Gaussian curvature, mean curvature and directional curvature for point cloud attention detection, the detection of concave and wrinkle type defects is improved, early browning topography defect detection including bitter pustule disease and brown spot disease is realized, and various agricultural product varieties can be adapted, realizing microscopic agricultural product appearance defect detection.

[0027] Further, the process of generating an early browning region segmentation map of the agricultural product by the cross-modal feature fusion model based on the browning topography feature map and the browning texture feature map comprises:

[0028] performing convolution operation based on the splicing feature of the browning topography feature map and the browning texture feature map to generate a suspicious feature matrix, and performing gating fusion mechanism based on the browning topography feature map and the browning texture feature map through the suspicious feature matrix to generate a fusion feature map;

[0029] generating a key vector and a value vector based on the fusion feature map, generating a query vector based on the browning texture feature map, generating a standard attention value through attention calculation based on the cross-attention mechanism of the key vector and the query vector, and generating an attention feature through weighted fusion based on the standard attention value, the value vector and the suspicious feature matrix;

[0030] generating the early browning region segmentation map of the agricultural product through the decoder based on the key vector, the attention feature, the browning topography feature map and the browning texture feature map.

[0031] Further, the process of generating the attention feature based on the standard attention value and the value vector includes:

[0032] The product of the suspicious feature matrix and the learnable weight matrix plus one is element-wise multiplied with the standard attention value to generate a modulation value, which is used to amplify the standard attention value when the suspicious feature matrix value is greater than zero;

[0033] The product of the modulation value and the value vector is taken as the attention feature.

[0034] Further, the process of generating the early browning region segmentation map of the agricultural product includes:

[0035] The key vector and the attention feature are output through an upsampling layer to generate an upsampling value;

[0036] The upsampling value, the browning topographic feature map, and the browning texture feature map are concatenated to generate a decoding feature through a first convolutional layer;

[0037] The decoding feature is generated through a second convolutional layer and a sigmoid function to generate the early browning region segmentation map of the agricultural product, wherein the convolutional kernel size of the first convolutional layer is larger than that of the second convolutional layer.

[0038] Further, the process of calculating the browning index and enhancing the texture based on the collected color image of the agricultural product includes:

[0039] The browning index is calculated based on the difference between the red component and the green component and the blue component of the color image of the agricultural product;

[0040] The coordinates of the color image of the agricultural product are filtered through a Gabor filter to generate the enhanced texture.

[0041] Further, the process of collaboratively optimizing and training the point cloud branch, the image branch, and the cross-modal feature fusion model includes:

[0042] A region segmentation loss term based on Dice loss is constructed to optimize the range of the early browning region of the agricultural product to be consistent with the sample;

[0043] A color segmentation loss term based on SSIM loss is constructed to optimize the color degree of the early browning region of the agricultural product to be consistent with the sample;

[0044] A curvature browning topography constraint term based on regularization is constructed to optimize the deformation of the early browning region of the agricultural product to be consistent with the sample;

[0045] A comprehensive loss function is constructed based on a weighted sum of the region segmentation loss term, the color segmentation loss term and the curvature brown morphology constraint term, and the comprehensive loss function is used for collaborative optimization training of the point cloud branch, the image branch and the cross-modal feature fusion model.

[0046] In particular, the sensitivity to the weak color change and the weak texture change of early brown is realized through brown index enhancement and Gabor texture enhancement. Through the gating fusion mechanism, the weight of geometric depression and texture is balanced to realize adaptive detection of brown defects in shadow and highlight areas. Through the cross-attention mechanism, the recall rate of the small brown area is improved, and the accurate positioning of the brown area boundary is realized through the decoder.

[0047] The application further provides an intelligent appearance defect detection system for agricultural products, which applies the intelligent appearance defect detection method for agricultural products.

[0048] Compared with the prior art, the application has the beneficial effects that the point cloud branch is constructed by embedding the PointTransformer network with curvature similarity to accurately extract the surface geometric roughness features highly related to brown, effectively perceive the three-dimensional surface morphology defects of early brown of agricultural products, combine uncertainty perception, geometric guidance and cross-modal attention to establish the association between color image texture features and curvature up-sampling color texture defect features and roughness features, avoid false reports of early brown of single features, realize accurate identification and positioning of the early brown defect area of agricultural products at the pixel level, and further make the early appearance defect detection of agricultural products jump from visible to microscopic, thereby providing support for unmanned sorting.

[0049] In particular, the application improves the detection of concave and wrinkled defects by combining Gaussian curvature, average curvature and directional curvature for point cloud attention detection, realizes early brown morphology defect detection including bitter pustule disease and brown spot disease, and is adaptable to various agricultural product varieties, thereby realizing microscopic appearance defect detection of agricultural products.

[0050] In particular, the sensitivity to the weak color change and the weak texture change of early brown is realized through brown index enhancement and Gabor texture enhancement. Through the gating fusion mechanism, the weight of geometric depression and texture is balanced to realize adaptive detection of brown defects in shadow and highlight areas. Through the cross-attention mechanism, the recall rate of the small brown area is improved, and the accurate positioning of the brown area boundary is realized through the decoder. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The flowchart of the intelligent appearance defect detection method for agricultural products of the embodiments of the application is shown in the figure.

[0052] Figure 2 A flowchart of a point cloud branch of the intelligent appearance defect detection method for agricultural products according to an embodiment of the present application;

[0053] Figure 3 A flowchart of a cross-modal feature fusion model of the intelligent appearance defect detection method for agricultural products according to an embodiment of the present application;

[0054] Figure 4 A flowchart of collaborative optimization training of the intelligent appearance defect detection method for agricultural products according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objects and advantages of the present application clearer, the present application will be further described in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0056] The preferred embodiments of the present application will be described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present application and are not used to limit the protection scope of the present application.

[0057] It should be noted that in the description of the present application, the terms "upper", "lower", "left", "right", "inner", "outer" and the like indicating the direction or positional relationship are based on the direction or positional relationship shown in the drawings, which is only for the convenience of description, and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present application.

[0058] In addition, it should also be noted that in the description of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the communication between the two elements inside. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0059] As Figures 1 to 4As shown, the present application provides an intelligent appearance defect detection method and system for agricultural products. A point cloud branch is constructed by embedding a Point Transformer network with curvature similarity to accurately extract surface geometric roughness features highly related to browning, effectively perceive three-dimensional surface topography defects of early browning of agricultural products, and establish the association between color image texture features and curvature up-sampling to establish color texture defect features and roughness features by combining uncertainty perception, geometric guidance, and cross-modal attention. This avoids false positives of early browning caused by a single feature and realizes accurate identification and positioning of pixel-level early browning defect areas of agricultural products, thereby enabling the transition of early appearance defect detection of agricultural products from visible to microscopic and providing support for unmanned sorting.

[0060] As shown, Figure 1 The present embodiment proposes an intelligent appearance defect detection method for agricultural products, which includes:

[0061] The collected agricultural product point cloud image is processed through a point cloud branch constructed based on a Point Transformer network architecture embedded with curvature similarity to generate a browning topography feature map, wherein the curvature similarity is adjusted based on the variety of the agricultural product.

[0062] The collected agricultural product color image is calculated to obtain a browning index and enhanced texture, and the browning index and enhanced texture are processed through an image branch constructed based on a UNet network architecture to generate a browning texture feature map.

[0063] The browning topography feature map and the browning texture feature map are processed through a gating fusion mechanism, a cross-attention mechanism, and a decoder of a cross-modal feature fusion model to generate an early browning region segmentation map of the agricultural product.

[0064] As shown, Figure 2 In the present embodiment, the process of generating a browning topography feature map from the collected agricultural product point cloud image through a point cloud branch includes:

[0065] A dynamic radius is determined based on the variety of the agricultural product.

[0066] A Gaussian curvature, an average curvature, and a directional curvature are calculated based on local point cloud data within the dynamic radius of the detected point cloud in the agricultural product point cloud image.

[0067] The curvature similarity is calculated based on the Gaussian curvature, the average curvature, and the directional curvature.

[0068] The curvature similarity and an attention generation value are fused to generate the browning topography feature map.

[0069] Specifically, the process of determining the dynamic radius can be represented as:

[0070]

[0071] In the formula, represents the dynamic radius, represents the minimum defect area of the detection target, and the apple is , represents the product variety correlation coefficient, the product variety correlation coefficient of the smooth surface of the apple is 0.8, the product variety correlation coefficient of the rough surface of the pineapple is 1.2, and the product variety correlation coefficient of the porous surface of the apple is 1.5. Therefore, the inherent defect area of different types of agricultural products is removed.

[0072] As shown in Figure 2 , in the embodiment, the process of calculating the Gaussian curvature, the average curvature, and the directional curvature based on the local point cloud data within the judgment radius of the point cloud detected in the agricultural product point cloud image includes:

[0073] calculating a covariance matrix based on the local point cloud data;

[0074] performing matrix eigenvalue decomposition based on the covariance matrix to generate a plurality of eigenvalues;

[0075] calculating the surface shape ratio of the agricultural product based on the plurality of eigenvalues to generate the Gaussian curvature, the average curvature, and the directional curvature.

[0076] Specifically, the process of calculating the covariance matrix can be represented as:

[0077]

[0078] In the formula, represents the covariance matrix of the given point cloud detection point , K represents the number of point cloud data points within the spherical detection range composed of the dynamic radius of the given point cloud detection point , represents the point cloud data within the spherical detection range, that is , , represents the centroid of the spherical detection range of the given point cloud detection point , that is .

[0079] Specifically, the process of generating a plurality of eigenvalues can be represented as:

[0080]

[0081] In the formula, represents the covariance matrix of the given point cloud detection point , represent three-dimensional feature vectors of a given point cloud detection point , is a feature vector of a given point cloud detection point , including sequentially increasing to detect topographic defects in different directions.

[0082] Therefore, by calculating the statistical feature vectors from the point cloud data, the microscopic geometric features of the surfaces of different types of agricultural products can be reflected.

[0083] Specifically, the process of calculating the Gaussian curvature, the mean curvature and the directional curvature can be represented as:

[0084]

[0085] In the formula, , , represent the Gaussian curvature, the mean curvature and the directional curvature respectively, represent three feature vectors of a given point cloud detection point .

[0086] As shown in Figure 2 , in the present embodiment, the process of calculating the curvature similarity based on the Gaussian curvature, the mean curvature and the directional curvature includes:

[0087] calculating the query point curvature feature and the key point curvature feature based on the Gaussian curvature, the mean curvature and the directional curvature;

[0088] generating a gating weight based on the query point curvature feature through a learnable parameter and a sigmoid activation function;

[0089] calculating the distance based on the query point curvature feature and the key point curvature feature;

[0090] determining the curvature similarity based on the product of the gating weight and the exponential function of the distance.

[0091] Specifically, the process of generating the gating weight can be represented as:

[0092]

[0093] In the formula, represents the gating weight, represents the sigmoid function, represents a learnable parameter. Therefore, according to the surface characteristics of the query point, the sensitivity of the point to the curvature similarity is dynamically adjusted, represents the query point curvature feature, i.e. including the Gaussian curvature, the mean curvature and the directional curvature. Therefore, the sensitivity of the current point to the curvature similarity is adjusted through the gating weight.

[0094] Specifically, the process of generating the gating weight can be represented as:

[0095] ;

[0096] In the formula, denotes the distance, denotes the query point curvature feature, denotes the key point curvature feature, wherein the query point curvature feature and the key point curvature feature are both from three feature vectors The query point curvature determines the sensitivity of the current point to the geometric change, and the key point curvature feature provides the geometric reference information of the neighborhood point, so that the relative relationship between the points can be quantified by calculating the Mahalanobis distance according to the above formula, and the greater the distance value, the more similar the geometric characteristics.

[0097] Specifically, the process of generating the curvature similarity can be represented as:

[0098]

[0099] In the formula, denotes the curvature similarity, denotes the distance, denotes the query point curvature feature, denotes the key point curvature feature, denotes the attenuation coefficient, preferably 0.8.

[0100] Specifically, the process of generating the browning topography feature map can be represented as:

[0101]

[0102] In the formula, denotes the browning topography feature map, Q K denotes the key point curvature feature, d denotes the vector dimension scaling factor, V denotes the value point curvature feature, and softmax denotes the softmax function, wherein denotes the attention generation value.

[0103] Especially, by combining the Gaussian curvature, the average curvature and the directional curvature for point cloud attention detection, the detection of concave and wrinkled defects is improved, the early browning topography defect detection including the bitter pit and the brown spot is realized, and various agricultural product varieties can be adapted, and the microscopic agricultural product appearance defect detection is realized.

[0104] In the embodiment, the process of calculating the browning index and the enhanced texture from the collected agricultural product color image includes:

[0105] The browning index is calculated based on the differences between the red component and the green and blue components of the color image of the agricultural product, respectively.

[0106] The coordinates of the agricultural product color image are filtered using Gabor to generate the enhanced texture.

[0107] Specifically, the process of calculating the browning index can be expressed as follows:

[0108]

[0109] In the formula, Indicates the browning index. These represent the red, green, and blue components of the color image of agricultural products, respectively. It should be a minimum value, preferably 0.1, to prevent the denominator from being zero.

[0110] Specifically, the process of calculating enhanced textures can be represented as:

[0111]

[0112] In the formula, T represents enhanced texture. Indicates Gabor filtering Multiple filter directions selected For example, if the agricultural product is an apple, and the browning of the fruit stem cracks is radial, then the filter directions are 30 degrees, 90 degrees, and 150 degrees. If the agricultural product is a banana, and the browning occurs along the ridge direction, then the filter directions are 0 degrees and 90 degrees. If the agricultural product is a citrus, and the browning of the oil cell ruptures spreads obliquely, then the filter directions are 45 degrees and 135 degrees.

[0113] like Figure 3 As shown, in this embodiment, the process of generating an early browning region segmentation map of agricultural products by using the browning morphology feature map and the browning texture feature map through a cross-modal feature fusion model includes:

[0114] A convolution operation is performed on the spliced ​​features of the brown morphology feature map and the brown texture feature map to generate a suspicious feature matrix. Based on the suspicious feature matrix, the brown morphology feature map and the brown texture feature map are subjected to a gated fusion mechanism to generate a fused feature map.

[0115] Based on the fused feature map, a key vector and a value vector are generated. Based on the browning texture feature map, a query vector is generated. Based on the cross-attention mechanism of the key vector and the query vector, a standard attention value is generated through attention calculation. Based on the standard attention value, the value vector, and the suspicious feature matrix, a weighted fusion is performed to generate attention features.

[0116] The decoder is used to generate the early browning region segmentation map of the agricultural product based on the key vector, the attention feature, the browning topographic feature map and the browning texture feature map.

[0117] Specifically, the process of generating the fusion feature map can be represented as:

[0118]

[0119]

[0120] In the formula, denotes the browning topographic feature map and the browning texture feature map spliced features, denotes a convolution layer using a 3x3 convolution kernel, denotes a convolution layer using a 1x1 convolution kernel, denotes an activation function, denotes a sigmoid function, and U denotes a suspicious feature matrix, denotes element-wise multiplication, denotes a fusion feature map. Thus, uncertainty-aware gating fusion is achieved, reliability compensation is performed on shadow and highlight regions, and the false detection rate is reduced.

[0121] As shown in Figure 3 , in the embodiment, the process of generating the attention feature based on the standard attention value and the value vector includes:

[0122] The product of the suspicious feature matrix and the learnable weight matrix is added to one and multiplied element-wise with the standard attention value to generate a modulation value, which is used to amplify the standard attention value when the suspicious feature matrix value is greater than zero;

[0123] The product of the modulation value and the value vector is taken as the attention feature.

[0124] Specifically, the process of generating the attention feature can be represented as:

[0125]

[0126]

[0127]

[0128] In the formula, denotes a standard attention value, Q and V respectively denote a key vector and a value vector generated based on a fusion feature map, and K denotes a query vector generated based on a browning texture feature map, denotes a vector dimension scaling factor of the standard attention, denotes element-wise multiplication, denotes a learnable weight matrix, U denotes a suspicious feature matrix, denotes a modulation value, denotes an attention feature.

[0129] Therefore, the above formula projects the suspicious region of the agricultural product to a local high-weight region through the learnable weight matrix, when is greater than zero, The value of combines the variety of the image of the agricultural product and the detection part of the agricultural product, for example, the apple stem corresponds to The element value of is greater than zero, and the potato sprout eye corresponds to The element value of is less than zero, which realizes forced attention to the suspicious region. At this time, when U is less than or equal to zero, the low-confidence region is shielded, and at this time is zero, .

[0130] Preferably, the process of generating the attention feature can be extended to a multi-head attention mechanism, and the multi-head attention can be represented as:

[0131]

[0132] In the formula, denotes an attention feature component of the hth attention head, , denote the key vector and the value vector of the hth attention head generated based on the fused feature map respectively, K denotes the query vector of the hth attention head generated based on the brown texture feature map, denotes a vector dimension scaling factor of the standard attention, denotes element-wise multiplication, denotes a learnable weight matrix of the hth attention head, and U denotes a suspicious feature matrix, denotes a spliced feature of the attention feature components of the first to hth attention heads, denotes a learnable matrix, denotes an attention feature.

[0133] Therefore, by extending the process of generating the attention feature to the multi-head attention mechanism, the multi-head attention learns the discrimination criteria of different defect types, such as apple radial cracks and orange oblique oil cell rupture, through independent parameters, thereby reducing the sensitivity of a single attention mechanism to specific noise. Meanwhile, after splicing and linear transformation, the outputs of each head can fuse multi-view feature information, thereby improving the generalization ability of the model to different varieties and different maturity of agricultural products.

[0134] As shown in Figure 3 , in the present embodiment, the process of generating the early brown region segmentation map of the agricultural product includes:

[0135] output an up-sampling value through an up-sampling layer;

[0136] generate a decoding feature through a first convolution layer on the spliced feature of the up-sampling value, the brown topographic feature map and the brown texture feature map;

[0137] generate the early brown area segmentation map of the agricultural product through a second convolution layer and a sigmoid function, wherein the convolution kernel size of the first convolution layer is larger than that of the second convolution layer.

[0138] Specifically, the process of generating the early brown area segmentation map of the agricultural product can be represented as:

[0139]

[0140] In the formula, indicates the attention feature, K indicates the key vector, indicates the up-sampling calculation, indicates the up-sampling value , the spliced feature of the brown topographic feature map and the brown texture feature map indicates the first convolution layer with a 5x5 convolution kernel, indicates the decoding feature, indicates the second convolution layer with a 1x1 convolution kernel, indicates the sigmoid function, indicates the early brown area segmentation map of the agricultural product.

[0141] Therefore, the up-sampling layer retains the coordinates of the micro-defects such as recesses and cracks for subsequent splicing operations, preventing small-size defects from being diluted in feature fusion. The first convolution layer solves the problem of point cloud feature biasing local geometry and image feature biasing global texture, providing fusion features rich in context for segmentation map generation. The second convolution layer generates a refined segmentation boundary, avoiding the edge blur problem caused by large convolution kernels and ensuring accurate positioning of early micro-brown areas.

[0142] As shown in Figure 4 In the embodiment, the process of collaboratively optimizing and training the point cloud branch, the image branch and the cross-modal feature fusion model includes:

[0143] A region segmentation loss term based on Dice loss is constructed to optimize the range of the early brown area of the agricultural product to be consistent with the sample;

[0144] A color segmentation loss term based on SSIM loss is constructed to optimize the color degree of the early brown area of the agricultural product to be consistent with the sample;​

[0145] A regularization-based curvature browning morphology constraint term is constructed to optimize the deformation of early browning regions of agricultural products and ensure consistency with the samples.

[0146] A comprehensive loss function is constructed by weighted summation of the region segmentation loss term, the color segmentation loss term, and the curvature brown deformation morphology constraint term. This comprehensive loss function is then used to perform collaborative optimization training on the point cloud branch, the image branch, and the cross-modal feature fusion model.

[0147] Specifically, the region segmentation loss term can be expressed as:

[0148]

[0149] In the formula, Represents the region segmentation loss term. This represents the ground truth mask for early browning regions in the sample's agricultural products. This represents the prediction mask for the segmentation map of early browning regions in agricultural products output by the cross-modal feature fusion model.

[0150] Specifically, the color segmentation loss term can be expressed as:

[0151]

[0152] In the formula, Represents the color segmentation loss term. These represent the mean window brightness of the sample and the output segmentation map of the cross-modal feature fusion model, respectively. These represent the window contrast variance of the sample and the output segmentation map of the cross-modal feature fusion model, respectively. This represents the feature range covariance of the segmentation map output by the sample and the cross-modal feature fusion model. , These represent the luminance stability constant and the contrast stability constant, respectively, preferably 6.5 and 58.

[0153] Specifically, the curvature brown deformation morphology loss term can be expressed as:

[0154] ,

[0155] In the formula, This represents the curvature brown deformation morphology loss term. The set of points i representing the browning regions of the sample. The normalization factor, The curvature values ​​of electrical points i in the browned region output by the point cloud branch include Gaussian curvature, mean curvature, and directional curvature. Let j represent the set of points in the normal region. curvature values including Gaussian curvature, mean curvature and directional curvature of normal region points j output by the point cloud branch, denotes the Euclidean norm, embodying regularization on the curvature prediction error, denotes the relative curvature deviation.

[0156] Especially, the sensitivity to the weak color change and texture change of early brown spots is realized through the brown index enhancement and Gabor texture enhancement. Through the gating fusion mechanism, the weight of geometric concave and texture is balanced to realize adaptive detection of brown defects in shadow and highlight areas. Through the cross attention mechanism, the recall rate of the micro brown area is improved, and the accurate positioning of the brown area boundary is realized through the decoder.

[0157] Specifically, the hyperparameters of the point cloud branch, the image branch and the cross-modal feature fusion model described in the embodiment include: point cloud resolution 0.2mm / point, number of Transformer layers 6, number of attention heads 8, curvature similarity 5-20, feature dimension 256, image branch input resolution 1024x1024, number of Gabor directions 8, Gabor wavelength [3, 5, 7], UNet depth 5, initial channel number 64, gating fusion dimension 512, number of attention heads 8, up-sampling multiple 4, first convolution kernel 5x5, second convolution kernel 1x1, skip connection weight [0.3, 0.4, 0.3], output threshold of decision segmentation map 0.3, training hyperparameters: learning rate 3e-4, batch size 16, optimizer AdamW.

[0158] The embodiment also provides an intelligent appearance defect detection system for agricultural products, which applies the intelligent appearance defect detection method for agricultural products.

[0159] In this embodiment, the point cloud branch is constructed by embedding the Point Transformer network with curvature similarity to accurately extract the surface geometric roughness features highly related to browning, effectively perceive the three-dimensional surface topography defects of early browning of agricultural products, and establish the association between the color texture defect features and the roughness features by combining uncertainty perception, geometric guidance and cross-modal attention, and by upsampling the curvature to avoid false positives of early browning caused by a single feature, so as to accurately identify and locate the pixel-level early browning defect area of agricultural products, and further enable the early appearance defect detection of agricultural products to jump from visible to microscopic, thereby providing support for unmanned sorting. In particular, by combining the Gaussian curvature, average curvature and directional curvature for point cloud attention detection, the detection of concave and wrinkled defects is improved, early browning topography defect detection including bitter pustule and brown spot is realized, and the method is suitable for various agricultural product varieties, thereby realizing microscopic appearance defect detection of agricultural products. In particular, the sensitivity to weak color changes and texture changes of early browning is realized through browning index enhancement and Gabor texture enhancement. Through the gating fusion mechanism, the weights of geometric concave and texture are balanced, and the browning defects can be adaptively detected in shadow and highlight areas. Through the cross-attention mechanism, the recall rate of the small browning area is improved, and the accurate positioning of the browning area boundary is realized through the decoder.

[0160] Those skilled in the art can appreciate that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0161] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to related technical features without departing from the principles of the present application, and the technical solutions after such changes or replacements will fall within the protection scope of the present application.

[0162] The above description is only the preferred embodiments of the present application and is not intended to limit the present application; for those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for intelligent external defect detection of agricultural produce, characterized in that, The method comprises the following steps: The collected agricultural product point cloud image is input into a point cloud branch constructed based on a Point Transformer network architecture with embedded curvature similarity to generate a browning topographic feature map, wherein the curvature similarity is adjusted based on the variety of the agricultural product; The browning index and enhanced texture of the collected agricultural product color image are calculated, and the browning index and enhanced texture are input into an image branch constructed based on a UNet network architecture to generate a browning texture feature map; The browning topographic feature map and the browning texture feature map are input into a cross-modal feature fusion model to generate an early browning region segmentation map of the agricultural product.

2. The intelligent external appearance defect detection method for agricultural products according to claim 1, characterized in that, The process of inputting the collected agricultural product point cloud image into the point cloud branch to generate the browning topographic feature map comprises the following steps: Determine the dynamic radius based on the variety of the agricultural product in the agricultural product point cloud image; Calculate the Gaussian curvature, mean curvature and directional curvature based on the local point cloud data within the dynamic radius of the detected point cloud in the agricultural product point cloud image; Calculate the curvature similarity based on the Gaussian curvature, mean curvature and directional curvature; Fuse the curvature similarity and attention generation value to generate the browning topographic feature map.

3. The intelligent external appearance defect detection method for agricultural products according to claim 2, characterized in that, The process of calculating the Gaussian curvature, mean curvature and directional curvature based on the local point cloud data within the judgment radius of the detected point cloud in the agricultural product point cloud image comprises the following steps: Calculate the covariance matrix based on the local point cloud data; Perform matrix eigenvalue decomposition based on the covariance matrix to generate a plurality of eigenvalues; Calculate the surface shape ratio of the agricultural product based on a plurality of the eigenvalues to generate the Gaussian curvature, mean curvature and directional curvature.

4. The intelligent external appearance defect detection method for agricultural products according to claim 2, characterized in that, The process of calculating the curvature similarity based on the Gaussian curvature, mean curvature and directional curvature comprises the following steps: Calculate the query point curvature feature and key point curvature feature based on the Gaussian curvature, mean curvature and directional curvature; Generate the gating weight based on the query point curvature feature through a learnable parameter and a sigmoid activation function; Calculate the distance based on the query point curvature feature and the key point curvature feature; Determine the curvature similarity based on the product of the gating weight and the exponential function of the distance.

5. The intelligent external appearance defect detection method for agricultural products according to claim 1, characterized in that, The process of inputting the browning topographic feature map and the browning texture feature map into the cross-modal feature fusion model to generate the early browning region segmentation map of the agricultural product comprises the following steps: Perform convolution operation based on the spliced features of the browning topographic feature map and the browning texture feature map to generate a suspicious feature matrix, and input the browning topographic feature map and the browning texture feature map into a gating fusion mechanism based on the suspicious feature matrix to generate a fusion feature map; Generate a key vector and a value vector based on the fusion feature map, generate a query vector based on the browning texture feature map, generate a standard attention value through attention calculation based on the cross-attention mechanism of the key vector and the query vector, and generate an attention feature through weighted fusion based on the standard attention value, the value vector and the suspicious feature matrix; Input the key vector, the attention feature, the browning topographic feature map and the browning texture feature map into a decoder to generate the early browning region segmentation map of the agricultural product.

6. The intelligent external appearance defect detection method for agricultural products according to claim 5, characterized in that, The process of generating the attention feature based on the weighted fusion of the standard attention value and the value vector includes: multiplying the product of the suspicious feature matrix and the learnable weight matrix plus one with the standard attention value element by element to generate a modulation value, which is used to amplify the standard attention value when the suspicious feature matrix value is greater than zero; multiplying the modulation value and the value vector as the attention feature.

7. The intelligent external appearance defect detection method for agricultural products according to claim 6, characterized in that, The process of generating the early browning region segmentation map of the agricultural product includes: outputting an up-sampling value by up-sampling the key vector and the attention feature; generating a decoding feature by the first convolution layer based on the spliced feature of the up-sampling value, the browning topographic feature map and the browning texture feature map; generating the early browning region segmentation map of the agricultural product by the second convolution layer and the sigmoid function based on the decoding feature, wherein the convolution kernel size of the first convolution layer is larger than that of the second convolution layer.

8. The intelligent external defect detection method for agricultural products according to any one of claims 1 to 7, characterized in that, The process of calculating the browning index and the enhanced texture based on the collected color image of the agricultural product includes: calculating the browning index based on the difference between the red component and the green component and the blue component of the color image of the agricultural product, respectively; generating the enhanced texture by Gabor filtering the coordinates of the color image of the agricultural product.

9. The intelligent external defect detection method for agricultural products according to any one of claims 1 to 7, characterized in that, The process of collaboratively optimizing and training the point cloud branch, the image branch and the cross-modal feature fusion model includes: constructing a region segmentation loss term based on the Dice loss to optimize the range of the early browning region of the agricultural product to be consistent with the sample; constructing a color segmentation loss term based on the SSIM loss to optimize the color degree of the early browning region of the agricultural product to be consistent with the sample; constructing a curvature browning topography constraint term based on regularization to optimize the deformation of the early browning region of the agricultural product to be consistent with the sample; constructing a comprehensive loss function based on the weighted sum of the region segmentation loss term, the color segmentation loss term and the curvature browning topography constraint term, and using the comprehensive loss function to collaboratively optimize and train the point cloud branch, the image branch and the cross-modal feature fusion model.

10. An intelligent external appearance defect detection system for agricultural products, characterized by, The intelligent appearance defect detection system applies the intelligent appearance defect detection method for agricultural products as claimed in any one of claims 1 to 9.

Images