Agricultural product grade classification method based on image recognition and control system thereof

Abstract

The invention relates to the field of image recognition, in particular to an agricultural product grade classification method based on image recognition and a control system thereof, and the method comprises the following steps: obtaining internal quality data and depth map data of agricultural products through a multi-modal sensor, and forming multi-modal data with image data; establishing a dynamic grading model based on market rules and historical data with labels, and inputting multi-modal data to respectively obtain a size score, an internal quality score, an appearance score and a defect score; performing weighted calculation on the size score, the internal quality score, the appearance score and the defect score by using a dynamic weight distribution mechanism, and outputting a grade score; triggering a sorting instruction based on the grade score, and controlling sorting equipment to distribute the agricultural products; and generating a verifiable traceability label and pasting the traceability label on the agricultural product. According to the method, non-contact detection of internal indexes is realized through multi-modal data, and the defect that in the prior art, surface features are analyzed only by depending on images is overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to a kind of agricultural product grade classification method based on image recognition and control system thereof. BACKGROUND

[0002] Image recognition technology is an important field of artificial intelligence, which is a technology for identifying objects in an image to identify different targets or characteristics of objects in the image.Agricultural product grade classification is a key link in the agricultural industry chain, directly affecting product pricing, market circulation and consumer trust.The traditional method mainly relies on manual experience sorting, and there are problems such as low efficiency, strong subjectivity and non-uniform standards.In recent years, automatic grading technology based on image recognition has been gradually applied, but there are still the following defects: The existing technology mainly relies on image analysis of surface features, which cannot detect internal quality indicators such as sugar content, moisture and pesticide residues, resulting in a disconnect between grading results and market demand;Natural light changes and background noise can easily cause image segmentation and feature extraction errors, affecting classification accuracy;The grading process lacks transparent records, consumers cannot verify the basis for grade determination, and quality disputes are likely to arise.Therefore, it is necessary to involve a kind of agricultural product grade classification method based on image recognition and control system thereof. SUMMARY

[0003] In view of the technical defects in the background art, the present application proposes an agricultural product grade classification method based on image recognition and control system thereof, which solves the above technical problems and meets the actual needs, and the specific technical solution is as follows: The agricultural product grade classification method based on image recognition comprises the following steps: Step S1, obtain multi-angle image data of agricultural products through array industrial camera, normalize and background segmentation process multi-angle image data based on real-time collected environmental light data, obtain internal quality data and depth map data of agricultural products through multi-modal sensor and form multi-modal data with image data; Step S2, establish a dynamic grading model based on market rules and labeled historical data, the dynamic grading model includes size regression branch, internal quality branch, appearance branch and defect branch, input multi-modal data to obtain size score, internal quality score, appearance score and defect score respectively; Step S3, based on the emphasis score points of different agricultural products, use dynamic weight distribution mechanism to weight calculate size score, internal quality score, appearance score and defect score, and output grade score; Step S4, trigger sorting instruction based on grade score, control sorting equipment to sort agricultural products; Step S5, store the grade score and multi-modal data association to the blockchain network, generate a verifiable traceability label attached to the agricultural products.

[0004] Further, the specific steps of the normalization processing and background segmentation processing in step S1 include: Based on real-time ambient light data, use Retinex algorithm to dynamically supplement multi-angle image data, enhance image contrast through adaptive histogram equalization, align multi-angle image data based on spatial position parameters of array industrial camera, realize normalization processing of multi-angle image data; Input the normalized multi-angle image data and depth sensor data into the pre-trained image segmentation model, preliminarily separate foreground and background through threshold segmentation, and form the preliminary segmentation result of segmentation; Apply morphological operation to the preliminary segmentation result to eliminate small holes and edge burrs, use connected domain analysis to remove small area noise, and form the multi-angle segmentation result of completed segmentation; Fuse the multi-angle segmentation results of the same agricultural product, remove abnormal areas caused by occlusion of viewing angle, and integrate the multi-angle segmentation results through voting mechanism to retain the target area overlapped by multiple viewing angles to form the multi-angle image data after background cutting processing.

[0005] Further, the image segmentation model enhances feature reuse through nested dense connection, and the specific segmentation steps are as follows: Based on the depth map data obtained by the depth sensor, preliminarily circle out the position of the agricultural product on the image to form contrast data; Through the encoder module, the image extraction features are gradually reduced to form a feature compression package containing all key features and a set of original feature maps of different sizes; Based on the contrast data, the feature compression package is continuously enlarged through the decoder module, and the original feature map of the corresponding size is spliced into an enlarged feature map until the image is restored to the input size, generating a pixel-level binary mask segmentation image.

[0006] Further, the specific steps of the size regression branch for scoring the agricultural product to obtain the size score are as follows: Geometric correction is performed on the multi-angle image, and different angle images are aligned through feature point matching; Based on a single image of multi-angle image, the outline of agricultural product is identified through edge detection, and the two-dimensional size information of agricultural product is calculated, based on depth map data, three-dimensional point cloud model is constructed by using parallax principle of multi-view image, three-dimensional size information of agricultural product is calculated through three-dimensional point cloud model; The size standard threshold interval of different agricultural products is established based on market rules, and the size score is obtained by comparing the two-dimensional size information and the three-dimensional size information with the size standard threshold interval after combination; The size regression branch is assisted by a size loss function during training, and the size loss function is:

[0007] wherein, is a size regression loss value, is a sample number, is a model prediction value, is a real measurement value.

[0008] Further, the specific steps of the internal quality branch for scoring the agricultural products to obtain an internal quality score are as follows: The input multi-modal sensor data is preprocessed, and specific features are extracted, and then the feature data is fused; A regression model is established based on a partial least squares method algorithm and a support vector regression algorithm, the fused feature data is input, and the regression model outputs a quality parameter, and the regression model adopts an attention mechanism to focus on a key wavelength region; The output quality parameter is scored based on market rules to obtain a quality score; Wherein, the multi-modal sensor data includes a continuous spectrum curve, a three-dimensional data cube, sound wave detection data and a multi-dimensional gas response vector.

[0009] Further, the specific steps of the appearance branch for scoring the agricultural products to obtain an appearance score are as follows: Based on multi-angle image data, color features, texture features and shape features of the agricultural products are extracted, and the color features, texture features and shape features are judged respectively; Based on the color features, a color space analysis is performed to determine which level of the grade standard in the market rules the color conforms to, and a color grade is output; Based on the texture features, a surface smoothness is detected by a filter, a number of surface rough spots of the agricultural products is extracted, and a surface texture score of the agricultural products is output according to the corresponding standard in the market rules; Based on the shape features, whether the shape conforms to the corresponding shape in the market rules is determined by contour matching, and a shape score is output according to the matching result; The color grade, the surface texture score and the shape score are weighted according to the rating characteristics of different agricultural products, and a weighted appearance score is output.

[0010] Further, the specific steps of the defect branch for scoring the agricultural products to obtain a defect score are as follows: The defect recognition model is established based on historical data, multi-angle image data after background segmentation processing is input, and multi-angle image data with labels is output; The defect area ratio is calculated through the multi-angle image data with labels, and the defect is rated; The agricultural products are scored based on the defect rating and the defect area ratio, and a defect score is generated; When the defect score is lower than the set threshold, the agricultural products are determined as unqualified products, and are prohibited from flowing into the market.

[0011] Further, the dynamic weight distribution mechanism in step S3 dynamically adjusts the weight proportion of the size score, the internal quality score, the appearance score and the defect score according to the variety characteristics, market demand and sorting standard of different agricultural products, and the distribution steps are as follows: According to the variety of different agricultural products, a corresponding preset weight template data set is established; The variety of the agricultural products is determined through multi-angle image data, and the corresponding preset weight template is called according to the variety of the agricultural products; Based on the called preset weight template, market feedback data and environmental parameters are input into the preset distribution model, and the weight combination with maximum economic benefit is output.

[0012] The agricultural product grade classification control system based on image recognition comprises: The data acquisition module comprises an array type industrial camera unit and a multi-modal sensor unit, and is used for acquiring multi-angle image data, depth map data and internal quality data of agricultural products; The preprocessing module comprises a normalization processing unit and a background segmentation processing unit, and is used for preprocessing the multi-angle image data; The dynamic scoring module comprises a size regression branch unit, an internal quality branch unit, an appearance branch unit, a defect branch unit and a weight distribution unit, and is used for outputting the score of the agricultural products; The execution control module is connected with the agricultural product diversion system, generates corresponding sorting instructions according to the grade score, and controls the agricultural product diversion system to divert the agricultural products; The blockchain storage module is used for storing the grading results and the original multi-modal data, and generating a verifiable traceability label.

[0013] Further, the weight distribution unit comprises: The variety identification module is internally provided with a multi-modal variety identification model and a preset weight template database, and the corresponding preset weight template in the preset weight template database is called by identifying the agricultural products; The market data interface is connected with an e-commerce platform, a supply chain system and a bulk transaction market, and automatically acquires e-commerce sales data, logistics data and real-time agricultural product futures prices; Dynamic optimization module: multiple target optimization model is arranged inside, connected with market data interface and variety identification module, and optimal weight distribution is outputted; Abnormal processing module: with self-healing mechanism and fluctuation detection function.

[0014] Compared with the prior art, the agricultural product grade classification method and control system based on image recognition provided by the application have the following beneficial effects: The application obtains multi-angle images of agricultural products through an array type industrial camera, combines with depth data and internal quality data to form multi-modal data, realizes non-contact detection of internal indexes such as sugar content, moisture content, pesticide residues and internal decay, makes up for the defects of the prior art which only relies on image analysis of surface features, uses multi-angle images and depth data to construct a three-dimensional point cloud model of agricultural products, improves size measurement accuracy and improves the accuracy of shape regularity judgment, and solves the perspective deviation problem of traditional two-dimensional detection.

[0015] Before analyzing the multi-angle image data, the application uses real-time illumination data to realize normalization processing of illumination intensity, and uses Retinex algorithm and adaptive histogram equalization to eliminate the influence of illumination fluctuation, solves the problem of reduced classification accuracy caused by natural illumination intensity, and improves the accuracy of feature extraction during image segmentation.

[0016] The application hashes the grading results and original multi-modal data on a chain, supports consumers to verify by scanning codes, and reduces the rate of quality disputes. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flowchart of the agricultural product grade classification method based on image recognition in the application.

[0018] Figure 2 The module schematic diagram of the agricultural product grade classification control system based on image recognition in the application. DETAILED DESCRIPTION

[0019] In the description of the present application, it needs to be understood that the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "middle", "inner" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, it needs to be explained that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0020] The embodiments of the present application will be described below in conjunction with the drawings and related embodiments. The embodiments of the present application are not limited to the following examples, and the present application relates to the necessary components in the related technical field, which should be regarded as the known technology in the technical field, and can be known and mastered by the person skilled in the art.

[0021] Referring to Figure 1 A method for classifying agricultural products based on image recognition, comprising the following steps: Step S1, acquiring multi-angle image data of agricultural products through an array industrial camera, performing normalization processing and background segmentation processing on the multi-angle image data based on real-time collected environmental light data, and acquiring internal quality data and depth map data of the agricultural products through a multi-modal sensor to form multi-modal data with the image data; Step S2, establishing a dynamic grading model based on market rules and labeled historical data, the dynamic grading model including a size regression branch, an internal quality branch, an appearance branch and a defect branch, and inputting the multi-modal data to obtain size scores, internal quality scores, appearance scores and defect scores respectively; Step S3, based on the scoring points of different agricultural products, using a dynamic weight distribution mechanism to perform weighted calculation on the size scores, internal quality scores, appearance scores and defect scores, and outputting grade scores; Step S4, triggering a sorting instruction based on the grade scores to control a sorting device to sort the agricultural products; Step S5, store the grade score and multi-modal data association to the blockchain network, generate a verifiable traceability label attached to the agricultural products.

[0022] It should be noted that in step S1, the agricultural products moving on the conveyor belt are photographed in 360° surround view by the array of industrial cameras arranged in a ring. The images obtained include multiple perspectives such as the front, side, and calyx end. Each camera is equipped with a high-resolution image sensor, with a frame rate of up to 30 fps, ensuring clear imaging of fast-moving agricultural products. Taking apples as an example, the collected images cover the morphological characteristics of different varieties such as Red Fuji and Gala, and at least 6-8 images of different angles are obtained for each apple, providing sufficient visual information for subsequent analysis. Multi-modal sensors include near-infrared spectrometers, hyperspectral imagers, acoustic detectors, and electronic noses. The near-infrared spectrometer can obtain continuous spectral curves 2-5 mm below the surface of the agricultural product, which can be used to analyze internal quality data such as sugar content, acidity, and moisture; the hyperspectral imager simultaneously collects image and spectral information, forming a three-dimensional data cube, which can detect internal lesions such as brown areas in apples; the acoustic detector emits and receives ultrasonic waves to obtain acoustic detection data, which can determine whether there are hollows or rotting problems inside the agricultural product; the electronic nose detects the multi-dimensional gas response vector released by the agricultural product based on an array of gas sensors, and evaluates its maturity and fermentation level. The data obtained by these sensors are time-stamped and aligned with the processed image data, and the features are fused to form multi-modal data containing visual, spectral, acoustic, and gas information, providing comprehensive data support for subsequent grading. Depth map data is generated by ToF depth sensors, which can generate millimeter-precision depth maps and are mainly used for 3D volume reconstruction.

[0023] It should be noted that the sorting equipment executes the sorting instructions through actuators, including high-speed pneumatic nozzles and servo motor-driven mechanical arms. The high-speed pneumatic nozzles blow the agricultural products into the corresponding channels according to the grade score, and the mechanical arms are used to handle fragile agricultural products.

[0024] It should be noted that after the grade score calculation in step S3, the system compares the score result with the preset grade threshold, for example, the grade score of first-class quality needs to be more than 90 points, the grade score of second-class quality is between 80 and 90 points, and so on. According to the comparison result, step S4 is executed, in which the system generates corresponding sorting instructions according to the comparison result, sends the instructions to the sorting equipment through network communication protocol, and the sorting equipment accurately sorts the agricultural products into different channels or areas according to the received instructions, realizing classification storage or packaging according to grades, and providing convenience for subsequent sales and processing.

[0025] In one embodiment of the present application, the specific steps of the normalization processing and background segmentation processing in step S1 include: Based on real-time ambient light data, the multi-angle image data is dynamically supplemented using the Retinex algorithm, the image contrast is enhanced through adaptive histogram equalization, the multi-angle image data is aligned based on the spatial position parameters of the array industrial camera, and the normalization processing of the multi-angle image data is realized; the specific operation of aligning the multi-angle image data includes camera calibration and geometric transformation, wherein the camera calibration is to pre-calibrate the internal and external parameters of the array industrial camera using a checkerboard calibration board, and the geometric transformation is to map the multi-view image into a unified coordinate system through a Honography matrix based on the spatial position parameters of the camera, and then extract SIFT feature points, and realize sub-pixel level alignment after removing the mis-matching points through the RANSAC algorithm. The adaptive histogram equalization only enhances the brightness channel in the LAB color space, keeps the chroma unchanged, and prevents color distortion.

[0026] The normalized multi-angle image data and the depth sensor data are input into the pre-trained image segmentation model, the foreground and the background are preliminarily separated through threshold segmentation, and the preliminary segmented segmentation result is formed; The morphological operation is applied to the preliminary segmented segmentation result, the small holes and the edge burrs are eliminated, the small area noise is removed through the connected domain analysis, and the multi-angle segmentation result after segmentation is formed; The multi-angle segmentation results of the same agricultural product are fused to remove abnormal regions caused by occlusion of the viewing angle, and the multi-angle segmentation results are integrated through a voting mechanism to retain the target regions overlapped by multiple viewing angles to form the multi-angle image data after background segmentation. At least three viewing angle overlapping regions are retained to ensure the integrity of the target, wherein the multi-view segmentation result is combined with the depth map to generate a 3D point cloud through perspective projection transformation, and the DBSCAN algorithm is used to separate discrete noise points, and the method for removing abnormal regions is as follows: if a region is segmented as foreground only in a single viewing angle, while the other viewing angles are background, it is determined as an occlusion abnormal point and removed. The 3D point cloud fusion and the voting mechanism eliminate single-view errors and improve the integrity of the target segmentation, and through multi-modal fusion, the original multi-angle data, the depth map data and the light data are jointly optimized, and the problems of complex background and reflection interference are solved.

[0027] It should be noted that the Retinex algorithm inputs the original multi-angle image and real-time ambient light data, and outputs the dynamically adjusted multi-angle image data, three Gaussian kernels are used for multi-scale filtering of the multi-angle image, the light separation and reflection components are separated, and the weight is dynamically adjusted according to the ambient light intensity, and the formula is as follows:

[0028] wherein, represents the input multi-angle original image, in RGB format, raw data from an array industrial camera, which may contain uneven illumination and background noise, is a Gaussian filter kernel function, where represents the standard deviation of the Kth Gaussian kernel, represents the convolution operation of the input image and the Gaussian kernel, which represents the low-pass filtering of the image, separates the illumination component of the image by blurring, and retains the reflection component, represents the multi-scale estimation of the reflection component, represents the weight coefficient of the Kth scale, satisfying =1, represents the output image after illumination normalization. The Retinex parameter is dynamically adjusted to ensure illumination robustness.

[0029] It should be noted that the real-time ambient light data is obtained by a light sensor deployed in the collection area at a frequency of 10 times per second, covering parameters such as illumination intensity, color temperature, and light source direction. The Retinex algorithm is based on the characteristic that the human visual system is not sensitive to changes in light. The algorithm decomposes the image into a reflection component and an illumination component. When processing agricultural product images, the algorithm dynamically adjusts the brightness caused by strong light and shadows by analyzing the gray value distribution of the pixel points in different channels. For example, for an apple image in the shadow, the algorithm can enhance the details of the dark area, making the skin texture and color closer to the real state, and avoiding feature misjudgment caused by light differences. After light compensation, adaptive histogram equalization technology is used to further optimize image quality. Adaptive histogram equalization technology divides the image into multiple sub-blocks and performs histogram equalization processing on each sub-block independently, avoiding the problem of over-enhancement or detail loss caused by global equalization.

[0030] In an embodiment of the present application, the image segmentation model enhances feature reuse through nested dense connection, and the specific segmentation steps are as follows: Based on the depth map data obtained by the depth sensor, the position of the agricultural product on the image is preliminarily circled, forming contrast data; Through the encoder module, the image is gradually reduced to extract features to form a feature compression package containing all key features and a set of original feature maps of different sizes; Based on the contrast data, the feature compression package is continuously enlarged through the decoder module, combined with the original feature map of the corresponding size to splice into an enlarged feature map, until the image is restored to the input size, generating a pixel-level binary mask segmentation image.

[0031] It should be noted that the pre-processing is performed by the depth map data, the effective distance range is set based on the depth sensor data, the foreground target area is extracted, the distance threshold is automatically calculated by the adaptive threshold method, the depth value threshold range is set according to the actual size of the agricultural products and the shooting distance, and the background interference objects at long or short distances are quickly filtered out. The pixels with similar distances in the depth map are taken as seeds, the depth difference of adjacent pixels is calculated, and the continuous foreground area is gradually expanded to form an initial mask. The initial mask is subjected to an open operation to remove isolated pixels or small area holes caused by depth noise, so that the foreground area boundary is smoother, and the error of subsequent model processing is reduced.

[0032] The encoder module adopts a hierarchical convolution structure, gradually reduces the image size to realize feature extraction and compression, introduces nested dense connections, the output of each convolution layer is not only transmitted to the next layer, but also directly connected with all subsequent layers to form a dense connection path, different size convolution kernels are set in different convolution layer groups to capture features of different scales, at the end of the encoder, the feature map is compressed into a fixed length feature vector through a global average pooling layer to form a feature compression package, while the original feature maps of different sizes are retained to provide rich feature materials for the decoder. The decoder module is based on the feature compression package and the original feature map output by the encoder, and gradually restores the image size and generates the final segmentation result under the guidance of the contrast data of the initial depth map positioning.

[0033] In an embodiment of the present application, the specific steps of scoring the agricultural products by the size regression branch are as follows: The multi-angle images are geometrically corrected, and different view images are aligned through feature point matching; the geometric correction includes camera calibration and distortion correction, specifically, a checkerboard calibration board is used to collect calibration images of different poses, the intrinsic matrix and distortion coefficient of each camera are solved through Zhang Zhengyou calibration method, radial and tangential distortion correction is applied to the original image to eliminate image bending and deformation, and straight lines in the image are restored to true straight lines, for example, the curved banana profile caused by lens distortion is restored to a straight line form, ensuring the accuracy of subsequent size measurement. The ORB algorithm is used for feature point extraction and matching of multi-angle images.

[0034] Based on a single image of multi-angle images, the outline of the agricultural products is recognized through edge detection, the two-dimensional size information of the agricultural products is calculated, based on the depth map data, a three-dimensional point cloud model is constructed using the parallax principle of multi-view images, and the three-dimensional size information of the agricultural products is calculated through the three-dimensional point cloud model; wherein the Canny algorithm is used for edge detection to extract the target outline and remove background interference, and the parallax principle of multi-view images refers to matching corresponding pixels of left and right views based on the binocular vision principle, and then converting the depth map into a point cloud and removing outliers through the Open3D library.

[0035] The size standard threshold interval of different agricultural products is established based on market rules, and the size score is obtained by comparing the two-dimensional size information and the three-dimensional size information with the size standard threshold interval after combination; the threshold interval is established according to the national standard or the enterprise standard of different agricultural products, and the final size score is obtained after dynamic weighting processing of the scores of the two-dimensional size information and the three-dimensional size information, and the weight comparison is adjusted according to the characteristics of the agricultural products, for example, the score weight of the two-dimensional size information is reduced for the easily deformed agricultural products.

[0036] The size regression branch is assisted by a size loss function during training, which can combine the robustness and convergence speed of the regression task, and the size loss function is as follows:

[0037] Wherein, is the size regression loss value, is the sample number, is the model prediction value, is the true measurement value.

[0038] In an embodiment of the present application, the specific steps of the internal quality branch for scoring the agricultural products to obtain the internal quality score are as follows: The input multi-modal sensor data is preprocessed, specific features are extracted, and then the feature data is fused; the preprocessing includes preprocessing of spectral data, preprocessing of acoustic wave data and preprocessing of gas data.

[0039] A regression model is established based on a partial least squares algorithm and a support vector regression algorithm, the fused feature data is input, and the regression model outputs the quality parameters; the regression model adopts an attention mechanism to focus on the key wavelength region; the partial least squares algorithm is used to establish a regression model for the multicollinearity problem of spectral data and quality parameters, and the support vector regression algorithm adopts a radial basis kernel function to process the nonlinear relationship between acoustic wave data and internal decay degree; a channel attention module is introduced in the spectral data processing, the importance weight of each wavelength channel is calculated through global average pooling, the key wavelength region is focused, the weight of the key wavelength is increased, and the error is reduced.

[0040] The output quality parameters are scored based on market rules to obtain the quality score; the base score is given to the standard parameters during scoring, and the extra score is given to the excellent parameters; the quality parameters are divided into multiple grade intervals according to the industry standard in the market rules, and the corresponding scores are defined in sequence, and the score corresponding to the grade is obtained after the quality parameter reaches the grade.

[0041] Wherein, the multi-modal sensor data includes a continuous spectral curve, a three-dimensional data cube, acoustic wave detection data and a multi-dimensional gas response vector.

[0042] It should be noted that the pre-processing of the spectral data specifically includes the following steps: The standard normal variable transformation is used to eliminate the influence of the roughness of the agricultural product surface on the spectrum. The filter is applied to smooth the spectral curve and reduce the intensity of random noise signals. The key wavelengths are screened through the continuous projection algorithm.

[0043] The pre-processing of the acoustic data includes the following steps: The time-domain waveform in the acoustic data is subjected to fast Fourier transform to generate a frequency spectrum and extract a mel frequency cepstral coefficient. The four-part interval method is used to remove abnormal waveforms caused by environmental noise such as mechanical vibration.

[0044] The pre-processing of the gas data specifically includes the following steps: The gas response value of the electronic nose is subjected to Z-score standardization to eliminate the influence of sensor baseline drift. The gas data sequences of different collection rates are aligned.

[0045] In an embodiment of the present application, the appearance branch scores the agricultural product to obtain an appearance score, and the specific steps are as follows: Based on multi-angle image data, color features, texture features and shape features of the agricultural product are extracted, and the color features, texture features and shape features are judged respectively. Based on the color features, the color space analysis is performed to determine which level of the color grade in the market rule the color conforms to, and the color grade is output. Based on the texture features, the surface smoothness is detected by the filter, the number of rough spots on the surface of the agricultural product is extracted, and the surface texture score of the agricultural product is output according to the corresponding standard in the market rule. Based on the shape features, the contour matching is performed to determine whether the shape conforms to the corresponding shape in the market rule, and the shape score is output according to the matching result. The color grade, surface texture score and shape score are weighted according to the rating characteristics of different agricultural products, and the weighted appearance score is output. The weighting is allocated by dynamic weights, and the weights are allocated according to the emphasis of different agricultural products.

[0046] It should be noted that for the analysis of color features, the HSV color space is selected, which can separate color hue and brightness, avoid the interference of uneven light on color judgment, and can use histogram equalization technology to enhance color contrast. The color threshold interval is set according to the market rule, the hue in the color space is compared with the color threshold interval, and the color grade of the agricultural product is confirmed.

[0047] It should be noted that for the analysis of the texture feature, first, the multi-angle image data is smoothed by using a Gaussian filter, the mean square error of the smoothed image and the original image is calculated, the smoothness is scored based on the mean square error, the smaller the mean square error, the higher the smoothness score, and the higher the relative agricultural product grade, and then the number of spots of the image is obtained through the filter, and the agricultural product is scored according to the number of spots, the spot score is obtained, and the smoothness score and the spot score are weighted to obtain the comprehensive score of the surface texture.

[0048] It should be noted that for the analysis of the shape feature, first, the Fourier algorithm is used to extract the target contour point sequence, the shape feature vector is constructed, the similarity is calculated according to the matching of the shape feature vector and the preset standard template, and the shape score is obtained by comparing the calculation result of the similarity with the preset scoring standard.

[0049] In an embodiment of the present application, the specific steps of scoring the agricultural product by the defect branch to obtain the defect score are as follows: A defect recognition model is established based on historical data, multi-angle image data after background segmentation processing is input, and multi-angle image data with labels is output; the historical data includes historical image data with multiple defect types, covers different light, angle and variety scenes, and is pixel-level labeled to generate a label file containing defect categories, positions and areas, a defect sample library is constructed, and the defect recognition model is established according to the defect sample library. The defect recognition model uses YOLOv7-tiny real-time positioning to locate the defect area, supports the recognition of multiple defect types, realizes pixel-level defect labeling based on DeepLabV3+, and outputs a binary mask. The multi-view defect mask is mapped to a three-dimensional point cloud, aligned through an ICP algorithm, and a global defect distribution heat map is generated. If a certain area is labeled as a defect in multiple views, it is determined to be a real defect, avoiding single-view misjudgment.

[0050] The defect area ratio is calculated based on the multi-angle image data with labels, and the defect is rated; the defect area ratio is obtained by calculating the ratio of the number of defect pixels to the total number of pixels, and the area ratio is pre-rated based on market rules to obtain the defect rating.

[0051] The agricultural product is scored based on the defect rating and the defect area ratio to generate a defect score; the weights of the defect rating and the defect area ratio are distributed according to market demand.

[0052] When the defect score is lower than the set threshold, the agricultural product is determined to be unqualified and is prohibited from flowing into the market. When the defect score is lower than the set threshold, a sorting instruction is triggered, and the agricultural product is blown into a waste channel.

[0053] In one embodiment of the present application, the dynamic weight distribution mechanism in step S3 dynamically adjusts the weight proportions of the size score, the intrinsic quality score, the appearance score and the defect score according to the variety characteristics, market demand and sorting standards of different agricultural products, and the distribution steps are as follows: According to the variety of different agricultural products, a corresponding preset weight template data set is established; each preset weight template includes a basic weight matrix and an adjustable parameter range, the basic weight matrix defines the initial weights of the four branches of size, intrinsic quality, appearance and defect, and the adjustable parameter range refers to the dynamic adjustment of each weight within the specified range.

[0054] The variety of the agricultural product is determined through multi-angle image data, and the corresponding preset weight template is called according to the variety of the agricultural product; the variety of the agricultural product is confirmed by constructing a multi-modal variety identification model, the multi-modal variety identification model identifies the variety of the agricultural product according to the combination of multi-angle image data, depth map data and spectral data, outputs a variety label, and combines a fuzzy matching algorithm to call the most similar preset weight template in the preset weight template data set according to the variety label.

[0055] Based on the called preset weight template, market feedback data and environmental parameters are input into the preset distribution model to output a weight combination that maximizes economic benefits. The market feedback data includes sales data of an e-commerce platform and logistics data of a supply chain system.

[0056] Referring to Figure 2 , an agricultural product grade classification control system based on image recognition, comprising: A data acquisition module: including an array industrial camera unit and a multi-modal sensor unit, for acquiring multi-angle image data, depth map data and intrinsic quality data of agricultural products; having a data synchronization mechanism, the collected data is aligned with time stamp and fused into multi-modal data.

[0057] A preprocessing module: for preprocessing multi-angle image data, including a normalization processing unit and a background segmentation processing unit; the normalization processing unit is to normalize the light data based on real-time acquisition to output image data with uniform light; the background segmentation unit is used to cut the background in the image, and output a pixel-level binary mask with clear background and foreground.

[0058] A dynamic scoring module: including a size regression branch unit, an intrinsic quality branch unit, an appearance branch unit, a defect branch unit and a weight distribution unit, for outputting the score of the agricultural product; the size regression branch unit, the intrinsic quality branch unit, the appearance branch unit and the defect branch unit are respectively used for scoring the size, the intrinsic quality, the appearance and the defect of the agricultural product, and then the size, the intrinsic quality, the appearance and the defect scores are dynamically distributed according to the weight distribution unit, and finally the grade score of the agricultural product is output.

[0059] An execution control module is connected with the agricultural product distribution system, generates corresponding sorting instructions according to the grade score, and controls the agricultural product distribution system to distribute the agricultural products; the agricultural product distribution system comprises a pneumatic nozzle array and an auxiliary mechanical arm, and can respond to the sorting instructions through a real-time control protocol.

[0060] A blockchain storage module is used to store the grading results and the original multi-modal data, and generate a verifiable traceability label.

[0061] In an embodiment of the present application, the weight distribution unit comprises: A variety identification module automatically identifies the variety of the agricultural products based on multi-angle image data, and calls a corresponding preset weight template; a multi-modal variety identification model and a preset weight template database are internally provided, and the preset weight template database is called through identification of the agricultural products.

[0062] A market data interface is used to access market data in real time; the interface is connected with an e-commerce platform, a supply chain system and a bulk transaction market, and automatically obtains e-commerce sales data, logistics data and real-time agricultural product futures prices.

[0063] A dynamic optimization module takes the maximization of economic benefits as an objective function, iteratively optimizes the weight combination of each branch of the dynamic scoring module, and internally provides a multi-objective optimization model connected with the market data interface and the variety identification module to output an optimal weight distribution.

[0064] An abnormality processing module has a self-healing mechanism and a fluctuation detection function; when the fluctuation of a specific quality indicator of a batch of agricultural products exceeds a threshold value, the weight proportion of the indicator is automatically increased to enhance the robustness of the system; the self-healing mechanism comprises an incremental learning module and a fuse strategy mechanism that can be updated in real time, and when the loss rate is greater than a preset value after weight adjustment, the sorting work is suspended and manual intervention is notified.

[0065] The present application obtains multi-angle images of agricultural products through an array industrial camera, and forms multi-modal data in combination with depth map data and internal quality data, realizes non-contact detection of internal indicators such as sugar content, moisture content, pesticide residues and internal decay, makes up for the defects of the prior art which only relies on image analysis of surface features, constructs a three-dimensional point cloud model of agricultural products by using multi-angle images and depth data, improves the size measurement accuracy, improves the accuracy of shape regularity judgment, and solves the perspective deviation problem of traditional two-dimensional detection.

[0066] Before analyzing the multi-angle image data, the present application uses real-time illumination data to realize normalization processing of illumination intensity, and uses Retinex algorithm and adaptive histogram equalization to eliminate the influence of illumination fluctuation, solves the problem of reduced classification accuracy caused by natural illumination intensity, and improves the accuracy of feature extraction during image segmentation.

[0067] The application chains the grading results and the original multi-modal data on the hash, supports consumers to verify by scanning the code, and reduces the quality dispute rate.

[0068] The above only describes the preferred embodiments of the present application, and it should be pointed out that, for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can also be made, which should be considered as the protection scope of the present application.

Claims

1. A method for classifying agricultural products by grade based on image recognition, characterized in that, Includes the following steps: Step S1: Acquire multi-angle image data of agricultural products through an array-type industrial camera; perform normalization and background segmentation processing on the multi-angle image data based on real-time acquired ambient light data; acquire the intrinsic quality data and depth map data of agricultural products through a multi-modal sensor and form multi-modal data with the image data. Step S2: Establish a dynamic grading model based on market rules and labeled historical data. The dynamic grading model includes a size regression branch, an intrinsic quality branch, an appearance branch, and a defect branch. Input multimodal data to obtain size scores, intrinsic quality scores, appearance scores, and defect scores respectively. Step S3: Based on the key scoring points of different agricultural products, use a dynamic weight allocation mechanism to perform weighted calculations on size score, internal quality score, appearance score and defect score, and output the grade score. Step S4: Trigger sorting instructions based on grade rating to control sorting equipment to divert agricultural products; Step S5: Link and store the rating and multimodal data to the blockchain network, and generate verifiable traceability labels to be affixed to agricultural products.

2. The method for classifying agricultural products based on image recognition according to claim 1, characterized in that, The specific steps of normalization processing and background segmentation processing in step S1 include: Based on real-time ambient lighting data, the Retinex algorithm is used to dynamically supplement multi-angle image data, and adaptive histogram equalization is used to enhance image contrast. Based on the spatial position parameters of the array industrial camera, the multi-angle image data is aligned to achieve normalization processing of multi-angle image data. Normalized multi-angle image data and depth sensor data are input into a pre-trained image segmentation model. Threshold segmentation is used to initially separate the foreground and background, forming a preliminary segmentation result. Morphological operations are applied to the initial segmentation results to eliminate small holes and edge burrs, and connected component analysis is used to remove small area noise, resulting in a multi-angle segmentation result after the segmentation is completed. 3D point cloud fusion is performed on the multi-angle segmentation results of the same agricultural product to remove abnormal areas caused by viewpoint occlusion. The multi-angle segmentation results are integrated through a voting mechanism, and the target areas with multiple overlapping views are retained to form multi-angle image data after background cutting processing.

3. The method for classifying agricultural products based on image recognition according to claim 2, characterized in that, The image segmentation model enhances feature reuse through nested dense connections, and its specific segmentation steps are as follows: Based on the depth map data obtained by the depth sensor, the location of agricultural products on the image is initially delineated to form comparative data; The encoder module progressively reduces the image size to extract features, forming a feature compressed package containing all key features and a set of original feature maps of different sizes. Based on the comparative data, the feature compressed package is continuously enlarged by the decoder module and stitched together with the original feature map of the corresponding size to form an enlarged feature map until the image is restored to the input size, generating a segmented image of a pixel-level binary mask.

4. The method for classifying agricultural products based on image recognition according to claim 1, characterized in that, The specific steps for obtaining size scores for agricultural products using the size regression branch are as follows: Geometric correction is performed on multi-angle images, and images from different perspectives are aligned through feature point matching; Based on a single image from multiple angles, the outline of agricultural products is identified through edge detection, and the two-dimensional size information of agricultural products is calculated. Based on depth map data, a three-dimensional point cloud model is constructed using the parallax principle of multi-view images, and the three-dimensional size information of agricultural products is calculated through the three-dimensional point cloud model. Based on market rules, establish size standard threshold ranges for different agricultural products, and obtain size scores by combining two-dimensional and three-dimensional size information and comparing it with size standard threshold ranges. The size regression branch is assisted during training by a size loss function, which is: in, This represents the size regression loss value. For the sample size, These are the model's predicted values. These are actual measured values.

5. The method for classifying agricultural products based on image recognition according to claim 1, characterized in that, The specific steps for scoring agricultural products using the intrinsic quality branch to obtain intrinsic quality scores are as follows: The input multimodal sensor data is preprocessed to extract specific features, and then the feature data is fused. A regression model is established based on partial least squares algorithm and support vector regression algorithm. The input is fused feature data, and the regression model outputs quality parameters. The regression model uses an attention mechanism to focus on key wavelength regions. Based on market rules, the output quality parameters are scored to obtain a quality score; The multimodal sensor data includes continuous spectral curves, three-dimensional data cubes, acoustic detection data, and multidimensional gas response vectors.

6. The method for classifying agricultural products based on image recognition according to claim 1, characterized in that, The specific steps for scoring agricultural products using the appearance branch to obtain appearance scores are as follows: Based on multi-angle image data, the color features, texture features, and shape features of agricultural products are extracted, and the color features, texture features, and shape features are judged separately. Based on color characteristics, color space analysis is used to determine which level of the market rule standard a color conforms to, and the color level is output. Based on texture features, surface smoothness is detected by a filter, the number of rough spots on the surface of agricultural products is extracted, and the surface texture score of agricultural products is output according to the corresponding standard in the market rules. Based on shape features, contour matching is used to determine whether the shape conforms to the corresponding shape in the market rules, and a shape score is output based on the matching result. The color grade, surface texture score, and shape score are weighted according to the rating characteristics of different agricultural products, and the weighted appearance score is output.

7. The method for classifying agricultural products based on image recognition according to claim 1, characterized in that, The specific steps for scoring agricultural products using the defect branch to obtain a defect score are as follows: A defect identification model is established based on historical data. The input is multi-angle image data after background segmentation, and the output is labeled multi-angle image data. The defect area percentage is calculated using labeled multi-angle image data, and the defect is rated accordingly. Agricultural products are scored based on defect rating and defect area percentage to generate defect scores; When the defect score is lower than the set threshold, the agricultural product is judged to be unqualified and prohibited from entering the market.

8. The method for classifying agricultural products based on image recognition according to claim 1, characterized in that, The dynamic weight allocation mechanism in step S3 dynamically adjusts the weight ratios of size score, internal quality score, appearance score, and defect score based on the variety characteristics, market demand, and sorting standards of different agricultural products. The allocation steps are as follows: Establish corresponding pre-defined weighted template datasets based on different agricultural product varieties; The variety of agricultural products is determined by multi-angle image data, and the corresponding preset weight template is called according to the variety of agricultural products. Based on the pre-defined weight template, market feedback data and environmental parameters are input into the pre-defined allocation model, and the weight combination that maximizes economic benefits is output.

9. An agricultural product grading and classification control system based on image recognition, characterized in that, To implement the method according to any one of claims 1-8, comprising: Data acquisition module: includes an array-type industrial camera unit and a multimodal sensor unit, used to acquire multi-angle image data, depth map data and intrinsic quality data of agricultural products; Preprocessing module: Includes a normalization processing unit and a background segmentation processing unit, used to preprocess multi-angle image data; Dynamic scoring module: includes size regression branch unit, internal quality branch unit, appearance branch unit, defect branch unit and weight allocation unit, used to output scores for agricultural products; Execution control module: Connects to the agricultural product distribution system, generates corresponding sorting instructions based on the grade rating, and controls the agricultural product distribution system to distribute agricultural products; Blockchain storage module: Used to store hierarchical results and raw multimodal data, generating verifiable traceability tags.

10. The control system according to claim 9, characterized in that, The weight allocation unit includes: Variety identification module: It has a multimodal variety identification model and a preset weight template database. It can call the corresponding preset weight template in the preset weight template database by identifying agricultural products. Market data interface: Connects with e-commerce platforms, supply chain systems and bulk commodity trading markets to automatically acquire e-commerce sales data, logistics data and real-time agricultural futures prices; Dynamic optimization module: It has a built-in multi-objective optimization model, which connects to the market data interface and the product identification module to output the optimal weight allocation; Anomaly handling module: It has a self-healing mechanism and fluctuation detection function.

Images