Fruit and vegetable surface state recognition method based on computer vision

Abstract

The invention discloses a fruit and vegetable surface state recognition method based on computer vision, belongs to the technical field of fruit and vegetable storage state monitoring, and aims at solving the problems that in traditional fruit and vegetable surface state recognition, subjectivity is high, efficiency is low, and abnormal changes are difficult to find in time. The method comprises the steps that fruit and vegetable surface images are collected in a triggering mode and subjected to standardized preprocessing; extracting foreground and background images of fruits and vegetables according to background features of the storage space, and dividing independent fruit and vegetable areas by combining fruit and vegetable types; feature points are extracted to compare and match independent fruit and vegetable areas in the continuous images, newly added, updated and matched areas are identified, and fruit and vegetable change and maintenance areas are divided; establishing a classification evaluation database for the fruit and vegetable change region, and constructing a region index set; judging the state of the fruit and vegetable maintaining region through similarity, and updating or continuing to use the region index set; and performing abnormal trend analysis and state early warning according to the regional index set. Automatic and precise monitoring and early warning of the surface states of the fruits and vegetables are achieved.

Description

Technical Field

[0001] This invention relates to the field of fruit and vegetable storage status monitoring technology, and more specifically to a method for recognizing the surface status of fruits and vegetables based on computer vision. Background Technology

[0002] In the storage and distribution of fruits and vegetables, effective identification and monitoring of their surface condition is crucial for ensuring quality and reducing spoilage. Currently, the identification of fruit and vegetable surface condition still relies primarily on manual observation. Inspectors judge color changes, texture features, and defects on the surface of fruits and vegetables with the naked eye. This method is not only time-consuming and labor-intensive, but also prone to subjective and inconsistent judgments due to differences in individual experience and fatigue, making it difficult to accurately reflect the true condition of the fruits and vegetables.

[0003] Even though some solutions incorporate image recognition technology, they still have certain limitations: when processing continuously acquired fruit and vegetable images, it is difficult to accurately distinguish which areas are newly appearing and which areas have changed, often resulting in indiscriminate processing of all areas, leading to data redundancy and insufficient targeting; in the evaluation process, there is a lack of differentiated standards for different types of fruits and vegetables, making it difficult to scientifically quantify the degree of abnormality for various types of fruits and vegetables; at the same time, there is a lack of effective analysis of the development trend of abnormalities on the surface of fruits and vegetables, which can usually only be detected when the abnormality is already quite obvious, making it impossible to detect signs of the spread or aggravation of abnormalities in advance, and making it difficult to take measures at the optimal time for intervention, which may lead to a rapid decline in the quality of fruits and vegetables and increased losses. Therefore, in order to overcome these limitations, this invention proposes a method for identifying the surface state of fruits and vegetables based on computer vision. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method for recognizing the surface state of fruits and vegetables based on computer vision, which solves the technical problems of traditional fruit and vegetable surface state recognition, such as strong subjectivity, low efficiency, difficulty in accurately distinguishing dynamic changes in fruit and vegetable areas, inability to make differentiated assessments for different state areas, and inability to capture abnormal trends and issue early warnings in a timely manner.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Methods for recognizing the surface condition of fruits and vegetables based on computer vision include:

[0007] Images of fruit and vegetable surfaces are acquired via trigger-based acquisition and then subjected to standardized preprocessing.

[0008] Based on the background features of the storage space, the fruit and vegetable foreground image and the fruit and vegetable background image of the standardized preprocessed fruit and vegetable surface image are extracted. The fruit and vegetable regions are classified according to the macroscopic image features of the fruit and vegetable foreground image, and independent fruit and vegetable regions are divided in combination with the fruit and vegetable type.

[0009] Feature points of independent fruit and vegetable regions are extracted. Based on the pixel coordinates and feature point descriptors of the feature points, the independent fruit and vegetable regions in continuously acquired fruit and vegetable surface images are compared and matched to identify newly added, updated and matched independent fruit and vegetable regions, and to divide the fruit and vegetable change region and the fruit and vegetable maintenance region.

[0010] For areas of fruit and vegetable change, a classification and evaluation database is established according to the type of fruit and vegetable to obtain the grade score of unobstructed individual fruit and vegetable areas in the areas of fruit and vegetable change, identify abnormal fruit and vegetable areas, and construct a regional indicator set for fruit and vegetable change areas.

[0011] Locate the corresponding matching region of the fruit and vegetable maintenance region in the previous fruit and vegetable surface image, determine whether the state of the fruit and vegetable maintenance region has changed by measuring similarity, and update or reuse the region index set of the matching region according to the state determination result as the region index set of the fruit and vegetable maintenance region.

[0012] Based on the regional indicator set of fruit and vegetable surface images, perform abnormal trend analysis and early warning of fruit and vegetable surface conditions.

[0013] Specifically, the steps for identifying newly added, updated, and matched independent fruit and vegetable areas include:

[0014] Extract feature points from independent fruit and vegetable regions in fruit and vegetable surface images, generate feature point descriptors, and record the pixel position coordinates of each feature point;

[0015] Based on the pixel position coordinates of feature points in independent fruit and vegetable regions, a coordinate set of independent fruit and vegetable regions is constructed. This set is then compared with the coordinate set of feature points in the previously acquired fruit and vegetable surface image to calculate the degree of overlap between the coordinate sets.

[0016] If the overlap between the feature point coordinate set of the current independent fruit and vegetable region and the feature point coordinate set of all independent fruit and vegetable regions in the previously collected fruit and vegetable surface image is less than the preset low overlap threshold, then it is determined to be a current independent fruit and vegetable region that does not exist in the previously collected fruit and vegetable surface image and is marked as a newly added independent fruit and vegetable region.

[0017] Remove newly added independent fruit and vegetable regions from the current fruit and vegetable surface image. For the remaining unlabeled independent fruit and vegetable regions, select typical feature points, match the descriptors of the typical feature points with the feature point descriptors of all independent fruit and vegetable regions in the previously collected fruit and vegetable surface image, calculate the matching degree of the feature point pairs, and filter out potentially matching independent fruit and vegetable regions.

[0018] The feature point descriptors of the independent fruit and vegetable regions in the current fruit and vegetable surface image are compared one by one with the corresponding potential matching independent fruit and vegetable regions to quantify the region matching degree. If the region matching degree is greater than the preset high matching threshold, the current independent fruit and vegetable region is marked as a matching independent fruit and vegetable region; otherwise, it is marked as an updated independent fruit and vegetable region.

[0019] Specifically, the steps for dividing the fruit and vegetable change zone into the fruit and vegetable maintenance zone include:

[0020] For updating independent fruit and vegetable regions, based on the feature point matching degree between them and the corresponding independent fruit and vegetable regions in the previously collected fruit and vegetable surface images, the matching and non-matching feature points in the updated independent fruit and vegetable regions are identified.

[0021] Configure the grid size to adaptively divide the grid for updating individual fruit and vegetable areas;

[0022] Calculate the proportion of matching feature points in each grid to the total number of feature points in the grid, in response to labeling each grid type, including matching subgrids and updating subgrids;

[0023] Merge the continuous matching sub-grids and updated sub-grids to form preliminary matching sub-region outlines and updated sub-region outlines;

[0024] Locate the boundary grid between the contour of the matching sub-region and the contour of the updated sub-region, and determine the boundary dividing line between the matching sub-region and the updated sub-region by calculating the gradient change of the matching degree of the feature points at the boundary of the boundary grid.

[0025] Based on the boundary dividing line, the scope of the matching sub-region outline and the updating sub-region outline are clearly defined, and the updating independent fruit and vegetable region is divided into the updating sub-region and the matching sub-region.

[0026] The newly added independent fruit and vegetable area and the updated sub-area of ​​the updated independent fruit and vegetable area are designated as the fruit and vegetable change area, and the matching sub-area of ​​the matched independent fruit and vegetable area and the updated independent fruit and vegetable area are designated as the fruit and vegetable maintenance area.

[0027] Specifically, the steps for constructing a regional indicator set for fruit and vegetable variation areas include:

[0028] Identify the types of fruits and vegetables in areas of fruit and vegetable variation, establish a classification and evaluation database according to fruit and vegetable types, and store the status evaluation parameters and multi-level standard thresholds for each type of fruit and vegetable.

[0029] Multi-level standard thresholds are used to divide each state evaluation parameter into multiple severity levels and assign a level score;

[0030] Extract the edge texture features of all fruits and vegetables within the fruit and vegetable variation area; use a region growing algorithm based on edge texture feature clustering to delineate the outlines of individual fruit and vegetable regions within the fruit and vegetable variation area;

[0031] Based on the type of fruit and vegetable in the area of ​​fruit and vegetable change, a preset standard outline template of the same type of fruit and vegetable is called. The matching degree of the outline of the divided individual fruit and vegetable area is calculated with the standard outline template. If the outline is closed and the matching degree is greater than the preset outline threshold, it is marked as an unobstructed individual fruit and vegetable area; otherwise, it is marked as an obstructed individual fruit and vegetable area.

[0032] Specifically, the steps for constructing a regional indicator set for fruit and vegetable variation zones also include:

[0033] For unshaded individual fruit and vegetable areas, the corresponding status evaluation parameters are retrieved from the classification evaluation database according to the fruit and vegetable type to which they belong, and the status evaluation parameters of fruits and vegetables in the unshaded individual fruit and vegetable areas are extracted.

[0034] The extracted state evaluation parameters are compared with the corresponding multi-level standard thresholds in the classification and evaluation database to obtain the level score of each state evaluation parameter.

[0035] The overall score of an unobstructed individual fruit and vegetable area is obtained by summing the grade scores. If the overall score is greater than the preset abnormal threshold, it is marked as an abnormal fruit and vegetable area.

[0036] The observation window is configured based on the comprehensive score of the abnormal fruit and vegetable area. The side length of the observation window is determined by the ratio of the comprehensive score to the average size of the same type of fruit and vegetable. The average comprehensive score of the individual fruit and vegetable area that is not obscured within the observation window is used as the abnormal diffusion index.

[0037] The proportion of abnormal fruit and vegetable areas within the fruit and vegetable change area is statistically analyzed, and the abnormal diffusion index of each abnormal fruit and vegetable area is obtained. The proportion of unobstructed individual fruit and vegetable areas under each severity level of each state evaluation parameter is used to construct a regional indicator set for the fruit and vegetable change area.

[0038] Specifically, the steps for constructing the regional indicator set for fruit and vegetable maintenance areas include:

[0039] Locate the matching region of the fruit and vegetable maintenance area from the previously acquired fruit and vegetable surface images;

[0040] Individual fruit and vegetable areas within the fruit and vegetable maintenance area and the matching area are identified and numbered to establish a preliminary correspondence between individual fruit and vegetable areas and construct pairs of individual fruit and vegetable areas.

[0041] Calculate the similarity of individual fruit and vegetable region pairs. If the similarity of an individual fruit and vegetable region pair is less than the preset fruit and vegetable similarity threshold, it is determined that the state of the fruit and vegetable maintenance region has changed; otherwise, it is determined that the state of the fruit and vegetable maintenance region has not changed.

[0042] If the status of the fruit and vegetable maintenance area changes, the status evaluation parameters of the unshaded individual fruit and vegetable areas in the fruit and vegetable maintenance area are extracted again, the comprehensive score of the unshaded individual fruit and vegetable areas is updated, abnormal fruit and vegetable areas are identified, and the abnormal diffusion index of abnormal fruit and vegetable areas is updated to construct the regional index set of the fruit and vegetable maintenance area.

[0043] If the status of the fruit and vegetable maintenance area remains unchanged, retrieve the regional indicator set of the matching area and use it as the regional indicator set of the fruit and vegetable maintenance area.

[0044] Specifically, the steps for extracting the foreground image of fruits and vegetables from the standardized preprocessed surface image include:

[0045] Based on the structure of the storage space, the background features of the storage space are extracted. The background features include shelf geometry features, color grayscale features, and texture features. A background feature library containing templates of typical background pixel features in the storage space is established.

[0046] Configure the image patch segmentation size to divide the collected fruit and vegetable surface images into non-overlapping image patches, extract the geometric shape features, color grayscale features and texture features of each image patch, and calculate the similarity between each image patch and typical feature templates in the background feature library.

[0047] Configure a similarity threshold to mark image patches that have a similarity greater than the threshold with typical feature templates in the background feature library as candidate background regions, and otherwise mark them as candidate foreground regions;

[0048] By employing regional connectivity analysis, all image patches are traversed, and adjacent image patches with the same label are merged into continuous regions, forming a background region set and a fruit and vegetable region set.

[0049] Area filtering and morphological dilation and erosion are performed on the background region set and the fruit and vegetable region set to extract the fruit and vegetable foreground image and the fruit and vegetable background image.

[0050] Specifically, the steps for dividing the area into separate fruit and vegetable zones include:

[0051] The initial fruit and vegetable regions are determined based on the spatial continuity of pixels and color channel similarity in the fruit and vegetable foreground image.

[0052] For each initial fruit and vegetable region, the similarity within the region is measured, macroscopic image features within the initial fruit and vegetable region are extracted and their coefficient of variation is calculated. Macroscopic image features include color features, texture features and shape features.

[0053] If the coefficient of variation of the macroscopic image features is lower than the preset classification threshold, the initial fruit and vegetable region is determined to be a single type of fruit and vegetable; otherwise, the initial fruit and vegetable region is determined to contain multiple types of fruit and vegetables.

[0054] For an initial fruit and vegetable region that is determined to be a single type of fruit and vegetable, it is treated as an independent fruit and vegetable region and compared with a preset fruit and vegetable type feature library to match and label the fruit and vegetable type.

[0055] For an initial fruit and vegetable region that is determined to contain multiple types of fruits and vegetables, the initial fruit and vegetable region is traversed by a sliding window. The macroscopic image feature difference between the pixels in the sliding window and the pixels in the adjacent window is calculated. Potential boundary points with macroscopic image feature differences greater than a preset difference threshold are marked. Physical boundary lines are formed by polynomial curve fitting to divide the region into multiple independent fruit and vegetable regions.

[0056] The macroscopic image features of each independent fruit and vegetable region are compared with a pre-set fruit and vegetable type feature library to match and label the fruit and vegetable types.

[0057] Specifically, the steps for trigger-based acquisition of images of fruit and vegetable surfaces include:

[0058] Based on the preset acquisition frequency, auxiliary data in the storage space is synchronously acquired through auxiliary sensors to construct a time-series monitoring dataset, and the change characteristics of the auxiliary data in the time-series monitoring dataset are extracted.

[0059] Configure trigger thresholds for auxiliary data change characteristics for image acquisition triggering judgment;

[0060] The change characteristics of the extracted auxiliary data are compared with the trigger threshold. If the change characteristics of the auxiliary data are greater than the corresponding trigger threshold, the acquisition of fruit and vegetable surface images is triggered.

[0061] When the acquisition of fruit and vegetable surface images is triggered, the light adjustment module is linked according to the preset shooting light parameters, and a start command is sent to the vision sensor and the light adjustment module. The light adjustment module turns on the supplementary light according to the preset shooting light parameters, and the vision sensor acquires fruit and vegetable surface images under the preset shooting light parameters.

[0062] After the images of the fruit and vegetable surfaces are acquired, the illumination adjustment module turns off the supplementary lighting equipment and restores the basic illumination parameters to those before acquisition. At the same time, the vision sensor stops acquiring images.

[0063] Specifically, the steps for conducting abnormal trend analysis and early warning of fruit and vegetable surface conditions include:

[0064] Key parameters were extracted from the regional index set of fruit and vegetable surface images, including the proportion of abnormal fruit and vegetable regions for each fruit and vegetable type, the proportion of each severity level of the status evaluation parameters, and the abnormal diffusion index of abnormal fruit and vegetable regions. A status dataset was constructed according to the order of fruit and vegetable surface image acquisition.

[0065] A trend fitting algorithm is used to fit the changing trends of the key parameters in the state dataset, and the slope of the changes in the key parameters is extracted.

[0066] For each type of fruit and vegetable, a threshold for the slope of change of key parameters is configured. If the slope of change of a key parameter is greater than the corresponding threshold, it is marked as an abnormal key parameter. The abnormal key parameter type, the type of fruit and vegetable to which it belongs, and the slope data are integrated to generate an abnormal key parameter early warning information and trend analysis report.

[0067] The beneficial effects of this invention are:

[0068] This application, through precise processing and dynamic analysis of fruit and vegetable surface images, can effectively improve the efficiency and accuracy of fruit and vegetable surface condition identification, and reduce the interference of subjective factors; it can accurately distinguish the dynamic changes of fruit and vegetable areas, realize differentiated evaluation of different state areas, and avoid data redundancy; it can conduct targeted analysis for different types of fruits and vegetables, and accurately quantify the degree of abnormality; at the same time, it can capture abnormal trends in a timely manner and issue early warnings, providing support for timely intervention, thereby reducing fruit and vegetable losses, ensuring fruit and vegetable quality, and meeting the needs of large-scale and refined fruit and vegetable storage and circulation management. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of the structure of the fruit and vegetable surface state recognition method based on computer vision of the present invention;

[0070] Figure 2 This is a flowchart illustrating the process of extracting the fruit and vegetable foreground image from the standardized preprocessed fruit and vegetable surface image according to the present invention.

[0071] Figure 3 A flowchart illustrating the specific steps involved in dividing the area into independent fruit and vegetable zones according to the present invention;

[0072] Figure 4 This is a flowchart illustrating the specific steps involved in identifying newly added, updated, and matched independent fruit and vegetable regions according to the present invention.

[0073] Figure 5 A flowchart illustrating the specific steps involved in dividing the fruit and vegetable change zone and the fruit and vegetable maintenance zone according to the present invention.

[0074] Figure 6 This is a flowchart illustrating the specific steps involved in obtaining a set of regional indicators for fruit and vegetable maintenance areas according to the present invention. Detailed Implementation

[0075] Please see Figure 1 This embodiment introduces a computer vision-based method for recognizing the surface state of fruits and vegetables, including:

[0076] Step S1: By deploying visual sensors and auxiliary sensors in the storage space and establishing a linkage mechanism, images of the fruit and vegetable surface are acquired through triggered image acquisition to reduce the impact of light on the fruit and vegetable.

[0077] In this embodiment, a multi-view visual sensor is fixedly installed in the storage cabinet or storage room to cover all fruit and vegetable storage areas. A light adjustment module is also configured to maintain basic lighting that meets the preservation requirements of fruits and vegetables during non-collection periods, avoiding the impact of continuous strong light on the produce. A linkage mechanism is established between the visual sensor and gas, temperature, and humidity sensors. During non-collection periods, these auxiliary sensors monitor environmental parameters in real time. When abnormal fluctuations in parameters are detected, such as an increase in the concentration of specific volatile organic compounds or a sudden temperature change, the visual sensor is automatically triggered to acquire images. After acquisition, the supplemental lighting system is immediately shut down to reduce the continuous impact of light on the fruits and vegetables.

[0078] Preferably, the specific steps for acquiring images of fruit and vegetable surfaces through triggered image acquisition include:

[0079] Based on the preset acquisition frequency, auxiliary data in the storage space is synchronously acquired through auxiliary sensors to construct a time-series monitoring dataset. The time-series monitoring dataset contains gas concentration data, temperature data, humidity data and corresponding acquisition timestamps from different acquisition nodes. The change characteristics of the auxiliary data in the time-series monitoring dataset are extracted through time series analysis algorithms, including data change rate, fluctuation amplitude, and duration of continuous anomalies.

[0080] Trigger thresholds for auxiliary data change characteristics are configured according to the types of fruits and vegetables and storage and preservation standards, which are used to trigger image acquisition. The change characteristics of the auxiliary data extracted in real time are compared with the trigger thresholds. If the change characteristics of the auxiliary data are greater than the corresponding trigger threshold, it indicates that the surface state of the fruits and vegetables in the storage space may have changed, triggering the acquisition of fruit and vegetable surface images.

[0081] When the acquisition of fruit and vegetable surface images is triggered, the light adjustment module is linked according to the preset shooting light parameters, and a start command is sent to the vision sensor and the light adjustment module. The light adjustment module turns on the supplementary light according to the preset shooting light parameters, and the vision sensor completes the acquisition of multi-view fruit and vegetable surface images under the preset shooting light parameters.

[0082] After the images of the fruit and vegetable surfaces are acquired, the illumination adjustment module turns off the supplementary lighting equipment and restores the basic illumination parameters to those not acquired during the acquisition period. At the same time, the vision sensor stops acquiring images.

[0083] Step S2: Standardize and preprocess the collected fruit and vegetable surface images to optimize image quality and retain key visual features.

[0084] In this embodiment, the acquired raw images undergo standardized preprocessing. Noise reduction algorithms are used to eliminate noise caused by environmental interference, and illumination equalization techniques are employed to correct uneven brightness in the fruit and vegetable surface images caused by localized light differences. Simultaneously, the image size and color space are standardized to ensure consistency of image features in subsequent processing. The preprocessing process must preserve key visual information of the fruit and vegetable surface, such as color, texture, and contours, to provide high-quality image data for subsequent segmentation and recognition.

[0085] Step S3: Based on the background features of the storage space, extract the fruit and vegetable foreground images of the standardized preprocessed fruit and vegetable surface images, classify the fruit and vegetable regions according to the macroscopic image features of the fruit and vegetable foreground images, and divide independent fruit and vegetable regions based on the fruit and vegetable types.

[0086] In this embodiment, based on the standardized preprocessed fruit and vegetable surface image, the background features of the storage space are first extracted to establish a background feature library containing typical background pixel feature templates, providing a benchmark reference for accurate distinction between background and foreground. Then, the image block separation size is configured according to the complexity of background features and the typical size of fruits and vegetables, the image is divided into non-overlapping image blocks and feature parameters are extracted. By calculating the similarity with the background feature library, and combining the threshold to mark candidate background regions and candidate foreground regions, the initial separation of background regions and fruit and vegetable regions is achieved. Adjacent and identically labeled image blocks are merged through region connectivity analysis. After area screening and optimization through morphological dilation and erosion operations, effective image regions containing only fruits and vegetables are extracted as fruit and vegetable foreground images, eliminating isolated noise interference and optimizing the continuity of region boundaries. Based on fruit and vegetable foreground images, macroscopic image features such as color, texture, and shape are extracted. Initial fruit and vegetable regions are preliminarily divided according to pixel spatial continuity and color channel similarity, providing basic region units for subsequent classification. For each initial fruit and vegetable region, similarity measurement is performed, and the coefficient of variation of macroscopic image features is calculated to determine whether it is a single type of fruit or vegetable, improving the accuracy of type identification. For single-type regions, the region is compared with a fruit and vegetable type feature database, and the K-nearest neighbor algorithm is used to identify the label type, achieving fast and accurate type matching. For multi-type regions, feature difference boundary points are marked by a sliding window, and the region is segmented into independent fruit and vegetable regions by polynomial curve fitting before type matching, ensuring accurate separation and identification of different types of fruits and vegetables. Finally, the independent fruit and vegetable region segmentation results that correspond one-to-one with the actual fruit and vegetable types are output.

[0087] Preferably, the specific steps for extracting the fruit and vegetable foreground image from the standardized preprocessed fruit and vegetable surface image and dividing it into independent fruit and vegetable regions include:

[0088] Please see Figure 2Based on the structure of the storage space, background features of the storage space are extracted, including: the geometric features of the storage space shelves, such as the rectangular frame ratio of the shelves and the spacing of the beams; color grayscale features, such as the RGB mean of the white partition and the grayscale distribution range of the wooden partition; and texture features, such as the LBP texture of the container edge, in order to establish a background feature library containing typical feature templates of background pixels in the storage space.

[0089] The image block segmentation size is configured based on the complexity of the background features in the storage space and the typical size of fruits and vegetables. This ensures that the image block size can fully encompass local background features such as beam width and partition texture units, while also covering the smallest individual fruit and vegetable recognition unit. The collected fruit and vegetable surface images are divided into non-overlapping image blocks according to the image block segmentation size. Feature parameters of each image block are extracted based on the background features of the storage space, including geometric shape features, color grayscale features, and texture features. The similarity between each image block and typical feature templates in the background feature library is calculated using Euclidean distance or cosine similarity algorithms.

[0090] By statistically analyzing the feature fluctuation range of typical background feature templates and combining the minimum feature difference between fruits and vegetables and the background, a similarity threshold is configured. If the similarity between the image patch and the typical feature template in the background feature library is greater than the set similarity threshold, it indicates that the image patch mainly contains background information and is marked as a candidate background region. Otherwise, it indicates that the image patch may contain fruit and vegetable information and is marked as a candidate foreground region.

[0091] For candidate background and candidate foreground regions, region connectivity analysis is used to traverse all image blocks and merge adjacent and labeled image blocks into continuous regions, forming a background region set and a fruit and vegetable region set.

[0092] The background region set and the fruit and vegetable region set are filtered by area, and isolated candidate background regions and isolated candidate foreground regions with an area smaller than a preset area threshold are removed; the smoothness of the region boundary is optimized by morphological dilation and erosion operations to eliminate the jagged edges generated by merging; finally, the fruit and vegetable foreground image and fruit and vegetable background image are extracted from the fruit and vegetable surface image, and the pixels corresponding to all background region sets are excluded to obtain the effective image region containing only fruits and vegetables.

[0093] Please see Figure 3 Based on the spatial continuity and color channel similarity of the pixels in the fruit and vegetable foreground image, the fruit and vegetable foreground image is initially divided to obtain initial fruit and vegetable regions. Each region corresponds to a continuous fruit and vegetable distribution area.

[0094] For each initial fruit and vegetable region, a similarity measurement is performed within the region. Macroscopic image features, including color, texture, and shape features, are extracted from the initial fruit and vegetable region. The coefficient of variation of the macroscopic image features within the region is calculated, such as the standard deviation of the color histogram and the variance of the texture features. The consistency of features within the region is comprehensively evaluated. If the coefficient of variation of the macroscopic image features is lower than a preset classification threshold, the initial fruit and vegetable region is determined to be a single type of fruit and vegetable; otherwise, the initial fruit and vegetable region is determined to contain multiple types of fruit and vegetables.

[0095] For an initial fruit and vegetable region that is determined to be a single type of fruit and vegetable, it is treated as an independent fruit and vegetable region. Its macroscopic image features are compared with a preset fruit and vegetable type feature library. The fruit and vegetable type feature library contains typical color, texture, and shape templates of various fruits and vegetables. The feature matching degree is calculated by the K nearest neighbor classification algorithm to identify the fruit and vegetable type of the independent fruit and vegetable region and mark it.

[0096] For an initial fruit and vegetable region that is determined to contain multiple types of fruits and vegetables, the initial fruit and vegetable region is traversed through a sliding window with a preset step size. The macroscopic image feature difference between the pixels in the sliding window and the pixels in the adjacent windows is calculated. When the macroscopic image feature difference between the adjacent windows is greater than a preset difference threshold, it is marked as a potential boundary point.

[0097] Polynomial curve fitting is used to form physical boundary lines between different types of fruits and vegetables for potential boundary points, dividing the initial fruit and vegetable region into multiple independent fruit and vegetable regions, each corresponding to a single type of fruit and vegetable; the macroscopic image features of each independent fruit and vegetable region are compared with a preset fruit and vegetable type feature library, and their fruit and vegetable types are matched and labeled, finally outputting the segmentation results of all independent fruit and vegetable regions that correspond one-to-one with the actual fruit and vegetable types.

[0098] Step S4: Extract feature points of independent fruit and vegetable regions. Based on the pixel coordinates and feature point descriptors of the feature points, compare and match the independent fruit and vegetable regions in the continuously acquired fruit and vegetable surface images to identify newly added, updated and matched independent fruit and vegetable regions. Then, mark the status of the independent fruit and vegetable regions in the current fruit and vegetable surface image and divide the updated sub-region and matched sub-region of the updated independent fruit and vegetable regions.

[0099] In this embodiment, stable feature points of independent fruit and vegetable regions are extracted and feature point descriptors are generated to ensure good feature stability, unaffected by factors such as angle and distance during fruit and vegetable collection, thus laying a solid foundation for subsequent comparison and matching. A coordinate set is constructed based on the pixel position coordinates of the feature points. By comparing the coordinate sets, regions not present in the previously collected image can be quickly identified, effectively improving the efficiency of identifying newly added regions. For the remaining unmarked independent fruit and vegetable regions, potential matching regions are first screened using typical feature point matching. Then, a comprehensive comparison of the feature point descriptors of both regions is performed to quantify the matching degree and obtain the region matching degree. Based on the region matching degree, they are marked as either matched independent fruit and vegetable regions or updated independent fruit and vegetable regions. This staged matching method significantly improves overall matching efficiency while ensuring matching accuracy. For updated independent fruit and vegetable regions, matching and unmatched feature points are identified and their coordinates are recorded. Through operations such as grid division, sub-grid marking, contour merging, and segmentation line determination, the range of updated and matched sub-regions is clearly defined. This series of operations can accurately capture local changes within the updated region, providing a reliable basis for analyzing changes in fruit and vegetable regions caused by partial handling and replacement. It accurately identifies the addition, matching, and update status of independent fruit and vegetable regions in continuously collected fruit and vegetable surface images, meticulously divides the updated region into sub-regions, and comprehensively and accurately reflects the dynamic changes of independent fruit and vegetable regions, providing strong support for fruit and vegetable management, monitoring, and other applications.

[0100] Please see Figure 4 Preferably, the specific steps for identifying newly added, updated, and matched independent fruit and vegetable areas include:

[0101] The scale-invariant feature transformation (SIN) algorithm is used to extract feature points of independent fruit and vegetable regions in fruit and vegetable surface images. The SIN descriptor is generated by the scale-invariant feature transformation to ensure that the features are scale and rotation invariant. At the same time, the pixel position coordinates of each feature point are recorded.

[0102] Based on the pixel coordinates of feature points in the independent fruit and vegetable region, a coordinate set for the independent fruit and vegetable region is constructed. This set is then compared with the coordinate set of feature points in the previously acquired fruit and vegetable surface image. The overlap of the coordinate sets is calculated by determining the proportion of the number of feature points with overlapping positions in the two coordinate sets to the total number of feature points in the current independent fruit and vegetable region.

[0103] If the overlap between the feature point coordinate set of the current independent fruit and vegetable region and the feature point coordinate set of all independent fruit and vegetable regions in the previously acquired fruit and vegetable surface image is less than a preset low overlap threshold, then it is determined to be a current independent fruit and vegetable region that does not exist in the previously acquired fruit and vegetable surface image and is marked as a newly added independent fruit and vegetable region; otherwise, no processing is performed.

[0104] Remove newly added independent fruit and vegetable regions from the current fruit and vegetable surface image. For the remaining unlabeled independent fruit and vegetable regions, select any typical feature point. Match the descriptor of the typical feature point with the feature point descriptors of all independent fruit and vegetable regions in the previously collected fruit and vegetable surface image. Quantify the feature point matching degree by calculating the Euclidean distance between feature point pairs. Select potential matching independent fruit and vegetable regions based on the feature point matching degree.

[0105] For the independent fruit and vegetable regions in the current fruit and vegetable surface image and their corresponding potential matching independent fruit and vegetable regions, a brute-force matching algorithm is used to compare the feature point descriptors of the two one by one. The feature point matching degree is quantified again by calculating the Euclidean distance of the feature point pairs. The region matching degree is obtained by statistically analyzing the proportion of the number of successfully matched feature points to the total number of feature points in the current independent fruit and vegetable region. If the region matching degree is greater than the preset high matching threshold, it indicates that the two regions are completely matched, and the current independent fruit and vegetable region is marked as a matched independent fruit and vegetable region; otherwise, it is marked as an updated independent fruit and vegetable region.

[0106] Please see Figure 5 Preferably, the specific steps for dividing the fruit and vegetable change zone and the fruit and vegetable maintenance zone include:

[0107] For updating independent fruit and vegetable regions, based on the feature point matching degree between them and the corresponding independent fruit and vegetable regions in the previously collected fruit and vegetable surface images, matching and unmatched feature points in the updated independent fruit and vegetable regions are identified, and the pixel position coordinates of the two types of feature points are recorded.

[0108] An adaptive grid is used to update the independent fruit and vegetable regions. The grid size is dynamically configured according to the distribution density of feature points. Smaller grids are used in densely populated areas and larger grids are used in sparse areas to ensure that each grid contains a sufficient number of feature points to support the analysis.

[0109] Calculate the ratio of the number of matching feature points in each grid to the total number of feature points in the grid. If the ratio is higher than the preset sub-region matching threshold, then mark the grid as a matching sub-grid; otherwise, mark it as an update sub-grid.

[0110] Based on the grid type of adjacent grids, a region growing algorithm is used to merge consecutive matching sub-grids to form a preliminary matching sub-region outline; consecutive updating sub-grids are also merged to form a preliminary updating sub-region outline. Grid types include matching sub-grids and updating sub-grids.

[0111] Locate the boundary grid between the matching sub-region contour and the updated sub-region contour. Determine the boundary dividing line by calculating the gradient change of the matching degree of the boundary feature points of the boundary grid: generate a smooth curve along the direction of the most significant gradient change as the boundary dividing line between the matching sub-region and the updated sub-region.

[0112] The boundaries of the matching and updated sub-regions are defined based on the boundary segmentation lines. Within the matching sub-region, the proportion of matched feature points in all grids meets a preset standard, and the shape and size deviations from the corresponding areas in the previously acquired fruit and vegetable surface image are within acceptable limits. Within the updated sub-region, the proportion of unmatched feature points in all grids meets a preset standard, or there are significant changes in shape and size. The boundary coordinates, feature point matching details, and parameter change data of both the matching and updated sub-regions are recorded to complete the division of sub-regions within the updated independent fruit and vegetable region.

[0113] The newly added independent fruit and vegetable area and the updated sub-area of ​​the updated independent fruit and vegetable area are designated as the fruit and vegetable change area, and the matching sub-area of ​​the matched independent fruit and vegetable area and the updated independent fruit and vegetable area are designated as the fruit and vegetable maintenance area.

[0114] Step S5: For areas experiencing fruit and vegetable changes, firstly, the types of fruits and vegetables in the area are determined using image recognition algorithms. Then, a classification and evaluation database is established based on the identified fruit and vegetable types. Using this database, edge texture features of all fruits and vegetables within the affected area are extracted. Individual fruit and vegetable regions are then defined, distinguishing between unobstructed and obstructed regions. State evaluation parameters for unobstructed regions are extracted and compared with corresponding multi-level standard thresholds in the classification and evaluation database to obtain a grade score. A comprehensive score is obtained by summing the grade scores. Unobstructed regions with a comprehensive score greater than a preset anomaly threshold are marked as abnormal regions. Finally, the grade distribution of abnormal regions is statistically analyzed, and the proportion of abnormal regions, average anomaly grade, etc., are calculated to construct a regional index set for the affected fruit and vegetable areas.

[0115] In this embodiment, image recognition algorithms are used to determine fruit and vegetable types, laying the foundation for establishing appropriate evaluation standards and ensuring that the evaluation dimensions match the characteristics of the fruits and vegetables. A classification evaluation database is established according to fruit and vegetable types, providing exclusive state evaluation parameters and multi-level standard thresholds to ensure the relevance and accuracy of the evaluation. Using this classification evaluation database, edge texture features of fruits and vegetables are extracted, providing a reliable basis for dividing individual fruit and vegetable regions and improving the accuracy of region division. Dividing individual fruit and vegetable regions and distinguishing between unobstructed and obstructed areas clarifies the scope of the evaluation object, ensuring that the evaluation of each individual fruit and vegetable is as comprehensive as possible. The state of the unobstructed areas is extracted. Evaluation parameters are compared with multi-level standard thresholds to obtain grade scores, which can quantitatively assess the state of unshaded fruits and vegetables and unify the measurement standards. A comprehensive score is obtained by summing the grade scores, and areas with comprehensive scores exceeding the preset abnormality threshold are marked as abnormal fruit and vegetable areas, which can quickly and accurately identify abnormal fruits and vegetables and provide clear targets for subsequent risk assessment and treatment. The distribution of abnormal fruit and vegetable area grades, the proportion of abnormal fruit and vegetable areas and the average abnormality level are calculated, and a regional indicator set is constructed, which can comprehensively reflect the abnormal status of the entire fruit and vegetable change area, provide data support for the overall regional status assessment, and facilitate the understanding of the surface status of fruits and vegetables from a macro perspective.

[0116] Preferably, the specific steps for constructing a regional indicator set for fruit and vegetable variation areas include:

[0117] Image recognition algorithms are used to identify the types of fruits and vegetables in areas of change. A classification and evaluation database is then established based on these types. This database stores state evaluation parameters and multi-level standard thresholds for each type, with different parameters corresponding to different fruit and vegetable types to ensure a high degree of alignment between the evaluation dimensions and the characteristics of the fruits and vegetables. For example, apples are evaluated based on the size and number of surface blemishes, leafy vegetables on the degree of yellowing and the distribution of insect holes, and citrus fruits on the gloss and depth of dents in the peel, ensuring a high degree of alignment between the evaluation dimensions and the characteristics of the fruits and vegetables.

[0118] Multi-level standard thresholds refer to multiple thresholds set for each state evaluation parameter in order of severity from low to high, including the upper limit of normal threshold, the threshold for slight abnormality, the threshold for moderate abnormality, and the threshold for severe abnormality. They are used to clarify the division boundary of each state evaluation parameter under different degrees of severity, divide each state evaluation parameter into multiple severity levels, and assign a level score.

[0119] The edge texture features of all fruits and vegetables within the changing area are extracted using a multi-scale texture analysis algorithm and an edge detection operator.

[0120] A region growing algorithm based on edge texture feature clustering divides individual fruit and vegetable regions within a changing fruit and vegetable area. Specifically, it performs similarity clustering based on extracted edge texture features, aggregating pixels with similar edge texture features into the same initial cluster unit. Using each initial cluster unit as the starting point for growth, a feature similarity threshold is set, and pixels that meet the threshold condition are continuously included in the growth range. At the same time, the growth boundary is controlled by combining fruit and vegetable edge features to prevent growth across fruit and vegetable regions until a complete region outline is formed, completing the initial division of individual fruit and vegetable regions.

[0121] Based on the type of fruit and vegetable in the area of ​​fruit and vegetable change, a preset standard outline template of the same type of fruit and vegetable is called. The matching degree of the initially divided individual fruit and vegetable area outline is calculated with the standard outline template. If the individual fruit and vegetable area outline is closed and the matching degree with the standard outline template is greater than the preset outline threshold, it is determined to be a complete individual fruit and vegetable area and marked as an unobstructed individual fruit and vegetable area. Otherwise, it is determined to be incomplete and marked as an obstructed individual fruit and vegetable area.

[0122] For unshaded individual fruit and vegetable areas, the corresponding status evaluation parameters are retrieved from the classification evaluation database according to the type of fruit and vegetable. The feature extraction algorithm is used to extract the status evaluation parameters of fruits and vegetables in the unshaded individual fruit and vegetable areas, such as the size and number of fruit surface defects of apples and the degree of yellowing of leafy vegetables.

[0123] The extracted status evaluation parameters are compared with the corresponding multi-level standard thresholds in the classification and evaluation database to obtain the grade score for each status evaluation parameter in an unobstructed individual fruit and vegetable area. For example, if the status evaluation parameter does not exceed the upper limit of the normal threshold, the grade score is 0; if the parameter exceeds the upper limit of the normal threshold but does not reach the slight abnormality threshold, the grade score is 1; if the parameter reaches the slight abnormality threshold but does not reach the moderate abnormality threshold, the grade score is 2; if the parameter reaches the moderate abnormality threshold but does not reach the severe abnormality threshold, the grade score is 3; and if the parameter reaches or exceeds the severe abnormality threshold, the grade score is 4.

[0124] Based on the rating of each status evaluation parameter of the unobstructed individual fruit and vegetable area, the comprehensive score of the unobstructed individual fruit and vegetable area is obtained by summing them. If the comprehensive score is greater than the preset abnormal threshold, the unobstructed individual fruit and vegetable area is marked as an abnormal fruit and vegetable area.

[0125] Based on the comprehensive score of the abnormal fruit and vegetable area, an observation window is configured. The side length of the observation window is determined by the ratio of the comprehensive score of the abnormal fruit and vegetable area to the average size of the same type of fruit and vegetable. The higher the comprehensive score, the larger the side length is, so as to cover a wider range of potential diffusion. The average comprehensive score of the unobstructed individual fruit and vegetable areas within the observation window is obtained as the abnormal diffusion index.

[0126] The proportion of abnormal fruit and vegetable areas within the fruit and vegetable change area is statistically analyzed, and the abnormal diffusion index of each abnormal fruit and vegetable area is obtained. The proportion of unobstructed individual fruit and vegetable areas under each severity level of each state evaluation parameter is used to construct a regional indicator set for the fruit and vegetable change area.

[0127] Step S6: For the fruit and vegetable maintenance area, locate the matching region corresponding to the previously collected fruit and vegetable surface image from the stored data; determine whether the state of the fruit and vegetable maintenance area has changed by measuring the similarity between the fruit and vegetable maintenance area and the matching region. If a state change is determined, update the original region index set of the matching region as the region index set of the fruit and vegetable maintenance area; if no state change is determined, directly use the region index set of the matching region as the region index set of the fruit and vegetable maintenance area.

[0128] In this embodiment, the matching sub-regions of the matched independent fruit and vegetable regions and the updated independent fruit and vegetable regions are defined as fruit and vegetable maintenance regions. This accurately defines the range of fruit and vegetable regions whose status has not changed significantly, providing a clear target for subsequent status comparison analysis, avoiding invalid processing and improving the process's focus. The corresponding matching region is located from the previously collected fruit and vegetable surface images that have been stored, providing a benchmark for comparing the current and historical states, ensuring accurate judgment and clear correspondence. By constructing individual fruit and vegetable region pairs and calculating similarity to determine whether the state has changed, the focus can be placed on the differences of each individual fruit and vegetable, objectively quantifying surface state changes, avoiding subjective errors, and making the results more reliable and consistent. When the state changes, the regional indicator set is reconstructed, which can reflect the latest state in a timely manner by updating the state evaluation parameters, comprehensive scores, and anomaly diffusion index, ensuring the timeliness and accuracy of the indicator set and providing the latest data support for subsequent evaluation. When the state has not changed, the regional indicator set of the matched regions is retrieved, which can reduce redundant processing, improve efficiency, and maintain data continuity and consistency.

[0129] Please see Figure 6 Preferably, the specific steps for obtaining the regional indicator set of the fruit and vegetable maintenance area include:

[0130] From the previously collected images of fruit and vegetable surfaces, locate the matching region of the fruit and vegetable maintenance area; identify and number the individual fruit and vegetable regions within the maintenance area and the matching region, establish a preliminary correspondence between the individual fruit and vegetable regions, and construct pairs of individual fruit and vegetable regions;

[0131] The structural similarity analysis method is used to calculate the similarity of individual fruit and vegetable region pairs. If the similarity of an individual fruit and vegetable region pair is less than the preset fruit and vegetable similarity threshold, it is determined that the state of the fruit and vegetable maintenance region has changed, indicating that the surface state of the fruit and vegetable has changed significantly, and the region index set of the fruit and vegetable maintenance region needs to be updated. Otherwise, it is determined that the state of the fruit and vegetable maintenance region has not changed, indicating that the surface state of all individual fruit and vegetable region pairs has not changed significantly, and there is no need to adjust the region index set.

[0132] If the status of the fruit and vegetable maintenance area changes, the status evaluation parameters of the unshaded individual fruit and vegetable areas in the fruit and vegetable maintenance area are extracted again, the comprehensive score of the unshaded individual fruit and vegetable areas is updated, abnormal fruit and vegetable areas are identified, and the abnormal diffusion index of abnormal fruit and vegetable areas is updated to construct the regional index set of the fruit and vegetable maintenance area.

[0133] If the status of the fruit and vegetable maintenance area remains unchanged, retrieve the regional indicator set of the matching area and use it as the regional indicator set of the fruit and vegetable maintenance area.

[0134] Step S7: Based on the regional indicator set of fruit and vegetable surface images, conduct abnormal trend analysis and fruit and vegetable surface condition early warning, including: based on the regional indicator set of fruit and vegetable surface images, analyze the changing trends of indicators such as the proportion of abnormal fruit and vegetable areas and the distribution of abnormalities at each level, determine whether the abnormality has spread or intensified, and then issue corresponding fruit and vegetable surface condition early warning based on the trend analysis results.

[0135] In this embodiment, analysis based on a regional indicator set can help grasp abnormal dynamics with specific quantitative indicators and avoid subjective judgment bias; analyzing abnormal trends can identify signs of spread or aggravation in advance, giving time for intervention; issuing corresponding warnings can enable relevant personnel to be aware of the risks to the surface condition of fruits and vegetables in a timely manner, making it easier to take targeted measures to reduce losses.

[0136] Preferably, the specific steps for conducting abnormal trend analysis and early warning of fruit and vegetable surface conditions include:

[0137] Key parameters were extracted from the indicator set of fruit and vegetable surface image regions, including the proportion of abnormal fruit and vegetable regions for each fruit and vegetable type, the proportion of each severity level of the status evaluation parameters, and the abnormal diffusion index of abnormal fruit and vegetable regions. A status dataset was constructed according to the order of fruit and vegetable surface image acquisition.

[0138] A trend fitting algorithm is used to fit the changing trends of the key parameters in the state dataset, and the slope of the changes in the key parameters is extracted.

[0139] Based on each type of fruit and vegetable, configure the slope threshold of key parameter changes, and clarify the trend characteristic standards corresponding to the proportion of abnormal areas, the proportion of status evaluation parameters of severity level, and the abnormal diffusion index under different fruit and vegetable types.

[0140] If the slope of change of a key parameter is greater than the corresponding slope threshold, it is marked as an abnormal key parameter. The type of the abnormal key parameter, the type of fruit or vegetable to which it belongs, and the slope data are integrated through the early warning information template to generate an early warning information for the abnormal key parameter, which is used to indicate that there is an abnormal trend in the key parameter. Based on the overall change of key parameters in the fruit and vegetable surface image and the distribution characteristics of abnormal key parameters, the data is integrated and analyzed through the report generation tool to generate a trend analysis report.

[0141] For different types of fruits and vegetables, the system automatically pushes warning information containing abnormal trend charts for each type of fruit and vegetable, high-risk area locations, and suggested intervention measures to relevant terminals according to the configured push rules, thus completing the warning process for the surface condition of fruits and vegetables.

[0142] Working principle and its effects:

[0143] This application utilizes computer vision technology to automate the entire process of identifying and monitoring the surface condition of fruits and vegetables, achieving precise and efficient processing from image acquisition to anomaly warning. This not only improves the objectivity and timeliness of identification but also provides a scientific basis for fruit and vegetable quality management.

[0144] By triggering image acquisition and preprocessing, combined with auxiliary sensor data to trigger image acquisition, invalid shots are reduced and image quality is optimized. This reduces the impact of light on fruits and vegetables and provides high-quality data for subsequent analysis. Foreground images are extracted based on background features of the storage space, and independent fruit and vegetable regions are divided. Feature matching accurately distinguishes different types of fruits and vegetables, effectively solving the problem of fruit and vegetable region segmentation in complex backgrounds and improving the accuracy of region division. Comparative matching of independent fruit and vegetable regions in continuous images identifies newly added, updated, and matched independent fruit and vegetable regions and divides them into changing and maintaining regions, achieving refined tracking of dynamic changes in fruits and vegetables and avoiding the inefficiency caused by indiscriminate processing. A classification and evaluation database is established for changing regions. Anomalies are identified by combining the grade scores of unobstructed regions and a regional indicator set is constructed. For maintaining regions, the regional indicator set is updated or reused based on similarity judgment, ensuring the targeting and accuracy of the evaluation and reducing redundant calculations. Anomaly trend analysis and early warning are performed based on the regional indicator set. Anomalies are judged by the slope of key parameter changes and early warning information is generated, which can promptly capture the spread or aggravation of anomalies and buy time for the formulation of intervention measures.

[0145] In summary, this method, through multi-stage technical optimization, achieves automation, precision, and intelligence in the identification of fruit and vegetable surface conditions, effectively reducing labor costs and subjective errors, improving the timeliness of anomaly monitoring, and ultimately reducing fruit and vegetable losses, thus ensuring the quality of fruits and vegetables during storage and distribution.

[0146] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for recognizing the surface state of fruits and vegetables based on computer vision, characterized in that, include: Images of fruit and vegetable surfaces are acquired via trigger-based acquisition and then subjected to standardized preprocessing. Based on the background features of the storage space, the fruit and vegetable foreground image and the fruit and vegetable background image of the standardized preprocessed fruit and vegetable surface image are extracted. The fruit and vegetable regions are classified according to the macroscopic image features of the fruit and vegetable foreground image, and independent fruit and vegetable regions are divided in combination with the fruit and vegetable type. Feature points of the independent fruit and vegetable regions are extracted. Based on the pixel position coordinates and feature point descriptors of the feature points, the independent fruit and vegetable regions in the continuously acquired fruit and vegetable surface images are compared and matched to identify newly added, updated and matched independent fruit and vegetable regions, and to divide the fruit and vegetable change regions and the fruit and vegetable maintenance regions. For the fruit and vegetable change area, a classification and evaluation database is established according to the fruit and vegetable type to obtain the grade score of the unobstructed individual fruit and vegetable area in the fruit and vegetable change area, identify abnormal fruit and vegetable areas, and construct a regional index set for the fruit and vegetable change area. Locate the corresponding matching region of the fruit and vegetable maintenance region in the previous fruit and vegetable surface image, determine whether the state of the fruit and vegetable maintenance region has changed by measuring similarity, and update or reuse the region index set of the matching region according to the state determination result as the region index set of the fruit and vegetable maintenance region. Based on the regional index set of the fruit and vegetable surface images, perform abnormal trend analysis and early warning of fruit and vegetable surface conditions.

2. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 1, characterized in that, The steps for identifying newly added, updated, and matched independent fruit and vegetable regions include: Extract feature points from independent fruit and vegetable regions in fruit and vegetable surface images, generate feature point descriptors, and record the pixel position coordinates of each feature point; Based on the pixel position coordinates of feature points in independent fruit and vegetable regions, a coordinate set of independent fruit and vegetable regions is constructed. This set is then compared with the coordinate set of feature points in the previously acquired fruit and vegetable surface image to calculate the degree of overlap between the coordinate sets. If the overlap between the feature point coordinate set of the current independent fruit and vegetable region and the feature point coordinate set of all independent fruit and vegetable regions in the previously collected fruit and vegetable surface image is less than the preset low overlap threshold, then it is determined to be a current independent fruit and vegetable region that does not exist in the previously collected fruit and vegetable surface image and is marked as a newly added independent fruit and vegetable region. Remove newly added independent fruit and vegetable regions from the current fruit and vegetable surface image. For the remaining unlabeled independent fruit and vegetable regions, select typical feature points, match the descriptors of the typical feature points with the feature point descriptors of all independent fruit and vegetable regions in the previously collected fruit and vegetable surface image, calculate the matching degree of the feature point pairs, and filter out potentially matching independent fruit and vegetable regions. The feature point descriptors of the independent fruit and vegetable regions in the current fruit and vegetable surface image are compared one by one with the corresponding potential matching independent fruit and vegetable regions to quantify the region matching degree. If the region matching degree is greater than the preset high matching threshold, the current independent fruit and vegetable region is marked as a matching independent fruit and vegetable region; otherwise, it is marked as an updated independent fruit and vegetable region.

3. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 2, characterized in that, The specific steps for dividing the fruit and vegetable change zone and the fruit and vegetable maintenance zone include: For updating independent fruit and vegetable regions, based on the feature point matching degree between them and the corresponding independent fruit and vegetable regions in the previously collected fruit and vegetable surface images, the matching and non-matching feature points in the updated independent fruit and vegetable regions are identified. Configure the grid size to adaptively divide the grid for updating individual fruit and vegetable areas; Calculate the proportion of matching feature points in each grid to the total number of feature points in the grid, in response to labeling each grid type, including matching subgrids and updating subgrids; Merge the continuous matching sub-grids and updated sub-grids to form preliminary matching sub-region outlines and updated sub-region outlines; Locate the boundary grid between the contour of the matching sub-region and the contour of the updated sub-region, and determine the boundary dividing line between the matching sub-region and the updated sub-region by calculating the gradient change of the matching degree of the feature points at the boundary of the boundary grid. Based on the boundary dividing line, the scope of the matching sub-region outline and the updating sub-region outline are clearly defined, and the updating independent fruit and vegetable region is divided into the updating sub-region and the matching sub-region. The newly added independent fruit and vegetable area and the updated sub-area of ​​the updated independent fruit and vegetable area are designated as the fruit and vegetable change area, and the matching sub-area of ​​the matched independent fruit and vegetable area and the updated independent fruit and vegetable area are designated as the fruit and vegetable maintenance area.

4. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 1, characterized in that, The specific steps for constructing the regional indicator set for fruit and vegetable variation areas include: Identify the types of fruits and vegetables in areas of fruit and vegetable variation, and establish a classification and evaluation database according to the types of fruits and vegetables. The classification and evaluation database stores the status evaluation parameters of each type of fruit and vegetable and the multi-level standard thresholds of the status evaluation parameters. The multi-level standard threshold is used to divide each state evaluation parameter into multiple severity levels and assign a level score; Extract the edge texture features of all fruits and vegetables within the fruit and vegetable variation area; use a region growing algorithm based on edge texture feature clustering to delineate the outlines of individual fruit and vegetable regions within the fruit and vegetable variation area; Based on the type of fruit and vegetable in the changing fruit and vegetable area, a preset standard contour template of the same type of fruit and vegetable is called. The matching degree of the divided individual fruit and vegetable area contour is calculated with the standard contour template. If the contour is closed and the matching degree is greater than the preset contour threshold, it is marked as an unobstructed individual fruit and vegetable area; otherwise, it is marked as an obstructed individual fruit and vegetable area.

5. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 4, characterized in that, The specific steps for constructing the regional indicator set for fruit and vegetable variation areas also include: For the unobstructed individual fruit and vegetable area, the corresponding status evaluation parameters are retrieved from the classification evaluation database according to the fruit and vegetable type to which it belongs, and the status evaluation parameters of the fruits and vegetables in the unobstructed individual fruit and vegetable area are extracted; The extracted state evaluation parameters are compared with the corresponding multi-level standard thresholds in the classification and evaluation database to obtain the level score of each state evaluation parameter. The comprehensive score of the unobstructed individual fruit and vegetable area is obtained by summing the grade scores. If the comprehensive score is greater than the preset abnormal threshold, it is marked as an abnormal fruit and vegetable area. An observation window is configured based on the comprehensive score of the abnormal fruit and vegetable area. The side length of the observation window is determined by the ratio of the comprehensive score to the average size of similar fruits and vegetables. The average comprehensive score of the individual fruit and vegetable area that is not obscured within the observation window is obtained as the abnormal diffusion index. The proportion of abnormal fruit and vegetable areas within the fruit and vegetable change area is statistically analyzed, and the abnormal diffusion index of each abnormal fruit and vegetable area is obtained. The proportion of unobstructed individual fruit and vegetable areas under each severity level of each state evaluation parameter is used to construct the regional index set of the fruit and vegetable change area.

6. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 5, characterized in that, The specific steps for constructing the regional indicator set for the fruit and vegetable maintenance area include: Locate the matching region of the fruit and vegetable maintenance area from the previously acquired fruit and vegetable surface images; Individual fruit and vegetable areas within the fruit and vegetable maintenance area and the matching area are identified and numbered to establish a preliminary correspondence between individual fruit and vegetable areas and construct pairs of individual fruit and vegetable areas. Calculate the similarity of individual fruit and vegetable region pairs. If the similarity of an individual fruit and vegetable region pair is less than the preset fruit and vegetable similarity threshold, it is determined that the state of the fruit and vegetable maintenance region has changed; otherwise, it is determined that the state of the fruit and vegetable maintenance region has not changed. If the status of the fruit and vegetable maintenance area changes, the status evaluation parameters of the unshaded individual fruit and vegetable areas in the fruit and vegetable maintenance area are extracted again, the comprehensive score of the unshaded individual fruit and vegetable areas is updated, abnormal fruit and vegetable areas are identified, and the abnormal diffusion index of abnormal fruit and vegetable areas is updated to construct the regional index set of the fruit and vegetable maintenance area. If the status of the fruit and vegetable maintenance area remains unchanged, retrieve the regional indicator set of the matching area and use it as the regional indicator set of the fruit and vegetable maintenance area.

7. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 1, characterized in that, The specific steps for extracting the foreground image of fruits and vegetables from the standardized preprocessed surface image include: Based on the structure of the storage space, the background features of the storage space are extracted. The background features include shelf geometry features, color grayscale features, and texture features. A background feature library containing templates of typical background pixel features in the storage space is established. Configure the image block separation size to divide the collected fruit and vegetable surface images into non-overlapping image blocks, extract the geometric shape features, color grayscale features and texture features of each image block, and calculate the similarity between each image block and the typical feature templates in the background feature library; Configure a similarity threshold, and mark image blocks whose similarity to typical feature templates in the background feature library is greater than the similarity threshold as candidate background regions, otherwise mark them as candidate foreground regions; By employing regional connectivity analysis, all image patches are traversed, and adjacent image patches with the same label are merged into continuous regions, forming a background region set and a fruit and vegetable region set. Area filtering and morphological dilation and erosion are performed on the background region set and the fruit and vegetable region set to extract the fruit and vegetable foreground image and the fruit and vegetable background image.

8. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 1, characterized in that, The specific steps for dividing the area into independent fruit and vegetable zones include: The initial fruit and vegetable regions are determined based on the spatial continuity of pixels and color channel similarity in the fruit and vegetable foreground image. For each initial fruit and vegetable region, the similarity within the region is measured, macroscopic image features within the initial fruit and vegetable region are extracted and their coefficient of variation is calculated. The macroscopic image features include color features, texture features and shape features. If the coefficient of variation of the macroscopic image features is lower than the preset classification threshold, the initial fruit and vegetable region is determined to be a single type of fruit and vegetable; otherwise, the initial fruit and vegetable region is determined to contain multiple types of fruit and vegetables. For an initial fruit and vegetable region that is determined to be a single type of fruit and vegetable, it is treated as an independent fruit and vegetable region and compared with a preset fruit and vegetable type feature library to match and label the fruit and vegetable type. For an initial fruit and vegetable region that is determined to contain multiple types of fruits and vegetables, the initial fruit and vegetable region is traversed by a sliding window. The macroscopic image feature difference between the pixels in the sliding window and the pixels in the adjacent window is calculated. Potential boundary points with macroscopic image feature differences greater than a preset difference threshold are marked. Physical boundary lines are formed by polynomial curve fitting to divide the region into multiple independent fruit and vegetable regions. The macroscopic image features of each independent fruit and vegetable region are compared with a pre-set fruit and vegetable type feature library to match and label the fruit and vegetable types.

9. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 1, characterized in that, The specific steps for trigger-based acquisition of fruit and vegetable surface images include: Based on the preset acquisition frequency, auxiliary data in the storage space is synchronously acquired through auxiliary sensors to construct a time-series monitoring dataset, and the change characteristics of the auxiliary data in the time-series monitoring dataset are extracted. Configure trigger thresholds for auxiliary data change characteristics for image acquisition triggering judgment; The change characteristics of the extracted auxiliary data are compared with the trigger threshold. If the change characteristics of the auxiliary data are greater than the corresponding trigger threshold, the acquisition of fruit and vegetable surface images is triggered. When the acquisition of fruit and vegetable surface images is triggered, the light adjustment module is linked according to the preset shooting light parameters, and a start command is sent to the vision sensor and the light adjustment module. The light adjustment module turns on the supplementary light according to the preset shooting light parameters, and the vision sensor acquires fruit and vegetable surface images under the preset shooting light parameters. After the images of the fruit and vegetable surfaces are acquired, the illumination adjustment module turns off the supplementary lighting equipment and restores the basic illumination parameters to those before acquisition. At the same time, the vision sensor stops acquiring images.

10. The method for recognizing the surface state of fruits and vegetables based on computer vision as described in claim 1, characterized in that, The specific steps for conducting abnormal trend analysis and early warning of fruit and vegetable surface conditions include: Key parameters were extracted from the regional index set of fruit and vegetable surface images, including the proportion of abnormal fruit and vegetable regions for each fruit and vegetable type, the proportion of each severity level of the status evaluation parameters, and the abnormal diffusion index of abnormal fruit and vegetable regions. A status dataset was constructed according to the order of fruit and vegetable surface image acquisition. A trend fitting algorithm is used to fit the changing trends of the key parameters in the state dataset, and the slope of the changes in the key parameters is extracted. For each type of fruit and vegetable, a threshold for the slope of change of key parameters is configured. If the slope of change of a key parameter is greater than the corresponding threshold, it is marked as an abnormal key parameter. The abnormal key parameter type, the type of fruit and vegetable to which it belongs, and the slope data are integrated to generate an abnormal key parameter early warning information and trend analysis report.

Images