Durian assembly line automatic quality inspection method and system based on image recognition

Abstract

The invention discloses a durian assembly line automatic quality inspection method and system based on image recognition, relates to the technical field of intelligent visual inspection, and is suitable for automatic detection requirements of durian surface crack recognition and risk prediction. The method comprises the following steps: extracting a target image segment with potential crack risk characteristics from an image sequence containing a state before a crack occurs based on a historically collected image, and carrying out risk label labeling; performing structure expression optimization processing on the image sequence; extracting characterization features of the image segments, and constructing a feature vector sequence reflecting fruit crack risk evolution in combination with training set label information; and analyzing the sequence to assess a fruit crack risk evolution trend, obtaining a crack risk assessment value, and positioning a corresponding high-risk area in the image sequence based on an assessment result. According to the technical scheme, early recognition and accurate positioning of the cracking risk of the durian are achieved, and intelligence and reliability of assembly line quality inspection are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent visual detection, and in particular to an automated quality inspection method and system for a durian production line based on image recognition. Background Art

[0002] Currently, image recognition and automatic sorting technologies are widely used in fruit quality inspection, especially for standard fruits such as apples, citrus fruits, and bananas. Image recognition algorithms can automatically determine the fruit's color, size, shape, and surface defects, enabling real-time grading and efficient sorting. However, for fruits with complex structures and irregular surface features, such as durian, existing quality inspection methods still face significant technical bottlenecks in image recognition accuracy, feature extraction capabilities, and grading strategies.

[0003] Durian has a complex appearance, with irregular protrusions and grooves on its surface. It is easily affected by the external environment, such as cracks that may appear during transportation and storage. Cracks not only affect the appearance of the durian but also accelerate its decay, becoming a gateway for microorganisms and bacteria to enter, severely impacting its edible value and shelf life. Therefore, crack detection is not only a critical step in durian quality control but also a necessary measure to ensure durian safety and extend its shelf life.

[0004] Currently, most existing technologies for detecting durian cracks focus on fruit surface defect identification methods based on image recognition. These methods typically rely on traditional image processing techniques, such as edge detection and grayscale analysis, to determine whether the peel has cracks through preset rules or feature extraction. However, these methods have significant shortcomings: on the one hand, the texture of durian peel is complex and irregular, and cracks often appear as subtle changes, making it difficult for traditional methods to accurately identify micro-cracks or potential cracks; on the other hand, existing image processing technology has difficulty coping with the dynamic changes in cracks in durian fruit during transportation and storage, and is unable to assess the evolution trend of crack risks in real time. As a result, crack detection is limited to the state where cracks have already occurred in the fruit, lacking early warning of potential cracks.

[0005] Furthermore, because durian is a non-standardized fruit with varying surface characteristics and crack states, existing standardized processing methods fail to fully account for individual differences in the fruit. For example, factors such as crack size, depth, and stage of occurrence all affect the fruit's crack risk. Existing methods struggle to dynamically and individually assess crack risk for each durian, leading to significant errors in grading and determining the quality of the fruit, which in turn affects the accuracy and efficiency of the sorting and rejection processes.

[0006] Therefore, the durian crack detection method based on image recognition still faces many challenges and requires technical breakthroughs in crack detection accuracy, dynamic crack risk assessment and personalized processing. Summary of the Invention

[0007] In response to the above problems, the present invention proposes an automated quality inspection method and system for durian production lines based on image recognition. Through technical means such as crack risk evolution assessment and crack risk area positioning, it can accurately identify and assess the risk of cracks on the surface of durian fruits, and through real-time monitoring of the changing trend of fruit cracks, it can achieve early warning and precise graded control management of durian cracks.

[0008] To achieve the above objectives, in a first aspect, the present invention provides an automated quality inspection method for durian production line based on image recognition, comprising the following steps: Acquire multiple consecutive frames of images of the fruit as it moves along the conveyor path, and perform feature enhancement processing on the images to highlight the surface structural features of the peel associated with the risk of cracking. The resulting image sequence is then transformed into a structure-optimized image sequence. Based on historically collected durian image data, target image segments with potential crack risk characteristics are extracted from the image sequence containing the state before cracking, and crack risk labels are annotated on the target image segments. The labeled target image segments are organized in chronological order to construct a training set containing crack risk evolution labels. Each sample in the training set reflects the change in crack risk during the fruit's structural evolution. Extracting a peel surface structure feature vector related to the crack risk from the structure optimization image sequence, and forming a feature vector sequence reflecting the evolution of the fruit crack risk based on the feature vector; Based on the crack risk evolution labels in the training set, the feature vector sequence reflecting the evolution of fruit crack risk is analyzed to evaluate the evolution trend of fruit crack risk. Determine the fruit crack risk assessment value based on the assessment results, and locate the corresponding crack risk area in the structure optimization image sequence; Fruits with crack risk assessment values ​​exceeding a preset threshold are marked as crack risk objects, and the fruits are graded and controlled or removed based on the located crack risk areas.

[0009] In a second aspect, the present invention provides an automated quality inspection system for a durian production line based on image recognition, comprising: The image acquisition and processing module is used to obtain continuous multi-frame images of the fruit during its movement along the conveyor path, and perform feature enhancement processing on the images to highlight the surface structural features of the peel related to the risk of cracking. The processing results form a structure-optimized image sequence; An image segment extraction and annotation module is used to extract target image segments with potential crack risk characteristics from an image sequence containing the state before cracking based on historically collected durian image data, and annotate the target image segments with crack risk labels; The training set construction module is used to organize the annotated target image segments in chronological order to construct a training set containing crack risk evolution labels. Each sample in the training set reflects the change in crack risk of the fruit during the structural evolution process. a feature vector extraction module, configured to extract a peel surface structure feature vector related to the crack risk from the structure optimization image sequence, and form a feature vector sequence reflecting the evolution of the fruit crack risk based on the feature vector; The evolution trend assessment module is used to analyze the feature vector sequence reflecting the evolution of fruit crack risk based on the crack risk evolution labels in the training set, and evaluate the evolution trend of fruit crack risk; a crack risk assessment module, for determining a crack risk assessment value of the fruit based on the assessment results, and locating a corresponding crack risk area in the structure optimization image sequence; The hierarchical control execution module is used to mark fruits whose crack risk assessment values ​​exceed a preset threshold as crack risk objects, and to perform hierarchical control or culling on the fruits according to the located crack risk areas.

[0010] One or more technical solutions provided in the present invention have at least the following technical effects or advantages: for the problem of durian crack detection, through real-time monitoring, accurate assessment of the changing trend of fruit crack risk and accurate positioning of crack risk areas, the occurrence of potential cracks can be predicted in advance, and the prediction and real-time classification of durian cracks in a dynamic environment can be realized, providing a more accurate and efficient solution for durian quality control. Specifically, the collected image sequence is subjected to structural expression optimization processing to enhance the continuity and recognition sensitivity of the changes in the surface state of the fruit in the image, and improve the recognizability of the crack risk evolution characteristics; using historical image data containing the state before the crack occurs, target image segments with potential crack risk characteristics are extracted and labeled with crack risk labels, and a training data set reflecting the crack evolution process is constructed to lay a data foundation for subsequent crack risk analysis; based on the structural optimization image sequence, the representational features of the image segments are extracted, and combined with the risk evolution label information in the training set, a feature vector sequence reflecting the evolution of the fruit crack risk is extracted to characterize the potential evolution trend of the fruit from the uncracked to the pre-crack state; by analyzing the feature vector sequence, the evolution trend of the fruit crack risk is evaluated, and the crack risk assessment value of each fruit sample is obtained, so as to realize the early risk judgment of the fruit that has not cracked; according to the crack risk assessment value, the corresponding high-risk area in the structural optimization image sequence is located, and the location where the crack risk may be concentrated is clarified from the image space, providing a basis for sorting, grading or early warning control. Compared with the existing technology, a crack risk identification mechanism based on the evolution of image temporal features has been constructed, which can identify the development trend of cracks before they are obviously formed, breaking through the limitation of traditional surface inspection that can only identify defects that have occurred; at the same time, by locating high-risk areas, the accuracy of subsequent sorting and rejection is improved. It is suitable for assembly line-style rapid detection and automated quality inspection tasks, and is particularly suitable for high-value-added fruits such as durian with complex skin and prone to surface changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic diagram of a process flow for an automated quality inspection method for durian production lines based on image recognition provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of an automated quality inspection system for a durian production line based on image recognition is provided in an embodiment of the present application.

[0013] Explanation of the reference numerals: image acquisition and processing module 11 , image segment extraction and annotation module 12 , training set construction module 13 , feature vector extraction module 14 , evolution trend assessment module 15 , rift risk assessment module 16 , hierarchical control execution module 17 . DETAILED DESCRIPTION

[0014] The present invention introduces a dynamic crack risk assessment mechanism, multi-dimensional crack risk assessment and crack risk area positioning, and proposes an automated quality inspection method and system for durian production lines based on image recognition. This solves the problem that the quality inspection of durian fruits on traditional production lines is limited to identifying cracks that have already occurred and lacks early warning of potential crack risks, resulting in insufficient grading accuracy and affecting the rejection and sorting efficiency.

[0015] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] Example 1, as Figure 1 As shown, a durian production line automated quality inspection method based on image recognition includes the following steps: S1. Acquire multiple consecutive frames of images of the fruit as it moves along the conveyor path and perform feature enhancement processing on the images to highlight surface structural features of the peel associated with crack risk. The resulting image sequences are then converted into structure-optimized images. Specifically, this step involves image denoising, edge enhancement, and texture detail enhancement. The resulting image sequence removes background noise, enhances details and edges, and forms a structurally optimized image sequence. This structurally optimized image sequence has higher detail resolution and clearer crack precursor features, providing more accurate image data for subsequent crack risk assessment. The surface structural features of the peel associated with crack risk are shown in Table 1. Table 1 Furthermore, in this implementation, the method for performing feature enhancement processing on an image includes: S101. Denoising the acquired multiple consecutive frames of images to preserve edge features and enhance texture details on the fruit peel surface. During the implementation, each frame of the image is denoised using a Gaussian filter method. A Gaussian kernel is used to weighted average the value of each pixel in the image with the values ​​of surrounding pixels, thereby smoothing the image and removing high-frequency noise. After denoising, a Canny edge detection algorithm is used to highlight areas of the image with large grayscale variations, preserving details and edge information. Finally, local areas of the image (such as the fruit peel surface texture or detailed areas near cracks) are enhanced using histogram equalization to make the texture features clearer in the local area and highlight subtle surface changes. S102. A Laplacian high-pass filter is used to enhance edge features in the image by adjusting its parameters, thereby highlighting the peel surface structure associated with crack risk. During implementation, a standard Laplacian high-pass filter is used to calculate edge and texture changes in the image using second-order derivatives. Based on the actual surface conditions of the durian peel, the parameters of the Laplacian high-pass filter are adjusted, and a 3×3 Laplacian kernel is selected to highlight texture details, particularly subtle changes before cracking. The kernel is: , Select a 5×5 Laplacian kernel to highlight larger crack areas or overall texture structure. The core kernel is: , After being processed by the Laplacian high-pass filter, the edge features related to the crack in the image will be significantly enhanced, the edges of the crack area will be more obvious, and the image details will be more prominent; S103. The image is processed based on a weighted filter, and the weights are adjusted to enhance the texture features of the slight changes before the crack, and finally a structure-optimized image sequence is formed. In the implementation process, based on the weighted filter, the texture details related to the crack risk are highlighted by adjusting the weights. The core of the design of the weighted filter is to adjust the weights of each pixel in the filter to ensure that the area with slight changes before the crack is highlighted in the image. In order to achieve this goal, a custom weight matrix is ​​used. Its design principle is as follows: in the weight matrix of the filter, the weight of the center pixel is set to the maximum value, usually 1; the weights of neighboring pixels are larger, and the weights of pixels far from the center are smaller; the weight adjustment value will be dynamically adjusted according to different areas of the image. The specific design is to select 3×3 and 5×5 window sizes as the basis to balance the relationship between image details and noise smoothing. The 3×3 window is suitable for processing relatively small changes, and its weight matrix is , The center value of 1.0 is used to emphasize the small change area before the crack, and the neighborhood values ​​of 0.2 and 0.4 are used to emphasize the texture detail changes in the adjacent area. The 5×5 window is suitable for processing the crack risk of a larger area. Its weight matrix is , The center value of 1.0 is used to target subtle changes in the crack risk area, and the neighborhood value of 0.1-0.5 with larger values ​​is used to enhance the details of the area around the crack. Using the above-mentioned weight matrix, a weighted filter is used to perform weighted calculation on the surrounding neighborhood at each pixel point in the image. The weighted pixel value will be updated according to the value of the surrounding neighborhood. The tiny cracks and details in the image will be enhanced, especially in the early stage of the crack, the tiny texture changes around the crack will appear, which is convenient for subsequent crack risk assessment. In the process of continuous multi-frame image processing, each frame of the image will be processed by the weighted filter, and finally an image sequence containing optimized texture features is formed, that is, a structurally optimized image sequence. This sequence reflects the dynamic process of the evolution of the crack risk on the fruit surface, and provides reliable data support for subsequent crack trend analysis.

[0018] S2. Based on historically collected durian image data, extract target image segments with potential crack risk characteristics from the image sequence containing the state before cracking, and annotate the target image segments with crack risk labels; Specifically, a comprehensive image database was constructed by shooting several durian fruit samples under different conditions. The image database not only covers images of cracks that have occurred, but also includes images of subtle changes before the cracks occur and images of areas with potential crack risk. The image data in the image database are processed using the denoising, edge enhancement, and texture detail enhancement technologies applied in S1. According to the surface structural features of the peel related to the crack risk, image segments with potential crack risk are extracted from the entire image sequence and organized in chronological order to construct an image sequence containing the state before the crack occurs, reflecting the structural evolution process of the durian peel. Target image segments with crack risk are extracted from this image sequence. Based on the experience of judging the crack risk, each target image segment is annotated with a crack risk label. The crack risk label classification is shown in Table 2. Table 2 S3. Organize the labeled target image segments in chronological order to construct a training set containing crack risk evolution labels. Each sample in the training set reflects the change in crack risk during the fruit's structural evolution. Specifically, the target image segments after organization and annotation will first be extracted according to the continuous motion trajectory of the durian fruit during transportation. Each image segment will be marked according to its acquisition timestamp; then each frame of the image will be sorted by the timestamp to form a continuous image sequence, representing the dynamic state of the durian fruit in the transportation path; finally, each target image segment will be matched with the timestamp and organized into an image sequence in chronological order. Each group of image sequences is a "time window", and the images in each time window reflect the structural evolution of the durian peel in a specific time period. After the target image segments are organized in chronological order, the next step is to construct a training set containing crack risk evolution labels based on these image segments. Each training set sample not only contains a single image data, but also needs to label the crack risk changes of the image segment during the entire structural evolution process. Furthermore, in this implementation process, the construction of the training set of the crack risk evolution label includes the following steps: S301. In the process of historically collecting durian image data, the length of each time window is determined based on a continuous image sequence of durian fruit during transportation, wherein the time window is an image sequence formed by organizing the target image segments in chronological order to reflect the structural evolution of the durian peel within a specific time period. Specifically, the length of the time window is set according to the speed of the durian fruit moving on the conveyor belt and the image acquisition frequency during data acquisition. First, a fixed number of frames is set for each time window. Based on experience, each time window is usually given 5-10 frames of image. Then, the time length corresponding to each time window is determined by the image acquisition frequency. Finally, the actual length of each time window that needs to be defined is calculated based on the speed of the durian fruit moving on the conveyor belt. S302. Match the crack risk label of each target image segment with each frame image in the time window to construct a training sample, ensuring that each time window is associated with a corresponding crack risk label. Specifically, first, the timestamp is used to locate which target image segment each frame belongs to. Since each target image segment has been labeled with a crack risk label, the label is mapped to the corresponding image frame. If target image segment 1 is labeled with a slight risk, all image frames in the time window are associated with this label. If a time window contains multiple target image segments, the label of each target image segment will be assigned to the image frame corresponding to the target image segment until the label of the next target image segment. For example, suppose there are 6 frames of images in a time window, which come from two target image segments, and 3 frames of images in target image segment 1 are labeled with "slight risk". , the three frames in the target image segment 2 are labeled as "medium risk", then in this time window, the 1st, 2nd, and 3rd frames will be labeled as "minor risk", and the 4th, 5th, and 6th frames will be labeled as "medium risk". The label of each target image segment will only affect the image frames corresponding to the target image segment until the next target image segment is encountered. For each time window, record each frame and its corresponding crack risk label, and then use each time window as an independent training sample. These training samples contain each frame in the time window and its corresponding label. Finally, all constructed training samples are sequentially combined into a complete training set; S303. Arrange all time windows and crack risk labels in chronological order, and pre-process the target image segments within the time window to ensure the size consistency and value range adaptation of the image data. Specifically, according to the timestamp of image acquisition, ensure that each time window is arranged in chronological order. For each time window, confirm that each frame image in the window is consistent with the corresponding rift risk label. Suppose we have 3 time windows, time window 1 contains 3 frames of images, and the rift risk label is "minor risk", time window 2 contains 4 frames of images, and the rift risk label is "medium risk", time window 3 contains 5 frames of images, and the rift risk label is "high risk". These time windows should be sorted in sequence according to the timestamp to ensure temporal continuity. All images of the target image segment in the time window are uniformly scaled to 224×224 pixels to ensure that the input data has the same size during training. The pixel values ​​of the images are normalized and scaled to the range of 0-1 so that the model can better adapt to the input data. S304. Store all training samples in a unified data structure and classify the data according to the training set usage scenario. Specifically, use pandas's DataFrame to store data. You can define a DataFrame to store image data and labels, with each row representing a training sample. Based on the rift risk label, the training samples are divided into different data sets, and the samples are classified using pandas' groupby() and saved as different data sets. Then, use the HDF5 format to improve data reading and writing efficiency. During the training process, pandas can be used to efficiently read and load training data.

[0019] S4. Extracting a peel surface structure feature vector associated with the risk of cracking from the structure-optimized image sequence, and forming a feature vector sequence reflecting the evolution of the risk of cracking of the fruit based on the feature vector, Specifically, texture feature vectors, edge feature vectors, deformation feature vectors and color difference feature vectors are extracted from the structure-optimized image sequence, wherein the texture feature is extracted using the Gabor filter method, and each extracted feature vector represents the texture feature of a specific area in the image; the edge feature is extracted using the Canny edge detection method, and each extracted feature vector represents the edge feature of a specific area in the image; the deformation feature is extracted by performing an expansion operation on the image to highlight the tiny structural changes on the peel surface, and then performing an erosion operation to detect and highlight the tiny concave areas on the peel surface caused by internal pressure, force or external influence. , apply morphological gradient to calculate their difference, obtain gradient image, highlight the change of structure, combine Canny edge detection to extract features related to deformation features, extract color difference features, first convert the image from RGB color space to HSV or Lab color space, focus on analyzing the changes in hue, saturation and brightness, then perform differential operation on continuous image frames to identify areas with obvious color difference, highlight the slight color difference changes of the peel before the crack, finally use edge detection method, combine color difference information, extract linear color difference bands, that is, features related to color difference features, based on the extraction of the above feature vectors, form a feature vector sequence reflecting the evolution of fruit crack risk, Furthermore, the method for forming a feature vector sequence includes: S401. Combining the feature vectors of the peel surface structure features extracted and related to the risk of cracks to form a multi-dimensional comprehensive feature vector for each frame image, In the implementation process, different types of single-frame feature vectors are actually extracted from the structure-optimized image sequence, including texture feature vectors, edge feature vectors, deformation feature vectors, and color difference feature vectors. These feature vectors are combined to form a multidimensional comprehensive feature vector. This vector contains information about different types of features and is used to describe the overall structure of the image. The feature combination adopts a splicing method to splice multiple feature vectors together to form a longer feature vector. For example, only two feature vectors are extracted, texture features [f_1, f_2, f_3] and edge features [f_4, f_5, f_6], and the synthesized feature vector is [f_1, f_2, f_3, f_4, f_5, f_6]. S402. The feature vectors of the single frame image are spliced ​​into a complete feature vector sequence of the time window in chronological order. During the implementation process, we must first determine the length of the time window. I will not go into details here on how to determine the length of the time window. Assuming that a time window contains 3 frames of images, the final feature vector sequence will be: the first frame [f_1, f_2, f_3, f_4, f_5, f_6] + the second frame [f_7, f_8, f_9, f_10, f_11, f_12] + the third frame [f_13, f_14, f_15, f_16, f_17, f_18]. Since each time window corresponds to an image sequence of a period of time, all time windows will eventually form a complete feature vector sequence of the time window. S403. Based on the feature vector sequence of the time window, through feature optimization and selection, redundant features are removed and the features that best reflect the risk of the breach are retained. During implementation, since the features extracted from the image may be redundant or irrelevant to the breach risk, the principal component analysis (PCA) method is used to reduce the dimensionality of the features, remove redundant features, and retain the most discriminative features. All feature vectors are combined into a matrix, with each row being the feature vector of a sample. By calculating the covariance matrix and extracting the principal components, high-dimensional features are mapped to lower dimensions. In addition, a mutual information-based method is required to determine the features most relevant to the evolution of breach risk and select the most discriminative features. In feature selection, the mutual information value between each feature and the breach risk label is calculated to determine which features are most relevant. The mutual information value between all features and the breach risk label is also calculated. The features most relevant to the breach risk are screened out based on the size of the mutual information, the most important features are retained, and the features with small mutual information are eliminated. S404. Combine each time window with the corresponding crack risk label to finally obtain a feature vector sequence reflecting the evolution of fruit crack risk. During the implementation process, the feature vector sequence of each time window is labeled with a crack risk label and combined with the crack risk label to finally obtain a feature vector sequence reflecting the evolution of fruit crack risk and also containing the corresponding crack risk information.

[0020] S5. Based on the crack risk evolution labels in the training set, analyze the feature vector sequence reflecting the evolution of fruit crack risk and evaluate the evolution trend of fruit crack risk. Specifically, the crack risk evolution labels in the training set provide basic data for analysis. In each time window, the peel surface structure is a dynamic change process, which is reflected in the feature vector sequence. Each time window corresponds to the peel surface structure characteristics related to the crack risk within a period of time, and these feature vector sequences reflect the changes in the peel surface structure. By analyzing the evolution of the feature vector sequence, the evolution trend of the crack risk can be revealed. By analyzing the change trend of the feature vector sequence, the evolution trend of the crack risk can be evaluated. This process is not only an observation of the peel surface structure characteristics related to the crack risk, but also includes how to quantify the evolution of the crack risk by calculating the rate and difference of change of the peel surface structure characteristics related to the crack risk. Furthermore, the process of analyzing the feature vector sequence reflecting the evolution of the fruit crack risk and evaluating the evolution trend of the fruit crack risk includes: S501. Based on the feature vector sequence reflecting the evolution of the risk of fruit cracking, the difference and change rate of the feature vector of each frame in each time window are calculated to monitor the change rate of the risk of fruit cracking. During the implementation process, within each time window, the feature vectors of continuous image frames represent the structure of the fruit peel at different time points. The difference values ​​of these feature vectors can be calculated by the Euclidean distance method, and then the changes in each frame of the image can be quantified to monitor the changes in the surface structure of the fruit peel. The rate of change of the fruit peel surface structure is represented by calculating the rate of change between adjacent frames in each time window. The rate of change = difference value / time interval, where the time interval is the time difference between image frames. By calculating the difference value and the rate of change, the rate of change of the fruit peel surface structure can be monitored. The risk of cracking is usually related to the rapid change of the surface characteristics of the fruit peel. The average value of the rate of change within the time window is taken to represent the rate of change of the risk of fruit cracking. S502. Analyze the evolution trend of the fruit crack risk, and infer the change stage of the fruit crack risk based on the feature evolution trend under different crack risk labels in the training set. During the implementation process, According to the data characteristics of the characteristic vector sequence of the fruit crack risk evolution, the ARIMA model is selected to analyze the trend of the time series, determine the parameters of the ARIMA model, and select p, d, and q as the main parameters of the ARIMA model. p (autoregressive term): indicates how many time steps of data are used to predict the current value. The PACF graph is used to select the appropriate p value. The PACF graph shows the relationship between each time lag and the current value. The significant lag is selected as the p value. d (number of differences): indicates how many times the difference needs to be made in order to make the data stable. The unit root test is used to determine whether the data needs to be differentiated and to determine the d value. q (moving average term): represents the relationship between the current data point and the previous q residuals. The ACF plot is used to determine the q value. The ACF plot shows the relationship between the residuals and past data points. The significant lag is selected as the q value. The feature vector of each image frame in the feature vector sequence reflecting the evolution of the risk of fruit cracking is input into the ARIMA model for fitting. The residual of the ARIMA model is checked to see if it conforms to white noise. Based on the fitted ARIMA model, the trend of the feature vector changes in the next few frames of images is predicted, i.e., the trend of the feature vector sequence of the evolution of the risk of fruit cracking. Combining the crack risk labels in the training set and the results of the evolution trend of the fruit crack risk, the fruit crack risk change stages are inferred, including no risk stage, slight risk stage, medium risk stage, and high risk stage. At the same time, an intensity value is assigned to each fruit crack risk change stage. The intensity value is between 0 and 1.0. The no risk stage corresponds to an intensity value of 0, the slight risk stage corresponds to an intensity value of 0.25, the medium risk stage corresponds to an intensity value of 0.5, and the high risk stage corresponds to an intensity value of 1.0. S503. Analyze the similarity between the feature vector of the current time window and each historical feature vector in the training set to determine the feature correlation. During the implementation process, the feature correlation degree represents the similarity between the current fruit crack feature and the historical data (or known crack risk status), reflecting the relative intensity of the current crack risk. The cosine similarity algorithm is used to calculate the cosine similarity value between the feature vector of the current time window and each historical feature vector in the training set. The obtained cosine similarity value is usually between -1 and 1. The larger the cosine similarity value, the more similar it is. The average of the top three cosine similarity values ​​is the feature correlation degree of the time window t.

[0021] S6. Determine the risk assessment value of the fruit crack according to the assessment results, and locate the corresponding high-risk area in the structure optimization image sequence. Furthermore, during the implementation process, a comprehensive crack risk assessment value is constructed based on the evaluation results of the evolution trend of the fruit crack risk. The crack risk assessment value is calculated by weighted summing the feature changes in each time window and combining the feature evolution trends under different crack risk labels to quantify the intensity and evolution trend of the fruit crack risk. The calculation formula of the crack risk assessment value is: , Where R(t) is the breach risk assessment value in time window t, F(t) is the fruit crack risk change rate in time window t, S(t) is the intensity value of the fruit crack risk change stage in time window t, C(t) is the feature correlation degree in time window t, ɑ is the fruit crack risk change rate weight, β is the fruit crack risk change stage weight, and γ is the feature correlation degree weight. The fruit crack risk change rate weight ɑ, fruit crack risk change stage weight β, and feature correlation weight γ are calculated by The weight coefficient is determined by analyzing the importance of data features F(t), the intensity value S(t) of the fruit crack risk change stage, and the feature correlation C(t). The decision tree model is used to evaluate the change rate of fruit crack risk. F(t), the intensity value S(t) of the fruit crack risk change stage and the feature correlation C(t), the contribution of the target variable prediction, and the weights of these risk assessment indicators are assigned accordingly. Based on the feature vector of each time window, the rate of change of fruit crack risk, the intensity value of the fruit crack risk change stage, the feature correlation, and the crack risk label, data preprocessing is performed, including standardization, missing value processing, and categorical variable encoding. A decision tree model is constructed, and a decision tree classifier is used to train the model. The importance of each risk assessment indicator is evaluated. During the evaluation process, the importance score of each risk assessment indicator is returned. This score reflects the contribution of the risk assessment indicator when the model makes a prediction. The importance score of each risk assessment indicator evaluated by the decision tree is a relative value. They need to be normalized to obtain a standardized weight coefficient to ensure that the sum of the weight coefficients of all risk assessment indicators is 1 and they reflect the contribution of each feature to the crack risk. To set the threshold of the breach risk assessment value, first determine the threshold cutoff point. According to the breach risk assessment value calculated based on historical data, the distribution of the breach risk assessment value R(t) calculated based on the historical data corresponding to the breach risk label is statistically analyzed. Based on the distribution, the threshold cutoff point is determined using the Otsu method. The risk label of the gap is no risk, R(t)<0.2 The risk label of the breach is slight risk, 0.2≤R(t)<0.4, The risk label of the breach is medium risk, 0.4≤R(t)<0.6 The risk label of the breach is high risk, R(t)≥0.6, Furthermore, during the implementation process, the corresponding crack risk area in the structure optimization image sequence is simultaneously located. The method includes: The spatial gradient of the crack risk assessment value in each frame of the structural optimization image sequence is calculated to identify areas with significant crack risk changes. Specifically, the spatial gradient refers to the rate of change of the crack risk assessment value at a point in the image relative to its surrounding neighborhood points. The spatial gradient can be used to identify areas in the image where crack risk changes dramatically, namely, areas of significant change. The gradient of each pixel in each image is calculated using the Sobel operator. Based on the gradient amplitude distribution of the training set images, the pixel point with the 95th percentile gradient amplitude is selected as the threshold. If the gradient amplitude of a region is greater than this threshold, the crack risk in that region is considered to have changed significantly. The areas with significant changes in crack risk are grouped to obtain multiple risk-similar areas. The areas with significant changes in crack risk in the image are identified and classified as crack risk areas. Specifically, cluster analysis is performed on the areas with significant changes in crack risk based on their area size, boundary coordinates, and average gradient amplitude. The K-means clustering algorithm is used to group the areas based on gradient amplitude and spatial position similarity. The significantly changed areas belonging to the same category are grouped into a risk-similar area group, and area markers are generated in the image. Areas with higher risks are screened out based on the threshold of the breach risk assessment value. If the risk index of an area exceeds the threshold of the breach risk assessment value, the area is determined to be a breach risk area. Specifically, for each risk-similar area in the clustering results, its average gradient amplitude is calculated and compared with the breach risk assessment value threshold. If the mean value of the area exceeds the breach risk assessment value threshold, it is determined to be a breach risk area. Mark the crack risk areas in the image and record the size, boundary coordinates and crack risk label of each crack risk area.

[0022] S7. Marking fruits whose crack risk assessment value exceeds a preset threshold as crack risk objects, and performing graded control or culling on the fruits according to the crack risk areas located, During the implementation process, when the crack risk assessment value R(t)<0.2, the fruit is assessed and marked as a non-crack risk object; when 0.2≤R(t)<0.4, the fruit is marked as a slight crack risk object; when 0.4≤R(t)<0.6, the fruit is marked as a medium crack risk object; when R(t)≥0.6, the fruit is marked as a high crack risk object; for fruits with a slight crack risk or a medium crack risk, further graded control treatment is carried out, and these fruits are classified and placed in different areas or storage areas, and given different storage conditions, such as temperature and humidity control, oxygen circulation, etc., to delay the possibility of cracking; for objects with a high crack risk, they are directly eliminated to prevent them from affecting the overall fruit quality or causing other fruits to rot.

[0023] Example 2, based on the same inventive concept as the above-mentioned embodiment of a method for automated quality inspection of durian production line based on image recognition, as Figure 2 As shown, the present application provides an automated quality inspection system for durian production lines based on image recognition. The system and method embodiments in the present application are based on the same inventive concept. The system includes: The image acquisition and processing module 11 is used to obtain multiple frames of continuous images of the fruit during its movement along the conveying path, and perform feature enhancement processing on the images to highlight the surface structural features of the peel related to the risk of cracking. The processing results form a structure-optimized image sequence; An image segment extraction and labeling module 12 is configured to extract target image segments with potential crack risk characteristics from an image sequence containing a state before cracking based on historically collected durian image data, and label the target image segments with crack risk labels; The training set construction module 13 is used to organize the labeled target image segments in chronological order to construct a training set containing crack risk evolution labels, where each sample in the training set reflects the change in crack risk of the fruit during the structural evolution process; a feature vector extraction module 14 for extracting a peel surface structure feature vector related to the crack risk from the structure optimization image sequence, and forming a feature vector sequence reflecting the evolution of the fruit crack risk based on the feature vector; An evolution trend assessment module 15 is used to analyze the feature vector sequence reflecting the evolution of the fruit crack risk based on the crack risk evolution labels in the training set, and assess the evolution trend of the fruit crack risk; a crack risk assessment module 16 for determining a crack risk assessment value of the fruit according to the assessment result and locating a corresponding crack risk area in the structure optimization image sequence; The hierarchical control execution module 17 is used to mark fruits whose crack risk assessment values ​​exceed a preset threshold as crack risk objects, and perform hierarchical control or culling on the fruits according to the located crack risk areas.

[0024] The automatic quality inspection system for durian production lines based on image recognition provided by an embodiment of the present invention can execute the automatic quality inspection method for durian production lines based on image recognition provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0025] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention.

[0026] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A durian production line automated quality inspection method based on image recognition, characterized in that, The following steps are included: Acquire multiple consecutive frames of images of the fruit as it moves along the conveyor path, and perform feature enhancement processing on the images to highlight the surface structural features of the peel associated with the risk of cracking. The resulting image sequence is then transformed into a structure-optimized image sequence. Based on historically collected durian image data, target image segments with potential crack risk characteristics are extracted from the image sequence containing the state before cracking, and crack risk labels are annotated on the target image segments. The labeled target image segments are organized in chronological order to construct a training set containing crack risk evolution labels. Each sample in the training set reflects the change in crack risk during the fruit's structural evolution. Extracting a peel surface structure feature vector related to the crack risk from the structure optimization image sequence, and forming a feature vector sequence reflecting the evolution of the fruit crack risk based on the feature vector; Based on the crack risk evolution labels in the training set, the feature vector sequence reflecting the evolution of fruit crack risk is analyzed to evaluate the evolution trend of fruit crack risk. Determine the fruit crack risk assessment value based on the assessment results, and locate the corresponding crack risk area in the structure optimization image sequence; Fruits with crack risk assessment values ​​exceeding a preset threshold are marked as crack risk objects, and the fruits are graded and controlled or removed based on the located crack risk areas.

2. The automated quality inspection method according to claim 1, characterized in that: The method for performing feature enhancement processing on the image includes: De-noising is performed on the acquired continuous multi-frame images to retain the edge features in the images and enhance the texture details of the peel surface; A Laplacian high-pass filter is used to enhance edge features in the image by adjusting the filter parameters, thereby highlighting the surface structure of the peel related to the risk of cracking. The image is processed based on a weighted filter, and the weights are adjusted to enhance the texture features of slight changes before the crack, ultimately forming a structure-optimized image sequence.

3. The automated quality inspection method according to claim 2, characterized in that: The construction of the training set of the rift risk evolution label includes: In the process of historically collected durian image data, the length of each time window is determined based on a continuous image sequence of durian fruits during transportation. The time window is an image sequence formed by organizing the target image segments in chronological order to reflect the structural evolution of the durian peel within a specific time period. Match the crack risk label of each target image segment with each frame image in the time window to construct a training sample, ensuring that each time window is associated with the corresponding crack risk label; Arrange all time windows and crack risk labels in chronological order, and preprocess the target image segments within the time window to ensure the size consistency and value range adaptation of the image data; All training samples are stored in a unified data structure, and the data are classified and stored according to the usage scenarios of the training set.

4. The automated quality inspection method according to claim 3, characterized in that: The method for forming a feature vector sequence reflecting the evolution of the risk of fruit cracking includes: The feature vectors of the peel surface structural features related to the risk of cracking are combined to form a multi-dimensional comprehensive feature vector for each frame of image; The feature vectors of a single frame image are spliced ​​into a complete feature vector sequence of a time window in chronological order through the time window; Based on the feature vector sequence of the time window, through feature optimization and selection, redundant features are removed and the features that best reflect the breach risk are retained; Each time window is combined with the corresponding crack risk label to finally obtain a feature vector sequence reflecting the evolution of fruit crack risk.

5. The automated quality inspection method according to claim 4, characterized in that: The evolving trends in assessing fruit crack risk include: Based on the feature vector sequence reflecting the evolution of fruit crack risk, the difference and change rate of the feature vector of each frame in each time window are calculated to monitor the change rate of fruit crack risk. Analyze the evolution trend of fruit crack risk and infer the change stage of fruit crack risk based on the feature evolution trend under different crack risk labels in the training set; Analyze the similarity between the feature vector of the current time window and each historical feature vector in the training set to determine the feature correlation.

6. The automated quality inspection method according to claim 5, characterized in that: Determining the risk assessment value of the fruit crack according to the assessment result includes: According to the evaluation results of the evolution trend of the fruit crack risk, the crack risk assessment value is calculated, the evolution trend of the fruit crack risk is quantified, and the threshold of the crack risk assessment value is set. The calculation formula of the breach risk assessment value is: , Where R(t) is the breach risk assessment value in time window t, F(t) is the fruit crack risk change rate in time window t, S(t) is the intensity value of the fruit crack risk change stage in time window t, C(t) is the feature correlation in time window t, ɑ is the fruit crack risk change rate weight, β is the fruit crack risk change stage weight, and γ is the feature correlation weight.

7. The automated quality inspection method according to claim 6, characterized in that: The method for locating the corresponding crack risk area in the structure optimization image sequence includes: Calculate the spatial gradient of the crack risk assessment value in each frame of the structure optimization image sequence to identify areas with significant crack risk changes; The areas with significant changes in crack risk are grouped to obtain multiple areas with similar risks, and the areas with significant changes in crack risk in the image are identified and classified as crack risk areas; Screen out areas with higher risks based on the threshold of the breach risk assessment value. If the risk index of an area exceeds the threshold of the breach risk assessment value, the area is determined to be a breach risk area. Mark the crack risk areas in the image and record the size, boundary coordinates and crack risk label of each crack risk area.

8. A durian production line automated quality inspection system based on image recognition, characterized in that: The method for implementing the automatic quality inspection method for durian production line based on image recognition according to any one of claims 1 to 7 comprises: The image acquisition and processing module is used to obtain continuous multi-frame images of the fruit during its movement along the conveyor path, and perform feature enhancement processing on the images to highlight the surface structural features of the peel related to the risk of cracking. The processing results form a structure-optimized image sequence; An image segment extraction and annotation module is used to extract target image segments with potential crack risk characteristics from an image sequence containing the state before cracking based on historically collected durian image data, and annotate the target image segments with crack risk labels; The training set construction module is used to organize the annotated target image segments in chronological order to construct a training set containing crack risk evolution labels. Each sample in the training set reflects the change in crack risk of the fruit during the structural evolution process. a feature vector extraction module, configured to extract a peel surface structure feature vector related to the crack risk from the structure optimization image sequence, and form a feature vector sequence reflecting the evolution of the fruit crack risk based on the feature vector; The evolution trend assessment module is used to analyze the feature vector sequence reflecting the evolution of fruit crack risk based on the crack risk evolution labels in the training set, and evaluate the evolution trend of fruit crack risk; a crack risk assessment module, for determining a crack risk assessment value of the fruit based on the assessment results, and locating a corresponding crack risk area in the structure optimization image sequence; The hierarchical control execution module is used to mark fruits whose crack risk assessment values ​​exceed a preset threshold as crack risk objects, and to perform hierarchical control or culling on the fruits according to the located crack risk areas.

Images