A durian pipeline automatic quality inspection method and system based on image recognition

Abstract

The application discloses a kind of based on image recognition's durian flow line automatic quality inspection method and system, it is related to intelligent visual detection technical field, it is suitable for the automation detection needs of durian surface crack identification and risk prediction.The method comprises: based on historical acquisition image, from the image sequence containing the state before crack occurrence Extract target image segment with potential crack risk characteristics and carry out risk label annotation;Image sequence is carried out structure expression optimization processing;The characteristic feature of image segment is extracted, and the feature vector sequence reflecting the evolution of fruit crack risk is constructed in combination with training set label information;The sequence is analyzed to evaluate the evolution trend of fruit crack risk, obtain crack risk evaluation value, and locate the corresponding high-risk area in image sequence based on the evaluation result.Through the above technical scheme, early identification and accurate positioning of durian crack risk are realized, and the intelligence and reliability of flow line quality inspection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent visual detection, and particularly relates to a durian pipeline automatic quality inspection method and system based on image recognition. BACKGROUND

[0002] At present, fruit quality inspection has widely adopted image recognition and automatic sorting technology. Especially in standard fruits such as apples, citrus and bananas, image recognition algorithms can automatically judge the color, size, shape and surface defects of the fruits, and then realize real-time grading and efficient sorting. However, for durians with complex structure and irregular surface features, the existing quality inspection methods still have great technical bottlenecks in image recognition accuracy, feature extraction capability and grading determination strategy.

[0003] The appearance characteristics of durians are relatively complex, and the epidermis presents irregular protrusions, grooves and other structures, and is easily affected by the external environment. For example, cracks may occur on the fruit skin during transportation and storage. The cracks not only affect the appearance quality of the durians, but also accelerate the decay process of the durians, become a channel for microorganisms and bacteria to invade, and seriously affect the edible value and shelf life of the durians. Therefore, crack detection is not only a key link in durian quality control, but also a necessary measure to ensure the safety of durians and prolong their shelf life.

[0004] At present, for the detection of durian cracks, most of the existing technologies focus on fruit surface defect recognition methods based on image recognition. These methods usually rely on traditional image processing techniques such as edge detection, grayscale analysis, etc., and determine whether the fruit skin has cracks through pre-set rules or feature extraction. However, these methods have obvious shortcomings: on the one hand, the texture of durian skin is complex and irregular, and cracks often show subtle changes, making it difficult for traditional methods to accurately identify micro-cracks or potential cracks; on the other hand, existing image processing techniques cannot cope with the dynamic changes of durian skin cracks during transportation and storage, and cannot evaluate the evolution trend of crack risk in real time, resulting in that the detection of cracks is only limited to the state where the fruit cracks have occurred, and lacks early warning of potential cracks.

[0005] In addition, since durians are non-standard fruits, their surface features and crack states are different, and the unified processing method of existing technologies fails to fully consider the individual differences of the fruits. For example, the size, depth and stage of occurrence of cracks will affect the crack risk level of the fruits. Existing methods are difficult to dynamically and individually assess the crack risk of different durians, which leads to large errors in grading determination, and further affects the accuracy and efficiency of the sorting and rejection process.

[0006] Therefore, the durian crack detection method based on image recognition still faces many challenges, and technical breakthroughs are needed in terms of crack detection accuracy, dynamic and personalized processing of crack risk assessment, etc. SUMMARY

[0007] To solve the above problems, the present application provides a durian pipeline automatic quality inspection method and system based on image recognition, which can accurately identify and assess the risk of durian fruit surface cracks through crack risk evolution assessment and crack risk area positioning, and realize early warning and accurate grading control and management of durian cracks by monitoring the change trend of fruit cracks in real time.

[0008] To achieve the above purpose, in a first aspect, the present application provides a durian pipeline automatic quality inspection method based on image recognition, comprising the following steps,

[0009] Obtain continuous multiple frames of images of the fruit during its movement in the conveying path, and perform feature enhancement processing on the images to highlight the skin surface structure features related to crack risk, and the processing result forms a structure optimized image sequence;

[0010] Based on the historical durian image data, extract target image segments with potential crack risk features from the image sequence containing the pre-crack state, and label the target image segments with crack risk labels;

[0011] Organize the labeled target image segments in chronological order to construct a training set containing crack risk evolution labels, and each sample in the training set reflects the change of crack risk of the fruit in the structure evolution process;

[0012] Extract the skin surface structure feature vector related to crack risk from the structure optimized image sequence, and form a feature vector sequence reflecting the evolution of fruit crack risk based on the feature vector;

[0013] Based on the crack risk evolution labels in the training set, analyze the feature vector sequence reflecting the evolution of fruit crack risk to assess the evolution trend of fruit crack risk;

[0014] Determine the crack risk assessment value of the fruit according to the evaluation result, and locate the corresponding crack risk area in the structure optimized image sequence;

[0015] Mark the fruit with a crack risk assessment value exceeding a preset threshold as a crack risk object, and perform grading control or rejection processing on the fruit according to the located crack risk area.

[0016] In a second aspect, the present application provides a durian pipeline automatic quality inspection system based on image recognition, comprising,

[0017] An image acquisition and processing module is configured to acquire a plurality of continuous images of the fruit during its movement along the conveying path, and to perform feature enhancement processing on the images to highlight the surface structure features of the fruit skin that are related to the crack risk, and the processing result forms a sequence of structure-optimized images;

[0018] An image segment extraction and labeling module is configured to extract target image segments with potential crack risk features from the image sequence containing the pre-crack state based on the historical acquisition of durian image data, and to label the target image segments with crack risk labels;

[0019] A training set construction module is configured to organize the labeled target image segments in chronological order to construct a training set containing crack risk evolution labels, and each sample in the training set reflects the change of the crack risk of the fruit in the structure evolution process;

[0020] A feature vector extraction module is configured to extract the surface structure feature vector of the fruit skin related to the crack risk from the sequence of structure-optimized images, and to form a feature vector sequence reflecting the evolution of the crack risk of the fruit based on the feature vector;

[0021] An evolution trend evaluation module is configured to analyze the feature vector sequence reflecting the evolution of the crack risk of the fruit based on the crack risk evolution labels in the training set, and to evaluate the evolution trend of the crack risk of the fruit;

[0022] A crack risk evaluation module is configured to determine the crack risk evaluation value of the fruit according to the evaluation result, and to locate the corresponding crack risk area in the sequence of structure-optimized images;

[0023] A hierarchical control execution module is configured to mark the fruit with a crack risk evaluation value exceeding a preset threshold as a crack risk object, and to perform hierarchical control or rejection processing on the fruit according to the located crack risk area.

[0024] The one or more technical solutions provided in the present application have at least the following technical effects or advantages: for durian crack detection, by real-time monitoring, accurate evaluation of the change trend of fruit crack risk, and accurate positioning of the crack risk area, the occurrence of potential cracks can be predicted in advance, the prediction and real-time grading processing of durian cracks in a dynamic environment are realized, and a more accurate and efficient solution for durian quality control is provided. Specifically, the collected image sequence is subjected to structural expression optimization processing, the continuity and recognition sensitivity of the image representing the change of the fruit surface state are enhanced, and the degree of distinguishability of the crack risk evolution feature is improved; historical image data containing the state before the crack occurs is used to extract target image segments with potential crack risk features and label the crack risk labels, and a training data set reflecting the crack evolution process is constructed, laying a data foundation for subsequent crack risk analysis; the structural optimization image sequence is used to extract the feature of the image segment, and the risk evolution label information in the training set is combined to extract a feature vector sequence reflecting the evolution of the fruit crack risk, which is used to represent the potential evolution trend of the fruit from the uncracked state to the pre-crack state; by analyzing the feature vector sequence, the evolution trend of the fruit crack risk is evaluated, and a crack risk evaluation value of each fruit sample is obtained, realizing early risk judgment of uncracked fruits; according to the crack risk evaluation value, the high-risk area in the structural optimization image sequence is located, and the position where the crack risk may concentrate is determined from the image space, providing a basis for sorting, grading or early warning control. Compared with the prior art, a crack risk recognition mechanism based on image timing feature evolution is constructed, which can recognize the development trend of the crack before it is obviously formed, breaking through the limitation of traditional surface detection that can only recognize the defects that have occurred; at the same time, through the positioning of the high-risk area, the accuracy of subsequent sorting and rejection is improved, which is suitable for pipeline type rapid detection and automatic quality inspection tasks, and is particularly suitable for high-value fruits such as durians with complex skin and prone to surface changes. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 A flowchart of a durian pipeline automatic quality inspection method based on image recognition provided by the embodiment of the present application is shown in the figure.

[0027] Figure 2 A structural schematic diagram of a durian pipeline automatic quality inspection system based on image recognition provided by the embodiment of the present application is shown in the figure.

[0028] Explanation of reference signs: image acquisition and processing module 11, image segment extraction and labeling module 12, training set construction module 13, feature vector extraction module 14, evolution trend evaluation module 15, crack risk evaluation module 16, hierarchical control execution module 17. DETAILED DESCRIPTION

[0029] The present application introduces a dynamic crack risk evaluation mechanism, multi-dimensional crack risk evaluation, and crack risk area positioning, and proposes a durian pipeline automatic quality inspection method and system based on image recognition, which solves the problem that traditional pipeline quality inspection for durian fruits is only limited to identifying cracks that have occurred, lacks early warning of potential crack risks, and leads to insufficient grading accuracy, affecting rejection and sorting efficiency.

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and the above drawings of the present application are used to distinguish similar objects, and do not necessarily mean a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment one, as shown in a durian pipeline automatic quality inspection method based on image recognition, comprising the following steps, Figure 1

[0033] S1. Obtain continuous multiple frames of images of the fruit during the movement process in the conveying path, and perform feature enhancement processing on the images to highlight the skin surface structure features related to crack risk, and the processing result forms a structure optimized image sequence;

[0034] ​Specifically, the step includes denoising, edge enhancement, and texture detail enhancement processing of the images, and the obtained image sequence removes background noise, enhances details and edges, and forms a structure-optimized image sequence. The structure-optimized image sequence has higher detail resolution and clearer crack precursor features, providing more accurate image data for subsequent crack risk assessment. The fruit peel surface structure features related to crack risk are shown in Table 1,

[0035]

[0036] Table 1

[0037] Further, in the present embodiment, the method of image feature enhancement processing includes,

[0038] S101. Denoising the obtained continuous multiple frames of images, retaining the edge features in the images, and enhancing the texture details of the fruit peel surface; in the implementation process, a Gaussian filter method is used to denoise each frame of image, the value of each pixel point in the image is weighted and averaged with the surrounding neighborhood pixel values using a Gaussian kernel, thereby smoothing the image and removing high-frequency noise. After denoising, the Canny edge detection algorithm is used to highlight the areas with large gray scale changes in the image, retaining the detail and edge information. Finally, histogram equalization is used to enhance the local area of the image (if the peel surface texture or crack detail area) so that the texture features become clearer in the local range, highlighting the small surface changes;

[0039] S102. Use a Laplacian high-pass filter to enhance the edge features in the image by adjusting the parameters of the filter, thereby highlighting the fruit peel surface structure related to crack risk; in the implementation process, a standard Laplacian high-pass filter is used to calculate the edge and texture changes in the image by second-order derivative, and the parameters of the Laplacian high-pass filter are adjusted according to the actual situation of the durian peel surface. A 3x3 Laplacian kernel is selected to highlight the detail texture, especially the subtle changes before the crack occurs, and the core kernel is:

[0040] ,

[0041] A 5x5 Laplacian kernel is selected to highlight the larger crack area or overall texture structure, and the core kernel is:

[0042] ,

[0043] After processing by the Laplacian high-pass filter, the edge features related to the crack in the image will be significantly enhanced, and the edges of the crack area will be more obvious, and the image details will be more prominent;

[0044] S103. The image is processed based on a weighted filter, and the weights are adjusted to enhance the texture features of the slight changes before the crack, and finally a structure-optimized image sequence is formed. In the implementation process, based on the weighted filter, the texture details related to the crack risk are highlighted by adjusting the weights. The core of the design of the weighted filter is to adjust the weights of each pixel in the filter to ensure that the area with slight changes before the crack is highlighted in the image. In order to achieve this goal, a custom weight matrix is ​​used. Its design principle is as follows: in the weight matrix of the filter, the weight of the center pixel is set to the maximum value, usually 1; the weights of neighboring pixels are larger, and the weights of pixels far from the center are smaller; the weight adjustment value will be dynamically adjusted according to different areas of the image. The specific design is to select 3×3 and 5×5 window sizes as the basis to balance the relationship between image details and noise smoothing. The 3×3 window is suitable for processing relatively small changes, and its weight matrix is

[0045] ,

[0046] The center value of 1.0 is used to emphasize the small change area before the crack, and the neighborhood values ​​of 0.2 and 0.4 are used to emphasize the texture detail changes in the adjacent area. The 5×5 window is suitable for processing the crack risk of a larger area. Its weight matrix is

[0047] ,

[0048] The center value of 1.0 is used to target subtle changes in the crack risk area, and the neighborhood value of 0.1-0.5 with larger values ​​is used to enhance the details of the area around the crack. Using the above-mentioned weight matrix, a weighted filter is used to perform weighted calculation on the surrounding neighborhood at each pixel point in the image. The weighted pixel value will be updated according to the value of the surrounding neighborhood. The tiny cracks and details in the image will be enhanced, especially in the early stage of the crack, the tiny texture changes around the crack will appear, which is convenient for subsequent crack risk assessment. In the process of continuous multi-frame image processing, each frame of the image will be processed by the weighted filter, and finally an image sequence containing optimized texture features is formed, that is, a structurally optimized image sequence. This sequence reflects the dynamic process of the evolution of the crack risk on the fruit surface, and provides reliable data support for subsequent crack trend analysis.

[0049] S2. Based on historically collected durian image data, extract target image segments with potential crack risk characteristics from the image sequence containing the state before cracking, and annotate the target image segments with crack risk labels;

[0050] Specifically, a comprehensive image database is constructed by shooting a plurality of durian fruit samples under different conditions, which not only covers images of cracks that have occurred, but also includes images of slight changes before the cracks occur and potential crack risk areas, and the image data of the image database is processed by using the denoising, edge enhancement, and texture detail enhancement application techniques in S1, according to the fruit skin surface structure features related to crack risk, the image segments with potential crack risk are extracted from the entire image sequence and organized in time sequence to construct an image sequence containing the pre-crack state, reflecting the structural evolution process of the durian fruit skin, and the target image segments with crack risk are extracted from the image sequence, and crack risk labels are labeled for each target image segment by judging crack risk experience, and the crack risk label classification is shown in Table 2,

[0051]

[0052] Table 2

[0053] S3. The labeled target image segments are organized in time sequence to construct a training set containing crack risk evolution labels, and each sample in the training set reflects the change of crack risk in the structural evolution process of the fruit,

[0054] Specifically, the labeled target image segments are organized, first, according to the continuous motion trajectory of the durian fruit in the conveying process, each image segment is labeled according to its time stamp; then each frame of image is sorted by time stamp to form a continuous image sequence, representing the dynamic state of the durian fruit in the conveying path; finally, each target image segment is matched with the time stamp and organized in time sequence to form an image sequence, each group of image sequences is a "time window", and the images in each time window reflect the structural evolution of the durian fruit skin in a specific time period,

[0055] After the target image segments are organized in time sequence, the next step is to construct a training set containing crack risk evolution labels based on these image segments, each training set sample not only contains single image data, but also needs to label the crack risk change of the image segment in the entire structural evolution process,

[0056] Further, in the present embodiment, the construction of the crack risk evolution label training set includes the following steps,

[0057] S301. In the process of historical durian image data collection, according to the continuous image sequence of the durian fruit in the conveying process, the length of each time window is determined, and the time window is the image sequence reflecting the structural evolution of the durian fruit skin in a specific time period formed after the target image segments are organized in time sequence,

[0058] Specifically, the length of the time window is set according to the speed of the durian fruit moving on the conveyor belt and the image acquisition frequency at the time of data acquisition. First, a fixed number of frames is set for each time window, and according to experience, 5-10 frames of images are usually included in each time window. Then the corresponding time length of each time window is determined by the image acquisition frequency. Finally, the actual length of each time window is calculated according to the speed of the durian fruit moving on the conveyor belt.

[0059] S302. Match the crack risk label of each target image segment with each frame of image in the time window, and construct a training sample to ensure that each time window is associated with the corresponding crack risk label,

[0060] Specifically, first, locate which target image segment each frame of image belongs to through the timestamp. Since each target image segment has been labeled with a crack risk label, the label is mapped to the corresponding image frame. If target image segment 1 is labeled as "minor risk", all image frames in the time window are associated with this label. If a time window contains multiple target image segments, the label of each target image segment will be assigned to the image frame corresponding to the target image segment until the label of the next target image segment. For example, assuming that there are 6 frames of image in a time window, which come from two target image segments, 3 frames of image in target image segment 1 are labeled as "minor risk", and 3 frames of image in target image segment 2 are labeled as "moderate risk". Then in this time window, the 1st, 2nd and 3rd frames are labeled as "minor risk", and the 4th, 5th and 6th frames are labeled as "moderate risk". The label of each target image segment will only affect the image frame corresponding to the target image segment until the next target image segment is encountered. For each time window, record each frame of image and its corresponding crack risk label. Then each time window is taken as an independent training sample. These training samples contain each frame of image in the time window and its corresponding label. Finally, all the constructed training samples are combined in order to form a complete training set;

[0061] S303. Arrange all time windows and crack risk labels in chronological order, and preprocess the target image segments in the time window to ensure the size consistency and numerical range adaptation of the image data,

[0062] Specifically, according to the timestamp of image acquisition, ensure that each time window is arranged in chronological order, for each time window, confirm that each frame of image in the window is consistent with the corresponding crack risk label, assuming that we have 3 time windows, time window 1 contains 3 frames of images, the crack risk label is "slight risk", time window 2 contains 4 frames of images, the crack risk label is "moderate risk", and time window 3 contains 5 frames of images, the crack risk label is "high risk", the order of these time windows in the dataset should be arranged in chronological order according to the timestamp, ensuring that they are continuous in time, all images of the target image segment in the time window are uniformly scaled to 224x224 pixels, ensuring that the input data has the same size when training, and the pixel values of the images are standardized to the range of 0-1, so that the model can better adapt to the input data;

[0063] S304. Store all training samples in a unified data structure, and store the data in different categories according to the training set usage scenario,

[0064] Specifically, use pandas' DataFrame to store data, you can define a DataFrame to store image data and labels, each row represents a training sample, and according to the crack risk label, the training samples are divided into different data sets, and the samples are classified by pandas' groupby(), which are saved as different data sets, then use HDF5 format to improve data read and write efficiency, and use pandas to efficiently read and load training data during training.

[0065] S4. Extract the peel surface structure feature vector related to the crack risk from the structure optimization image sequence, and form a feature vector sequence reflecting the evolution of the crack risk of the fruit based on the feature vector,

[0066] Specifically, the texture feature vector, the edge feature vector, the deformation feature vector and the color difference feature vector are extracted from the structure-optimized image sequence, wherein the texture feature is extracted by using the Gabor filter method, each extracted feature vector represents the texture feature of a certain region in the image, the edge feature is extracted by using the Canny edge detection method, each extracted feature vector represents the edge feature of a certain region in the image, the deformation feature is obtained by performing the expansion operation on the image to highlight the small structural changes on the surface of the peel, then performing the erosion operation to detect and highlight the small concave regions on the surface of the peel caused by the internal pressure, stress or external influence, calculating the difference of the morphological gradients to obtain the gradient image, highlighting the structural changes, combining the Canny edge detection to extract the features related to the deformation feature, and the color difference feature is extracted by first converting the image from the RGB color space to the HSV or Lab color space, focusing on analyzing the changes of the hue, saturation and brightness, then performing the difference operation on the continuous image frames to identify the regions with obvious color difference, highlighting the small color difference changes of the pre-split peel, and finally using the edge detection method to extract the linear color difference band, i.e. the features related to the color difference feature, and based on the extracted feature vectors, the feature vector sequence reflecting the evolution of the fruit split risk is formed,

[0067] Further, the method for forming the feature vector sequence comprises,

[0068] S401. The feature vectors of the peel surface structure features related to the split risk are combined to form a multi-dimensional comprehensive feature vector of each image frame,

[0069] In the implementation process, different types of single-frame feature vectors are actually extracted from the structure-optimized image sequence, including the texture feature vector, the edge feature vector, the deformation feature vector and the color difference feature vector, and these feature vectors are combined to form a multi-dimensional comprehensive feature vector, which contains information of different types of features and is used to describe the overall structure of the image. The feature combination adopts the splicing method to splice multiple feature vectors together to form a longer feature vector. For example, only two types of feature vectors are extracted, the texture feature [f_1, f_2, f_3] and the edge feature [f_4, f_5, f_6], and the synthesized feature vector is [f_1, f_2, f_3, f_4, f_5, f_6];

[0070] S402. The feature vectors of the single-frame images are spliced into a complete feature vector sequence of the time window in time sequence through the time window,

[0071] In the implementation process, first, the length of the time window is determined, and how to determine the length of the time window is not described here. Assuming that a time window contains 3 frames of images, the final feature vector sequence will be: first frame [f_1, f_2, f_3, f_4, f_5, f_6]+second frame [f_7, f_8, f_9, f_10, f_11, f_12]+third frame [f_13, f_14, f_15, f_16, f_17, f_18], since each time window corresponds to a sequence of images for a period of time, all time windows will form a complete time window feature vector sequence;

[0072] S403. Based on the feature vector sequence of the time window, through feature optimization and selection, remove redundant features, and retain the features that best reflect the crack risk,

[0073] In the implementation process, since the features extracted from the image may be redundant or irrelevant to the crack risk, principal component analysis (PCA) method is used to reduce the dimension of the features, remove redundant features, and retain the most discriminative features. All feature vectors are combined into a matrix, each row is a sample feature vector. By calculating the covariance matrix and extracting the principal components, the high-dimensional features are mapped to a lower dimension. In addition, based on the mutual information method, the features most related to the crack risk evolution are determined, and the most discriminative features are selected. In feature selection, the mutual information value between each feature and the crack risk label is calculated to determine which features are most relevant. The mutual information values between all features and the crack risk label are calculated, and the features most related to the crack risk are selected according to the size of the mutual information. The most important features are retained, and the features with small mutual information are removed.

[0074] S404. Combine each time window with the corresponding crack risk label to obtain a feature vector sequence reflecting the evolution of fruit crack risk,

[0075] In the implementation process, the feature vector sequence of each time window is labeled with crack risk label, and combined with crack risk label to obtain a feature vector sequence reflecting the evolution of fruit crack risk, and also contains corresponding crack risk information.

[0076] S5. Based on the crack risk evolution label in the training set, analyze the feature vector sequence reflecting the evolution of fruit crack risk, and evaluate the evolution trend of fruit crack risk,

[0077] Specifically, the crack risk evolution labels in the training set provide basic data for analysis. In each time window, the peel surface structure is a dynamic change process, which is reflected in the feature vector sequence. Each time window corresponds to the peel surface structure characteristics related to the crack risk within a period of time, and these feature vector sequences reflect the changes in the peel surface structure. By analyzing the evolution of the feature vector sequence, the evolution trend of the crack risk can be revealed. By analyzing the change trend of the feature vector sequence, the evolution trend of the crack risk can be evaluated. This process is not only an observation of the peel surface structure characteristics related to the crack risk, but also includes how to quantify the evolution of the crack risk by calculating the rate and difference of change of the peel surface structure characteristics related to the crack risk.

[0078] Furthermore, the process of analyzing the feature vector sequence reflecting the evolution of the fruit crack risk and evaluating the evolution trend of the fruit crack risk includes:

[0079] S501. Based on the feature vector sequence reflecting the evolution of the risk of fruit cracking, the difference and change rate of the feature vector of each frame in each time window are calculated to monitor the change rate of the risk of fruit cracking.

[0080] During the implementation process, within each time window, the feature vectors of continuous image frames represent the structure of the fruit peel at different time points. The difference values ​​of these feature vectors can be calculated by the Euclidean distance method, and then the changes in each frame of the image can be quantified to monitor the changes in the surface structure of the fruit peel. The rate of change of the fruit peel surface structure is represented by calculating the rate of change between adjacent frames in each time window. The rate of change = difference value / time interval, where the time interval is the time difference between image frames. By calculating the difference value and the rate of change, the rate of change of the fruit peel surface structure can be monitored. The risk of cracking is usually related to the rapid change of the surface characteristics of the fruit peel. The average value of the rate of change within the time window is taken to represent the rate of change of the risk of fruit cracking.

[0081] S502. Analyze the evolution trend of the fruit crack risk and infer the change stage of the fruit crack risk based on the feature evolution trend under different crack risk labels in the training set.

[0082] During the implementation process,

[0083] According to the data characteristics of the characteristic vector sequence of the fruit crack risk evolution, the ARIMA model is selected to analyze the trend of the time series, determine the parameters of the ARIMA model, and select p, d, and q as the main parameters of the ARIMA model.

[0084] p (autoregressive term): indicates how many time steps of data are used to predict the current value, use PACF plot to select appropriate p value, PACF plot shows the relationship between each time lag and the current value, select significant lag as p value,

[0085] d (difference number): indicates how many times of difference are needed to make the data stationary, determine d value by unit root test to judge whether the data needs to be differenced,

[0086] q (moving average term): indicates the relationship between the current data point and the first q residual, use ACF plot to determine q value, ACF plot shows the relationship between residual and past data points, select significant lag as q value,

[0087] Each image frame feature vector of the feature vector sequence reflecting the evolution of fruit crack risk is input into the ARIMA model for fitting, and it is checked whether the ARIMA model residual meets white noise, based on the fitted ARIMA model, the trend of the feature vector of the future several image frames is predicted, that is, the trend of the feature vector sequence of the evolution of fruit crack risk,

[0088] Combined with the crack risk label in the training set and the result of the evolution trend of fruit crack risk, the fruit crack risk change stage is inferred, including no risk stage, slight risk stage, medium risk stage, high risk stage, and at the same time, each fruit crack risk change stage is assigned a strength value, the strength value is between 0 and 1.0, the no risk stage corresponds to a strength value of 0, the slight risk stage corresponds to a strength value of 0.25, the medium risk stage corresponds to a strength value of 0.5, and the high risk stage corresponds to a strength value of 1.0,

[0089] S503. Analyze the similarity between the feature vector of the current time window and each historical feature vector in the training set to determine the feature correlation degree,

[0090] In the implementation process, the feature correlation degree represents the similarity between the current fruit crack feature and the historical data (or known crack risk state), reflecting the relative strength of the current crack risk, the cosine similarity algorithm is used to calculate the cosine similarity value between the feature vector of the current time window and each historical feature vector in the training set, the obtained cosine similarity value is usually between -1 and 1, the greater the obtained cosine similarity value, the more similar it is, the greater the obtained cosine similarity value, the more similar it is, and the average value of the top three calculated cosine similarity values is the feature correlation degree of the time window t.

[0091] S6. Determine the crack risk evaluation value of the fruit according to the evaluation result, and locate the corresponding high-risk area in the structure optimization image sequence,

[0092] Further, in the implementation process, according to the evaluation result of the evolution trend of the fruit crack risk, a comprehensive crack risk evaluation value is constructed, the crack risk evaluation value is obtained by weighted summation of the feature changes in each time window, and the feature evolution trend under different crack risk labels is combined to quantify the intensity and evolution trend of the fruit crack risk, and the calculation formula of the crack risk evaluation value is ,

[0093] wherein R(t) is the crack risk evaluation value of the time window t, F(t) is the fruit crack risk change rate of the time window t, S(t) is the intensity value of the fruit crack risk change stage of the time window t, C(t) is the feature correlation degree of the time window t, and a is the fruit crack risk change rate weight, β is the fruit crack risk change stage weight, and γ is the feature correlation degree weight.

[0094] The fruit crack risk change rate weight a, the fruit crack risk change stage weight β, and the feature correlation degree weight γ are determined by analyzing the data feature importance of the fruit crack risk change rate F(t), the intensity value of the fruit crack risk change stage S(t), and the feature correlation degree C(t), the contribution degree of the target variable prediction is evaluated by using the decision tree model, F(t), the intensity value of the fruit crack risk change stage S(t), and the feature correlation degree C(t), the contribution degree of the target variable prediction is evaluated by using the decision tree model,

[0095] Based on the feature vector of each time window, the fruit crack risk change rate, the intensity value of the fruit crack risk change stage, the feature correlation degree, and the crack risk label, data preprocessing is performed, including standardization processing, missing value processing, and category variable coding, a decision tree model is constructed, a decision tree classifier is used to train the model, the importance of each risk evaluation index is evaluated, and the importance score of each risk evaluation index is returned in the evaluation process. The score reflects the contribution of the risk evaluation index when the model makes a prediction. The importance score of each risk evaluation index evaluated by the decision tree is a relative value, which needs to be normalized to obtain the standardized weight coefficient, ensuring that the sum of all risk evaluation index weight coefficients is 1, and they reflect the contribution of each feature to the crack risk.

[0096] The threshold of the crack risk evaluation value is set, the threshold boundary point is first determined, the distribution of the crack risk evaluation value R(t) corresponding to the historical data calculated by the crack risk label is calculated according to the crack risk evaluation value calculated by the historical data, and the threshold boundary point is determined based on the distribution by using the Otsu method,

[0097] The crack risk label is no risk, and R(t) < 0.2

[0098] The crack risk label is slight risk, 0.2≤R(t)<0.4,

[0099] The crack risk label is moderate risk, 0.4≤R(t)<0.6

[0100] The crack risk label is high risk, R(t)≥0.6,

[0101] Further, in the implementation process, the corresponding crack risk area in the structure optimization image sequence is located at the same time, and the method comprises the following steps:

[0102] The spatial gradient of the crack risk evaluation value in each frame of the structure optimization image sequence is calculated to identify the area with significant crack risk change. Specifically, the spatial gradient refers to the change rate of the crack risk evaluation value of a point in the image relative to its surrounding neighborhood points. Through the spatial gradient, the part of the image with dramatic crack risk change, i.e., the significant change area, can be determined. The gradient of each pixel point in each image is calculated by the Sobel operator. According to the gradient amplitude distribution of the training set image, the pixel points with 95% gradient amplitude are selected as the threshold value. If the gradient amplitude of a certain area is greater than the threshold value, it is considered that the crack risk change of the area is significant.

[0103] The areas with significant crack risk change are grouped to obtain a plurality of risk similar areas, and the areas with significant crack risk change in the image are identified. These areas are classified as crack risk areas. Specifically, the significant change areas are clustered and analyzed based on the area size, boundary coordinates and average gradient amplitude of the significant change areas. The K-means clustering algorithm is used to group the areas according to the gradient amplitude and spatial position similarity. The significant change areas belonging to the same category are grouped into a risk similar area group, and the area markers are generated in the image.

[0104] The areas with higher risk are screened according to the threshold value of the crack risk evaluation value. If the risk index of a certain area exceeds the threshold value of the crack risk evaluation value, the area is determined as a crack risk area. Specifically, for each risk similar area in the clustering result, the average gradient amplitude is calculated and compared with the threshold value of the crack risk evaluation value. If the average value of the area exceeds the threshold value of the crack risk evaluation value, it is determined as a crack risk area.

[0105] The crack risk areas in the image are marked, and the size, boundary coordinates and crack risk label of each crack risk area are recorded.

[0106] S7. Marking the fruits with crack risk evaluation values exceeding the preset threshold value as crack risk objects, and performing grading control or rejection processing on the fruits according to the located crack risk areas,

[0107] In the implementation process, when the crack risk assessment value R(t) < 0.2, the fruit is marked as a non-crack risk object, when 0.2≤R(t)<0.4, the fruit is marked as a slight crack risk object, when 0.4≤R(t)<0.6, the fruit is marked as a medium crack risk object, and when R(t)≥0.6, the fruit is marked as a high crack risk object; for the fruits with slight crack risk or medium crack risk, further grading control processing is performed, and the fruits are classified and placed in different areas or storage areas, and different storage conditions such as temperature and humidity control, oxygen flow, etc. are given to delay the possibility of crack occurrence. For high crack risk objects, direct rejection processing is performed to avoid affecting the overall fruit quality or causing other fruits to rot.

[0108] In the second embodiment, based on the same inventive concept as the image recognition-based durian pipeline automatic quality inspection method in the foregoing embodiments, as shown in the following table, the present application provides an image recognition-based durian pipeline automatic quality inspection system. The system and method embodiments in the present application are based on the same inventive concept. The system comprises: Figure 2

[0109] The image acquisition and processing module 11 is configured to acquire continuous multiple frames of images of the fruit during the movement of the fruit on the conveying path, and perform feature enhancement processing on the images to highlight the skin surface structure features related to the crack risk, and the processing result forms a structure-optimized image sequence.

[0110] The image segment extraction and labeling module 12 is configured to extract a target image segment with potential crack risk features from the image sequence containing the pre-crack occurrence state based on the historical durian image data, and label the target image segment with a crack risk label.

[0111] The training set construction module 13 is configured to organize the labeled target image segments in chronological order to construct a training set containing crack risk evolution labels, and each sample in the training set reflects the change of the crack risk of the fruit in the structure evolution process.

[0112] The feature vector extraction module 14 is configured to extract a skin surface structure feature vector related to the crack risk from the structure-optimized image sequence, and form a feature vector sequence reflecting the crack risk evolution of the fruit based on the feature vector.

[0113] The evolution trend evaluation module 15 is configured to analyze the feature vector sequence reflecting the crack risk evolution of the fruit based on the crack risk evolution labels in the training set, and evaluate the evolution trend of the crack risk of the fruit.

[0114] ​The crack risk assessment module 16 is configured to determine a crack risk assessment value of the fruit according to the assessment result, and locate a corresponding crack risk region in the structure optimization image sequence;

[0115] The hierarchical control execution module 17 is configured to mark the fruit with the crack risk assessment value exceeding a preset threshold as a crack risk object, and perform hierarchical control or rejection processing on the fruit according to the located crack risk region.

[0116] The image recognition-based durian pipeline automatic quality inspection system provided in the embodiments of the present application can execute the image recognition-based durian pipeline automatic quality inspection method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0117] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, however, any number of different modules can be used and run on the user terminal and / or server, and each unit and module included is only divided according to the functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy mutual differentiation, and do not limit the protection scope of the present application.

[0118] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An image recognition-based automatic quality inspection method for durian pipeline, characterized in that, The method comprises the following steps, acquire a plurality of continuous image frames of the fruit during the movement of the conveying path, and perform feature enhancement processing on the image to highlight the surface structure features of the fruit skin related to the crack risk, and the processing result forms a structure optimization image sequence; based on the historical image data of durians, extract target image segments with potential crack risk features from the image sequence containing the pre-crack state, and label the target image segments with crack risk labels; organize the labeled target image segments in chronological order to construct a training set containing crack risk evolution labels, and each sample in the training set reflects the change of the crack risk of the fruit in the structure evolution process; extract the surface structure feature vector of the fruit skin related to the crack risk from the structure optimization image sequence, and form a feature vector sequence reflecting the evolution of the crack risk of the fruit based on the feature vector; based on the crack risk evolution labels in the training set, analyze the feature vector sequence reflecting the evolution of the crack risk of the fruit, and evaluate the evolution trend of the crack risk of the fruit; determine the crack risk evaluation value of the fruit according to the evaluation result, and locate the corresponding crack risk area in the structure optimization image sequence; mark the fruit with a crack risk evaluation value exceeding a preset threshold as a crack risk object, and perform grading control or rejection processing on the fruit according to the located crack risk area.

2. The automated quality inspection method of claim 1, wherein, The method for performing feature enhancement processing on the image comprises, perform denoising processing on the acquired plurality of continuous image frames, retain the edge features in the image, and enhance the texture details of the fruit skin; use a Laplacian high-pass filter to enhance the edge features in the image by adjusting the parameters of the filter, thereby highlighting the surface structure of the fruit skin related to the crack risk; based on a weighted filter, process the image by adjusting the weight to strengthen the texture features of the small changes before the crack, and finally form a structure optimization image sequence.

3. The automated quality inspection method of claim 2, wherein, The construction of the training set of the crack risk evolution labels comprises, during the process of historical collection of durian image data, determine the length of each time window according to the continuous image sequence of durian fruit during the conveying process, and the time window is an image sequence reflecting the structure evolution of durian skin in a specific time period after the target image segments are organized in chronological order; match the crack risk label of each target image segment with each frame of image in the time window to construct a training sample, and ensure that each time window is associated with the corresponding crack risk label; organize all time windows and crack risk labels in chronological order, and preprocess the target image segments in the time window to ensure the size consistency and numerical range adaptation of the image data; store all training samples in a unified data structure, and classify and store the data according to the training set usage scenarios.

4. The automated quality inspection method of claim 3, wherein, The method for forming a feature vector sequence reflecting the evolution of the crack risk of the fruit comprises, combine the feature vectors extracted from the surface structure features of the fruit skin related to the crack risk to form a multi-dimensional comprehensive feature vector of each frame of image; splice the feature vectors of single frame images into a complete feature vector sequence of a time window in chronological order through a time window; Based on the time window-based feature vector sequence, redundant features are removed through feature optimization and selection, and the features that can best reflect the crack risk are retained. Each time window is combined with the corresponding crack risk label to ultimately obtain a feature vector sequence reflecting the evolution of fruit crack risk.

5. The automated quality inspection method of claim 4, wherein, The evaluation of the evolution trend of the fruit crack risk includes, Based on the feature vector sequence reflecting the evolution of the fruit crack risk, the difference and change rate of each frame feature vector in each time window are calculated to monitor the change rate of the fruit crack risk; The evolution trend of the fruit crack risk is analyzed, and based on the feature evolution trend under different crack risk labels in the training set, the change stage of the fruit crack risk is inferred. The similarity between the feature vector of the current time window and each historical feature vector in the training set is analyzed to determine the feature correlation degree.

6. The automated quality inspection method of claim 5, wherein, The evaluation result of the evolution trend of the fruit crack risk includes, According to the evaluation result of the evolution trend of the fruit crack risk, the crack risk evaluation value is calculated, the evolution trend of the fruit crack risk is quantified, and the threshold value of the crack risk evaluation value is set, The formula for calculating the crack risk assessment value is , Wherein, R(t) is the crack risk assessment value of the time window t, F(t) is the crack risk change rate of the time window t, S(t) is the intensity value of the crack risk change stage of the time window t, C(t) is the feature correlation degree of the time window t, a is the crack risk change rate weight, β is the crack risk change stage weight, and γ is the feature correlation degree weight.

7. The automated quality inspection method of claim 6, wherein, The structure optimization image sequence corresponding crack risk area positioning method includes, The spatial gradient of the crack risk evaluation value in each frame of the structure optimization image sequence is calculated to identify the area where the crack risk changes significantly; The areas where the crack risk changes significantly are grouped to obtain multiple risk similar areas, and the areas where the crack risk changes significantly in the image are identified, and these areas are classified as crack risk areas; According to the threshold value of the crack risk evaluation value, the areas with higher risk are screened out, and if the risk index of a certain area exceeds the threshold value of the crack risk evaluation value, the area is determined as a crack risk area; The crack risk areas in the image are marked, and the size, boundary coordinates and crack risk label of each crack risk area are recorded.

8. An image recognition-based durian pipeline automatic quality inspection system, characterized in that, The steps for implementing the image recognition-based durian pipeline automatic quality inspection method according to any one of claims 1 to 7, the image recognition-based durian pipeline automatic quality inspection system includes, An image acquisition and processing module is configured to acquire continuous multiple frames of images of the fruit during movement on a conveying path, and to perform feature enhancement processing on the images to highlight the skin surface structure features related to crack risk, and the processing result forms a structure optimization image sequence; An image segment extraction and labeling module is configured to extract target image segments with potential crack risk features from the image sequence containing the state before crack occurrence based on historical durian image data, and to label the target image segments with crack risk labels; A training set construction module is configured to organize the labeled target image segments in chronological order to construct a training set containing crack risk evolution labels, and each sample in the training set reflects the change of crack risk during the structure evolution of the fruit; A feature vector extraction module is configured to extract skin surface structure feature vectors related to crack risk from the structure optimization image sequence, and to form a feature vector sequence reflecting the evolution of fruit crack risk based on the feature vectors; A feature vector extraction module is configured to extract skin surface structure feature vectors related to crack risk from the structure optimization image sequence, and to form a feature vector sequence reflecting the evolution of fruit crack risk based on the feature vectors; an evolution trend evaluation module, configured to analyze the sequence of feature vectors reflecting the evolution of the fruit crack risk based on the crack risk evolution labels in the training set, and evaluate the evolution trend of the fruit crack risk; a crack risk evaluation module, configured to determine a crack risk evaluation value of the fruit according to the evaluation result, and locate a corresponding crack risk region in the structure optimization image sequence; a hierarchical control execution module, configured to mark the fruit with the crack risk evaluation value exceeding a preset threshold as a crack risk object, and perform hierarchical control or rejection processing on the fruit according to the located crack risk region.

Images