Cubilose form grading and defect detection method and system based on machine vision

Abstract

The invention provides a bird's nest shape grading and defect detection method and system based on machine vision, and the method comprises the steps: constructing a first data set composed of four-side image groups of swallow calyx, constructing high-quality swallow calyx without defects into a second data set, and setting the text information of swallow calyx attributes; inputting the first data set image group into a pre-constructed defect identification model, and obtaining the defect identification model according to a convergence condition; obtaining each target swallow calyx image group by adopting an algorithm, obtaining swallow calyx size data through a parameter extraction algorithm, taking the size data and text information as input, and taking level attributes as output to train an initial high-quality swallow calyx classifier; and inputting a to-be-recognized swallow-calyx four-side picture and text information of swallow-calyx attributes into the defect recognition model, if a prediction result is a high-quality swallow-calyx, obtaining size data corresponding to the swallow-calyx image group by adopting an algorithm and an edge recognition algorithm, inputting the size data and the text information into a high-quality swallow-calyx classifier, and predicting to obtain a high-quality swallow-calyx level.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a method, system, equipment and medium for bird's nest morphology grading and defect detection based on machine vision. Background Art

[0002] Modern medical research has found that the main components of bird's nest are: water-soluble protein, carbohydrates; trace elements: calcium, phosphorus, iron, sodium, potassium and amino acids (lysine, cystine and arginine) that play an important role in promoting human vitality. Bird's nest is rich in nutrients and has always been considered a precious tonic, with many beneficial effects such as beautifying the skin, improving immunity, and regulating intestinal flora.

[0003] The processed bird's nests are currently sold in the form of nests or ready-to-eat cans, especially nests in nest form (hereinafter referred to as nests). The market prices of nests vary according to their grades, such as triangular nests, round nests, and high-quality nests (including 3A, 4A, 5A, super), etc. However, the type determination and grading of bird's nests are now basically determined manually with the help of tools, which makes the grading time-consuming and labor-intensive, and the artificial differences in the measurement results can easily lead to inaccurate grading.

[0004] In view of this, there is an urgent need to provide a bird's nest morphology grading and defect detection method and system based on machine vision that is convenient for bird's nest classification and grading and accurate in grading. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides a bird's nest morphology grading and defect detection method, system, equipment and medium based on machine vision to solve the technical problems of inaccurate bird's nest grading and laborious and time-consuming technical problems in the related art.

[0006] One or more embodiments of this specification provide a method for bird's nest morphology grading and defect detection based on machine vision, comprising the following steps:

[0007] Constructing a first data set consisting of a four-sided image group of bird's nest; wherein the first data set includes a four-sided image group of bird's nest with defect type labels set, and a four-sided image group of bird's nests of various levels and forms without defects and with bird's nest type labels set;

[0008] The four-side image groups of defect-free high-quality bird's nests of various grades and shapes in the first data set are constructed into a second data set, and text information representing the attributes of the bird's nest is set, wherein the attributes of the bird's nest include the grade;

[0009] The image group of the first data set is input into a pre-built defect recognition model for model training, and a defect recognition model is obtained according to a convergence condition; the image group in the second data set is segmented by a target image segmentation algorithm to obtain each target bird's nest image group; and the parameter extraction algorithm is used to extract parameters of each target bird's nest image to obtain the size data corresponding to each bird's nest image group, wherein the size data includes the length, height, length-to-height ratio and height of the bird's nest;

[0010] Using the size data and other attribute text information of each high-quality bird's nest except the grade attribute as input data, and using the grade attribute as output data to train an initial high-quality bird's nest classifier to obtain a high-quality bird's nest classifier; and

[0011] The four-sided image of the bird's nest to be identified and the text information of the bird's nest attributes are input into the defect recognition model to determine the defect type or bird's nest type. If the prediction result is a high-quality bird's nest, the target image segmentation algorithm is used to segment the four-sided image of the bird's nest to obtain each target bird's nest image group. The parameter extraction algorithm is then used to extract parameters from each image in the target bird's nest image group to obtain the size data corresponding to the bird's nest image group. The size data and text information are input into the high-quality bird's nest classifier to predict the high-quality bird's nest level.

[0012] One or more embodiments of this specification provide a bird's nest morphology grading and defect detection system based on machine vision, including:

[0013] A first data set establishment module is used to construct a first data set consisting of a four-side image group of a bird's nest; wherein the first data set includes a four-side image group of a bird's nest with a defect type label set, and a four-side image group of a bird's nest of various levels and forms without defects and with a bird's nest type label set;

[0014] A second data set establishment module is used to construct a second data set from four-side image groups of defect-free high-quality bird's nests of various grades and shapes in the first data set, and to set text information characterizing bird's nest attributes, wherein the bird's nest attributes include grades;

[0015] A defect recognition model construction module is used to input the image group of the first data set into a pre-built defect recognition model for model training, and obtain a defect recognition model according to a convergence condition; a target segmentation module is used to segment the image group in the second data set using a target image segmentation algorithm to obtain each target bird's nest image group;

[0016] The target size determination module is used to extract parameters of each target bird's nest image through a parameter extraction algorithm to obtain the size data corresponding to each bird's nest image group, and the size data includes length, height, length-to-height ratio and nest height;

[0017] A high-quality bird's nest classifier construction module is used to take the size data and other attribute text information of each high-quality bird's nest except the grade attribute as input data, and use the grade attribute as output data to train an initial high-quality bird's nest classifier to obtain a high-quality bird's nest classifier;

[0018] The prediction module is used to construct a high-quality bird's nest classifier module, which is used to input the four-sided image of the bird's nest to be identified and the text information of the bird's nest attributes into the defect recognition model to determine the defect type or the bird's nest type. If the prediction result is a high-quality bird's nest, the target image segmentation algorithm is used to segment the four-sided image of the bird's nest to obtain each target bird's nest image group, and then the parameter extraction algorithm is used to extract parameters from each image in the target bird's nest image group to obtain the size data corresponding to the bird's nest image group. The size data and text information are input into the high-quality bird's nest classifier to predict the quality of the bird's nest level.

[0019] One or more embodiments of the present specification provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the machine vision-based bird's nest morphology grading and defect detection method as described above when executing the computer program.

[0020] One or more embodiments of the present specification provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the machine vision-based bird's nest morphology grading and defect detection method as described above.

[0021] The present disclosure provides a method, system, equipment and medium for morphological grading and defect detection of bird's nests based on machine vision. The advantage is that, considering that the shapes of triangular nests and round nests are essentially different from those of high-quality bird's nests, defective nests and non-defective nests, as well as the categories of non-defective nests, are identified and classified through a defect recognition model. The bird's nest size data determined after processing by a target image segmentation algorithm, combined with the attributes of high-quality bird's nests, are classified according to the grading standards through a Bayesian model or a neural network model. The method of this embodiment not only realizes the identification of bird's nest defects, but also can realize the category of defect-free bird's nests. The target image segmentation algorithm is then used to segment the bird's nests in the image, thereby facilitating the determination of the bird's nest size. The Bayesian model or the neural network model is used according to other pre-set attribute parameters to realize the classification according to the grade of high-quality bird's nests based on the grading standards. This detection method through machine learning can realize accurate and rapid classification of bird's nests, and saves time and effort. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A flow chart of a method for bird's nest morphology grading and defect detection based on machine vision provided for one or more embodiments of this specification;

[0024] Figure 2 A schematic diagram of providing different levels of high-quality bird's nest for one or more embodiments of this specification;

[0025] Figure 3 A schematic side view of high-quality bird's nests of different levels taken at a uniform shooting angle and distance provided in one or more embodiments of this specification;

[0026] Figure 4 A threshold segmentation histogram of the existing two-dimensional Otsu algorithm provided in one or more embodiments of this specification;

[0027] Figure 5 A threshold segmentation histogram of the improved Otsu algorithm provided in one or more embodiments of this specification;

[0028] Figure 6 A block diagram of a bird's nest morphology grading and defect detection system based on machine vision provided for one or more embodiments of this specification;

[0029] as well as

[0030] Figure 7 A schematic diagram of the structure of a computer device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] The present invention is described in detail below in conjunction with specific implementation methods and the accompanying drawings.

[0033] Method Embodiment

[0034] According to an embodiment of the present invention, a method for bird's nest morphology grading and defect detection based on machine vision is provided. Figure 1 As shown, it is a flow chart of the bird's nest morphology grading and defect detection method based on machine vision provided in this embodiment. According to the bird's nest morphology grading and defect detection method based on machine vision in an embodiment of the present invention, the steps include:

[0035] Step S1, constructing a first data set consisting of a four-sided image group of bird's nest; wherein the first data set includes a four-sided image group of bird's nest with a defect type label set, and a four-sided image group of bird's nests of various levels and shapes without defects and with a bird's nest type label set, the four-sided images include a front view, a rear view, a left view and a right view, wherein the bird's nest types include triangular nests, round nests, high-quality bird's nests, etc.

[0036] Step S2, constructing a second data set from four-side image groups of defect-free high-quality bird's nests of various levels and forms (or shapes) in the first data set, and also setting text information characterizing the attributes of each high-quality bird's nest, wherein reference Figure 2 Shown and Figure 3 High-quality bird's nest includes 3A, 4A, 5A, 6A, super and other grades; bird's nest properties include grade, sialic acid content, nitrite, sulfur dioxide content, protein content, hydrogen peroxide, cleanliness and water content, and you can also choose parameters such as origin, color, weight, clarity and so on according to your needs. I will not go into details here.

[0037] The first and second data sets are constructed separately to train the network model because the bird's nests on the market include not only various types, but some types of bird's nests are also divided into multiple levels according to attributes such as size. In order to make the trained model applicable, the training of the model not only needs to let it learn the morphological characteristics of bird's nests of different types and levels as well as different sizes or shapes of the same level, because for example, the size of a 3A-grade bird's nest only needs to meet the requirements of length ≥ 6.0, height ≥ 3.0 and the height of the nest is 3 to 3.5 cm. Therefore, the model needs to learn the features of different size combinations to improve the prediction accuracy of the model; in addition, it is also necessary to learn the feature expressions of the same type of bird's nests from different shooting angles. For bird's nests with grade distinctions, it is necessary to set the corresponding text information according to the bird's nest grading evaluation criteria to provide model learning.

[0038] In this embodiment, the bird's nest grading and evaluation (enterprise) standards are exemplified as follows, and can also be set according to industry standards or national standards.

[0039] Specifications of Super (special grade) bulbs: moisture content ≤ 20, bulb height > 5cm, sialic acid content ≥ 10, protein content ≥ 50, length ≥ 11.0, height ≥ 3.0, standard number of bulbs / 1000g / approximately 150 to 172 bulbs.

[0040] Specifications of 5A large teapot: moisture content ≤20, teapot height 4.2~5cm, sialic acid content ≥10, protein content ≥50, length ≥9.0, height ≥3.0, standard number of teapots / 1000g / approximately 170~200 teapots.

[0041] Specifications of 4A medium lamp: moisture content ≤20, lamp height 3.5-4.2cm, sialic acid content ≥10, protein content ≥50, length ≥7.0, height ≥3.0, standard lamp number / 1000g / approximately 190-220 lamps.

[0042] Specifications of 3A small cups: moisture content ≤20, cup height 3-3.5cm, sialic acid content ≥10, protein content ≥50, length ≥6.0, height ≥3.0, standard number of cups / 1000g / approximately 210-280 cups.

[0043] Oral (round bottom) cup specifications: moisture content ≤20, cup height 3.6~4.2cm, standard cup number / 1000 grams / approximately 170~220 cups.

[0044] Super C (large bird's nest strip) specifications: 25≤moisture≤20, body height 2.5~3.1cm, standard number of pieces / 1000g / about 190~300 pieces.

[0045] Step S3, inputting the first data set image group into a pre-built defect recognition model for model training, and obtaining a defect recognition model according to a convergence condition;

[0046] In this embodiment, the defect recognition model may include a feature extraction layer, a feature screening layer and a softmax layer, and the prediction results are defective bird's nests and corresponding defect types, or non-defective bird's nests and corresponding bird's nest types.

[0047] Step S4, segment the image groups in the second data set using a target image segmentation algorithm to obtain each target bird's nest image group. In this embodiment, the image groups in the second data set are segmented using an Otsu (maximum between-class variance method) algorithm and an edge recognition algorithm.

[0048] Step S5, extracting parameters from each image in each target bird's nest image group obtained in step S4 by means of a parameter extraction algorithm, and obtaining the size data corresponding to each bird's nest image group, wherein the size data includes the length, height and length-to-height ratio of the bird's nest determined based on the front view and the rear view, and the height of the nest determined based on the left view and the right view; wherein the length is the length of the two points farthest from the nest base of the bird's nest, the height is the vertical height from the vertex of the bird's nest to the two points farthest from the nest base of the bird's nest, and the height of the nest is the length from the top of the bird's nest along the arc-shaped nest body to the edge of the base when the bird's nest is viewed from the side;

[0049] Step S6, using the predicted size data and other attribute text information of each high-quality bird's nest except the level attribute in step S2 as input data, and using the level attribute as output data to train an initial high-quality bird's nest classifier to obtain a high-quality bird's nest classifier; and

[0050] Step S7, input the four-sided image of the bird's nest to be identified and the text information of the bird's nest attributes into the defect recognition model to determine the defect type or bird's nest type. If the prediction result is a high-quality bird's nest, go to step S8;

[0051] Step S8, use the target image segmentation algorithm to segment the four-sided image of the bird's nest to obtain each target bird's nest image group, use the parameter extraction algorithm to extract parameters of each image in the target bird's nest image group, obtain the size data corresponding to the bird's nest image group, input the size data and text information into the high-quality bird's nest classifier, and predict the high-quality bird's nest level.

[0052] The method provided in this embodiment takes into account the essential difference between the shapes of triangular and round lamps and high-quality bird's nest lamps. Therefore, defective lamps and non-defective lamps, as well as the categories of non-defective lamps, i.e., triangular lamps, round lamps or high-quality bird's nest lamps, are identified and classified through a defect recognition model. Then, for the grade judgment of high-quality bird's nest lamps, the aspect ratio and lamp height of each bird's nest determined in steps S4 and S5 are combined with the attributes of high-quality bird's nest lamps to realize the grade classification of high-quality bird's nest lamps based on the grading standard through a Bayesian model or a neural network model. The method of this embodiment not only realizes the identification of bird's nest defects, but also realizes the category of defect-free bird's nest lamps. Then, the target image segmentation algorithm is used to realize the segmentation of the bird's nest lamps in the image, thereby facilitating the determination of the size of the bird's nest lamps, and the Bayesian model or the neural network model is used according to other pre-set attribute parameters to realize the grade classification of high-quality bird's nest lamps based on the grading standard. This detection method through machine learning can realize accurate and rapid classification of bird's nest lamps, and saves time and effort.

[0053] In the present embodiment, four-sided images of bird's nest are used as model training corpus because bird's nest is a three-dimensional object with curvature, and an image taken from a single perspective may not be able to capture the complete shape of the bird's nest, the height of the nest, defects (damage), etc. Therefore, in the present embodiment, four-sided images of the bird's nest are photographed at the same perspective and distance to obtain a first data set, which can improve the model recognition accuracy. In order to enhance the accuracy of the recognition model, the training data set in step S1 and the bird's nest with recognition are a four-sided image group consisting of the front view, rear view, left view and right view of the bird's nest photographed by a camera at a uniform perspective. However, for the training data set, four-sided images can be photographed at different perspectives to enrich the data set and make the trained model more applicable.

[0054] In this embodiment, image enhancement is performed on the collected bird's nest images. Enhancement methods such as random flipping, random cropping, and random rotation may be used to preprocess the images. After preprocessing, a 224×224 pixel RGB three-channel image is generated to reduce the model's dependence on the object position. Color and brightness changes are used to reduce the model's sensitivity to color, thereby obtaining the final data set.

[0055] In this embodiment, the defect recognition model can use Yolo, Vgg, SSD, Transform and other models as the basic model, or a fusion model integrated by multiple models, which can realize the defect recognition in this embodiment. For example, using the YOLOv3 network model, the 5 layers of the model: convolution layer, skip connection layer, upsampling layer, routing layer, YOLOv3 detection layer, and set network parameters; another example is CNN+Transform+feature fusion model, etc., which is not described in detail here.

[0056] In this embodiment, the Otsu algorithm and the edge recognition algorithm are selected in step S4 to perform image segmentation on the image so as to more completely retain the edge information of the image and avoid over-segmentation; in this embodiment, the edge recognition algorithm can use Sobel, Log and Canny algorithms to obtain the boundary between the target and background areas.

[0057] Based on the segmentation results of the image by the edge recognition algorithm, the maximum inter-class variance method (i.e., Otsu algorithm) can achieve good technical results for the segmentation of bird's nests. However, since the sizes of high-quality bird's nests of various levels (such as height, curvature and other parameters) are not particularly different, especially based on graphic recognition and comparison, if the segmentation of bird's nests is not accurate enough, the size of the bird's nest is a crucial parameter for the evaluation of bird's nest levels. If the prediction is inaccurate, the predicted level result will be inaccurate; and the existing Otsu algorithm is very ideal when processing images without noise interference and with obvious double peaks in the histogram. However, due to the great differences in image sources, images are easily interfered by noise when shooting, resulting in a variety of histogram distribution forms, and the simple Otsu algorithm cannot obtain satisfactory results. In order to improve the accuracy of the segmentation results, the two-dimensional Otsu algorithm based on genetic algorithms or the fast algorithm of two-dimensional adaptive thresholds can also be used to segment the target bird's nest in the image. However, these two methods can reduce the amount of calculation and improve the segmentation quality, but the anti-noise ability is still insufficient. The size of the band area of ​​the target and background is determined artificially, which generally requires multiple adjustments and repeated tests, and the adaptability is poor.

[0058] This embodiment uses the improved Otsu algorithm to achieve the segmentation of the target bird's nest in the image. The implementation process of the improved Otsu algorithm is as follows.

[0059] refer to Figure 2As shown, in this embodiment, it is assumed that the image resolution to be segmented is M×N and the image grayscale is L, then the grayscale of its neighborhood mean image (the neighborhood mean is obtained by using a 3×3 neighborhood template) is also L. Each pixel in the image corresponds to a binary group (i, j), i = 0, 1, ..., L-1; j = 0, 1, ..., L-1, where i is the pixel grayscale value; j is the neighborhood mean. Let the number of pixels with a pixel grayscale value of i and a neighborhood mean of j be r ij , then the two-dimensional joint probability density is:

[0060]

[0061] Any given threshold point (s, t) can divide the image into four regions in the two-dimensional histogram, such as Figure 2 As shown in Figure 1, D0 and D1 are the target and background regions respectively; A and B contain edges and noise.

[0062] The improvement of this embodiment is to use the asymmetric boundary histogram oblique division method to segment the two-dimensional histogram. Figure 3 As shown in the figure, D0 and D1 are the target and background areas respectively; A and B are the noise areas. The two boundaries gf±m=0 limit the bandwidth of D0 and D1, and the straight line gfT=0 perpendicular to the diagonal is the segmentation line, and T is the threshold. Only when m=n, since the grayscale value of the edge point of the image will be higher or lower than the grayscale value of the remaining pixels in its neighborhood to a certain extent, there is also a certain deviation between its grayscale value and the neighborhood mean. Therefore, in the two-dimensional histogram, the edge point will deviate from the diagonal appropriately and be between the noise area and the target background area. When the grayscale value of the edge point is greater than the neighborhood mean, the edge point is located below the diagonal; otherwise, the edge point is located above the diagonal. The improved Otsu algorithm obtains two straight lines by fitting the edge points on the upper and lower sides of the diagonal, namely the boundaries between the A, B areas and the D0, D1 areas.

[0063] In order to improve the anti-noise ability of the algorithm, for the noise points (x, y) in A and B, the gray value is replaced by the mean of its neighborhood, that is, when gfn = 0 or g-f + m < 0, let:

[0064] f(x, y) = g(x, y);

[0065] After replacing the noise, in the two-dimensional histogram, all noise points will be transferred from A and B to the diagonal line and become part of the target and background area. Then, the Otsu oblique segmentation method is used for threshold segmentation to improve the algorithm's anti-noise performance.

[0066] In a specific embodiment, before implementing image segmentation based on the improved Otsu, the embodiment first uses an edge segmentation algorithm to detect the edge of the original image f and obtain all edge points;

[0067] For each edge point, its gray value is i, and the neighborhood mean is j. Then in the two-dimensional histogram, the corresponding coordinate point (i, j) will be scattered on both sides of the diagonal. For the edge points on the upper left side of the diagonal, the straight line fitting method based on the least squares method is used to obtain the straight line gfn = 0. For the edge points on the lower right side of the diagonal, the straight line g-f+m = 0 is obtained in the same way, that is, the values ​​of m and n are obtained. Determining the size of the strip area in this way can not only enhance the algorithm's adaptive ability, but also improve the segmentation quality.

[0068] Based on obtaining all edge points, the improved Otsu algorithm realizes image segmentation through the following calculation process.

[0069] Assume ω0(T), μ 0i (T), μ 0j (T) represent the probability of the region D0 class and the two gray level means, ω1(T), μ 1i (T), μ 1j (T) represents the probability of the D1 class and the two gray level means, μ i , μ j Represents the total mean of 2 gray levels.

[0070] When 0≤T≤L-1, such as Figure 3 As shown, the recursive formula is as follows:

[0071]

[0072]

[0073] Similarly, the recursive formula when L-1≤T≤2L-2 is as follows:

[0074]

[0075] in,

[0076] ω0(0)=P(0,0), ω 1(2L-2)=P(L-1,L-1);

[0077] μ 0i (0)=0,μ 1i (2L-2) = (L-1)P(L-1,L-1);

[0078] μ 0j (0) = 0, μ 1j (2L-2) = (L-1)P(L-1,L-1);

[0079] The between-class variance is expressed as:

[0080]

[0081] The optimal threshold T′ is obtained when the inter-class variance is the largest, that is:

[0082] T′=agemax 0≤T≤2L-2 {t r S B (T)}.

[0083] In this embodiment, step S5 uses an extraction procedure of a parameter extraction algorithm to extract parameters from the target bird's nest image obtained in step S4, and the specific steps are as follows:

[0084] Import the target bird's nest image into the parameter extraction program, first convert the target bird's nest image into a grayscale image, then find the two points with the farthest distance between the left and right edges of the bird's nest through the front view and the back view, and build a line connecting the two points, which is the length. Similarly, find the two points with the farthest distance between the upper and lower edges of the bird's nest, and build a line connecting the two points, which is the height. Record the values ​​of length and height, and calculate the length-to-height ratio of the bird's nest. Then find the multiple points with the farthest distance from the top of the bird's nest along the arc to the edge of the base through the left view and the right view, and build an arc connecting line. The length of the arc is the height of the nest.

[0085] In this embodiment, the high-quality bird's nest classifier can be a neural network classifier, a Bayesian classifier, a support vector machine classifier, or a linear discriminant analysis classifier. The Bayesian classifier is used as the high-quality bird's nest classifier to illustrate the bird's nest grade classification process in this embodiment.

[0086] This embodiment applies the Bayesian network model to analyze the size data and text information as grade evaluation indicators, and evaluates the impact of the grade evaluation indicators on the grade probability of each bird's nest. The Bayesian network is a graphical model that represents variables through nodes and the probabilistic dependencies between variables through edges. In this process, the model calculates the conditional probability distribution based on prior knowledge and the grade evaluation indicators in the feature vector data set, and then infers the posterior probability of each level, providing a prediction basis for analyzing the grade of high-quality bird's nest under a large number of grade evaluation indicators, as follows.

[0087] In this embodiment, the size information obtained in step S4 and the text information characterizing the attributes of high-quality bird's nests in each image group provided in step S2 are used as grade evaluation indicators, and a bird's nest morphology grading network is constructed based on a Bayesian network according to the grade evaluation indicators and the correlation between the grade evaluation indicators. The size information, text information and bird's nest grade are used as prior knowledge to train the bird's nest morphology grading network to obtain a high-quality bird's nest classifier.

[0088] In the present embodiment, the Bayesian model is used to implement the prediction and classification of grades because the Bayesian training process has very little computational complexity. Compared with the neural network model, a large amount of training data will not take too long to train, which saves computing resources while making predictions fast. In addition, since the characteristics of each indicator are basically independent of each other, the change of one indicator will not affect the probability of occurrence of another indicator. Therefore, in the present embodiment, the prediction and classification of grades implemented using the Bayesian model has a high accuracy.

[0089] In another embodiment, a Clip (Contrastive language-image pre-training) model + classifier combination can also be used as a basic model, using the second data set and the corresponding size data obtained as input, and the bird's nest level as output for training to obtain a high-quality bird's nest classifier. The training process of the Clip model + classifier is a conventional technology and will not be elaborated here.

[0090] Device Embodiment

[0091] According to an embodiment of the present invention, a bird's nest morphology grading and defect detection system based on machine vision is provided. Figure 6 As shown, it is a block diagram of a bird's nest morphology grading and defect detection system based on machine vision provided in this embodiment. The bird's nest morphology grading and defect detection system based on machine vision according to an embodiment of the present invention includes:

[0092] The first data set establishment module 10 is used to construct a first data set consisting of a four-sided image group of bird's nest; wherein the first data set includes a four-sided image group of bird's nest with defect type labels set, and a four-sided image group of bird's nests of various levels and shapes without defects and with bird's nest type labels set, the four-sided images include a front view, a rear view, a left view and a right view, wherein the bird's nest types include triangular bird's nest, round bird's nest, high-quality bird's nest, etc.

[0093] The second data set establishment module 20 is used to construct the four-sided image group of defect-free high-quality bird's nests of various levels and forms (or nest shapes) in the first data set into a second data set, and also set text information characterizing the properties of each high-quality bird's nest, wherein high-quality bird's nest includes 3A, 4A, 5A, 6A, super and other levels; bird's nest properties include grade, sialic acid content, nitrite, sulfur dioxide content, protein content, hydrogen peroxide, cleanliness and water content, and the origin, color, gram weight, clarity and other parameters can also be selected according to needs, which will not be elaborated here.

[0094] The defect recognition model construction module 30 is used to input the first data set image group into a pre-constructed defect recognition model for model training, and obtain a defect recognition model according to a convergence condition; the defect recognition model may include a feature extraction layer, a feature screening layer and a softmax layer, and the prediction results are defective bird's nests and corresponding defect types, or non-defective bird's nests and corresponding bird's nest types.

[0095] The target segmentation module 40 is used to segment the image groups in the second data set using a target image segmentation algorithm to obtain each target bird's nest image group. In this embodiment, the Otsu algorithm and the edge recognition algorithm can be used to segment the four-sided bird's nest image.

[0096] The target size determination module 50 is used to extract parameters from each image in each target bird's nest image group obtained in the target segmentation module 40 through a parameter extraction algorithm to obtain size data corresponding to each bird's nest image group, wherein the size data includes the length, height and length-to-height ratio of the bird's nest determined based on the front view and the back view, and the height of the nest determined based on the left view and the right view; wherein the length is the length of the two points farthest from the nest base of the bird's nest, the height is the vertical height from the top vertex of the bird's nest to the two points farthest from the nest base of the bird's nest, and the nest height is the length from the top of the bird's nest along the arc-shaped nest body to the edge of the base when the bird's nest is viewed from the side;

[0097] A high-quality bird's nest classifier construction module 60 is used to use the predicted size data and other attribute text information of each high-quality bird's nest except the level attribute in step S2 as input data, and use the level attribute as output data to train an initial high-quality bird's nest classifier to obtain a high-quality bird's nest classifier; and

[0098] The prediction module 70 is used to input the four-sided image of the bird's nest to be identified and the text information of the bird's nest attributes into the defect recognition model to determine the defect type or the bird's nest type. If the prediction result is a high-quality bird's nest, it is used to segment the four-sided image of the bird's nest using a target image segmentation algorithm to obtain each target bird's nest image group, extract parameters from each image in the target bird's nest image group using a parameter extraction algorithm, obtain the size data corresponding to the bird's nest image group, input the size data and text information into a high-quality bird's nest classifier, and predict the high-quality bird's nest level.

[0099] In this embodiment, taking into account the essential difference between the shapes of triangular and round lamps and high-quality bird's nest lamps, defective lamps and non-defective lamps, as well as the categories of non-defective lamps, i.e. triangular lamps, round lamps or high-quality bird's nest lamps, are identified and classified through a defect recognition model. Then, for the grade judgment of high-quality bird's nest lamps, based on the aspect ratio and lamp height of each bird's nest lamp, combined with the attributes of high-quality bird's nest lamps, a Bayesian model or a neural network model is used to realize the grade classification of high-quality bird's nest lamps based on the grading standard. The device of this embodiment not only realizes the recognition of bird's nest defects, but also realizes the category of defect-free bird's nest lamps. Then, the target image segmentation algorithm is used to realize the segmentation of the bird's nest lamps in the image, thereby facilitating the determination of the size of the bird's nest lamps, and the Bayesian model or the neural network model is used according to other pre-set attribute parameters to realize the grade classification of high-quality bird's nest lamps based on the grading standard. This detection method through machine learning can realize accurate and rapid classification of bird's nest lamps, and saves time and effort.

[0100] In the preferred embodiment of the present invention, four-sided images of the bird's nest are photographed at the same viewing angle and distance to obtain the first data set, which can improve the model recognition accuracy and enhance the accuracy of the recognition model.

[0101] This embodiment also provides a preprocessing module 80 for performing image enhancement on the bird's nest images in the first data set. The images can be preprocessed using enhancement methods such as random flipping, random cropping, and random rotation. After preprocessing, a 224×224 pixel RGB three-channel image is generated to reduce the model's dependence on the object position. The color and brightness changes are used to reduce the model's sensitivity to color, thereby obtaining the preprocessed first data set.

[0102] In this embodiment, the defect recognition model can use Yolo, Vgg, SSD, Transform and other models as a basic model, or a fusion model integrated by multiple models.

[0103] The edge recognition algorithm of this embodiment can use Sobel, Log and Canny algorithms to obtain the boundary between the target and background areas. First, the edge segmentation algorithm is used to detect the edge of the original image to obtain all edge points; then the image segmentation is implemented based on the improved Otsu to obtain the target bird's nest image.

[0104] In this embodiment, the target size determination module 50 is specifically configured to perform the following steps to implement parameter extraction:

[0105] The extraction program of the parameter extraction algorithm is used to extract parameters of the target bird's nest image extracted by the target segmentation module 40. The image is imported into the parameter extraction program, and the image is converted into a grayscale image. The two points with the farthest distance between the left and right edges of the bird's nest are found through the front view and the back view, and a line connecting the two points is constructed, which is the length. Similarly, the two points with the farthest distance between the upper and lower edges of the bird's nest are found, and a line connecting the two points is constructed, which is the height. The values ​​of length and height are recorded, and the length-to-height ratio of the bird's nest can be calculated. Then, the multiple points with the farthest distance from the top of the bird's nest along the arc-shaped nest body to the edge of the base are found through the left view and the right view, and an arc-shaped line is constructed, and the length of the arc is the height of the nest body.

[0106] In this embodiment, the high-quality bird's nest classifier construction module 60 is configured to execute the following steps to construct a high-quality bird's nest classifier:

[0107] The size information obtained by the target segmentation module 40 and the text information characterizing the attributes of high-quality bird's nests in each image group provided in the second data set establishment module 20 are used as grade evaluation indicators, and a bird's nest morphology grading network is constructed based on a Bayesian network according to the grade evaluation indicators and the correlation between the grade evaluation indicators. The size information, text information and bird's nest grade are used as prior knowledge to train the bird's nest morphology grading network to obtain a high-quality bird's nest classifier.

[0108] The embodiment of the present invention is an apparatus embodiment corresponding to the above-mentioned method embodiment. The specific operations of the processing steps of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0109] like Figure 7 As shown, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the bird's nest morphology grading and defect detection method based on machine vision in the above embodiment is implemented, or when the computer program is executed by a processor, the bird's nest morphology grading and defect detection method based on machine vision in the above embodiment is implemented, and when the computer program is executed by the processor, the following method steps are implemented:

[0110] Step S1, constructing a first data set consisting of a four-sided image group of bird's nest; wherein the first data set includes a four-sided image group of bird's nest with defect type labels set, and a four-sided image group of bird's nest without defects and with bird's nest type labels set at various levels and in various forms.

[0111] Step S2, constructing a second data set from a group of four-sided images of defect-free high-quality bird's nests of various levels and forms (or shapes) in the first data set, and also setting text information characterizing the attributes of each high-quality bird's nest, wherein the attributes of high-quality bird's nests include the grade.

[0112] Step S3, input the first data set image group into a pre-built defect recognition model for model training, and obtain the defect recognition model according to the convergence condition; the defect recognition model includes a feature extraction layer, a feature screening layer and a softmax layer, and the prediction result is a defective bird's nest and the corresponding defect type, or a non-defective bird's nest and the corresponding bird's nest type.

[0113] Step S4, using a target image segmentation algorithm to segment the image groups in the second data set to obtain target bird's nest image groups.

[0114] Step S5, extracting parameters of each image in each target bird's nest image group obtained in step S4 by using a parameter extraction algorithm to obtain the size data corresponding to each bird's nest image group, the size data including the length, height, length-to-height ratio and nest height of the bird's nest.

[0115] Step S6, using the predicted size data and other attribute text information of each high-quality bird's nest except the level attribute in step S2 as input data, and using the level attribute as output data to train an initial high-quality bird's nest classifier to obtain a high-quality bird's nest classifier.

[0116] Step S7, input the four-sided image of the bird's nest to be identified and the text information of the bird's nest attributes into the defect recognition model to determine the defect type or bird's nest type. If the prediction result is a high-quality bird's nest, go to step S8;

[0117] Step S8, use the target image segmentation algorithm to segment the four-sided image of the bird's nest to obtain each target bird's nest image group, use the parameter extraction algorithm to extract parameters of each image in the target bird's nest image group, obtain the size data corresponding to the bird's nest image group, input the size data and text information into the high-quality bird's nest classifier, and predict the high-quality bird's nest level.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0119] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and the contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. The bird's nest morphology grading and defect detection method based on machine vision is characterized by The following steps are involved: Constructing a first data set consisting of a four-sided image group of bird's nest; wherein the first data set includes a four-sided image group of bird's nest with defect type labels set, and a four-sided image group of bird's nests of various levels and forms without defects and with bird's nest type labels set; The four-side image groups of defect-free high-quality bird's nests of various grades and shapes in the first data set are constructed as a second data set, and text information characterizing the attributes of the bird's nest is set, wherein the attributes of the bird's nest include the grade; Inputting the first data set image group into a pre-built defect recognition model for model training, and obtaining a defect recognition model according to a convergence condition; The target image segmentation algorithm is used to segment the image groups in the second data set to obtain target bird's nest image groups; the parameter extraction algorithm is then used to extract parameters of each target bird's nest image to obtain the size data corresponding to each bird's nest image group, the size data including the length, height, length-to-height ratio and height of the bird's nest; Using the size data and other attribute text information of each high-quality bird's nest except the grade attribute as input data, and using the grade attribute as output data to train an initial high-quality bird's nest classifier to obtain a high-quality bird's nest classifier; and The four-sided image of the bird's nest to be identified and the text information of the bird's nest attributes are input into the defect recognition model to determine the defect type or bird's nest type. If the prediction result is a high-quality bird's nest, the target image segmentation algorithm is used to segment the four-sided image of the bird's nest to obtain each target bird's nest image group. The parameter extraction algorithm is then used to extract parameters from each image in the target bird's nest image group to obtain the size data corresponding to the bird's nest image group. The size data and text information are input into the high-quality bird's nest classifier to predict the high-quality bird's nest level.

2. The method for bird's nest morphology grading and defect detection based on machine vision according to claim 1, characterized in that: The four views include a front view, a rear view, a left view and a right view, wherein the bird's nest types include a triangular nest, a round nest, and a high-quality bird's nest; wherein the high-quality bird's nest includes 3A, 4A, 5A, 6A and / or super grades; Bird's nest properties also include sialic acid content, nitrite, sulfur dioxide content, protein content, hydrogen peroxide, cleanliness and moisture content, origin, color, weight and / or clarity parameters.

3. The method for bird's nest morphology grading and defect detection based on machine vision as claimed in claim 1, characterized in that: The first data set is obtained by taking four-sided images of the bird's nest at the same viewing angle and distance.

4. The method for bird's nest morphology grading and defect detection based on machine vision as claimed in claim 1, characterized in that: The image preprocessing steps are as follows: The bird's nest images in the first data set are enhanced, and the images are preprocessed using enhancement methods such as random flipping, random cropping and / or random rotation. After preprocessing, RGB three-channel images of 224×224 pixels are generated to obtain the preprocessed first data set.

5. The method for bird's nest morphology grading and defect detection based on machine vision as claimed in claim 1, characterized in that: The defect recognition model uses Yolo, Vgg, SSD or Transform model as a basic model, or a fusion model integrating multiple models.

6. The method for bird's nest morphology grading and defect detection based on machine vision according to claim 1, characterized in that: The steps of extracting parameters from each image in each target bird's nest image group by using a parameter extraction algorithm are as follows: Import the target bird's nest image into the parameter extraction program, first convert the target bird's nest image into a grayscale image, then find the two points on the left and right edges of the bird's nest with the farthest distance through the front view and the back view, and construct a line connecting the two points, which is the length, then find the two points on the upper and lower edges of the bird's nest with the farthest distance, and construct a line connecting the two points, which is the height. According to the length and height values, calculate the aspect ratio; then find the multiple points on the top of the bird's nest along the arc-shaped nest body to the edge of the base with the farthest distance through the left view and the right view, and construct an arc connecting line, the length of the arc is the height of the nest body.

7. The method for bird's nest morphology grading and defect detection based on machine vision according to claim 1, characterized in that: The initial high-quality bird's nest classifier is a Bayesian model, or a Clip model + classifier combination model.

8. The bird's nest morphology grading and defect detection system based on machine vision is characterized by include: A first data set establishment module is used to construct a first data set consisting of a four-side image group of a bird's nest; wherein the first data set includes a four-side image group of a bird's nest with a defect type label set, and a four-side image group of a bird's nest of various levels and forms without defects and with a bird's nest type label set; A second data set establishment module is used to construct a second data set from four-side image groups of defect-free high-quality bird's nests of various grades and shapes in the first data set, and to set text information characterizing bird's nest attributes, wherein the bird's nest attributes include grades; A defect recognition model building module is used to input the first data set image group into a pre-built defect recognition model for model training, and obtain a defect recognition model according to a convergence condition; A target segmentation module, used to segment the image groups in the second data set using a target image segmentation algorithm to obtain target bird's nest image groups; The target size determination module is used to extract parameters of each target bird's nest image through a parameter extraction algorithm to obtain the size data corresponding to each bird's nest image group, and the size data includes length, height, length-to-height ratio and nest height; A high-quality bird's nest classifier construction module is used to take the size data and other attribute text information of each high-quality bird's nest except the grade attribute as input data, and use the grade attribute as output data to train an initial high-quality bird's nest classifier to obtain a high-quality bird's nest classifier; The prediction module is used to construct a high-quality bird's nest classifier module, which is used to input the four-sided image of the bird's nest to be identified and the text information of the bird's nest attributes into the defect recognition model to determine the defect type or the bird's nest type. If the prediction result is a high-quality bird's nest, the target image segmentation algorithm is used to segment the four-sided image of the bird's nest to obtain each target bird's nest image group, and then the parameter extraction algorithm is used to extract parameters from each image in the target bird's nest image group to obtain the size data corresponding to the bird's nest image group. The size data and text information are input into the high-quality bird's nest classifier to predict the quality of the bird's nest level.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the machine vision-based bird's nest morphology grading and defect detection method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the machine vision-based bird's nest morphology grading and defect detection method as described in any one of claims 1 to 7 is implemented.

Images