A finished pastry appearance analysis method and device based on AI image analysis recognition

Abstract

This invention discloses a method and apparatus for analyzing the appearance of finished pasta products based on AI image analysis and recognition, belonging to the field of food inspection technology. It includes: S1: acquiring multispectral images and preprocessing the multispectral images to obtain preprocessed multispectral images; S2: recognizing the preprocessed multispectral images using a set rotatable anchor frame to obtain a pasta position and posture detection box, and then cutting the pasta position and posture detection box using an image cutting algorithm to obtain a cutting result; S3: obtaining feature data based on the separation boundary, and determining the confidence level based on the feature data. This solves the problems of low efficiency and difficulty in handling random postures and background interference in traditional manual inspection, and achieves automated classification and grading of pasta fermentation status, foreign matter contamination, and appearance defects.

Description

Technical Field

[0001] This invention relates to the field of food testing technology, specifically to a method and apparatus for analyzing the appearance of finished pasta products based on AI image analysis and recognition. Background Technology

[0002] With the improvement of people's living standards and the increasing demands for food safety and quality, finished pastries, as an important part of traditional Chinese food, directly affect consumers' willingness to buy and brand image through their appearance quality. The appearance characteristics of finished pastries, such as shape, color, texture, and size consistency, are important indicators for measuring whether their processing technology meets standards and whether the production process is stable.

[0003] Currently, the appearance inspection of finished pasta products mainly relies on manual visual inspection. While manual inspection offers some flexibility and relies on experience-based judgment, it suffers from problems such as low efficiency, strong subjectivity, susceptibility to fatigue, and difficulty in standardization. Especially in large-scale industrial production, traditional manual inspection can no longer meet the demands for efficient, accurate, and real-time quality control.

[0004] In recent years, computer vision and artificial intelligence technologies have developed rapidly, especially in fields such as industrial quality inspection, agricultural product grading, and food defect identification. Deep learning-based image recognition technologies, such as convolutional neural networks (CNN) and object detection algorithms (YOLO, Faster R-CNN), have demonstrated excellent performance in image classification, object recognition, and defect detection. They can automatically extract key features from images and achieve intelligent recognition and evaluation of surface appearance features through model training.

[0005] Chinese invention patent CN108445007A discloses an image fusion-based detection method and apparatus. The method includes illuminating an object from different directions using multiple light-emitting modules to obtain multiple frames of images to be detected; performing image fusion processing on the multiple frames; and performing feature filtering processing on the resulting texture or height images. This method not only reduces the impact of poor image acquisition quality under low-contrast conditions but also obtains relatively complete surface feature information. This facilitates the fusion of surface feature information from each frame of the image to be detected into texture and height images, making it easier to identify and filter surface defect features from the distinguishing features of these images. The apparatus combined with the above surface defect detection method can minimize the influence of environmental factors during the detection process, synthesize feature images including surface defect features from a large number of images, enhance the filtering effect of surface defect features, and complete the surface defect detection work under low-contrast conditions.

[0006] However, on automated pastry production lines, finished pastries are typically distributed randomly on conveyor belts, exhibiting various issues such as unfixed positions, overlapping, tilting, or rotation. This makes accurate segmentation and positioning of the target area difficult. Furthermore, the complex industrial environment, often accompanied by background noise such as conveyor belt textures, uneven lighting, glare, and obstructions, further increases the difficulty of feature extraction during the target positioning stage, making it challenging to accurately identify individual pastries and assess their appearance quality. Summary of the Invention

[0007] The purpose of this invention is to provide a method and apparatus for analyzing the appearance of finished pasta products based on AI image analysis and recognition, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for analyzing the appearance of finished pastries based on AI image analysis and recognition, comprising:

[0009] S1: Data preprocessing: Multispectral images are acquired by using a multi-camera time-sharing imaging system, a polarized light imaging system, and a near-infrared imaging system, and the multispectral images are preprocessed to obtain preprocessed multispectral images.

[0010] S2: Obtaining Segmentation Results: Using a set rotatable anchor frame, the preprocessed multispectral image is identified to obtain surface position and pose detection boxes. These boxes are then segmented using an image segmentation algorithm to obtain segmentation results, including:

[0011] S2.1: Rotation detection: The preprocessed multispectral image is detected using the rotatable anchor frame to identify abnormal and retained facets;

[0012] S2.2: Obtain fused features: Based on the original RGB three-channel image, near-infrared band image and red channel image of the retained points, obtain the fused feature image;

[0013] S2.3: Constructing a spatial relationship graph: Based on the characteristics of the retained face points, set the edge weights between the retained face points, and treat each retained face point as a node. Then, connect the retained face points with edges according to the edge weights to construct a face point spatial relationship graph.

[0014] S2.4: Determine the separation boundary: Treat the retained face points as independent superpixels, and merge or divide the retained face points according to the edge weights, determine the ownership probability of each pixel in the face point spatial relationship graph, and determine the separation boundary according to the ownership probability.

[0015] S3: Quality assessment: Based on the separation boundary, obtain the feature data, and determine the confidence level based on the feature data.

[0016] Furthermore, acquiring multispectral images includes:

[0017] W1: Multi-camera time-division imaging: An industrial camera is set above the conveyor belt, and the light intensity is determined according to the exposure time of the industrial camera to obtain a multispectral image;

[0018] W2: Polarized light imaging: By using a polarization camera, the Stokes vector is obtained, the degree of polarization is determined, and the materials in the multispectral image are distinguished according to the degree of polarization.

[0019] W3: Near-infrared imaging: Using a near-infrared imaging sensor, the surface points on the conveyor belt are scanned to obtain the difference in reflectance between the dry and wet areas of the surface points. Simultaneously, the difference in reflectance between the dry and wet areas is compared with a preset difference threshold range, and based on the comparison result, the surface point area is divided, specifically as follows:

[0020] When the difference in reflectance between the dry and wet areas is less than the lower limit of the preset difference threshold range, the corresponding area is an uncooked area; when the difference in reflectance between the dry and wet areas is within the preset difference threshold range, the corresponding area is a normal area; when the difference in reflectance between the dry and wet areas is greater than the upper limit of the preset difference threshold range, the corresponding area is a transitionally dry area.

[0021] Furthermore, multispectral images are obtained, including:

[0022] W1.1: Setting up industrial cameras: Industrial cameras are set up on the top and both sides of the conveyor belt, and the number of pulses of the encoder trigger interval is determined according to the conveying speed of the conveyor belt and the triggering time interval of the industrial cameras.

[0023] W1.2: Determine the light source intensity: Based on the conveyor belt speed, the pixel size of the industrial camera, and the number of blurred pixels, determine the maximum exposure time, set the actual exposure time based on the maximum exposure time, and determine the required light source illuminance based on the actual exposure time and the effective light-transmitting area of ​​the industrial camera lens.

[0024] Furthermore, distinguishing the materials in the multispectral image includes:

[0025] W2.1: Determine the Stokes vector: The Stokes vector is obtained by synchronously acquiring data through a four-way mosaic filter in the polarization camera;

[0026] W2.2: Material Classification: Based on the Stokes vector, the degree of polarization is determined, and the degree of polarization is compared with a preset polarization threshold range. Based on the comparison result, the materials in the multispectral image are distinguished, specifically as follows:

[0027] When the polarization degree is less than the lower limit of the preset polarization threshold range, the material in the image area corresponding to the polarization degree is a surface point; when the polarization degree is within the preset polarization threshold range, the material in the image area corresponding to the polarization degree is a plastic foreign object; when the polarization degree is greater than the upper limit of the preset polarization threshold range, the material in the image area corresponding to the polarization degree is a metal tray.

[0028] Furthermore, the preprocessed multispectral image is obtained, including:

[0029] S1.1: Image denoising: The multispectral image is filtered by a notch filter, and the filtered multispectral image is denoised by wavelet coefficients to obtain a denoised reconstructed image.

[0030] S1.2: Illumination compensation: Extract the illumination component by setting the pixel kernel size, determine the reflection component, and obtain the reflection image based on the maximum and minimum reflection components.

[0031] S1.3: Multispectral Fusion: Based on the pixel grayscale values ​​in the near-infrared texture image, near-infrared texture feature values ​​are determined. These near-infrared texture feature values ​​are then compared with preset feature thresholds, and the final fused image is determined based on this comparison. Specifically:

[0032] When the near-infrared texture feature value is less than a preset feature threshold, the corresponding visible light edge map is the final fused image; otherwise, the visible light edge map and the near-infrared texture map are fused to obtain the final fused image.

[0033] Furthermore, anomalous facets were identified, including:

[0034] S2.1.1: Set rotation angle: Take the center of the surface point as the origin, the angle between the major axis and the horizontal axis as the rotation angle, and the vertical distance between the top of the surface point and the conveyor belt as the height. Set the anchor frame parameters of the rotatable anchor frame, and divide the rotatable angle of the rotatable anchor frame to determine the rotation angle.

[0035] S2.1.2: Feature map enhancement: The preprocessed multispectral image is compressed using a backbone network model to obtain a single-channel feature map, and an attention map is set. At the same time, the original feature map is enhanced using the attention map to obtain an enhanced feature map.

[0036] S2.1.3: Identify Abnormal Dough Points: Based on the rotation angle, a 3D prediction head is set up, and the enhanced feature map is identified using the 3D prediction head to obtain the actual volume of the dough points. Simultaneously, based on the actual volume and the standard volume of the dough points, the dough point expansion rate is determined, and the dough point expansion rate is compared with a preset expansion threshold range. Based on the comparison result, the fermentation result of the dough points is determined, specifically as follows:

[0037] When the expansion rate of the pastry is less than the lower limit of the preset expansion threshold range, the pastry fermentation is normal; when the expansion rate of the pastry is within the preset expansion threshold range, the pastry is slightly over-fermented and is marked with a warning; when the expansion rate of the pastry is greater than the upper limit of the preset expansion threshold range, the pastry is severely over-fermented and is discarded.

[0038] Furthermore, the separation boundary is determined, including:

[0039] S2.4.1: Determine the weight threshold: Based on the softness coefficient and viscosity coefficient of the retained dough, determine the merging threshold and the cutting threshold, specifically as follows:

[0040] ,in: The merging threshold, The cutting threshold, This is the softness coefficient of the pastry. The viscosity coefficient of the pastry;

[0041] S2.4.2: Obtaining Processed Nodes: The edge weights are compared with the merging threshold and the cutting threshold. Based on the comparison results, the nodes corresponding to the edge weights are processed, specifically as follows:

[0042] When the edge weight is less than the merging threshold, the corresponding nodes are merged; when the edge weight is greater than the cutting threshold, the corresponding nodes are split; otherwise, the corresponding nodes remain unchanged.

[0043] S2.4.3: Determine the optimal boundary: Based on the processed nodes, obtain the pixel belonging probability of each pixel in the feature image, and determine the marked nodes based on the pixel belonging probability. At the same time, connect the marked nodes to obtain the obtained node boundary.

[0044] Furthermore, the pixel assignment probability is compared with a preset assignment threshold, and the marked node is determined based on the comparison result, specifically as follows:

[0045] When the probability of a pixel being assigned to a node is greater than a preset assignment threshold, the corresponding node is a labeled node; otherwise, the corresponding node is not a labeled node.

[0046] Furthermore, determining the confidence level includes:

[0047] S3.1: Feature extraction: Divide the separation boundary into equal intervals, and obtain geometric, optical and texture indicators based on the data features of each sampling point;

[0048] S3.2: Obtaining the overall confidence score: Based on the geometric, optical, and texture indices, obtain the individual feature confidence scores for each feature, and determine the overall confidence score based on the individual feature confidence scores, specifically as follows:

[0049] ,in: To assess the overall confidence level, The total number of features, The weight of the l-th feature is... Index of features Let be the confidence level of the l-th feature.

[0050] A finished pastry appearance analysis device based on AI image analysis and recognition uses any one of the above-mentioned finished pastry appearance analysis methods based on AI image analysis and recognition.

[0051] Compared with the prior art, the beneficial effects of the present invention are:

[0052] Firstly, this invention distinguishes between pasta, plastic foreign objects, and metal trays through Stokes vector polarization analysis, and classifies pasta into uncooked, normal, and over-dried areas based on near-infrared reflectance differences. At the same time, it achieves graded early warning of fermentation abnormalities based on volume expansion rate.

[0053] Secondly, this invention uses a multi-camera time-sharing imaging system for collaborative acquisition, and effectively eliminates interference such as conveyor belt texture and environmental reflection through notch filtering and wavelet denoising algorithms, thereby improving the image signal-to-noise ratio and defect detection rate. At the same time, through the collaborative setting between the rotatable anchor frame and the 3D volume prediction head, it solves the problem of missed detection caused by surface tilt and stacking.

[0054] Thirdly, this invention combines graph cutting algorithm with dynamic threshold and automatically adjusts the segmentation strategy according to the stickiness coefficient and softness of the dots, thereby improving the segmentation accuracy of sticky dots. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the finished pastry appearance analysis method of the present invention;

[0056] Figure 2 This is a diagram illustrating the improvement effect of the dynamic threshold method in this invention;

[0057] Figure 3 This is a comparison chart of production efficiency and cost in this invention;

[0058] Figure 4 This is a distribution diagram of the steamed state of pastries in this invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] On automated pastry production lines, finished pastries are typically distributed randomly on conveyor belts, exhibiting various characteristics such as non-fixed positions, overlapping, tilting, or rotation, making accurate segmentation and positioning of target areas difficult. Furthermore, the complex industrial environment, often accompanied by conveyor belt textures, uneven lighting, reflective interference, and background noise from foreign objects, further increases the difficulty of feature extraction during target positioning, hindering accurate identification and appearance quality assessment of individual pastries. This application addresses this by using multispectral imaging to collaboratively acquire pastry image data and employing rotatable anchor frame positioning and image segmentation algorithms for precise pastry segmentation. Simultaneously, it establishes a dynamic threshold segmentation model based on softness and viscosity coefficients through multi-dimensional feature fusion, including 3D volume detection, material polarization characteristic analysis, and differences in reflectivity between dry and wet areas. Finally, it assesses the quality level of pastries using a comprehensive confidence score based on geometric, optical, and textural indicators. This solves the problems of low efficiency and difficulty in handling random postures and background interference inherent in traditional manual inspection, enabling automated classification and grading of pastry fermentation status, foreign object contamination, and appearance defects.

[0061] Example 1

[0062] refer to Figures 1-3 This embodiment provides a method for analyzing the appearance of finished pastries based on AI image analysis and recognition. The method specifically includes the following steps:

[0063] Step S1: Data Preprocessing. This involves acquiring multispectral images of the finished product's surface on the conveyor belt using a multi-camera time-sharing imaging system, a polarized light imaging system, and a near-infrared imaging system. These images include, but are not limited to, RGB images, polarized images, and near-infrared images. The acquired multispectral images are then preprocessed to obtain preprocessed multispectral images. Details are as follows:

[0064] Step S1.1: Image Denoising. This involves converting the acquired image data into corresponding frequency domain data, and then using a set notch filter to determine the filtered image. Specifically:

[0065] ,in: This is the frequency domain representation of the filtered image. For inverse Fourier transform operators, This is the frequency domain representation of the original image. This is the transfer function of the notch filter.

[0066] In this embodiment, the transfer function of the corresponding notch filter is set according to the texture feature frequency points in the image, specifically as follows:

[0067] ,in: Here is the transfer function of the notch filter. The number of interference frequencies that need to be suppressed. For the index of the interference frequency, The horizontal component of the image spatial frequency. The horizontal component of the k-th interference frequency, The vertical component of the spatial frequency of the image. The vertical component of the k-th interference frequency. These are the bandwidth control parameters for Gaussian notch filters.

[0068] Furthermore, by setting wavelet coefficients, the filtered image is denoised to obtain the corresponding denoised reconstructed image, specifically:

[0069] ,in: For the reconstructed image after denoising, It is the inverse discrete wavelet transform operator. These are the thresholded wavelet coefficients for all scales and orientations.

[0070] In this embodiment, the formula for obtaining the wavelet coefficients after thresholding for all scales and directions is as follows:

[0071] ,in: These are the wavelet coefficients in the j-th layer and q-th direction after thresholding. For symbolic functions, These are the original wavelet coefficients in the j-th layer and q-th direction. It is a positive part function. The threshold value is used.

[0072] Step S1.2: Illumination Compensation. This involves extracting the illumination components using a set 121×121 pixel kernel size, and then determining the corresponding reflection components based on the extracted illumination components. Specifically:

[0073] ,in: For the reflection component, The original brightness value. This represents the illumination component.

[0074] To elaborate further, by setting a low-pass filter, the corresponding illumination component is obtained from the original brightness value, specifically:

[0075] ,in: For light component, The original brightness value. It is a two-dimensional Gaussian kernel. This is a convolution operation.

[0076] In the specific implementation process, the surface of the dot is illuminated by strong light of 200 lux, and the average illumination around it is 180 lux, so the corresponding reflection component is 0.1. Furthermore, the shaded area of ​​the dot is illuminated by light of 50 lux, and the average illumination around the shaded area is 30 lux, so the corresponding reflection component is 0.5.

[0077] In this embodiment, based on the obtained reflection components, the maximum and minimum reflection components are determined, and based on the determined maximum and minimum reflection components, the corresponding reflection image is obtained, specifically as follows:

[0078] ,in: For reflection images, For the minimum reflection component, For the maximum reflection component, This is the reflection component.

[0079] Step S1.3: Optical Multispectral Fusion. This involves determining the corresponding near-infrared texture feature values ​​based on the pixel grayscale values ​​in the near-infrared texture map. Specifically:

[0080] ,in: These are near-infrared texture feature values. The pixel grayscale index of the neighborhood. Let p be the gray value of the p-th pixel in the neighborhood. The grayscale value of the center pixel. Binary weights, It is a step function.

[0081] In this embodiment, the obtained near-infrared texture feature values ​​are compared with a preset feature threshold (set specifically according to actual data, for example, 40), and the final fused image is determined based on the comparison. Specifically:

[0082] When the obtained near-infrared texture feature value is less than the preset feature threshold (i.e., 40), the corresponding visible light edge map is the final fused image. Conversely, when the obtained near-infrared texture feature value is not less than the preset feature threshold (i.e., 40), the corresponding visible light edge map and near-infrared texture map are fused proportionally to obtain the corresponding final fused image.

[0083] In this embodiment, the final fused image is obtained based on the visible light edge map and the near-infrared texture map, specifically as follows:

[0084] ,in: For the final merged image, These are near-infrared texture feature values. For visible light feature weights, The intensity at the edge of visible light.

[0085] Step S2: Obtain the segmentation results. This involves using the set YOLOv8 rotation detection model to identify the final fused image obtained in step S1.3, obtaining the facet position and pose detection boxes in the final fused image. Simultaneously, an image segmentation algorithm is used to segment the obtained facet position and pose detection boxes, obtaining the corresponding segmentation results. Details are as follows:

[0086] Step S2.1: Rotation Detection. This involves using a set rotatable anchor frame to detect abnormal facets on the enhanced feature image. Details are as follows:

[0087] Step S2.1.1: Set the rotation angle. This involves using the surface point as the origin, defining the angle between its major axis and the horizontal axis as the rotation angle of the rotatable anchor frame, and defining the vertical distance between the top of the surface point and the conveyor belt as the height of the rotatable anchor frame. Specifically, based on the center coordinates, width, height, rotation angle, and height of the anchor frame, set the corresponding anchor frame parameters.

[0088] Furthermore, based on the set rotatable anchor frame, the rotatable angle of the rotatable anchor frame is divided into multiple angle intervals. The corresponding rotation angle is determined based on the discretization step size and the corresponding angle fine-tuning amount for each angle interval. Specifically:

[0089] ,in: For the final predicted rotation angle, The step size for angle discretization. A categorized index for angle intervals. This is the amount of angle fine-tuning.

[0090] Step S2.1.2: Feature Map Enhancement. This involves compressing the final fused image obtained in Step S1.3 using a backbone network model to obtain single-channel feature maps. Based on these single-channel feature maps, the attention level corresponding to each pixel in the feature map is determined. Specifically:

[0091] ,in: For attention maps, The base of the natural logarithm, This is a single-channel feature map.

[0092] Furthermore, based on the obtained attention map, the original feature map is enhanced to obtain the corresponding enhanced feature map, specifically as follows:

[0093] ,in: To enhance the feature map, For the final merged image, This is an attention graph.

[0094] Step S2.1.3: Identify abnormal facets. Based on the final predicted rotation angle obtained in step S2.1.1, set the rotation angle of the 3D prediction head, and use the set 3D prediction head to identify facets in the enhanced feature map obtained in step S2.1.2, obtaining the volume of each facet. Further, based on the obtained facet volume and the standard facet volume, determine the corresponding facet expansion rate, specifically:

[0095] ,in: The expansion rate of the pastry. This refers to the actual volume of the pastry. This refers to the standard volume of pastries.

[0096] In this embodiment, the obtained dough expansion rate is compared with a preset expansion threshold range (specifically set based on actual data, such as 15%-25%), and the dough fermentation result is determined based on the comparison result. Specifically:

[0097] When the obtained dough expansion rate is less than the lower limit of the preset expansion threshold range (i.e., 15%), the dough is considered to have fermented normally. When the obtained dough expansion rate is within the preset expansion threshold range (i.e., 15%-25%), the dough is considered to have slightly over-fermented, and a warning mark is issued for the dough. When the obtained dough expansion rate is greater than the upper limit of the preset expansion threshold range (i.e., 25%), the dough is considered to have severely over-fermented, and the dough is discarded.

[0098] Step S2.2: Obtain the fusion features. Specifically, based on the original RGB three-channel image, near-infrared band image, and red channel image corresponding to the points retained in step S2.1.3, determine the corresponding fused feature image, as follows:

[0099] ,in: The input features are the fused features. This is the original RGB three-channel image. This is a near-infrared band image. This is the red channel image.

[0100] It is worth noting that for thick pastries (such as steamed cakes), the features of the three corresponding layers of images with different focal lengths are fused to obtain the corresponding fused feature image, specifically:

[0101] ,in: For multi-focal-length fusion features, Let be the fusion weight of the focal length of the i-th layer. To extract features from the original image at the i-th focal length using HRNet, The base of the natural logarithm, Let be the object distance of the i-th focal length. For optimal imaging object distance, Let be the object distance of the j-th focal length.

[0102] Step S2.3: Construct a spatial relationship graph. Each facet retained in step S2.1.3 is treated as a node. Based on the facet's color, texture, and geometric attributes, a feature vector is set for each node. Based on the apparent differences between different facests, a feature Euclidean distance is set between nodes. Based on the physical distance between different facests, a spatial Euclidean distance is set between nodes. Based on the texture continuity of different facests within the same region, a structural similarity index is set between nodes. Based on the shadow area of ​​the facet projected onto the conveyor belt, a shadow area mask is set for each node. In other words, by setting the feature vector, feature Euclidean distance, spatial Euclidean distance, structural similarity index, and shadow area mask, the edge weights between different facests are determined. Specifically:

[0103] ,in: Let the edge weight be the weight between nodes r and o. To adjust the weighting coefficient of distance, Let r be the feature vector of node r. Let o be the feature vector of node o. Let be the spatial Euclidean distance between node r and node o. The weighting coefficients for the texture. Let be the structural similarity index between node r and node o. The weighting coefficients for the shadow features are... This serves as a mask for the shadow region of node r. This is a mask for the shadow region of node o.

[0104] In this embodiment, each face point is treated as a node, and different face points are connected by edges according to the determined edge weights between them, thus constructing a corresponding face point spatial relationship graph.

[0105] Step S2.4: Determine the separation boundary. Each facet is treated as an independent superpixel, and based on the edge weights determined in Step S2.3, different facets are merged or segmented to obtain an optimized spatial relationship graph. Using this optimized spatial relationship graph, the assignment probability of each pixel is determined, and based on these probabilities, different pixels are connected to determine the corresponding separation boundary. Specifically:

[0106] Step S2.4.1: Determine the weight thresholds. Specifically, based on the softness and viscosity coefficients of the dough, determine the corresponding merging and cutting thresholds.

[0107] ,in: The merging threshold, The cutting threshold, This is the softness coefficient of the pastry. is the viscosity coefficient of the pastry.

[0108] In the specific implementation process, based on the softness coefficient and viscosity coefficient of different pastries (such as steamed buns, mochi, and shortbread), corresponding merging thresholds and cutting thresholds are set, as shown in Table 1 below:

[0109] Table 1: Threshold Allocation Table

[0110]

[0111] Step S2.4.2: Obtain the processed node. That is, based on the merging and cutting thresholds set in step S2.4.1, compare the edge weights determined in step S2.3 with the set merging and cutting thresholds, and process the node corresponding to the edge weight based on the comparison result. Specifically:

[0112] If the obtained edge weight is less than the merging threshold, the nodes corresponding to that edge weight are merged. If the obtained edge weight is greater than the splitting threshold, the nodes corresponding to that edge weight are split. Otherwise, the nodes remain unchanged.

[0113] Step S2.4.3: Determine the optimal boundary. That is, based on the processed nodes determined in step S2.4.2, determine the assignment probability corresponding to each pixel in the feature image, specifically:

[0114] ,in: For pixels Belongs to node The probability, For pixels eigenvectors, Let i be the feature vector of node i. The base of the natural logarithm, Let be the feature vector of node j.

[0115] In this embodiment, the obtained pixel assignment probability is compared with a preset assignment threshold (which is specifically set based on actual data, such as 0.8), and the marked node is determined based on the comparison result. Specifically:

[0116] When the obtained pixel assignment probability is greater than the preset assignment threshold (i.e., 0.8), the node corresponding to that pixel assignment probability is a marked node. Conversely, when the obtained pixel assignment probability is not greater than the preset assignment threshold (i.e., 0.8), the node corresponding to that pixel assignment probability is not a marked node.

[0117] Furthermore, by connecting the identified marked nodes in sequence, the resulting node boundary is the determined optimal boundary.

[0118] refer to Figure 2 ,Depend on Figure 2It can be seen that in this embodiment, the dynamic threshold segmentation accuracy (95%, 96%, 94%) of all tested categories (steamed buns, mochi, and shortbread) is higher than that of the fixed threshold (92%, 78%, 88%), and the segmentation accuracy of mochi increased from 78% to 96%, which is a relative improvement of 23%. In other words, for foods with irregular shapes and soft, sticky textures, setting a dynamic threshold can improve the segmentation accuracy.

[0119] Step S3: Quality Assessment. This involves obtaining the feature data within the optimal boundary determined in step S2.4.3, and then determining the corresponding confidence level based on the obtained feature data. Specifically:

[0120] Step S3.1: Feature Extraction. Based on the optimal boundary determined in step S2.4.3, this optimal boundary is divided into multiple sampling points at equal intervals (e.g., 128 sampling points). Based on the data features corresponding to each sampling point, the corresponding geometric indices (i.e., volumetric deformation rate), optical indices (i.e., polarized gloss and near-infrared moisture distribution), and texture indices (i.e., Fourier descriptor continuity) are obtained. Specifically:

[0121] ,in: The expansion rate of the pastry. This refers to the actual volume of the pastry. For standard volume of pastries The light intensity in the 0° linear polarization direction. The light intensity in the 90° linear polarization direction. For polarized gloss, The standard deviation of near-infrared moisture distribution. The grayscale values ​​of the near-infrared image. For the standard deviation operator, These are the Fourier coefficients. The outline continuity index, For the number of outline points, Index the contour points.

[0122] Step S3.2: Obtain the overall confidence score. That is, based on the geometric, optical, and textural indices obtained in step S3.1, determine the confidence score of each individual feature. Specifically:

[0123] ,in: Let be the confidence level of the l-th feature. The base of the natural logarithm, The slope coefficient of the Sigmoid function. Let l be the standardized eigenvalue. This is the threshold value for the l-th feature.

[0124] Furthermore, based on the confidence level of each obtained feature, the corresponding overall confidence level is determined, specifically as follows:

[0125] ,in: To assess the overall confidence level, The total number of features, The weight of the l-th feature is... Index of features Let be the confidence level of the l-th feature.

[0126] In this embodiment, the corresponding production line action is determined by obtaining the comprehensive confidence level based on the confidence interval set according to the actual production line action. In the specific implementation process, multiple confidence intervals are set as shown in Table 2 below:

[0127] Table 2: Confidence Interval Table

[0128]

[0129] refer to Figure 3 ,Depend on Figure 3 It can be seen that the manual inspection speed is 30 items / minute, while the AI ​​inspection speed is 1500 items / minute, meaning the AI ​​inspection speed is 50 times faster than the manual inspection speed. Meanwhile, the cost per item for manual inspection is 0.5 yuan, while the cost per item for AI inspection is 0.1 yuan, meaning the cost per item for AI inspection is reduced by 80%. In other words, AI inspection achieves both higher speed and lower cost, breaking the traditional trade-off between efficiency and cost in production.

[0130] This embodiment also provides a finished pastry appearance analysis device based on AI image analysis and recognition, which uses the above-mentioned finished pastry appearance analysis method based on AI image analysis and recognition.

[0131] Example 2

[0132] This embodiment provides a method for analyzing the appearance of finished pasta products based on AI image analysis and recognition. The specific implementation method is the same as in Embodiment 1, except that in step S1, a multispectral image of the finished pasta products on the conveyor belt is acquired using a multi-camera time-sharing imaging system, a polarized light imaging system, and a near-infrared imaging system. The invention will be illustrated below with specific examples of this embodiment.

[0133] In this embodiment, multispectral images of the finished pasta products on the conveyor belt are acquired, as detailed below:

[0134] Step W1: Multi-camera time-sharing imaging. This involves installing industrial cameras in different directions above the conveyor belt, determining the corresponding light intensity based on the exposure time of each camera, and acquiring the corresponding multispectral image under the determined light intensity. Details are as follows:

[0135] Step W1.1: Set up the industrial cameras. This involves installing three industrial cameras above the conveyor belt: one directly above the conveyor belt, and two more at 45° angles to either side. Specifically, the top camera should be a 20-megapixel camera with a global shutter and a frame rate of 120fps. The camera at the left 45° angle should be a 5-megapixel polarized camera with an adjustable polarization angle of 10°. The camera at the right 45° angle should be an 850nm near-infrared camera.

[0136] Furthermore, based on the conveyor belt speed and the trigger time interval between the industrial camera, the number of pulses required for the encoder trigger interval is determined, specifically as follows:

[0137] ,in: The number of pulses required for the encoder trigger interval. The conveyor belt speed, The trigger time interval between industrial cameras. This is the encoder's resolution.

[0138] In the specific implementation process, when the conveyor belt speed is 0.5 m / s, the trigger time interval between industrial cameras is 33 ms, and the encoder resolution is 500 pulses / m, the corresponding number of pulses required for the encoder trigger interval is approximately 8 pulses. Specifically, the trigger sequences corresponding to the three industrial cameras are shown in Table 3 below:

[0139] Table 3: Trigger Sequence List

[0140]

[0141] Step W1.2: Determine the light source intensity. This involves determining the maximum exposure time based on the conveyor belt speed, the camera's pixel size, and the set number of blur pixels. Specifically:

[0142] ,in: Maximum exposure time to avoid motion blur The number of blurred pixels allowed. For camera pixel size, This refers to the conveyor belt's conveying speed.

[0143] Furthermore, based on the determined maximum exposure time, the actual exposure time used is set; that is, the actual exposure time used must not be less than the determined maximum exposure time. In this embodiment, the required light source illuminance is determined based on the set actual exposure time, the effective light-transmitting area of ​​the camera lens, and the sensitivity of the sensor, specifically as follows:

[0144] ,in: For the required light source illuminance, For the sensitivity of the sensor, This refers to the actual exposure time used. The effective light-transmitting area of ​​the camera lens. For the efficiency of the optical system.

[0145] In the specific implementation process, the sensor sensitivity was set to 1V / (μJ / cm). 2 The actual exposure time was set to 50μs, and the effective light-gathering area of ​​the camera lens was set to 10mm². 2 If the optical system efficiency is set to 0.6, the corresponding light source illuminance is approximately 3333 lux.

[0146] Step W2: Polarized light imaging. This involves using a polarized camera with mosaic filters set in different directions to acquire the corresponding Stokes vectors. Based on these Stokes vectors, the degree of polarization is determined, and the materials within the image area are distinguished according to the determined degree of polarization. Specifically:

[0147] Step W2.1: Determine the Stokes vector. This involves setting a four-way mosaic filter inside the polarization camera (e.g., a 5-megapixel FLIR BFS-PGE-50S5P-C), specifically at 0°, 45°, 90°, and 135°. In other words, synchronous acquisition is performed through the four-way mosaic filter built into the polarization camera to obtain the corresponding Stokes vector, specifically:

[0148] ,in: For Stokes parameters, The first component of the Stokes parameters, The second component of the Stokes parameters, The third component of the Stokes parameters, The fourth component of the Stokes parameters, The light intensity in the 0° linear polarization direction. The light intensity in the 90° linear polarization direction. The light intensity in the 45° linear polarization direction. The light intensity in the 135° linear polarization direction. The intensity of right-handed circularly polarized light. This represents the intensity of left-handed circularly polarized light.

[0149] Step W2.2: Material Division. This involves obtaining the corresponding degree of polarization based on the Stokes vector determined in Step W2.1. Specifically:

[0150] ,in: For degree of polarization, The first component of the Stokes parameters, The second component of the Stokes parameters, This is the third component of the Stokes parameters.

[0151] In this embodiment, the obtained polarization degree is compared with a preset polarization threshold range (specifically set based on actual data, such as 0.4-0.6), and the material in the image area is distinguished based on the comparison result. Specifically:

[0152] When the obtained polarization degree is less than the lower limit of the preset polarization threshold range (i.e., 0.4), the material in the image region corresponding to that polarization degree is a surface point. When the obtained polarization degree is within the preset polarization threshold range (i.e., 0.4-0.6), the material in the image region corresponding to that polarization degree is a plastic foreign object. When the obtained polarization degree is greater than the upper limit of the preset polarization threshold range (i.e., 0.6), the material in the image region corresponding to that polarization degree is a metal tray.

[0153] Step W3: Near-infrared imaging. This involves scanning the surface points on the conveyor belt using an InGaAs sensor equipped with an 850nm narrowband filter to obtain the corresponding residual light intensity after penetration. Specifically:

[0154] ,in: For penetration depth The remaining light intensity afterward The initial incident light intensity, The base of the natural logarithm, The attenuation coefficient is... This represents the penetration depth.

[0155] Furthermore, based on the reflectivity and attenuation coefficient corresponding to different penetration depths, the magnitude of the reflection difference between dry and wet areas can be obtained, specifically:

[0156] ,in: The difference in reflectivity between dry and wet areas, As the reference reflectivity, The attenuation coefficient of the drying material. The attenuation coefficient of the wetted material. For penetration depth, is the base of the natural logarithm.

[0157] In this embodiment, the obtained difference in reflectance between dry and wet areas is compared with a preset difference threshold range (specifically set based on actual data, such as 60%-70%), and the surface area is divided according to the comparison result. Specifically:

[0158] When the obtained difference in reflectance between dry and wet areas is less than the lower limit of the preset difference threshold range (i.e., 60%), the area corresponding to this difference in reflectance is an undercooked area. When the obtained difference in reflectance between dry and wet areas is within the preset difference threshold range (i.e., 60%-70%), the area corresponding to this difference in reflectance is a normal area. When the obtained difference in reflectance between dry and wet areas is greater than the upper limit of the preset difference threshold range (i.e., 70%), the area corresponding to this difference in reflectance is a transitionally dry area.

[0159] refer to Figure 4 ,Depend on Figure 4 It can be seen that: the uncooked area has the lowest reflectance and high moisture content, the normally cooked area has a moderate reflectance, accounting for 60% of the total sample, and the overly dry area has the highest reflectance, indicating excessive moisture loss.

[0160] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.

Claims

1. A method for analyzing the appearance of a finished pastry product based on AI image analysis recognition, characterized by, Including: S1: Data preprocessing: Multispectral images are acquired by using a multi-camera time-sharing imaging system, a polarized light imaging system, and a near-infrared imaging system, and the multispectral images are preprocessed to obtain preprocessed multispectral images. S2: Obtaining Segmentation Results: Using a set rotatable anchor frame, the preprocessed multispectral image is identified to obtain surface position and pose detection boxes. These boxes are then segmented using an image segmentation algorithm to obtain segmentation results, including: S2.1: Rotation Detection: The preprocessed multispectral image is detected using the rotatable anchor frame to identify anomalous and retained facets, including: S2.1.1: Set rotation angle: Take the center of the surface point as the origin, the angle between the major axis and the horizontal axis as the rotation angle, and the vertical distance between the top of the surface point and the conveyor belt as the height. Set the anchor frame parameters of the rotatable anchor frame, and divide the rotatable angle of the rotatable anchor frame to determine the rotation angle. S2.1.2: Feature map enhancement: The preprocessed multispectral image is compressed using a backbone network model to obtain a single-channel feature map, and an attention map is set. At the same time, the original feature map is enhanced using the attention map to obtain an enhanced feature map. S2.1.3: Identify Abnormal Dough Points: Based on the rotation angle, a 3D prediction head is set up, and the enhanced feature map is identified using the 3D prediction head to obtain the actual volume of the dough points. Simultaneously, based on the actual volume and the standard volume of the dough points, the dough point expansion rate is determined, and the dough point expansion rate is compared with a preset expansion threshold range. Based on the comparison result, the fermentation result of the dough points is determined, specifically as follows: When the expansion rate of the pastry is less than the lower limit of the preset expansion threshold range, the pastry fermentation is normal; when the expansion rate of the pastry is within the preset expansion threshold range, the pastry is slightly over-fermented, and the pastry is marked with a warning. When the expansion rate of the pastry exceeds the upper limit of the preset expansion threshold range, the pastry is severely over-fermented and must be discarded. S2.2: Obtain fused features: Based on the original RGB three-channel image, near-infrared band image and red channel image of the retained points, obtain the fused feature image; S2.3: Constructing a spatial relationship graph: Based on the characteristics of the retained face points, set the edge weights between the retained face points, and treat each retained face point as a node. Then, connect the retained face points with edges according to the edge weights to construct a face point spatial relationship graph. S2.4: Determine the separation boundary: Treat the retained face points as independent superpixels, and merge or divide the retained face points according to the edge weights, determine the ownership probability of each pixel in the face point spatial relationship graph, and determine the separation boundary according to the ownership probability. S3: Quality assessment: Based on the separation boundary, obtain the feature data, and determine the confidence level based on the feature data.

2. The method for analyzing the appearance of finished pastries based on AI image analysis and recognition according to claim 1, characterized in that, Acquiring multispectral images includes: W1: Multi-camera time-division imaging: An industrial camera is set above the conveyor belt, and the light intensity is determined according to the exposure time of the industrial camera to obtain a multispectral image; W2: Polarized light imaging: By using a polarization camera, the Stokes vector is obtained, the degree of polarization is determined, and the materials in the multispectral image are distinguished according to the degree of polarization. W3: Near-infrared imaging: Using a near-infrared imaging sensor, the surface points on the conveyor belt are scanned to obtain the difference in reflectance between the dry and wet areas of the surface points. Simultaneously, the difference in reflectance between the dry and wet areas is compared with a preset difference threshold range, and based on the comparison result, the surface point area is divided, specifically as follows: When the difference in reflectance between the dry and wet areas is less than the lower limit of the preset difference threshold range, the corresponding area is an uncooked area; when the difference in reflectance between the dry and wet areas is within the preset difference threshold range, the corresponding area is a normal area; when the difference in reflectance between the dry and wet areas is greater than the upper limit of the preset difference threshold range, the corresponding area is a transitionally dry area.

3. The method for analyzing the appearance of finished pastries based on AI image analysis and recognition according to claim 2, characterized in that, The obtained multispectral images include: W1.1: Setting up industrial cameras: Industrial cameras are set up on the top and both sides of the conveyor belt, and the number of pulses of the encoder trigger interval is determined according to the conveying speed of the conveyor belt and the triggering time interval of the industrial cameras. W1.2: Determine the light source intensity: Based on the conveyor belt speed, the pixel size of the industrial camera, and the number of blurred pixels, determine the maximum exposure time, set the actual exposure time based on the maximum exposure time, and determine the required light source illuminance based on the actual exposure time and the effective light-transmitting area of ​​the industrial camera lens.

4. The method for analyzing the appearance of finished pastries based on AI image analysis and recognition according to claim 2, characterized in that, Distinguishing materials in the multispectral image includes: W2.1: Determine the Stokes vector: The Stokes vector is obtained by synchronously acquiring data through a four-way mosaic filter in the polarization camera; W2.2: Material Classification: Based on the Stokes vector, the degree of polarization is determined, and the degree of polarization is compared with a preset polarization threshold range. Based on the comparison result, the materials in the multispectral image are distinguished, specifically as follows: When the polarization degree is less than the lower limit of the preset polarization threshold range, the material in the image area corresponding to the polarization degree is a surface point; when the polarization degree is within the preset polarization threshold range, the material in the image area corresponding to the polarization degree is a plastic foreign object; when the polarization degree is greater than the upper limit of the preset polarization threshold range, the material in the image area corresponding to the polarization degree is a metal tray.

5. The method for analyzing the appearance of finished pastries based on AI image analysis and recognition according to claim 1, characterized in that, The preprocessed multispectral image is obtained, including: S1.1: Image denoising: The multispectral image is filtered by a notch filter, and the filtered multispectral image is denoised by wavelet coefficients to obtain a denoised reconstructed image. S1.2: Illumination compensation: Extract the illumination component by setting the pixel kernel size, determine the reflection component, and obtain the reflection image based on the maximum and minimum reflection components. S1.3: Multispectral Fusion: Based on the pixel grayscale values ​​in the near-infrared texture image, near-infrared texture feature values ​​are determined. These near-infrared texture feature values ​​are then compared with preset feature thresholds, and the final fused image is determined based on this comparison. Specifically: When the near-infrared texture feature value is less than a preset feature threshold, the corresponding visible light edge map is the final fused image; otherwise, the visible light edge map and the near-infrared texture map are fused to obtain the final fused image.

6. The method for analyzing the appearance of finished pastries based on AI image analysis and recognition according to claim 1, characterized in that, The separation boundary is determined, including: S2.4.1: Determine the weight threshold: Based on the softness coefficient and viscosity coefficient of the retained dough, determine the merging threshold and the cutting threshold, specifically as follows: in: The merging threshold, The cutting threshold, This is the softness coefficient of the pastry. The viscosity coefficient of the pastry; S2.4.2: Obtaining Processed Nodes: The edge weights are compared with the merging threshold and the cutting threshold. Based on the comparison results, the nodes corresponding to the edge weights are processed, specifically as follows: When the edge weight is less than the merging threshold, the corresponding nodes are merged; when the edge weight is greater than the cutting threshold, the corresponding nodes are split; otherwise, the corresponding nodes remain unchanged. S2.4.3: Determine the optimal boundary: Based on the processed nodes, obtain the pixel belonging probability of each pixel in the feature image, and determine the marked nodes based on the pixel belonging probability. At the same time, connect the marked nodes to obtain the obtained node boundary.

7. The method for analyzing the appearance of finished pastries based on AI image analysis and recognition according to claim 6, characterized in that, The pixel's attribution probability is compared with a preset attribution threshold, and the marked node is determined based on the comparison result, specifically as follows: When the probability of a pixel being assigned to a node is greater than a preset assignment threshold, the corresponding node is a labeled node; otherwise, the corresponding node is not a labeled node.

8. The method for analyzing the appearance of finished pastries based on AI image analysis and recognition according to claim 1, characterized in that, Determining the confidence level includes: S3.1: Feature extraction: Divide the separation boundary into equal intervals, and obtain geometric, optical and texture indicators based on the data features of each sampling point; S3.2: Obtaining the overall confidence score: Based on the geometric, optical, and texture indices, obtain the individual feature confidence scores for each feature, and determine the overall confidence score based on the individual feature confidence scores, specifically as follows: in: To assess the overall confidence level, The total number of features, The weight of the l-th feature is... Index of features Let be the confidence level of the l-th feature.

9. A finished pastry appearance analysis device based on AI image analysis and recognition, characterized in that, The method for analyzing the appearance of finished pastries based on AI image analysis and recognition, as described in any one of claims 1-8, was used.

Images