Computer vision-based glassware production defect detection method and system

Abstract

The application discloses a glassware production defect detection method and system based on computer vision, and relates to the related field of glass manufacturing.The method comprises the following steps: collecting multi-modal original image data of glassware on a production line; removing interference from the multi-modal original image data to obtain a preliminary pure feature map; performing volume defect detection on the transmission feature map to obtain a first defect candidate heat map; using a phase unfolding feature map to distinguish internal and external defects of the first defect candidate heat map to obtain a second enhanced defect distribution map; using a motion time sequence feature map to remove pseudo-defects from the second enhanced defect distribution map to obtain a third time consistency defect map; and using production prior features to confirm defects of the third time consistency defect map to output a defect detection result.The application solves the problem of low detection accuracy caused by complex imaging interference and various defect forms in existing glassware defect detection, and achieves the effects of improving detection accuracy and reliability.

Description

Technical Field

[0001] This application relates to the field of glass manufacturing, and in particular to a method and system for detecting defects in glassware production based on computer vision. Background Technology

[0002] Defect detection in the glassware production process is a crucial step in ensuring product quality and yield. Currently, industrial production lines commonly employ manual visual inspection or traditional machine vision methods. Manual inspection relies on quality inspectors directly observing the surface and interior of the glassware under specific light sources. Traditional machine vision typically uses fixed lighting combined with a single industrial camera to capture images, identifying abnormal areas through preset grayscale thresholds or edge detection operators. However, manual inspection is susceptible to visual fatigue and subjective judgment, resulting in poor consistency and stability, and it struggles to detect minute or instantaneous defects on high-speed production lines. Traditional machine vision methods are ill-suited to the high light transmittance of glass, curved surfaces, and diverse defect morphologies. They are easily affected by surface reflections under complex lighting conditions in industrial settings, and a single imaging modality cannot simultaneously capture the complete features of different types of defects, leading to high false negative and false positive rates, failing to meet the accuracy and stability requirements of large-scale production.

[0003] Currently, the detection of defects in glassware faces technical challenges due to complex imaging interference and diverse defect morphologies, resulting in low detection accuracy. Summary of the Invention

[0004] This application provides a computer vision-based method and system for detecting defects in glassware production. It acquires high dynamic range images under transmitted light, stripe images under ring structured light, and continuous frame images under rotation. The transmitted images undergo refraction distortion correction to obtain a transmission feature map. The structured light images are subjected to polarization difference and phase unwrapping to obtain a phase unfolding feature map. The continuous frames are registered to obtain a motion temporal feature map. The transmission feature map is used to detect candidate regions for volumetric defects. The surface morphology information from the phase unfolding feature map is combined to distinguish between internal defects and surface deformation. Furthermore, optical flow tracking from the motion temporal feature map is used to eliminate false defects such as dust and reflections. Visual features are fused with prior production features such as mold information and process parameters. The system outputs defect type, location, size, and traceability information via a verification detection network. This approach solves the technical problem of low detection accuracy in existing glassware defect detection methods due to complex imaging interference and diverse defect morphologies, achieving improved detection accuracy and reliability.

[0005] This application provides a computer vision-based method for detecting defects in glassware production, comprising: acquiring multimodal raw image data of glassware on a production line, including high dynamic range transmission images under a transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images acquired when the glassware is rotating; removing interference from the multimodal raw image data to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map; performing volumetric defect detection on the transmission feature map to obtain a first defect candidate heatmap; using the phase unfolding feature map to distinguish between internal and external defects in the first defect candidate heatmap to obtain a second enhanced defect distribution map; using the motion temporal feature map to remove false defects from the second enhanced defect distribution map to obtain a third time-consistent defect map; using production prior features to confirm defects in the third time-consistent defect map, and outputting the defect detection result.

[0006] In a possible implementation, multimodal raw image data of glassware on the production line is acquired, including high dynamic range transmission images under a transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images acquired when the glassware is rotating. The following processing is performed: the ring structured light is subjected to outgoing polarization processing, and the camera acquiring the structured light images is subjected to incident polarization processing, wherein the direction of the outgoing polarization processing is perpendicular to the direction of the incident polarization processing; at least one pair of structured light images with different polarization angles is acquired, and the structured light image pairs have temporal consistency.

[0007] In a possible implementation, interference removal is performed on the multimodal raw image data to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map. The following processing is then performed: the refractive distortion field is pre-calibrated based on the three-dimensional model of the glass vessel, and the high dynamic range transmission image is subjected to inverse refraction transformation using the refractive distortion field to obtain the transmission feature map; polarization difference operation is performed on the structured light image pair to remove the specular reflection component, resulting in a de-highlight structured light image; phase unwrapping operation is performed on the de-highlight structured light image to obtain the phase unfolding feature map; and inter-frame registration is performed on the continuous time-series images to obtain the motion temporal feature map.

[0008] In a possible implementation, volumetric defect detection is performed on the transmission feature map to obtain a first defect candidate heatmap, and the following processing is performed: the transmission feature map is input into multiple pre-trained parallel convolutional branches, each with a different receptive field size, and each branch outputs a defect confidence map at a corresponding scale; prior information on the wall thickness of the glassware is obtained, and multiple weight coefficients are calculated based on the prior information on the wall thickness, wherein each weight coefficient corresponds to a convolutional branch; the defect confidence map and the weight coefficients are summed pixel-by-pixel to obtain a fused confidence map; threshold segmentation and noise filtering are performed on the fused confidence map to output the first defect candidate heatmap.

[0009] In a possible implementation, the phase unfolding feature map is used to distinguish between internal and external defects in the first candidate defect heatmap to obtain a second enhanced defect distribution map. The following processing is then performed: pixels with a confidence level greater than a first threshold in the first candidate defect heatmap are mapped onto the three-dimensional surface mesh of the glassware to obtain a set of three-dimensional spatial locations of candidate defects; the phase gradient magnitude of each location in the phase unfolding feature map is calculated, and regions with a magnitude greater than a second threshold are extracted as surface geometric distortion regions; each candidate defect in the set of three-dimensional spatial locations of candidate defects is traversed to determine whether it falls within the surface geometric distortion region: if it falls within, it is marked as a surface deformation defect; if it does not fall within, it is marked as an internal volumetric defect, resulting in a set of marked candidate defects; the set of marked candidate defects is superimposed on the first candidate defect heatmap to generate the second enhanced defect distribution map.

[0010] In a possible implementation, the motion temporal feature map is used to remove false defects from the second enhanced defect distribution map to obtain a third temporally consistent defect map. The following processing is then performed: the optical flow field between frames is extracted from the motion temporal feature map to establish the continuous trajectory of each point on the glass surface during rotational motion; each candidate defect point in the second enhanced defect distribution map is mapped onto the continuous trajectory to obtain the position change sequence and grayscale change sequence of the candidate defect in the continuous time series frames; a position stability score is calculated based on the position change sequence; a grayscale stability score is calculated based on the grayscale change sequence; when the position stability score is greater than or equal to a preset position score threshold and the grayscale stability score is greater than or equal to a preset grayscale score threshold, the corresponding candidate defect is determined to be a temporally consistent defect; otherwise, it is determined to be a temporally inconsistent defect; the temporally consistent defect is retained, and the temporally inconsistent defect is removed from the second enhanced defect distribution map to obtain the third temporally consistent defect map.

[0011] In a possible implementation, the third time-consistent defect map is verified using prior production features, and the defect detection result is output. The following processing is performed: acquiring the prior production features of the glassware, including mold information and process parameters; mapping the mold information into a mold embedding vector and normalizing the process parameters into a process parameter vector; performing global feature extraction on the third time-consistent defect map to obtain a visual feature vector; fusing the visual feature vector, the mold embedding vector, and the process parameter vector to obtain a fused feature vector; inputting the fused feature vector into a pre-trained verification detection network, and outputting the defect type, location coordinates, and size parameters of each defect as the defect detection result.

[0012] In a possible implementation, the following processing is also performed: determining the region type to which the defect belongs based on the location coordinates; querying a preset region-defect type mapping table to determine whether the combination of the region type and the defect type is valid; if valid, matching the corresponding target process from the preset defect-process mapping rules based on the region type and the defect type; retrieving the corresponding production station information from the production execution system based on the target process; and adding the production station information as traceability information to the defect detection result.

[0013] In a possible implementation, the following processing is also performed: if the combination of the region type and the defect type is invalid, a correction mechanism is triggered; and according to the correction mechanism, a manual review process is initiated.

[0014] This application also provides a computer vision-based glassware production defect detection system, comprising: a multimodal raw image data acquisition module for acquiring multimodal raw image data of glassware on the production line, including high dynamic range transmission images under a transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images acquired when the glassware under test is rotating; an interference removal module for removing interference from the multimodal raw image data to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map; a volumetric defect detection module for performing volumetric defect detection on the transmission feature map to obtain a first defect candidate heatmap; an internal and external defect differentiation module for differentiating internal and external defects in the first defect candidate heatmap using the phase unfolding feature map to obtain a second enhanced defect distribution map; a pseudo-defect removal module for removing pseudo-defects from the second enhanced defect distribution map using the motion temporal feature map to obtain a third time-consistent defect map; and a defect verification module for verifying defects in the third time-consistent defect map using production prior features and outputting defect detection results.

[0015] The proposed computer vision-based method and system for detecting defects in glassware production first acquires multimodal raw image data of glassware on the production line, including high dynamic range transmission images under transmitted light, structured light images under ring structured light illumination, and time-series continuous frame images acquired while the glassware is rotating. Next, interference removal is performed on the multimodal raw image data to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map. Then, volumetric defect detection is performed on the transmission feature map to obtain a first defect candidate heatmap. The phase unfolding feature map is then used to distinguish between internal and external defects in the first defect candidate heatmap, resulting in a second enhanced defect distribution map. The motion temporal feature map is then used to remove false defects from the second enhanced defect distribution map, resulting in a third time-consistent defect map. Finally, production prior features are used to confirm the defects in the third time-consistent defect map, and the defect detection result is output. The proposed method and system achieve the technical effect of improving detection accuracy and reliability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0017] Figure 1 This is a flowchart illustrating a computer vision-based method for detecting defects in glassware production, as provided in an embodiment of this application.

[0018] Figure 2 A schematic diagram of the structure of a computer vision-based glassware production defect detection system provided in an embodiment of this application.

[0019] Figure labeling: 10 Multimodal raw image data acquisition module, 20 Interference removal module, 30 Volumetric defect detection module, 40 Internal and external defect differentiation module, 50 False defect removal module, 60 Defect confirmation module. Detailed Implementation

[0020] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below.

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, references to "some embodiments" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. The terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only.

[0023] This application provides a computer vision-based method for detecting defects in glassware production, such as... Figure 1 As shown, the method includes:

[0024] Step S100: Collect multimodal raw image data of the glassware on the production line, including high dynamic range transmission images under transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images collected when the glassware under test is rotating.

[0025] Specifically, a transmissive light source is placed at the acquisition station, positioned below the glass vessel. The light passes vertically upward through the glass vessel and is received by the camera above. High dynamic range (HDR) imaging technology is employed, using multiple exposures or a HDR camera to capture multiple frames at different exposure times from the same shooting position. These multiple frames are then combined into a single HDR transmissive image, which simultaneously preserves details of bubbles in the bright areas and details of stones in the dark areas inside the glass vessel.

[0026] A ring of structured light lamps is placed around the glassware. The lamp consists of multiple light-emitting units arranged in a ring. The light shines on the surface of the glassware from the side at a certain angle. The structured light image collected shows a striped pattern of light and dark, which is used to extract surface morphology information.

[0027] The glassware is placed on a rotatable support platform, which is then rotated uniformly around a vertical axis. A ring-shaped structured light is activated, and simultaneously, a camera is started to continuously acquire images at a fixed frame rate, resulting in a series of time-series continuous frame images. Each frame records the surface stripe pattern of the glassware at a certain rotation angle.

[0028] In one possible implementation, multimodal raw image data of the glassware on the production line is acquired, including high dynamic range transmitted images under a transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images acquired while the glassware is rotating. Step S100 further includes step S110, performing outgoing polarization processing on the ring structured light and incoming polarization processing on the camera acquiring the structured light images, wherein the direction of the outgoing polarization processing is perpendicular to the direction of the incoming polarization processing. Specifically, a first polarizer is installed in the outgoing light path of the ring structured light lamp, and the transmission axis of the polarizer is fixed in a first direction. A second polarizer is installed in front of the camera lens, and the transmission axis of the polarizer is set to a second direction perpendicular to the first direction. For example, the first direction is 0 degrees and the second direction is 90 degrees. With the above configuration, the light emitted by the ring structured light becomes linearly polarized light after passing through the first polarizer. When this linearly polarized light shines on the surface of the glassware, the specular reflection component maintains its original polarization direction, while the diffuse reflection component changes its polarization direction. The second polarizer in front of the camera filters out the light that is perpendicular to the second direction, that is, it filters out the specular reflection component that maintains its original polarization direction and only receives the diffuse reflection component that has changed its polarization direction.

[0029] Step S120: Acquire at least one pair of structured light images with different polarization angles, and the structured light image pair has temporal consistency. Specifically, a rotatable polarizer is set in front of the camera lens, or two cameras with fixed polarization angles are set. When using a rotatable polarizer, first rotate the polarizer to a first angle to acquire a first frame of structured light image, then rotate the polarizer to a second angle to acquire a second frame of structured light image, the first angle and the second angle differing by 90 degrees. When using two fixed cameras, the first camera lens is fixed at a first polarization angle, and the second camera lens is fixed at a second polarization angle. Both cameras are triggered simultaneously to acquire a pair of structured light images. The acquired pair of structured light images has temporal consistency, that is, the time difference between the acquisition times of the two frames is less than a preset time threshold, such as 1 millisecond, to ensure that the positions of the glassware in the two frames are basically the same.

[0030] Step S200: The multimodal raw image data is subjected to interference removal to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map.

[0031] Specifically, different interferences in the three original images are processed separately to obtain three clean primary feature maps. For the high dynamic range transmission image, inverse refraction transformation is performed using a pre-calibrated refractive distortion field to eliminate the light bending distortion caused by the curved glass surface, resulting in a geometrically correct transmission feature map. For the structured light image pair, polarization difference operation is performed. Utilizing the characteristic that the specular reflection component has the same intensity at different polarization angles while the diffuse reflection component has different intensities, specular reflection interference is canceled through difference to obtain a de-highlighted structured light image. Then, phase unwrapping operation is performed to convert the degree of fringe curvature into surface height information, resulting in a phase unfolded feature map. For the time-series continuous frame images, inter-frame registration is performed. Feature point matching and perspective transformation are used to spatially align all frames, resulting in a motion temporal feature map. The above processing eliminates refractive distortion, specular reflection, and inter-frame offset, providing clean feature input for subsequent processing.

[0032] In one possible implementation, interference removal is performed on the multimodal raw image data to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map. Step S200 further includes step S210, where the refractive distortion field is pre-calibrated based on the three-dimensional model of the glass vessel, and the high dynamic range transmission image is subjected to inverse refraction transformation using the refractive distortion field to obtain the transmission feature map. Specifically, a calibration process is performed before formal testing. A standard glass vessel with a known three-dimensional shape is placed at the testing station, and multiple markers with known spatial coordinates are affixed to the surface of the standard glass vessel. A high dynamic range transmission image of the standard glass vessel is acquired under a transmission light source, and the pixel coordinates of the markers in the image are extracted. Each marker's pixel coordinates are paired with its known real spatial coordinates. Based on all corresponding points, a mapping function is calculated using the least squares method. This mapping function maps each pixel position in the image to its corrected position in real space. This mapping function is stored in the form of a lookup table, which is the refractive distortion field. Each entry in the lookup table records the original pixel coordinates and their corresponding corrected coordinates.

[0033] During formal inspection, a high dynamic range transmission image of the glassware under inspection is acquired. An inverse refraction transformation is then performed on this image using a calibrated refractive distortion field: Each pixel of the input image is traversed, and according to the mapping relationship in the refractive distortion field, the grayscale value of that pixel is assigned to the corresponding corrected coordinate position in the output image. For pixel positions in the output image that are not assigned a value, a bilinear interpolation method is used to calculate the fill value based on the grayscale values ​​of adjacent pixels. After the inverse refraction transformation, a transmission feature map is output. This image eliminates the light bending distortion caused by the curved surface of the glass and is used for the detection of volumetric defects.

[0034] Step S220: Perform polarization difference operation on the structured light image pair to remove the specular reflection component, obtaining a de-highlighted structured light image. Specifically, acquire the pair of structured light images collected in step S120, denoted as the first polarization angle image as the first structured light image and the second polarization angle image as the second structured light image. Perform pixel-level difference operation on the two images: for each pixel position in the images, calculate the pixel value of the first structured light image minus the pixel value of the second structured light image to obtain the difference value. Since the specular reflection component has the same intensity in both images, it is canceled out after the difference operation. The diffuse reflection component has different intensities in the two images, and it is retained after the difference operation. After taking the absolute value of the difference result, normalize it to map the pixel value range to between 0 and 255, and output the de-highlighted structured light image.

[0035] Step S230: Perform phase unwrapping operation on the de-highlight structured light image to obtain the phase unfolded feature map. Specifically, the de-highlight structured light image is input into the phase unwrapping algorithm. First, the wrapped phase is extracted using the Fourier transform contour method: a two-dimensional Fourier transform is performed on the de-highlight structured light image to extract the fundamental frequency component in the frequency domain. An inverse Fourier transform is then performed on the fundamental frequency component to calculate the wrapped phase value of each pixel. This wrapped phase value is calculated using the arctangent function, and its value range is between -π and +π. Then, the spatial phase unwrapping algorithm is used to unfold the wrapped phase into a continuous phase: starting from the center pixel of the image, the flood fill method is used to compare the wrapped phase difference of adjacent pixels pixel by pixel. When the absolute value of the phase difference is greater than π, an integer multiple of 2π is added or subtracted from the phase value to make the phase difference of adjacent pixels fall within the range of -π to +π. After traversing the entire image, the absolute phase value of each pixel is output to form a phase unfolded feature map. The value of each pixel in the image represents the relative height or phase change at that position on the surface of the glassware, which is used to characterize the surface geometry. The phase value changes gradually in flat areas, while the phase value changes abruptly in concave or convex areas.

[0036] Step S240: Perform inter-frame registration on the continuous time-series images to obtain the motion temporal feature map. Specifically, acquire the continuous time-series images collected in step S100, denoting the first frame as the reference frame and subsequent frames as frames to be registered. For each frame to be registered, firstly, use the scale-invariant feature transform algorithm to extract feature points between the frame and the reference frame. Then, use the random sampling consensus algorithm to filter matching point pairs and eliminate mismatched points. Based on the filtered matching point pairs, calculate a perspective transformation matrix, which maps the pixel coordinates in the frame to be registered to the coordinate system of the reference frame. Transform the frame to be registered according to the perspective transformation matrix to generate the registered image. Repeat the above process for each frame to be registered to obtain a sequence of all registered images, denoted as the motion temporal feature map. Each frame in this sequence is spatially aligned with the previous frame, used to analyze the motion trajectory of each point on the surface of the glassware during rotation.

[0037] Step S300: Perform volumetric defect detection on the transmission feature map to obtain a first defect candidate thermal map.

[0038] Specifically, the location and confidence level of volumetric defects such as bubbles, stones, and air lines are detected in the transmission feature map. Due to the large differences in glass vessel wall thickness and the wide range of defect sizes, a multi-scale parallel detection strategy is adopted: the transmission feature map is simultaneously input into three convolutional branches with small, medium, and large receptive fields, each responding to defects of different sizes. Simultaneously, the weight coefficients of the three branches are dynamically adjusted based on the acquired prior information on wall thickness; the smaller the wall thickness, the higher the weight of the small receptive field branch to detect small near-surface defects, and the larger the wall thickness, the higher the weight of the large receptive field branch to detect deep, large defects. The defect confidence maps output by the three branches are then weighted and summed pixel-by-pixel, and after thresholding and noise filtering, a first defect candidate heatmap is output.

[0039] In one possible implementation, volumetric defect detection is performed on the transmission feature map to obtain a first defect candidate heatmap. Step S300 further includes step S310, whereby the transmission feature map is input into multiple pre-trained parallel convolutional branches. These parallel convolutional branches have different receptive field sizes, and each branch outputs a defect confidence map at a corresponding scale. Specifically, a multi-branch convolutional neural network is pre-constructed and trained. This network contains at least three parallel convolutional branches. The first branch uses a 3×3 convolutional kernel, with a receptive field size corresponding to small-sized defects; the second branch uses a 7×7 convolutional kernel, with a receptive field size corresponding to medium-sized defects; and the third branch uses a 15×15 convolutional kernel, with a receptive field size corresponding to large-sized defects. Each branch independently contains three convolutional layers and one output layer. The network is trained by collecting a large number of transmission feature map samples labeled with defect locations and categories, and using a cross-entropy loss function and a stochastic gradient descent optimization algorithm to train the network parameters until the loss function converges. The transmission feature map output in step S210 is simultaneously input into three branches. Each branch performs convolution operation, activation function processing, and output layer mapping on the image. Finally, each branch outputs a defect confidence map. The size of each defect confidence map is the same as the input transmission feature map, and the value of each pixel represents the confidence level that a defect exists at that location at that scale.

[0040] Step S320: Obtain the prior wall thickness information of the glassware, and calculate multiple weight coefficients based on the prior wall thickness information, where each weight coefficient corresponds to a convolution branch. Specifically, read the design wall thickness value of the currently inspected glassware from the production execution system, and record it as the prior wall thickness information, for example, this wall thickness value is 3 mm. Calculate three weight coefficients based on the prior wall thickness information, corresponding to the first branch, second branch, and third branch, respectively. Set the lower limit of the wall thickness threshold to 2 mm and the upper limit of the wall thickness threshold to 5 mm. When the wall thickness value is less than 2 mm, set the weight of the first branch to 0.6, the weight of the second branch to 0.3, and the weight of the third branch to 0.1. When the wall thickness value is greater than 5 mm, set the weight of the first branch to 0.1, the weight of the second branch to 0.3, and the weight of the third branch to 0.6. When the wall thickness value is between 2 mm and 5 mm, set the weight of the first branch to 0.2, the weight of the second branch to 0.6, and the weight of the third branch to 0.2.

[0041] Step S330: The defect confidence map and the weight coefficients are summed pixel-by-pixel to obtain a fused confidence map. Specifically, the three defect confidence maps output in step S310 are obtained and denoted as the first confidence map, the second confidence map, and the third confidence map, respectively. The three weight coefficients calculated in step S320 are obtained and denoted as the first weight, the second weight, and the third weight, respectively. For each pixel position in the image, the fused confidence value is calculated as: first confidence map value at that pixel × first weight + second confidence map value at that pixel × second weight + third confidence map value at that pixel × third weight. All pixel positions are traversed to generate a fused confidence map, where each pixel value is a weighted confidence score indicating the presence of a defect at that position.

[0042] Step S340: Threshold segmentation and noise filtering are performed on the fused confidence map to output the first defect candidate heatmap. Specifically, a confidence threshold is set, for example, to 0.5. Each pixel in the fused confidence map is traversed, and pixels with confidence values ​​less than or equal to the threshold are set to 0, while pixels with confidence values ​​greater than the threshold retain their original values. Then, a median filtering method is used: for each pixel in the image, a 3×3 pixel neighborhood window is taken centered on it, and the median of all pixels within the window is used as the new value of that pixel. Isolated noise points are removed by traversing the entire image. After threshold segmentation and noise filtering, the first defect candidate heatmap is output, where the value of each pixel in the image represents the confidence level of the presence of a volumetric defect at that location, with a value range from 0 to 1.

[0043] Step S400: Use the phase unfolding feature map to distinguish between internal and external defects in the first defect candidate heat map to obtain a second enhanced defect distribution map.

[0044] Specifically, the rejection criteria differ between candidate defects classified as internal volumetric defects and surface geometric deformation defects. First, high-confidence pixels in the first defect candidate heatmap are mapped onto the 3D surface mesh of the glassware to obtain the true spatial coordinates of the candidate defects. Simultaneously, the phase gradient magnitude at each location in the phase unfolded feature map is calculated. This magnitude reflects the severity of surface unevenness, and continuous regions with magnitudes exceeding a threshold are extracted as surface geometric distortion regions. Then, it is determined whether the coordinates of each candidate defect fall within a surface geometric distortion region. If they do, they are marked as surface deformation defects; otherwise, they are marked as internal volumetric defects. Finally, the type labels are overlaid back onto the heatmap to generate a second enhanced defect distribution map.

[0045] In one possible implementation, the phase unfolding feature map is used to distinguish between internal and external defects in the first defect candidate heatmap to obtain a second enhanced defect distribution map. Step S400 further includes step S410, mapping pixels with a confidence level greater than a first threshold in the first defect candidate heatmap onto the three-dimensional surface mesh of the glassware to obtain a set of candidate defect three-dimensional spatial locations. Specifically, a three-dimensional surface mesh model of the glassware currently being inspected is pre-obtained, which is either imported from a computer-aided design file or pre-scanned using a 3D scanner. A first threshold is set, for example, to 0.5. Each pixel in the first defect candidate heatmap is traversed, and pixels with a confidence level greater than 0.5 are selected. For each selected pixel, the ray equation passing through the pixel from the camera's optical center is calculated according to the camera calibration parameters, and the intersection point of the ray and the three-dimensional surface mesh is calculated. The spatial coordinates of this intersection point are the three-dimensional coordinates of the defect point on the actual surface of the glassware. The three-dimensional coordinates of all selected pixels are collected to form a set of candidate defect three-dimensional spatial locations, where each element contains the three-dimensional coordinates of the defect and its original confidence level.

[0046] Step S420: Calculate the phase gradient magnitude at each position in the phase unfolded feature map, and extract regions with magnitudes greater than a second threshold as surface geometric distortion regions. Specifically, obtain the phase unfolded feature map output in step S230. For each pixel position in the image, calculate the phase difference in the horizontal direction of the pixel, i.e., the phase value of the pixel to the right of the pixel minus the phase value of the pixel to the left of the pixel, divided by 2, and record it as the horizontal gradient. Calculate the phase difference in the vertical direction of the pixel, i.e., the phase value of the pixel below the pixel minus the phase value of the pixel above the pixel, divided by 2, and record it as the vertical gradient. Calculate the sum of the squares of the horizontal and vertical gradients, and then take the square root to obtain the phase gradient magnitude of the pixel. Traverse the entire image to obtain the phase gradient magnitude of each pixel. Set a second threshold, for example, 0.1 radians per pixel. Mark pixels with phase gradient magnitudes greater than 0.1 as distorted pixels, and connect all adjacent distorted pixels into a continuous region; each continuous region is called a surface geometric distortion region.

[0047] Step S430: Traverse each candidate defect in the set of candidate defect three-dimensional spatial locations and determine whether it falls within the surface geometric distortion region. If it does, mark it as a surface deformation defect; otherwise, mark it as an internal volumetric defect, thus obtaining a set of marked candidate defects. Specifically, traverse each defect in the set of candidate defect three-dimensional spatial locations generated in step S410. For each defect, compare its three-dimensional coordinates with the boundaries of each surface geometric distortion region extracted in step S420. Determine whether the three-dimensional coordinates of the defect are located inside the boundary of any surface geometric distortion region. If the three-dimensional coordinates of the defect fall within any surface geometric distortion region, mark the defect as a surface deformation defect. If the three-dimensional coordinates of the defect do not fall within any surface geometric distortion region, mark the defect as an internal volumetric defect. Combine the three-dimensional coordinates, original confidence level, and marking result of each defect as an entry to form a set of marked candidate defects.

[0048] Step S440: The marked candidate defect set is superimposed on the first defect candidate heatmap to generate the second enhanced defect distribution map. Specifically, the first defect candidate heatmap output in step S340 is obtained. For each defect in the marked candidate defect set generated in step S430, its marking result is appended to the corresponding pixel position in the first defect candidate heatmap. Specifically, a label layer with the same size as the first defect candidate heatmap is created. Each pixel position in this layer stores a type label value: 0 indicates no defect, 1 indicates an internal volumetric defect, and 2 indicates a surface deformation defect. This label layer is superimposed on the first defect candidate heatmap to obtain a new image. Each pixel in this image contains a confidence value and a type label value, which is the second enhanced defect distribution map. Each candidate defect point in this image is accompanied by a type label of either an internal volumetric defect or a surface deformation defect.

[0049] Step S500: Use the motion time sequence feature map to remove false defects from the second enhanced defect distribution map to obtain a third time consistency defect map.

[0050] Specifically, pseudo-defects caused by dust, oil stains, or momentary reflections are eliminated. These pseudo-defects are difficult to distinguish from real defects in a single frame, but they do not consistently appear in the same physical position during the rotation of the glassware. First, the optical flow field is extracted from the motion temporal feature map to establish the continuous motion trajectory of each point on the glassware surface. Then, each candidate defect point is mapped onto this trajectory, and its position change sequence and grayscale change sequence in each frame are obtained. The position stability score and grayscale stability score are calculated respectively. Real defects maintain stable position and grayscale during rotation, and both scores are high; pseudo-defects have at least one low score. A defect is retained only when both scores reach a preset threshold; otherwise, it is eliminated, resulting in a third temporal consistency defect map.

[0051] In one possible implementation, the motion temporal feature map is used to remove false defects from the second enhanced defect distribution map to obtain a third temporally consistent defect map. Step S500 further includes step S510, extracting the optical flow field between frames from the motion temporal feature map to establish the continuous trajectory of each point on the surface of the glassware during rotational motion. Specifically, the motion temporal feature map output in step S240 is obtained. This map is a sequence of registered consecutive frame images, with a total number of frames denoted as N. For two adjacent frames, the Lucas-Cannard optical flow method is used to calculate the optical flow field: the image is divided into multiple small windows, assuming consistent pixel motion within each window, and the optical flow equation is solved using the least squares method to obtain the horizontal and vertical displacement of each pixel from the first frame to the second frame. The optical flow fields are calculated for the first frame and the second frame, the second frame and the third frame, up to the (N-1)th frame and the Nth frame, respectively, resulting in N-1 optical flow fields. For a point on the surface of a glass vessel, its pixel position in the first frame is known. The position of the point in the second frame is obtained based on the optical flow fields of the first and second frames. Then, the position of the point in the third frame is obtained based on the optical flow fields of the second and third frames, and so on, to obtain the continuous trajectory of the point in all N frames. This trajectory is a position sequence, and each position contains the frame number, horizontal coordinate, and vertical coordinate.

[0052] Step S520: Map each candidate defect point in the second enhanced defect distribution map onto the continuous trajectory to obtain the position change sequence and grayscale change sequence of the candidate defects in the continuous time series frames. Specifically, obtain the second enhanced defect distribution map output in step S440, in which each candidate defect point has a confidence value and a type label value. For each candidate defect point, take its pixel position in the first frame as the starting point, and find the corresponding position of the starting point in each subsequent frame according to the continuous trajectory established in step S510. If there is no pre-calculated trajectory at the starting point, the nearest neighbor interpolation method is used to find the trajectory point closest to the starting point as its mapping position. For each candidate defect point, record its position coordinates in each frame to form a position change sequence. At the same time, for each frame, extract the pixel grayscale value at the corresponding position in the frame to form a grayscale change sequence.

[0053] Step S530: Calculate the position stability score based on the position change sequence; calculate the grayscale stability score based on the grayscale change sequence. Specifically, for each candidate defect point, obtain its position change sequence, which contains N coordinate values. Calculate the centroid coordinates of this sequence, i.e., the average of the horizontal and vertical coordinates of all frames. For each frame, calculate the Euclidean distance between the frame's position and the centroid, calculate the sum of the squares of the Euclidean distances of all frames, and then divide by the number of frames N to obtain the position variance. Define the position stability score as 1 divided by the position variance and a preset constant, or as the reciprocal of the position variance and normalized. For example, position stability score = 1 / (position variance + 0.01). The smaller the position variance, the higher the position stability score, indicating that the defect point's position changes less during rotation.

[0054] For each candidate defect point, its grayscale change sequence is obtained, containing N grayscale values. The average grayscale value of this sequence is calculated. For each frame, the difference between the grayscale value of that frame and the average grayscale value is calculated, and the square of the difference is taken. The sum of the squared differences of all frames is calculated and then divided by the number of frames N to obtain the grayscale variance. The grayscale stability score is defined as 1 divided by the sum of the grayscale variance and a preset constant, or as the reciprocal of the grayscale variance after normalization. The smaller the grayscale variance, the higher the grayscale stability score, indicating that the grayscale change of the defect point is small during rotation.

[0055] Step S540: When the position stability score is greater than or equal to a preset position score threshold and the grayscale stability score is greater than or equal to a preset grayscale score threshold, the corresponding candidate defect is determined to be a time-consistent defect; otherwise, it is determined to be a time-inconsistent defect. Specifically, a preset position score threshold is set, for example, to 0.7. A preset grayscale score threshold is set, for example, to 0.6. For each candidate defect point, its position stability score is compared with 0.7, and its grayscale stability score is compared with 0.6. If the position stability score is greater than or equal to 0.7 and the grayscale stability score is greater than or equal to 0.6, the candidate defect is determined to be a time-consistent defect. If the above conditions are not met, i.e., the position stability score is less than 0.7 or the grayscale stability score is less than 0.6, the candidate defect is determined to be a time-inconsistent defect.

[0056] Step S550: The time consistency defect is retained, and the time inconsistency defect is removed from the second enhanced defect distribution map to obtain the third time consistency defect map. Specifically, the determination result of step S540 is obtained. For candidate defect points determined to be time consistency defects, their confidence value and type label value are retained in the second enhanced defect distribution map. For candidate defect points determined to be time inconsistency defects, their confidence value is set to 0 and their type label value is set to no defect in the second enhanced defect distribution map. After traversing all candidate defect points, a new image is generated, which only retains the information of time consistency defects, i.e., the third time consistency defect map. This image only retains defect points with stable positions and stable grayscale in the time series.

[0057] Step S600: Use production prior features to verify the defects in the third time consistency defect map and output the defect detection results.

[0058] Specifically, the visual inspection results are combined with non-visual information from the production process to finally confirm defects in the third-time consistency defect map, and output a complete inspection result including defect type, location, and size. Because visual features are inherently ambiguous—the same visual morphology may be a defect or normal texture under different process conditions, and the visual appearance of the same defect varies greatly under different glass colors—relying solely on visual perception can easily lead to misjudgment. Therefore, prior production features are introduced as auxiliary criteria. The mold information and process parameters of the currently inspected glassware are obtained. The discrete mold information is converted into a continuous vector through embedding mapping, and the process parameters are normalized. Simultaneously, global feature extraction is performed on the third-time consistency defect map to obtain vectors that characterize the visual features of the defects. The visual feature vector, mold embedding vector, and process parameter vector are concatenated and fused before being input into a pre-trained confirmation detection network, which simultaneously outputs the type, location coordinates, and size parameters of each defect. Furthermore, to improve the interpretability and usability of the inspection results, source tracing is performed using the location coordinates and type of defects. The location coordinates determine the area type where the defect is located, and a pre-defined area-defect type mapping table is consulted to determine the validity of the combination. If valid, the corresponding target process is further matched, and the production station information for that process is retrieved from the production execution system and added to the inspection results. If the combination is invalid, a correction mechanism is triggered, initiating a manual review process. Through the above processing, a comprehensive inspection result containing defect type, location, size, and production source tracing information is finally output, achieving a closed loop from defect identification to process tracing.

[0059] In one possible implementation, the third time-consistent defect map is verified using production prior features, and the defect detection result is output. Step S600 further includes step S610, acquiring the production prior features of the glassware, including mold information and process parameters. Specifically, the production prior features refer to non-image information related to the currently inspected glassware, read from the production execution system or programmable logic controller, including mold information and process parameters. The mold information includes the mold number and cavity number used in the current production, for example, the mold number is M-07 and the cavity number is cavity 2. The process parameters include the droplet temperature, stamping pressure, mold temperature, and cooling time, for example, the droplet temperature is 1150 degrees Celsius, the stamping pressure is 0.6 MPa, the mold temperature is 450 degrees Celsius, and the cooling time is 3 seconds.

[0060] Step S620: Map the mold information to a mold embedding vector and normalize the process parameters to a process parameter vector. Specifically, for the mold number in the mold information, a mold embedding matrix is ​​pre-constructed. The number of rows in this matrix is ​​the total number of molds, and the number of columns is a preset embedding dimension, for example, 64. The parameters of the mold embedding matrix are learned through a neural network training process. The specific training method is as follows: collect a large number of glassware defect samples labeled with mold numbers, use the mold number as a supervision signal, and train the embedding matrix using contrastive loss or cross-entropy loss, so that samples of the same mold are close to each other in the embedding space, and samples of different molds are far apart. Obtain the row vector corresponding to the current mold number. This row vector is the mold embedding vector, with a length of 64, and is used to convert the discrete mold identifier into a representation in a continuous vector space.

[0061] For each process parameter, there are different dimensions and value ranges. Each process parameter is normalized: for droplet temperature, a preset value range is used, for example, 900°C to 1300°C; the normalized value is the actual value minus 900, then divided by 400. For stamping pressure, a preset value range is used, for example, 0.2 MPa to 1.0 MPa; the normalized value is the actual value minus 0.2, then divided by 0.8. For mold temperature, a preset value range is used, for example, 300°C to 600°C; the normalized value is the actual value minus 300, then divided by 300. For cooling time, a preset value range is used, for example, 1 second to 5 seconds; the normalized value is the actual value minus 1, then divided by 4. The normalized process parameter values ​​are arranged in a fixed order to form a process parameter vector, for example, a one-dimensional vector of length 4.

[0062] Step S630: Perform global feature extraction on the third time consistency defect map to obtain a visual feature vector. Specifically, obtain the third time consistency defect map output in step S550, and input the image into a pre-trained feature extraction backbone network. For example, the backbone network uses a ResNet-50 network pre-trained on the ImageNet dataset, with its last fully connected layer removed. After adjusting the third time consistency defect map to a size of 224×224 pixels, input it into the network. The network outputs a fixed-length feature vector with a dimension of 2048, which is the visual feature vector used to represent the visual features of the defect.

[0063] Step S640: The visual feature vector, the mold embedding vector, and the process parameter vector are fused to obtain a fused feature vector. Specifically, the visual feature vector output in step S630 is obtained, with a dimension of 2048. The mold embedding vector output in step S620 is obtained, with a dimension of 64. The process parameter vector output in step S620 is obtained, with a dimension of 4. The three vectors are concatenated end-to-end along their feature dimensions to form a fused feature vector of length 2116.

[0064] Step S650: The fused feature vector is input into a pre-trained confirmation detection network, which outputs the defect type, location coordinates, and size parameters of each defect as the defect detection result. Specifically, a confirmation detection network is pre-constructed and trained, consisting of three fully connected branches. The first branch is a classification branch, containing three fully connected layers with an input dimension of 2^116 and an output dimension equal to the number of defect categories. For example, the defect categories include five types: bubbles, stones, dents, cracks, and normal defects. The output layer uses a softmax activation function to output the probability distribution of each defect category. The second branch is a localization branch, containing three fully connected layers with an input dimension of 2^116 and an output dimension of 4, representing the x-coordinate of the top-left corner, y-coordinate of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner of the defect bounding box. The output layer uses a linear activation function. The third branch is a size branch, containing three fully connected layers with an input dimension of 2^116 and an output dimension of 2, representing the length and width of the defect. The output layer uses a linear activation function, and the absolute value of the output value ensures that the size is positive. The network is trained by collecting a large number of fused feature vector samples with labeled defect types, bounding boxes, and sizes. The weighted sum of classification loss, localization loss, and size loss is used as the total loss function. The backpropagation algorithm is used to optimize the network parameters until the total loss converges.

[0065] The fused feature vector output from step S640 is simultaneously input into the three branches mentioned above. The classification branch outputs the defect type, and the category corresponding to the highest probability is taken as the final type. The localization branch outputs the coordinates of the defect bounding box. The size branch outputs the length and width of the defect. The defect type, location coordinates, and size parameters are combined to output the defect detection result.

[0066] In one possible implementation, step S600 further includes step S660, determining the region type to which the defect belongs based on the location coordinates. Specifically, the location coordinates from the defect detection result output in step S650 are obtained, i.e., the four coordinate values ​​of the defect bounding box. The center point coordinates of the bounding box are calculated, i.e., the sum of the upper left and lower right horizontal coordinates divided by 2, and the sum of the upper left and lower right vertical coordinates divided by 2. Based on these center point coordinates, they are mapped onto the three-dimensional surface mesh of the glassware to obtain the three-dimensional coordinates of the defect in real space. Based on the position of these three-dimensional coordinates on the glassware, it is divided into one of the preset region types. For example, the region types include the mouth region, the body region, and the bottom region. If the center point of the defect is less than 20 mm from the mouth of the bottle, it is determined to be the mouth region; if it is less than 15 mm from the bottom of the bottle, it is determined to be the bottom region; otherwise, it is determined to be the body region.

[0067] Step S670: Query the preset region-defect type mapping table to determine if the combination of the region type and the defect type is valid. If valid, match the corresponding target process from the preset defect-process mapping rules based on the region type and the defect type. Specifically, a region-defect type mapping table is pre-constructed, which records the possible defect types corresponding to each region type. For example, possible defect types corresponding to the mouth region include cracks and notches; possible defect types corresponding to the bottle body region include bubbles, stones, dents, and material marks; possible defect types corresponding to the bottom region include bubbles, uneven bottom thickness, and bottom corner cracks. Obtain the region type determined in step S660 and the defect type output in step S650, and query the mapping table to see if the combination exists in the list of allowed defect types for that region. If it exists, the combination is determined to be valid. If valid, further match the corresponding target process from the preset defect-process mapping rules based on the region type and the defect type. For example, the defect-process mapping rule is: cracks in the mouth region correspond to the mouth forming process, dents in the bottle body region correspond to the shaping process, and bubbles in the bottom region correspond to the initial forming process.

[0068] Step S680: Based on the target process, retrieve the corresponding production station information from the production execution system. Specifically, obtain the target process matched in step S670, and send a query request to the production execution system, with the target process as the request parameter. The production execution system returns the production station information corresponding to the process based on the correspondence between processes and stations. The production station information includes station number, equipment number, operator, and other information. For example, if the target process is the mouth forming process, the returned station information would be station number 03, equipment number F-012, and operator Zhang San.

[0069] Step S690: The production station information is added as traceability information to the defect detection result. Specifically, the production station information retrieved in step S680 is obtained, and this information is added as key-value pairs to the defect detection result output in step S650 to form an expanded detection result. For example, the original defect detection result is defect type crack, location coordinates, and size parameters; after adding traceability information, it becomes defect type crack, location coordinates, size parameters, station number 03, and equipment number F-012.

[0070] In one possible implementation, step S600 further includes step S6100: if the combination of the region type and the defect type is invalid, a correction mechanism is triggered; according to the correction mechanism, a manual review process is initiated. Specifically, the judgment result of step S670 is obtained. If the combination is invalid, i.e., the region type determined in step S660 and the defect type output in step S650 do not exist in the mapping table, the correction mechanism is triggered, the defect is marked as requiring manual review, and a prompt message is generated. This message includes an image screenshot of the defect, the detected defect type, and the actual region type. The prompt message is sent to the quality inspector's terminal to initiate the manual review process. After review, the quality inspector inputs the review result into the system as the final judgment result for the defect.

[0071] This application embodiment acquires high dynamic range images under transmitted light, stripe images under ring structured light, and continuous frame images under rotation. It corrects refraction distortion in the transmitted images to obtain a transmission feature map, performs polarization difference and phase unwrapping on the structured light image pairs to obtain a phase unfolding feature map, and registers the continuous frames to obtain a motion temporal feature map. The transmission feature map is used to detect candidate regions for volumetric defects, and the surface morphology information from the phase unfolding feature map is combined to distinguish between internal defects and surface deformation. Furthermore, optical flow tracking of the motion temporal feature map is used to eliminate false defects such as dust and reflections. Visual features are fused with prior production features such as mold information and process parameters. The verification detection network outputs defect type, location, size, and traceability information. This approach solves the technical problem of low detection accuracy in existing glassware defect detection due to complex imaging interference and diverse defect morphologies, achieving improved detection accuracy and reliability.

[0072] In the above text, refer to Figure 1 A computer vision-based method for detecting defects in glassware production, according to embodiments of the present invention, is described in detail. Next, reference will be made to... Figure 2 A computer vision-based defect detection system for glassware production according to an embodiment of the present invention is described.

[0073] The computer vision-based glassware production defect detection system according to embodiments of the present invention addresses the technical problem of low detection accuracy in existing glassware defect detection methods due to complex imaging interference and diverse defect morphologies, thereby improving detection accuracy and reliability. The computer vision-based glassware production defect detection system includes: a multimodal raw image data acquisition module 10, an interference removal module 20, a volumetric defect detection module 30, an internal / external defect differentiation module 40, a false defect removal module 50, and a defect confirmation module 60.

[0074] The multimodal raw image data acquisition module 10 is used to acquire multimodal raw image data of glassware on the production line, including high dynamic range transmission images under a transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images acquired when the glassware under test is rotating; the interference removal module 20 is used to remove interference from the multimodal raw image data to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map; the volumetric defect detection module 30 is used to perform volumetric defect detection on the transmission feature map to obtain a first defect candidate heatmap; the internal and external defect differentiation module 40 is used to differentiate between internal and external defects in the first defect candidate heatmap using the phase unfolding feature map to obtain a second enhanced defect distribution map; the pseudo-defect removal module 50 is used to remove pseudo-defects from the second enhanced defect distribution map using the motion temporal feature map to obtain a third time-consistent defect map; and the defect verification module 60 is used to verify the third time-consistent defect map using production prior features and output the defect detection result.

[0075] The detailed description of the specific configuration of the multimodal raw image data acquisition module 10 is explained as follows: As mentioned above, it acquires multimodal raw image data of glassware on the production line, including high dynamic range transmission images under a transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images acquired when the glassware under test is rotating. The multimodal raw image data acquisition module 10 may further include: a polarization processing unit for performing outgoing polarization processing on the ring structured light and incoming polarization processing on the camera acquiring the structured light images, wherein the direction of the outgoing polarization processing is perpendicular to the direction of the incoming polarization processing; and a structured light image acquisition unit for acquiring at least one pair of structured light images with different polarization angles, wherein the structured light image pairs have temporal consistency.

[0076] The detailed description of the specific configuration of the interference removal module 20 is explained as follows: As mentioned above, interference removal is performed on the multimodal original image data to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map. The interference removal module 20 may further include: an inverse refraction transformation unit for pre-calibrating the refractive distortion field according to the three-dimensional model of the glassware, and using the refractive distortion field to perform an inverse refraction transformation on the high dynamic range transmission image to obtain the transmission feature map; a polarization difference operation unit for performing polarization difference operation on the structured light image pair to strip the specular reflection component and obtain a de-highlight structured light image; a phase unwrapping operation unit for performing phase unwrapping operation on the de-highlight structured light image to obtain the phase unfolding feature map; and an inter-frame registration unit for performing inter-frame registration on the time-series continuous frame images to obtain the motion temporal feature map.

[0077] The detailed description of the specific configuration of the volumetric defect detection module 30 is explained as follows: As mentioned above, the volumetric defect detection module 30 performs volumetric defect detection on the transmission feature map to obtain a first defect candidate heatmap. The volumetric defect detection module 30 may further include: a defect detection unit for inputting the transmission feature map into multiple pre-trained parallel convolution branches, each parallel convolution branch having a different receptive field size, and each branch outputting a defect confidence map at a corresponding scale; a weight coefficient calculation unit for obtaining prior information on the wall thickness of the glassware, and calculating multiple weight coefficients based on the prior information on the wall thickness, wherein each weight coefficient corresponds to a convolution branch; a fusion unit for performing pixel-by-pixel weighted summation of the defect confidence map and the weight coefficients to obtain a fused confidence map; and a first defect candidate heatmap output unit for performing threshold segmentation and noise filtering on the fused confidence map to output the first defect candidate heatmap.

[0078] The detailed description of the specific configuration of the internal and external defect differentiation module 40 is explained as follows: As mentioned above, the phase unfolding feature map is used to differentiate between internal and external defects in the first defect candidate heatmap to obtain a second enhanced defect distribution map. The internal and external defect differentiation module 40 may further include: a pixel mapping unit for mapping pixels with a confidence level greater than a first threshold in the first defect candidate heatmap to the three-dimensional surface mesh of the glassware to obtain a set of three-dimensional spatial positions of candidate defects; a phase gradient magnitude calculation unit for calculating the phase gradient magnitude at each position in the phase unfolding feature map and extracting the region with a magnitude greater than a second threshold as the surface geometric distortion region; a defect marking unit for traversing each candidate defect in the set of three-dimensional spatial positions of candidate defects and determining whether it falls into the surface geometric distortion region: if it falls into the region, it is marked as a surface deformation defect; if it does not fall into the region, it is marked as an internal volumetric defect to obtain a set of marked candidate defects; and a second enhanced defect distribution map generation unit for superimposing the set of marked candidate defects onto the first defect candidate heatmap to generate the second enhanced defect distribution map.

[0079] The specific configuration of the pseudo-defect removal module 50 is described in detail below: As mentioned above, the pseudo-defect removal module 50 is used to remove pseudo-defects from the second enhanced defect distribution map using the motion temporal feature map to obtain a third time-consistent defect map. The pseudo-defect removal module 50 may further include: a continuous trajectory establishment unit for extracting the optical flow field between frames from the motion temporal feature map and establishing a continuous trajectory of each point on the surface of the glassware during rotational motion; a change sequence acquisition unit for mapping each candidate defect point in the second enhanced defect distribution map onto the continuous trajectory and obtaining the position change sequence and grayscale change sequence of the candidate defect in the continuous time sequence frames; a stability score calculation unit for calculating a position stability score based on the position change sequence and a grayscale stability score based on the grayscale change sequence; a time consistency determination unit for determining the corresponding candidate defect as a time-consistent defect when the position stability score is greater than or equal to a preset position score threshold and the grayscale stability score is greater than or equal to a preset grayscale score threshold, otherwise determining it as a time-inconsistent defect; and a third time-consistent defect map generation unit for retaining the time-consistent defect and removing the time-inconsistent defect from the second enhanced defect distribution map to obtain a third time-consistent defect map.

[0080] The specific configuration of the defect verification module 60 is described in detail below: As mentioned above, the defect verification module 60 uses production prior features to verify the third time-consistent defect map and outputs the defect detection result. The defect verification module 60 may further include: a production prior feature acquisition unit for acquiring the production prior features of the glassware, including mold information and process parameters; a vector construction unit for mapping the mold information into a mold embedding vector and normalizing the process parameters into a process parameter vector; a global feature extraction unit for performing global feature extraction on the third time-consistent defect map to obtain a visual feature vector; a feature fusion unit for fusing the visual feature vector, the mold embedding vector, and the process parameter vector to obtain a fused feature vector; and a verification detection unit for inputting the fused feature vector into a pre-trained verification detection network and outputting the defect type, location coordinates, and size parameters of each defect as the defect detection result.

[0081] The defect confirmation module 60 may further include: a region type determination unit for determining the region type to which the defect belongs based on the location coordinates; a target process matching unit for querying a preset region-defect type mapping table to determine whether the combination of the region type and the defect type is valid, and if valid, matching the corresponding target process from the preset defect-process mapping rules based on the region type and the defect type; a production station information retrieval unit for retrieving the corresponding production station information from the production execution system based on the target process; and a traceability information addition unit for adding the production station information as traceability information to the defect detection result.

[0082] The defect verification module 60 may further include: a correction mechanism triggering unit for triggering a correction mechanism if the combination of the region type and the defect type is invalid; and a manual review process initiation unit for initiating a manual review process according to the correction mechanism.

[0083] The computer vision-based glassware production defect detection system provided in this embodiment of the invention can execute the computer vision-based glassware production defect detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0084] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0085] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for detecting defects in glassware production based on computer vision, characterized in that, The method includes: Collect multimodal raw image data of glassware on the production line, including high dynamic range transmission images under transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images of the glassware under test in a rotating state. The original multimodal image data is subjected to interference removal to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map; Volumetric defect detection is performed on the transmission feature map to obtain a first defect candidate thermal map; The phase unfolding feature map is used to distinguish between internal and external defects in the first defect candidate heatmap to obtain a second enhanced defect distribution map. The motion time sequence feature map is used to remove false defects from the second enhanced defect distribution map to obtain a third time consistency defect map. The third time-consistent defect map is verified using production prior features, and the defect detection results are output. The multimodal raw image data undergoes interference removal to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map, comprising: The refractive distortion field is pre-calibrated based on the three-dimensional model of the glassware, and the high dynamic range transmission image is subjected to inverse refraction transformation using the refractive distortion field to obtain the transmission feature map. Polarization difference operation is performed on the structured light image pair to remove the specular reflection component, resulting in a de-highlighted structured light image; Perform phase unwrapping operation on the de-highlight structured light image to obtain the phase unfolded feature map; Inter-frame registration is performed on the continuous frame images of the time series to obtain the motion temporal feature map; Specifically, the phase unfolding feature map is used to distinguish between internal and external defects in the first defect candidate heatmap to obtain a second enhanced defect distribution map, including: Pixels with a confidence level greater than a first threshold in the first defect candidate heatmap are mapped onto the three-dimensional surface mesh of the glassware to obtain a set of three-dimensional spatial locations of candidate defects. Calculate the phase gradient magnitude at each position in the phase unfolded feature map, and extract the region with a magnitude greater than the second threshold as the surface geometric distortion region; Traverse each candidate defect in the set of candidate defect three-dimensional spatial locations and determine whether it falls within the surface geometric distortion region: if it falls within the region, mark it as a surface deformation defect; if it does not fall within the region, mark it as an internal volumetric defect, thus obtaining a set of marked candidate defects. The set of marked candidate defects is superimposed on the first candidate defect heatmap to generate the second enhanced defect distribution map; Specifically, the motion time-series feature map is used to remove false defects from the second enhanced defect distribution map to obtain a third time-consistency defect map, including: The optical flow field between each frame is extracted from the motion temporal feature map to establish the continuous trajectory of each point on the surface of the glassware during rotational motion. Each candidate defect point in the second enhanced defect distribution map is mapped onto the continuous trajectory to obtain the position change sequence and grayscale change sequence of the candidate defect in the continuous time series frame; Calculate the position stability score based on the position change sequence; calculate the grayscale stability score based on the grayscale change sequence; When the position stability score is greater than or equal to the preset position score threshold and the grayscale stability score is greater than or equal to the preset grayscale score threshold, the corresponding candidate defect is determined to be a time-consistent defect; otherwise, it is determined to be a time-inconsistent defect. The time consistency defect is retained, and the time inconsistency defect is removed from the second enhancement defect distribution map to obtain the third time consistency defect map.

2. The method for detecting defects in glassware production based on computer vision as described in claim 1, characterized in that, Acquire multimodal raw image data of glassware on the production line, including high dynamic range transmitted images under transmitted light, structured light images under ring structured light illumination, and time-series continuous frame images acquired while the glassware is rotating, including: The ring structured light is subjected to outgoing polarization processing, and the camera that acquires the structured light image is subjected to incident polarization processing, wherein the direction of the outgoing polarization processing is perpendicular to the direction of the incident polarization processing. At least one pair of structured light images with different polarization angles are acquired, and the structured light image pairs have temporal consistency.

3. The method for detecting defects in glassware production based on computer vision as described in claim 1, characterized in that, Volumetric defect detection is performed on the transmission feature map to obtain a first defect candidate thermal map, including: The transmission feature map is input into multiple pre-trained parallel convolutional branches, each with a different receptive field size. Each branch outputs a defect confidence map at a corresponding scale. Obtain prior information on the wall thickness of the glassware, and calculate multiple weight coefficients based on the prior information on the wall thickness, wherein each weight coefficient corresponds to a convolution branch; The defect confidence map and the weighting coefficients are summed pixel-by-pixel to obtain the fused confidence map. Threshold segmentation and noise filtering are performed on the fused confidence map to output the first defect candidate heatmap.

4. The method for detecting defects in glassware production based on computer vision as described in claim 1, characterized in that, The third time-consistency defect map is verified using prior production features, and the defect detection results are output, including: Obtain the prior production characteristics of the glassware, including mold information and process parameters; The mold information is mapped into a mold embedding vector, and the process parameters are normalized into a process parameter vector. Global feature extraction is performed on the third time consistency defect map to obtain a visual feature vector; The visual feature vector, the mold embedding vector, and the process parameter vector are fused to obtain a fused feature vector. The fused feature vector is input into a pre-trained verification detection network, which outputs the defect type, location coordinates, and size parameters of each defect as the defect detection result.

5. The method for detecting defects in glassware production based on computer vision as described in claim 4, characterized in that, Also includes: The region type to which the defect belongs is determined based on the location coordinates; Query the preset region-defect type mapping table to determine whether the combination of the region type and the defect type is valid. If valid, match the corresponding target process from the preset defect-process mapping rules according to the region type and the defect type. Based on the target process, retrieve the corresponding production station information from the production execution system; The production station information is added as traceability information to the defect detection results.

6. The method for detecting defects in glassware production based on computer vision as described in claim 5, characterized in that, Also includes: If the combination of the region type and the defect type is invalid, a correction mechanism is triggered. Based on the aforementioned correction mechanism, a manual review process is initiated.

7. A computer vision-based defect detection system for glassware production, characterized in that, The system is used to implement the computer vision-based glassware production defect detection method according to any one of claims 1-6, the system comprising: The multimodal raw image data acquisition module is used to acquire multimodal raw image data of glassware on the production line, including high dynamic range transmission images under transmitted light source, structured light images under ring structured light illumination, and time-series continuous frame images acquired when the glassware under test is rotating. The interference removal module is used to remove interference from the multimodal raw image data to obtain a primary clean feature map, including a transmission feature map, a phase unfolding feature map, and a motion temporal feature map. A volumetric defect detection module is used to perform volumetric defect detection on the transmission feature map to obtain a first defect candidate heat map; The internal and external defect differentiation module is used to differentiate between internal and external defects in the first defect candidate heatmap using the phase unfolding feature map, and to obtain a second enhanced defect distribution map. The pseudo-defect removal module is used to remove pseudo-defects from the second enhanced defect distribution map using the motion time sequence feature map to obtain a third time consistency defect map. The defect verification module is used to verify the defects in the third time-consistent defect map using production prior features and output the defect detection results.

Images