A Smart Detection Method and System for Defects in Medical Polymer Consumables Based on Multi-Feature Fusion

Abstract

This invention discloses an intelligent detection method and system for defects in medical polymer consumables based on multi-feature fusion, belonging to the field of medical consumables detection technology. This intelligent detection method for defects in medical polymer consumables based on multi-feature fusion acquires two-dimensional surface images, infrared transmission images, and three-dimensional point cloud data of the consumables. It extracts surface texture features, calculates the attenuation rate and gradient of transmitted light intensity to construct dual-channel features for extracting transmission physical features, projects the point cloud into a depth map, and constructs a multi-scale curvature map to extract geometric contour features. After aligning the three types of feature spaces, they are weighted and fused by a channel and spatial attention module. The detection head outputs the defect category and location. This invention achieves intelligent detection of defects in medical polymer consumables by fusing three modal data—two-dimensional surface, infrared transmission, and three-dimensional point cloud—combining convolutional neural networks and attention mechanisms. It adapts to the needs of mass production in assembly lines, improves detection efficiency, and provides reliable support for the quality control of consumables.

Description

Technical Field

[0001] This invention relates to the field of medical consumables testing technology, specifically to an intelligent detection method and system for defects in medical polymer consumables based on multi-feature fusion. Background Technology

[0002] Medical polymer consumables, with their lightweight, biocompatibility, and ease of processing, have been widely used in various medical procedures, including clinical diagnosis, nursing, and surgery, and are an indispensable component of the modern healthcare system. As medical standards continue to improve, the market demand for medical polymer consumables continues to grow, while the requirements for their production quality are also constantly increasing. Ensuring that every piece of medical polymer consumable meets safety standards is crucial for safeguarding medical safety and reducing medical risks.

[0003] Currently, the production of medical polymer consumables mostly adopts a batch production model on assembly lines, which is highly efficient and produces large volumes. To meet the quality control requirements of large-scale production, various testing technologies are gradually being applied to the production lines. Among them, image detection technology and three-dimensional contour detection technology have become mainstream technical means in the production and testing of medical polymer consumables due to their advantages such as fast detection speed, non-contact, and non-destructive nature.

[0004] On existing production lines, industrial cameras are typically used to capture two-dimensional images of consumables to observe their surface appearance; infrared imaging technology is used to capture infrared transmission images of consumables, utilizing the penetrating properties of infrared light to capture internal structural information; and laser profilometers are used to collect three-dimensional point cloud data of consumables to obtain their spatial geometric structure, providing basic data support for consumable quality assessment.

[0005] Meanwhile, with the rapid development of artificial intelligence technology, related technologies such as convolutional neural networks and multimodal fusion are gradually being deeply integrated with the field of medical consumable testing. By extracting and fusing features from the collected multi-type data, intelligent judgment of consumable quality can be achieved, gradually replacing traditional manual testing methods, improving testing efficiency, adapting to the needs of real-time testing on the production line, and providing a guarantee for the large-scale, high-quality production of medical polymer consumables.

[0006] The limitations of existing technologies include at least the following problems: In the early stages of detection, grayscale threshold comparison or template matching is usually performed only on two-dimensional appearance images, without in-depth analysis of the transparent material characteristics and three-dimensional structural features of the consumables. As a result, although the grayscale values ​​of the images meet the preset standards, it is difficult to effectively distinguish between actual transparent bubbles and transparent foreign objects due to their similar refractive indices, and micron-level scratches are directly ignored due to the lack of depth information. Ultimately, this results in extremely high false detection and false negative rates, and unqualified products enter the clinical process, seriously threatening the safety of medical operations. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides an intelligent detection method and system for defects in medical polymer consumables based on multi-feature fusion. This solves the problems of existing technologies, which rely solely on two-dimensional appearance images and are unable to effectively distinguish between transparent bubbles and transparent foreign objects with similar refractive indices, or capture micron-level scratch depth information, resulting in high false positive and false negative rates and difficulty in ensuring the clinical safety of medical consumables.

[0008] To achieve the above objectives, the present invention provides the following technical solution: an intelligent detection method for defects in medical polymer consumables based on multi-feature fusion, comprising the following steps: acquiring two-dimensional appearance images, infrared transmission images, and three-dimensional point cloud data of the medical polymer consumables to be tested on the production line; inputting the two-dimensional appearance image into a convolutional neural network to extract appearance texture features; calculating the attenuation rate of transmitted light intensity relative to the background without consumables for pixel-by-pixel in the infrared transmission image to generate an attenuation rate map; calculating the horizontal and vertical gradients of the attenuation rate map to generate gradient magnitude maps; concatenating the attenuation rate map and the gradient magnitude map into a dual-channel feature map, and inputting it into a convolutional neural network to extract the transmitted material. The system first identifies the surface curvature features. It projects 3D point cloud data into a depth map, constructs a Gaussian pyramid on the depth map, calculates surface curvature at each scale based on the local neighborhood covariance matrix, and concatenates the multi-scale curvature map with the depth map to form a multi-channel feature map. This map is then input into a convolutional neural network to extract geometric contour features. The surface texture features, transmission physical features, and geometric contour features are then spatially aligned at the pixel level and input into a multimodal fusion network. A channel attention module generates global channel weights, and a spatial attention module generates a spatial attention mask. The features are then weighted layer by layer and output as a fused feature map. Based on this fused feature map, a detection head outputs the defect category and location information of the consumable material under test.

[0009] Furthermore, the specific steps for calculating the attenuation rate of transmitted light intensity relative to a background without consumables, pixel by pixel, to generate an attenuation rate map are as follows: Multiple frames of infrared images are pre-collected and averaged when no consumables are passing through the production line, serving as the initial background image; infrared images without consumables are collected at preset intervals during production to update the background image; brightness compensation is performed on the background image based on real-time light source monitoring data; for each frame of the infrared transmission image, the grayscale value of each pixel is compared with the grayscale value of the corresponding background image to calculate the attenuation rate of transmitted light intensity, generating an attenuation rate map.

[0010] Furthermore, the specific steps for calculating the horizontal and vertical gradients to generate the gradient magnitude map from the attenuation rate map are as follows: use gradient operators to calculate the horizontal and vertical gradient responses of the attenuation rate map respectively; pre-calculate the gradient weight coefficient for each pixel position based on the surface curvature of the consumable to be tested; multiply the horizontal and vertical gradient responses by the corresponding gradient weight coefficients respectively, and then calculate the gradient magnitude map.

[0011] Furthermore, the 3D point cloud data is projected into a depth map, and a Gaussian pyramid is constructed on the depth map. The specific steps for calculating the surface curvature based on the local neighborhood covariance matrix at each scale are as follows: After projecting the 3D point cloud data into a depth map, a Gaussian pyramid is constructed on the depth map to obtain depth maps at multiple scales; in the depth map at each scale, for each pixel, the local neighborhood radius is dynamically determined based on the depth value of the pixel and the variance of the depth values ​​of the surrounding pixels; depth values ​​are collected within the dynamically determined local neighborhood to construct a covariance matrix; eigenvalues ​​are obtained by eigenvalue decomposition of the covariance matrix, and the surface curvature is calculated based on the ratio of the eigenvalues.

[0012] Furthermore, the specific steps for concatenating the multi-scale curvature map and the depth map into a multi-channel feature map and inputting it into a convolutional neural network to extract geometric contour features are as follows: Normalize the curvature map at each scale to map the curvature values ​​to a uniform numerical range; concatenate the normalized multi-scale curvature map with the original depth map in the channel dimension to obtain a multi-channel feature map; input the concatenated multi-channel feature map into the convolutional neural network for convolution operation to extract geometric contour features.

[0013] Furthermore, the specific steps for generating global channel weights through the channel attention module are as follows: obtain the difficulty distribution of defective samples in the current training batch; dynamically adjust the compression ratio of the fully connected layer in the channel attention module according to the difficulty distribution; perform global average pooling and global max pooling on the multimodal feature maps respectively to obtain two channel descriptors; input the two channel descriptors into the fully connected layer after the compression ratio is adjusted to generate the channel weight vector.

[0014] Furthermore, the specific steps for generating a spatial attention mask using the spatial attention module are as follows: A two-dimensional positional encoding map is generated based on the geometry of the consumable to be tested; the aligned appearance texture feature map, transmission physical feature map, and geometric contour feature map are concatenated along the channel dimension to obtain a multimodal feature map; the multimodal feature map is concatenated with the two-dimensional positional encoding map along the channel dimension; convolution is used to compress the concatenated feature map to obtain a spatial feature map; an activation function is applied to the spatial feature map to generate a spatial attention mask.

[0015] Furthermore, the specific steps for outputting the fused feature map after performing layer-by-layer weighted processing on the features are as follows: multiply the multimodal feature map with the channel weight vector channel by channel to obtain the first weighted feature map; multiply the first weighted feature map with the spatial attention mask element by element to obtain the second weighted feature map; input the second weighted feature map into a lightweight convolutional layer for feature refinement processing, and output the final fused feature map.

[0016] Furthermore, the specific steps for outputting the defect category and location information of the consumable under test through the detection head based on the fused feature map are as follows: input the fused feature map into the region proposal network to generate candidate regions; perform secondary discrimination on candidate regions with confidence scores below a preset threshold, and input the features of the candidate regions from the secondary discrimination into an additional classifier; combine the outputs of the region proposal network and the additional classifier to determine the defect category and location information of the consumable under test.

[0017] A multi-feature fusion-based intelligent detection system for defects in medical polymer consumables includes: a data acquisition unit for acquiring two-dimensional appearance images, infrared transmission images, and three-dimensional point cloud data of the medical polymer consumables to be tested on the production line; an appearance feature extraction unit for inputting the two-dimensional appearance images into a convolutional neural network to extract appearance texture features; a transmission feature extraction unit for calculating the attenuation rate of transmitted light intensity relative to the background without consumables on a pixel-by-pixel basis in the infrared transmission image to generate an attenuation rate map, calculating the horizontal and vertical gradients of the attenuation rate map to generate gradient magnitude maps, and concatenating the attenuation rate map and the gradient magnitude map into a dual-channel feature map before inputting it into a convolutional neural network to extract transmission physical features; and a geometric feature extraction unit for... The 3D point cloud data is projected into a depth map. A Gaussian pyramid is constructed on the depth map, and the surface curvature is calculated based on the local neighborhood covariance matrix at each scale. The multi-scale curvature map and the depth map are concatenated into a multi-channel feature map, which is then input into a convolutional neural network to extract geometric contour features. The feature alignment and fusion unit is used to perform pixel-level spatial alignment of the appearance texture features, transmission physical features, and geometric contour features, and then input them into a multimodal fusion network. The channel attention module generates global channel weights, and the spatial attention module generates a spatial attention mask. The features are weighted layer by layer and then output as a fused feature map. The defect detection output unit is used to output the defect category and location information of the consumable under test through the detection head based on the fused feature map.

[0018] The present invention has the following beneficial effects:

[0019] (1) The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion simultaneously acquires two-dimensional appearance images, infrared transmission images and three-dimensional point cloud data. Existing technologies usually rely on only a single two-dimensional image, which is difficult to cope with the complex optical properties of transparent polymer materials. The infrared transmission image of this method can penetrate the interior of the consumable and reflect the changes in density and refractive index. The three-dimensional point cloud accurately records the surface undulations, and the two-dimensional appearance image retains conventional texture and color information. The three complement each other and effectively avoid misjudgment caused by missing information. Especially when dealing with transparent materials, this combination of multi-source information can fundamentally break through the limitations of traditional visual inspection and ensure that the acquired data can truly and comprehensively reflect the quality status of the consumable.

[0020] (2) The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion calculates the attenuation rate of transmitted light intensity and its gradient response from infrared transmission images, and generates a dual-channel feature map that fuses attenuation information and gradient amplitude. This accurately depicts the density anomalies and refractive index change boundaries inside the consumables. To address the problem that transparent bubbles and transparent foreign objects are difficult to distinguish in two-dimensional images due to their similar gray levels, this method uses dynamically updated background images and brightness compensation to calculate the accurate attenuation rate and eliminate ambient light interference. The method captures the location of refractive index abrupt changes through horizontal and vertical gradient calculations and corrects cylindrical imaging distortion using a gradient operator weighted by surface curvature. This allows bubbles and foreign objects that are originally difficult to distinguish visually to be effectively separated in the feature space. The attenuation rate reflects the density difference, and the gradient amplitude highlights the interface abrupt changes. The combination of the two achieves a refined characterization of internal defects and significantly reduces the probability of missing transparent foreign objects and misjudging bubbles.

[0021] (3) The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion projects three-dimensional point cloud data into a depth map and constructs a Gaussian pyramid. At different scales, the surface curvature is calculated based on the local neighborhood covariance matrix, and finally a multi-channel feature map of multi-scale curvature and depth fusion is formed, which effectively captures micron-level surface defects. For geometric defects such as extremely shallow scratches and fine depressions that lack depth contrast, a single depth map is difficult to reflect their existence. This method introduces multi-scale analysis to capture the overall contour changes at the coarse scale and highlight the small undulations at the fine scale. At the same time, the neighborhood radius is dynamically adjusted according to the depth value variance to adaptively adapt to different curvature change areas, making the curvature calculation more accurate. The curvature maps of multiple scales are stacked with the original depth map and input into the convolutional neural network. The network can learn absolute depth information and relative curvature changes at the same time, thereby forming a strong response to micron-level surface defects. This fundamentally solves the problem that two-dimensional images ignore such defects due to lack of depth information, ensuring that even small defects can be accurately detected.

[0022] (4) The intelligent detection system for defects in medical polymer consumables based on multi-feature fusion constructs a multi-modal fusion network and introduces a dual attention mechanism. It performs pixel-level alignment and adaptive weighted fusion of surface texture features, transmission physical features and geometric contour features. The channel attention module dynamically adjusts the compression ratio according to the difficulty of the training samples to generate channel weights that reflect the importance of each modality. The spatial attention module combines the position encoding map of the consumable's geometric prior to generate an attention mask that focuses on the defect area. The two work in cascade to first recalibrate the channels of the feature map, then strengthen the key spatial positions, and finally refine it through lightweight convolution to output high-quality fused features. This solves the feature conflict caused by simple splicing of multi-modal features and can automatically adjust the contribution of each modality for different defect types, so that the system can remain stable in complex production environments.

[0023] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0024] Figure 1 This is a flowchart of an intelligent detection method for defects in medical polymer consumables based on multi-feature fusion, according to the present invention.

[0025] Figure 2 This is a flowchart illustrating the specific steps involved in generating a gradient magnitude map by calculating the horizontal and vertical gradients of the attenuation rate map in an intelligent detection method for defects in medical polymer consumables based on multi-feature fusion, according to the present invention.

[0026] Figure 3 This is a block diagram of an intelligent detection system for defects in medical polymer consumables based on multi-feature fusion, according to the present invention. Detailed Implementation

[0027] Please see Figure 1 This invention provides a technical solution: an intelligent detection method for defects in medical polymer consumables based on multi-feature fusion, comprising the following steps: acquiring two-dimensional appearance images, infrared transmission images, and three-dimensional point cloud data of the medical polymer consumables to be tested on the production line; inputting the two-dimensional appearance images into a convolutional neural network to extract appearance texture features; calculating the attenuation rate of transmitted light intensity relative to the background without consumables for each pixel of the infrared transmission image to generate an attenuation rate map; calculating the horizontal and vertical gradients of the attenuation rate map to generate gradient magnitude maps; and stitching the attenuation rate map and the gradient magnitude map into a dual-channel feature map, which is then input into a convolutional neural network to extract transmission physical features. The 3D point cloud data is projected into a depth map, and a Gaussian pyramid is constructed on the depth map. Surface curvature is calculated based on the local neighborhood covariance matrix at each scale. The multi-scale curvature map and the depth map are stitched together to form a multi-channel feature map, which is then input into a convolutional neural network to extract geometric contour features. The apparent texture features, transmission physical features, and geometric contour features are spatially aligned at the pixel level and then input into a multimodal fusion network. A channel attention module generates global channel weights, and a spatial attention module generates a spatial attention mask. The features are then weighted layer by layer and output as a fused feature map. Based on the fused feature map, the defect category and location information of the consumable under test are output through a detection head.

[0028] Among them, medical polymer consumables include, but are not limited to, disposable infusion sets, syringes, catheters, and other transparent / semi-transparent medical consumables;

[0029] Two-dimensional apparent images, infrared transmission images, and three-dimensional point cloud data are acquired synchronously using an encoder to ensure that the acquisition timestamps of the three modal data are consistent and the acquisition areas are completely corresponding, thus avoiding feature alignment deviations caused by asynchronous acquisition.

[0030] All convolutional neural networks were fine-tuned using pre-trained models, which were trained on a dataset of defective medical polymer consumables to ensure the relevance and accuracy of feature extraction.

[0031] Specifically, the steps for calculating the attenuation rate of transmitted light intensity relative to a background without consumables on a pixel-by-pixel basis in the infrared transmission image, and generating the attenuation rate map, are as follows:

[0032] Multiple frames of infrared images are pre-collected on the production line when no consumables are passing through, and then averaged to serve as the initial background image. Specifically:

[0033] When the production line is idle, the infrared acquisition device continuously acquires N frames of infrared images, where N ranges from 50 to 100 frames. Let the i-th frame be the background image. , where i∈[1,N], x and y are the horizontal and vertical coordinates of the image respectively, x∈[1,W], y∈[1,H], and W and H are the width and height of the infrared image respectively;

[0034] Through formula The initial background image is calculated. ;

[0035] For example: if N=80, W=640, H=480, then the average value of each pixel of the 80 frames of unloaded infrared images of 640×480 pixels is taken to obtain the initial background image, which is used to eliminate the influence of random noise from the infrared light source.

[0036] Infrared images of the production process at preset intervals when there are no consumables are collected, and the background image is updated accordingly. Specifically:

[0037] The preset time interval ranges from 10 to 30 minutes. The M frames of consumable-free infrared images acquired at the update time are denoted as... j∈[1,M], where M ranges from 20 to 50 frames;

[0038] Through formula Update the background image, where The background image before the update. For the updated background image, To update the weights (distinguished from the transmitted light intensity attenuation rate discussed later). The value range is 0.7-0.9;

[0039] For example: if the preset time interval is 20 minutes, then M=30. Then, 30 frames of idle infrared images are collected every 20 minutes, the average value is taken, and then fused with the original background image with a weight of 0.8:0.2 to realize the dynamic update of the background image and adapt to the brightness drift of the infrared light source during the production process.

[0040] Brightness compensation for the background image is performed based on real-time monitoring data of the light source, specifically as follows:

[0041] The brightness value of the infrared backlight is collected in real time by a light source brightness sensor. t represents the data acquisition time, and the preset standard brightness value is [value missing]. Brightness compensation coefficient ;

[0042] Through formula The background image is brightness compensated to obtain the compensated background image. ,in The current background image to be compensated (i.e., the background image that has not been compensated after the update). );

[0043] For example: if standard brightness Real-time monitoring of brightness Then the compensation coefficient The grayscale value of each pixel in the background image is multiplied by 1.11 to achieve brightness compensation and ensure the stability of the background light intensity.

[0044] For each frame of the infrared transmission image, the grayscale value of each pixel is compared with the grayscale value of the corresponding background image, the attenuation rate of transmitted light intensity is calculated, and an attenuation rate map is generated. Specifically, let the infrared transmission image be... The compensated background image is Transmitted light intensity attenuation rate:

[0045] ;

[0046] in, (Different from the background update weight in the previous text) );

[0047] when At that time, take ;

[0048] All pixels Composition attenuation rate diagram ;

[0049] For example: if the background grayscale value of a certain pixel The grayscale value of the corresponding pixel in the transmission image Then the attenuation rate of that pixel If the grayscale value of the transmitted image pixel is 210, which is greater than the grayscale value of the background, then the attenuation rate is 0, and finally a 640×480 pixel attenuation rate map is formed, which is used to characterize the light intensity attenuation caused by transparent defects (such as bubbles and foreign objects) inside the consumable.

[0050] In this implementation scheme, by performing multi-step processing on infrared transmission images, effective information corresponding to internal defects in consumables can be stably obtained. During the acquisition phase, the initial background image is obtained by averaging multiple frames of images, which can reduce random interference during the acquisition process and make the basic reference data more reliable. During continuous production, the background image is updated at fixed time intervals, and the new and old data are fused with weights to adapt to changes in light source during equipment operation and avoid the impact of image reference offset. At the same time, compensation is performed based on the real-time monitored light source brightness to further ensure that the background data is consistent with the actual acquisition environment and improve the overall data stability. When calculating the attenuation rate, a pixel-by-pixel comparison method is used to obtain the defect-related distribution based on the difference between the background and the transmission image. Values ​​exceeding the reasonable range are constrained so that the final generated attenuation rate map can truly reflect the light intensity changes caused by internal defects in consumables, thereby improving the identification effect of internal defects in transparent consumables.

[0051] Specifically, such as Figure 2 As shown, the specific steps for calculating the horizontal and vertical gradients to generate the gradient magnitude map from the attenuation rate map are as follows:

[0052] The gradient operator is used to calculate the horizontal and vertical gradient responses of the decay rate map, respectively, as follows:

[0053] The decay rate map was calculated using the Sobel gradient operator. The horizontal and vertical gradients, and the Sobel operator in the horizontal direction are:

[0054] ;

[0055] The vertical Sobel operator is:

[0056] ;

[0057] Horizontal gradient response:

[0058] ;

[0059] Vertical gradient response:

[0060] ;

[0061] Where x∈[2,W-1], y∈[2,H-1] (W and H are consistent with the width and height of the infrared image), and the gradient response of the edge pixels is 0;

[0062] For example, for a pixel at coordinates (100, 100) in the decay rate map, its horizontal gradient response is the sum of the gray values ​​of the pixel and its 8 surrounding pixels multiplied by the corresponding position coefficient of the Sobel operator. If there is a sudden change in the decay rate around the pixel (such as the edge of a bubble), the gradient response value will increase significantly, which is used to capture the edge information of defects.

[0063] The gradient weight coefficients for each pixel location are pre-calculated based on the surface curvature of the consumable material to be tested. Specifically:

[0064] Obtain the surface curvature distribution of the medical polymer consumables to be tested. The curvature distribution is obtained through a three-dimensional design model of the consumable. For cylindrical consumables (such as infusion set tubing), its surface curvature... R is the radius of the consumable (unit: mm), and the preset curvature weighting coefficient is calculated using the following formula:

[0065] ;

[0066] in, The greater the curvature of a region (such as the inner wall of a pipe), the closer the weighting coefficient is to 1; the smaller the curvature of a region (such as the outer wall of a pipe), the closer the weighting coefficient is to 0.5.

[0067] For example: if the radius of the cylindrical consumable is R=5mm, then the surface curvature is... Maximum curvature Then the gradient weight coefficients of all pixels of the consumable ;

[0068] If the consumable material has a variable curvature structure, the curvature of a certain area is 0.15mm. -1 The weighting coefficient is 0.15 / 0.2=0.75, which is used to enhance the gradient response in areas with large curvature and improve the accuracy of defect detection.

[0069] After multiplying the horizontal and vertical gradient responses by their respective gradient weight coefficients, the gradient magnitude map is calculated as follows:

[0070] Calculate the weighted horizontal gradient response and weighted vertical gradient response ,in These are the pre-calculated gradient weight coefficients;

[0071] Through formula Calculate the gradient magnitude and compile the gradient magnitudes of all pixels into a gradient magnitude map. ;

[0072] For example: if the horizontal gradient response of a pixel Vertical gradient response Gradient weight coefficients Then the weighted horizontal gradient Weighted vertical gradient gradient magnitude In the gradient magnitude map, the larger the value, the more drastic the change in the decay rate, which means that there is a more likely defect edge.

[0073] In this implementation scheme, gradient calculation of the attenuation rate map using gradient operators can accurately extract image change information corresponding to defect edges. Using fixed operators to complete gradient response calculations in the horizontal and vertical directions can stably identify abrupt changes in the attenuation rate distribution, making defect edges clearly prominent. Combining the surface curvature distribution of the consumables to calculate gradient weight coefficients allows adjustment of gradient response intensity according to the structural characteristics of different regions, better preserving defect information in areas with greater curvature and avoiding loss of edge features due to structural differences. Combining the calculated gradient response with the weight coefficients to generate a gradient amplitude map can suppress irrelevant interference while preserving the real defect edges, making the final amplitude map more consistent with the actual structure and defect distribution characteristics of the consumables.

[0074] Specifically, the steps for projecting 3D point cloud data into a depth map, constructing a Gaussian pyramid on the depth map, and calculating surface curvature based on the local neighborhood covariance matrix at each scale are as follows:

[0075] After projecting the 3D point cloud data into a depth map, a Gaussian pyramid is constructed from the depth map to obtain depth maps at multiple scales, specifically:

[0076] Record the 3D point cloud data as k∈[1,K], where K is the total number of point clouds. Projecting the point cloud data onto a two-dimensional pixel plane yields a depth map. ,in The Z-coordinate value of the point cloud corresponding to pixel (x,y) (unit: mm);

[0077] Construct an L-layer Gaussian pyramid, where L ranges from 3 to 5 layers, and the depth map of the l-th layer is shown. From the depth map of layer l-1 The Gaussian filter kernel is obtained through Gaussian filtering and downsampling.

[0078] ;

[0079] in, The standard deviation is Gaussian (unitless), which increases with the number of layers. , The downsampling uses a 2x downsampling (i.e., an average value is taken for every 2×2 pixels);

[0080] For example: If L=4, the initial depth map If the dimensions are 640×480, then The dimensions are 320×240. The dimensions are 160×120. The dimensions are 80×60. This enables the extraction of multi-scale deep features, taking into account both global and local information about defects;

[0081] In the depth map at each scale, for each pixel, the local neighborhood radius is dynamically determined based on the depth value of that pixel and the variance of the depth values ​​of surrounding pixels. Specifically:

[0082] Record the depth map of layer l The depth value of the middle pixel (x,y) Calculate the variance of the depth values ​​within the 3×3 neighborhood surrounding the pixel:

[0083] ;

[0084] in, The average depth value within a 3×3 neighborhood. The depth variance (unit: mm²); local neighborhood radius. The calculation formula is:

[0085] ;

[0086] in, The base neighborhood radius, with a value ranging from 3 to 5 pixels. The range of values ​​is (Unit: pixels);

[0087] For example: if The depth variance of a pixel's 3×3 neighborhood Global maximum depth variance Then the local neighborhood radius of the pixel , rounded down to an integer of 5 pixels;

[0088] If the variance is 0 (uniform depth region), the neighborhood radius is 3 pixels, realizing dynamic adaptation of the neighborhood radius and avoiding curvature calculation deviation caused by a fixed neighborhood.

[0089] The covariance matrix is ​​constructed by collecting depth values ​​within a dynamically determined local neighborhood, specifically as follows:

[0090] For a pixel (x, y) in the l-th depth map, with that pixel as the center, ... Define a local neighborhood for the radius, and collect the depth values ​​of all pixels within that neighborhood. ;

[0091] in, Let M be the total number of pixels in the neighborhood.

[0092] Construct a 3×3 covariance matrix ;

[0093] in, , The three-dimensional coordinates corresponding to the i-th pixel in the neighborhood (and the point cloud data) (Coordinate system consistent) It is the average of the three-dimensional coordinates within the neighborhood;

[0094] For example, if there are 25 pixels in a local neighborhood, calculate the covariance between the X, Y, and Z coordinates to obtain a 3×3 covariance matrix. The elements of the covariance matrix reflect the distribution of the point cloud in each direction in the neighborhood and are used for subsequent curvature calculation.

[0095] The covariance matrix is ​​decomposed into eigenvalues, and the surface curvature is calculated based on the ratio of these eigenvalues. Specifically:

[0096] Eigenvalues ​​of the covariance matrix C are obtained by performing eigenvalue decomposition. ,in The largest eigenvalue, The smallest eigenvalue (the eigenvalue has no unit and is only used to characterize the distribution features of point clouds);

[0097] Surface curvature uses shape index and curvature amplitude Common characterization (distinct from the surface curvature used in gradient weight coefficient calculation) The final surface curvature value of this pixel is... ;

[0098] For example: if the eigenvalue decomposition yields ;

[0099] The shape index ;

[0100] Curvature amplitude ;

[0101] Surface curvature The larger the curvature value, the more obvious the surface undulations in the area, and the more likely there are defects such as micron-level scratches.

[0102] In this implementation scheme, by projecting 3D point cloud data into a depth map and constructing a Gaussian pyramid, depth features can be extracted from different levels, taking into account both the global distribution and local details of surface defects of consumables. This avoids the omission of defects caused by single-scale analysis. The local neighborhood radius is dynamically determined based on the pixel depth value and the surrounding variance, which can adapt to the differences in depth distribution in different regions and avoid the calculation deviation caused by fixed neighborhoods. This allows the neighborhood selection to better fit the actual depth changes. The covariance matrix is ​​constructed in the dynamic neighborhood, which can accurately capture the distribution characteristics of local point clouds and provide a reliable basis for curvature calculation. By decomposing the covariance matrix features and combining the shape index and curvature amplitude to calculate the surface curvature, the undulation changes of the consumable surface can be accurately characterized, so as to clearly identify subtle surface defects.

[0103] Specifically, the steps for concatenating multi-scale curvature maps and depth maps into a multi-channel feature map and inputting it into a convolutional neural network to extract geometric contour features are as follows:

[0104] The curvature maps at each scale are normalized to map the curvature values ​​to a uniform numerical range. Specifically:

[0105] Let the curvature diagram of the l-th layer be denoted as (i.e., the image formed by the calculated surface curvature values), l∈[1,L] (L is the number of Gaussian pyramid layers), calculate the maximum value of the curvature map for each layer. and minimum value Using linear normalization, the normalized curvature values ​​are:

[0106] ;

[0107] For example: if the maximum value of the first layer curvature diagram minimum value If the curvature value of a pixel is 25, then the normalized value is ;

[0108] If the maximum value of the second-layer curvature map is 60, the minimum value is 5, and the curvature value of a certain pixel is 35, then the normalized value is This ensures that curvature values ​​at different scales are on the same order of magnitude, avoiding scale bias during feature fusion.

[0109] The normalized multi-scale curvature map is concatenated with the original depth map along the channel dimension to obtain a multi-channel feature map, specifically:

[0110] Original depth map (The initial depth map obtained from point cloud projection) is first normalized. The normalization method is related to the curvature. Figure 1 To, received ;

[0111] Normalized L-layer curvature map Compared with the normalized original depth map The feature maps are concatenated along the channel dimension, resulting in a multi-channel feature map with the following dimensions: , where H and W are the image height and width (consistent with the dimensions of the infrared image and depth map);

[0112] For example, if L=4, and the dimensions of both the depth map and the curvature map are 640×480, then the dimensions of the stitched multi-channel feature map are 640×480×5. Channel 1 is the original depth map, and channels 2-5 are normalized curvature maps of 4 scales, respectively, to achieve the fusion of multi-scale geometric features.

[0113] The concatenated multi-channel feature map is input into a convolutional neural network for convolution to extract geometric contour features, specifically:

[0114] The convolutional neural network consists of 4 convolutional layers, 2 pooling layers, and 1 fully connected layer. The kernel size of each convolutional layer is 3×3, the stride is 1, and the padding method is SAME (SAME padding, which is edge padding to ensure that the size of the feature map after convolution is consistent with the input). Each convolutional layer is followed by a ReLU activation function.

[0115] The pooling layer uses max pooling with a pooling kernel size of 2×2 and a step size of 2.

[0116] The fully connected layer outputs a 256-dimensional geometric contour feature vector. ( (representing a 256-dimensional real vector).

[0117] For example, a 640×480×5 multi-channel feature map is processed by the first convolutional layer (32 convolutional kernels) to output a 640×480×32 feature map. After being processed by the pooling layer, it outputs a 320×240×32 feature map. After passing through subsequent convolutional and pooling layers, it finally outputs a 256-dimensional feature vector through a fully connected layer. This vector contains multi-scale geometric information of the consumable surface and is used to identify deep defects such as micron-level scratches.

[0118] In this implementation scheme, by normalizing the multi-scale curvature map and the original depth map, feature values ​​of different scales and ranges can be mapped to a unified range, avoiding feature fusion deviations caused by numerical differences and ensuring that features of each scale can participate equally in subsequent processing. By channel-splitting the normalized multi-scale curvature map and the original depth map, the multi-scale geometric features of the consumable surface and the original depth information can be integrated to achieve comprehensive fusion of geometric features and enrich the feature information. The spliced ​​multi-channel feature map is then input into a convolutional neural network for processing. Through operations such as convolution and pooling, key features that can characterize the geometric contour of the consumable can be effectively extracted, and irrelevant interference information can be filtered out.

[0119] Specifically, the steps for generating global channel weights using the channel attention module are as follows:

[0120] The difficulty distribution of defective samples in the current training batch is obtained as follows:

[0121] The current training batch contains N samples (N is the total number of samples in the training batch, with no fixed value, set according to actual training needs), and the difficulty of each sample is expressed by the loss value. The representation, i∈[1,N], the loss value is calculated by the classification loss function (using the cross-entropy loss function), the larger the loss value, the higher the difficulty of sample classification;

[0122] The samples are categorized into three types based on their loss values: easy samples, moderate samples, and difficult samples. The loss value range for easy samples is [value missing]. Medium sample Difficult samples ,in ( (The threshold value for the loss is set based on the training performance of the samples).

[0123] The proportions of the three types of samples are denoted as follows: ,satisfy That is, to obtain the distribution of the difficulty of defect samples;

[0124] For example: If the current batch has 100 samples, of which 30 are easy samples ( ), 50 medium-sized samples ( ), 20 difficult samples ( If the difficulty distribution of this batch of samples is 0.3:0.5:0.2, then the weight allocation strategy for dynamically adjusting channel attention is used.

[0125] Based on the distribution of difficulty levels, the compression ratio of the fully connected layer in the channel attention module is dynamically adjusted, specifically as follows:

[0126] The compression ratio *r* of a fully connected layer is defined as the ratio of the number of input channels to the number of intermediate channels, with a base compression ratio of... ;

[0127] The dynamic adjustment formula is The value of r is in the range of [8, 24]. If the calculated r exceeds this range, the nearest boundary value is taken.

[0128] For example: if ,but The integer value is 13, which means the ratio of the number of input channels to the number of intermediate channels in the fully connected layer is 13:1. By adjusting the compression ratio, the channel attention module can be better adapted to the difficulty of the current sample, thereby improving the feature extraction accuracy of difficult samples.

[0129] Global average pooling and global max pooling are performed on the multimodal feature maps respectively to obtain two channel descriptors, as follows:

[0130] Let the multimodal feature map be denoted as Where C is the number of channels, and H and W are the feature map sizes (consistent with the feature map size). Represents an H×W×C dimensional real-valued feature map;

[0131] Global average pooling calculates the average value for each channel, resulting in the average channel descriptor. The calculation formula is:

[0132] c∈[1,C];

[0133] Global max pooling is used to calculate the maximum value for each channel, resulting in the maximum channel descriptor. The calculation formula is:

[0134] ;

[0135] For example, if the multimodal feature map size is 32×32×256 and C=256, then global average pooling will result in a 256-dimensional feature map. Each element is the average value of 32×32 pixels in the corresponding channel;

[0136] Global max pooling yields a 256-dimensional result. Each element is the maximum value of 32×32 pixels in the corresponding channel, and the two descriptors capture the global average response and global maximum response of the channel, respectively.

[0137] The two channel descriptors are input into the fully connected layer after the compression ratio is adjusted to generate a channel weight vector, which is as follows:

[0138] Will and By concatenating along the channel dimension, we obtain the concatenated descriptor. ( (representing a 2C-dimensional real vector).

[0139] Will The first fully connected layer is input with an input dimension of 2C and an output dimension of 2C / r (where r is the adjusted compression ratio). After being processed by the ReLU activation function, it is input into the second fully connected layer with an output dimension of C.

[0140] The channel weight vector is obtained by processing with the Sigmoid activation function. ,in The calculation formula is:

[0141] ;

[0142] in, The Sigmoid activation function (values ​​range [0,1], used to normalize the output to the weight range);

[0143] For example, if C=256 and r=13, the input dimension of the first fully connected layer is 512 and the output dimension is 512 / 13≈39. After ReLU activation, it is input into the second fully connected layer, and the output dimension is 256. After Sigmoid activation, a 256-dimensional channel weight vector is obtained. The larger the weight value, the more important the feature of the corresponding channel, which is used to highlight the feature channels related to defects.

[0144] In this implementation, the compression ratio within the channel attention is adjusted according to the difficulty distribution of the training samples. This allows the network to automatically adjust the feature extraction method when processing samples of different difficulties, better focusing on defective samples that are difficult to identify. At the same time, a combination of global average pooling and global max pooling is used to extract channel descriptors, which can comprehensively obtain the feature information of each channel from different perspectives and avoid the information loss caused by a single pooling method. By concatenating the results of the two pooling methods and then calculating them through a fully connected layer, a more comprehensive channel feature representation can be obtained, improving the accuracy of weight judgment. By generating channel weight vectors through the dynamically adjusted network structure, the influence of channels related to defects can be automatically strengthened and the influence of irrelevant channels weakened.

[0145] Specifically, the steps for generating a spatial attention mask using the spatial attention module are as follows:

[0146] A two-dimensional position coding map is generated based on the geometry of the consumable to be tested, specifically as follows:

[0147] Obtain the two-dimensional projected profile of the medical polymer consumable to be tested. ,in This indicates that the pixel belongs to the consumable area. This indicates that it belongs to the background area;

[0148] Two-dimensional location coding map The calculation formula is:

[0149] ;

[0150] in, The distance (in pixels) from pixel (x,y) to the center of the consumable outline is the Euclidean distance. The maximum Euclidean distance from a pixel to the center of the consumable outline (unit: pixels);

[0151] For example: For cylindrical consumables, its two-dimensional projection is a rectangle, and the center of the outline is the geometric center of the rectangle. The distance from a pixel to the center is... Then the position code value of the pixel The location code value of the central area of ​​the consumable is close to 1, while that of the edge area is close to 0, reflecting the prior probability of defects in different areas of the consumable (the probability of defects in the central area is higher than that in the edge area).

[0152] The aligned apparent texture feature map, transmission physical feature map, and geometric contour feature map are concatenated along the channel dimension to obtain a multimodal feature map, which is as follows:

[0153] Let the aligned apparent texture feature map be... ( (Number of channels in the apparent texture feature map), and the transmission physical feature map is... ( (Number of channels in the transmission physical feature map), and geometric contour feature map. ( (where the number of channels in the geometric contour feature map is 1), These represent the number of channels in the three feature maps (all three can be equal, each with 256 dimensions, or can be adjusted according to actual feature extraction requirements);

[0154] The three feature maps are concatenated along the channel dimension to obtain a multimodal feature map. ;

[0155] For example: if If the feature map size is 32×32, then the size of the spliced ​​multimodal feature map is 32×32×768, which includes the appearance, transmission and geometric feature information of the consumables.

[0156] The multimodal feature map and the two-dimensional location encoding map are concatenated along the channel dimension, specifically as follows:

[0157] Two-dimensional location coding map Expand to The single-channel feature map (with the same size as the multimodal feature map), and the multimodal feature map By concatenating along the channel dimension, we obtain the concatenated feature map. ;

[0158] For example, after concatenating a 32×32×768 multimodal feature map with a 32×32×1 position encoding map, a 32×32×769 feature map is obtained, which integrates the position encoding information into the multimodal features and guides the spatial attention module to focus on the consumable area.

[0159] The concatenated feature maps are then subjected to channel compression using convolution to obtain spatial feature maps, specifically:

[0160] The concatenated feature maps are processed using 1×1 convolution kernels. Channel compression is performed, and the number of convolutional kernels is... (Round to the nearest integer, and round up if there is a remainder), with a step size of 1 and a padding method of SAME, the spatial feature map is obtained after processing with the ReLU activation function. ,in ;

[0161] For example: if The number of convolution kernels is 192 (769 / 4≈192). After the 32×32×769 feature map is processed by 1×1 convolution, the output is a 32×32×192 spatial feature map, which realizes the compression of the channel dimension and reduces the amount of subsequent computation.

[0162] An activation function is applied to the spatial feature map to generate a spatial attention mask, specifically as follows:

[0163] Using 3×3 convolution kernels to analyze spatial feature maps Perform a convolution operation with a kernel size of 1, a stride of 1, and SAME padding to obtain a single-channel feature map. ;

[0164] Applying the Sigmoid activation function to this single-channel feature map yields a spatial attention mask:

[0165] ;

[0166] in, ( (This is the Sigmoid activation function, the same Sigmoid function used in the channel weight vector generation mentioned earlier).

[0167] For example, a 32×32×192 spatial feature map is processed by a 3×3 convolution to output a 32×32×1 single-channel feature map. After Sigmoid activation, a 32×32 spatial attention mask is obtained. The larger the mask value, the more important the feature at the corresponding position, which is used to highlight the spatial location of the defect.

[0168] In this implementation scheme, a two-dimensional position encoding map is generated based on the geometry of the consumable. This map combines the probability of defects occurring in different regions of the consumable to guide the attention module to focus on areas more prone to defects, reducing interference from background areas. The aligned three types of feature maps are then stitched together to form a multimodal feature map, which integrates the appearance, transmission, and geometric features of the consumable, making the feature information more comprehensive and providing sufficient basis for spatial attention judgment. Stitching the position encoding map with the multimodal feature map integrates position information into the features, further clarifying the direction of attention focus. By compressing the channel dimension through convolution operations, the subsequent computational burden can be reduced while retaining key feature information. Finally, a spatial attention mask is generated through an activation function, which can accurately highlight the spatial location of the defect, strengthen the features of the defect area, and weaken the influence of irrelevant areas.

[0169] Specifically, the steps for outputting the fused feature map after performing layer-by-layer weighted processing on the features are as follows:

[0170] The multimodal feature map is multiplied channel by channel weight vector to obtain the first weighted feature map, which is as follows:

[0171] Record multimodal feature maps Channel weight vector c∈[1,C], where C is the number of channels in the multimodal feature map (i.e., );

[0172] First weighted feature map Weighted calculations are performed channel by channel;

[0173] For example: if the weight of a certain channel c in a multimodal feature map... If the feature value of a pixel in this channel is 0.5, then the weighted feature value of that pixel is... ;

[0174] If a certain channel has a weight of 0.2 and a pixel feature value of 0.6, then the weighted average will be 0.12. Channel weighting highlights important feature channels and suppresses interference from irrelevant channels.

[0175] The first weighted feature map is multiplied element-wise with the spatial attention mask to obtain the second weighted feature map, which is as follows:

[0176] Spatial attention mask Expand to (Same size and number of channels as the initial weighted feature map), same as the initial weighted feature map Element-wise multiplication yields a quadratic weighted feature map. ;

[0177] For example: if the feature value of a pixel (x, y, c) in the first weighted feature map is 0.4, and the mask value at the corresponding position in the spatial attention mask is 0.9, then the feature value of that pixel after the second weighting is... ;

[0178] If the mask value is 0.1, the weighted value is 0.04. Spatial weighting highlights the spatial location of the defect and suppresses interference from the background area.

[0179] The second-weighted feature map is input into a lightweight convolutional layer for feature refinement, and the final fused feature map is output, which is as follows:

[0180] The lightweight convolutional layer employs a depthwise separable convolutional structure, comprising both depthwise convolution and pointwise convolution.

[0181] The depthwise convolution uses a 3×3 kernel, with one kernel corresponding to each channel, a stride of 1, and SAME padding.

[0182] Pointwise convolution uses a 1×1 convolution kernel, with the number of kernels matching the number of channels in the second-weighted feature map, and a stride of 1.

[0183] After depthwise separable convolution, the model is normalized using a BatchNorm layer (a batch normalization layer used to accelerate model training and prevent overfitting), followed by ReLU activation, to output the final fused feature map. ;

[0184] For example, a 32×32×768 quadratic weighted feature map is processed by depthwise separable convolution (3×3 depthwise convolution + 1×1 pointwise convolution) to output a 32×32×768 fused feature map. This feature map has been refined to retain key features related to defects and suppress noise interference, providing high-quality feature input for subsequent defect detection.

[0185] In this implementation, multiplying the multimodal feature map with the channel weight vector channel by channel can strengthen the feature channels related to defects and weaken the interference of irrelevant channels, making the feature expression more focused on effective information. Multiplying the first weighted feature map with the spatial attention mask element by element can further highlight the spatial location of the defect, reduce the influence of background and irrelevant regions, and improve the targeting of features. The second weighted feature map is input into a lightweight convolutional layer and refined using a depthwise separable convolutional structure, which can retain the key features related to defects, filter noise interference, reduce computational burden, and avoid redundant operations.

[0186] Specifically, the steps for outputting the defect category and location information of the consumable under test through the detection head based on the fused feature map are as follows:

[0187] The fused feature map is input into a region proposal network to generate candidate regions, specifically as follows:

[0188] The region proposal network consists of one 3×3 convolutional layer and two 1×1 convolutional layers. The 3×3 convolutional layer is used to extract local features from the fused feature map, and the first 1×1 convolutional layer outputs the bounding box coordinate offsets of the candidate regions. ;

[0189] in, This is the offset of the bounding box center coordinates. This is the width and height offset of the bounding box (the offset is unitless and used to correct the anchor box coordinates).

[0190] Confidence of candidate regions output by the second 1×1 convolutional layer This indicates the probability that the region is a defective region. );

[0191] There are three preset anchor box sizes (small, medium, and large), each size corresponding to three aspect ratios (1:1, 1:2, and 2:1), for a total of nine anchor boxes. Each pixel of the fused feature map is traversed to generate candidate regions.

[0192] For example: the fused feature map size is 32×32, each pixel generates 9 anchor boxes, and there are a total of 32×32×9=9216 candidate regions. Each candidate region contains bounding box coordinates and confidence score. Candidate regions with a confidence score greater than a preset threshold (such as 0.5) are initially judged as possible defect regions.

[0193] Candidate regions with confidence levels below a preset threshold undergo secondary discrimination. The features of the candidate regions from this secondary discrimination are then input into an additional classifier. Specifically:

[0194] Preset reliability threshold The value ranges from 0.3 to 0.5 (this may differ from the initial threshold of 0.5 for candidate region determination, and can be set according to the required detection accuracy) to filter out the confidence levels. The candidate regions are used as regions to be further discriminated.

[0195] The ROI Align method (Region of Interest Alignment method, used to accurately extract candidate region features and avoid feature misalignment) is used to extract features of the region to be further discriminated, resulting in a feature map of a fixed size (e.g., 7×7). This feature map is then input into an additional classifier.

[0196] The additional classifier consists of two fully connected layers with an input dimension of 7×7×C (where C is the number of channels in the fused feature map) and an output dimension of the number of defect categories + 1 (including the background class). The additional classifier is specifically trained for hard-to-classify samples (samples with confidence close to the threshold).

[0197] For example: if Then, candidate regions with confidence levels between 0 and 0.4 are selected, their ROI features are extracted, and they are input into an additional classifier. The additional classifier performs secondary discrimination on these difficult-to-distinguish regions to avoid missed detections or false detections.

[0198] By combining the outputs of the region proposal network and the additional classifier, the defect category and location information of the consumable under test are determined, specifically as follows:

[0199] For confidence level The candidate regions are directly used as the defect categories by the classification output of the region proposal network, and the bounding box coordinates are used as the location information after offset correction.

[0200] For confidence level The candidate regions are selected, and the output of the additional classifier is used as the defect category. If the additional classifier determines that the region is a defect, the corrected bounding box is used as the location information. If the region is determined to be a background, the candidate region is removed.

[0201] Finally, non-maximum suppression (NMS) is applied to all candidate regions identified as defects to remove overlapping candidate regions. Candidate regions with an overlap greater than a preset threshold (e.g., 0.5) are removed. The final output shows the defect category of the consumable under test (e.g., transparent bubbles, transparent foreign objects, micron-level scratches) and the corresponding location information (boundary box coordinates, in the format (x1, y1, x2, y2), where x1 and y1 are the coordinates of the upper left corner, and x2 and y2 are the coordinates of the lower right corner).

[0202] For example: if the confidence of a candidate region proposal network is 0.35 (<0.4), and the additional classifier determines it as a transparent bubble, and the bounding box coordinates are corrected to (x1, y1, x2, y2), then the region is retained as a transparent bubble defect;

[0203] If the additional classifier classifies the area as background, the area is removed to ensure the accuracy of defect detection.

[0204] In this implementation scheme, the feature map is input into the region proposal network to generate candidate regions. By using various anchor boxes to adapt to defects of different sizes and shapes, it can comprehensively capture regions where defects may exist, providing a sufficient basis for subsequent discrimination. Candidate regions with low confidence are subject to secondary discrimination. Combined with a specially trained additional classifier, it can effectively identify defect samples that are difficult to judge, avoiding missed detections or false detections caused by a single judgment criterion. By combining the output results of the region proposal network and the additional classifier, a differentiated judgment strategy is adopted for candidate regions with different confidence levels. At the same time, non-maximum suppression is used to remove overlapping regions, thereby improving the accuracy of defect localization.

[0205] Please see Figure 3This invention provides a technical solution: an intelligent detection system for defects in medical polymer consumables based on multi-feature fusion, comprising: a data acquisition unit for acquiring two-dimensional appearance images, infrared transmission images, and three-dimensional point cloud data of the medical polymer consumables to be tested on the production line; an appearance feature extraction unit for inputting the two-dimensional appearance image into a convolutional neural network to extract appearance texture features; a transmission feature extraction unit for calculating the attenuation rate of transmitted light intensity relative to the background without consumables for pixel-by-pixel in the infrared transmission image to generate an attenuation rate map, calculating the horizontal and vertical gradients of the attenuation rate map to generate a gradient magnitude map, and concatenating the attenuation rate map and the gradient magnitude map into a dual-channel feature map before inputting it into a convolutional neural network to extract transmission physical features; and geometric features. The extraction unit projects 3D point cloud data into a depth map, constructs a Gaussian pyramid on the depth map, calculates surface curvature at each scale based on the local neighborhood covariance matrix, and concatenates the multi-scale curvature map with the depth map to form a multi-channel feature map, which is then input into a convolutional neural network to extract geometric contour features. The feature alignment and fusion unit performs pixel-level spatial alignment of appearance texture features, transmission physical features, and geometric contour features, and then inputs them into a multimodal fusion network. The channel attention module generates global channel weights, and the spatial attention module generates a spatial attention mask. The features are then weighted layer by layer and output as a fused feature map. The defect detection output unit outputs the defect category and location information of the consumable under test based on the fused feature map through a detection head.

[0206] The data acquisition unit includes an industrial CCD camera, a near-infrared camera, and a multi-line laser profilometer. The three are synchronously triggered by an encoder to ensure synchronous acquisition of data in three modes.

[0207] Industrial CCD cameras are used to acquire two-dimensional surface images. They are equipped with bright-field reflective light sources to capture surface defects (such as scratches and stains) on consumables.

[0208] Near-infrared cameras are used to acquire infrared transmission images and are equipped with infrared backlights to capture transparent defects (such as bubbles and foreign objects) inside consumables.

[0209] Multi-line laser profilometers are used to acquire 3D point cloud data to capture deep defects (such as micron-level depressions and protrusions) on the surface of consumables.

[0210] The convolutional neural networks used in the appearance feature extraction unit, transmission feature extraction unit, and geometric feature extraction unit are all pre-trained and fine-tuned based on a dataset of defective medical polymer consumables to ensure the targeted nature of feature extraction.

[0211] The multimodal fusion network in the feature alignment fusion unit integrates channel attention and spatial attention modules, which can dynamically adjust feature weights and improve the quality of fused features.

[0212] The detection head in the defect detection output unit integrates a region proposal network and an additional classifier, which can effectively improve the detection accuracy of difficult-to-distinguish samples and reduce the false detection rate and false negative rate.

[0213] The units are connected by a high-speed data bus adapted to industrial-grade real-time transmission requirements, enabling real-time data transmission and processing, and meeting the real-time detection needs of the production line.

[0214] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0215] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A smart detection method for defects in medical polymer consumables based on multi-feature fusion, characterized in that, Includes the following steps: Two-dimensional appearance images, infrared transmission images, and three-dimensional point cloud data of the medical polymer consumables to be tested on the production line were collected. Two-dimensional appearance images are input into a convolutional neural network to extract appearance texture features; The attenuation rate of transmitted light intensity relative to the background without consumables is calculated pixel by pixel for the infrared transmission image to generate an attenuation rate map. The horizontal and vertical gradients are calculated separately for the attenuation rate map to generate a gradient magnitude map. The attenuation rate map and the gradient magnitude map are concatenated into a dual-channel feature map and input into a convolutional neural network to extract the transmission physical features. The 3D point cloud data is projected into a depth map, a Gaussian pyramid is constructed on the depth map, and the surface curvature is calculated based on the local neighborhood covariance matrix at each scale. The multi-scale curvature map and the depth map are stitched together to form a multi-channel feature map, which is then input into a convolutional neural network to extract geometric contour features. After pixel-level spatial alignment of apparent texture features, transmission physical features, and geometric contour features, the network inputs them into a multimodal fusion network. The global channel weights are generated by the channel attention module, and the spatial attention mask is generated by the spatial attention module. The features are then weighted layer by layer and the fused feature map is output. Based on the fused feature map, the detection head outputs the defect category and location information of the consumables under test.

2. The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion according to claim 1, characterized in that, The specific steps for calculating the attenuation rate of transmitted light intensity relative to a background without consumables on a pixel-by-pixel basis in the infrared transmission image, and generating the attenuation rate map, are as follows: Multiple infrared images were pre-collected on the production line when no consumables passed through, and then averaged to serve as the initial background image. Infrared images of the production process without consumables are collected at preset intervals during the production process, and the background image is updated accordingly. Brightness compensation is applied to the background image based on real-time monitoring data of the light source. For each frame of infrared transmission image, the gray value of each pixel is compared with the gray value of the background image at the corresponding position, the transmission light intensity attenuation rate is calculated, and an attenuation rate map is generated.

3. The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion according to claim 1, characterized in that, The specific steps for calculating the horizontal and vertical gradients and generating the gradient magnitude map from the attenuation rate map are as follows: The gradient operator is used to calculate the horizontal and vertical gradient responses of the decay rate map, respectively. The gradient weight coefficients for each pixel position are pre-calculated based on the surface curvature of the consumable to be tested; The gradient magnitude map is calculated by multiplying the horizontal and vertical gradient responses by their respective gradient weight coefficients.

4. The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion according to claim 1, characterized in that, The specific steps for projecting 3D point cloud data into a depth map, constructing a Gaussian pyramid from the depth map, and calculating surface curvature based on the local neighborhood covariance matrix at each scale are as follows: After projecting the 3D point cloud data into a depth map, a Gaussian pyramid is constructed on the depth map to obtain depth maps at multiple scales. In the depth map at each scale, the local neighborhood radius is dynamically determined for each pixel based on the depth value of that pixel and the variance of the depth values ​​of surrounding pixels. Depth values ​​are collected within a dynamically determined local neighborhood to construct a covariance matrix; The covariance matrix is ​​decomposed into eigenvalues, and the surface curvature is calculated based on the ratio of the eigenvalues.

5. The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion according to claim 1, characterized in that, The specific steps for concatenating a multi-scale curvature map and a depth map into a multi-channel feature map and inputting it into a convolutional neural network to extract geometric contour features are as follows: The curvature maps at each scale are normalized to map the curvature values ​​to a uniform numerical range. The normalized multi-scale curvature map is concatenated with the original depth map along the channel dimension to obtain a multi-channel feature map. The stitched multi-channel feature map is input into a convolutional neural network for convolution operation to extract geometric contour features.

6. The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion according to claim 1, characterized in that, The specific steps for generating global channel weights using the channel attention module are as follows: Obtain the distribution of difficulty levels of defective samples in the current training batch; Based on the distribution of difficulty levels, dynamically adjust the compression ratio of the fully connected layer in the channel attention module; Global average pooling and global max pooling are performed on the multimodal feature maps respectively to obtain two channel descriptors; Input the two channel descriptors into the fully connected layer after the compression ratio is adjusted to generate the channel weight vector.

7. The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion according to claim 1, characterized in that, The specific steps for generating a spatial attention mask using the spatial attention module are as follows: A two-dimensional position coding map is generated based on the geometry of the consumable to be tested; The aligned appearance texture feature map, transmission physical feature map, and geometric contour feature map are stitched together along the channel dimension to obtain a multimodal feature map; The multimodal feature map and the two-dimensional location coding map are concatenated along the channel dimension; The concatenated feature maps are compressed using convolution to obtain spatial feature maps. An activation function is applied to the spatial feature map to generate a spatial attention mask.

8. The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion according to claim 1, characterized in that, The specific steps for outputting the fused feature map after performing layer-by-layer weighted processing on the features are as follows: The multimodal feature map is multiplied channel by channel weight vector to obtain the first weighted feature map; The first weighted feature map is multiplied element-wise with the spatial attention mask to obtain the second weighted feature map; The weighted feature map is input into a lightweight convolutional layer for feature refinement, and the final fused feature map is output.

9. The intelligent detection method for defects in medical polymer consumables based on multi-feature fusion according to claim 1, characterized in that, The specific steps for outputting the defect category and location information of the consumable under test based on the fused feature map through the detection head are as follows: The fused feature map is input into the region proposal network to generate candidate regions; Candidate regions with confidence scores below a preset threshold are subjected to secondary discrimination, and the features of the candidate regions from the secondary discrimination are input into an additional classifier. By combining the outputs of the region proposal network and the additional classifier, the defect category and location information of the consumable under test are determined.

10. A multi-feature fusion-based intelligent detection system for defects in medical polymer consumables, employing the multi-feature fusion-based intelligent detection method for defects in medical polymer consumables as described in any one of claims 1-9, characterized in that, include: The data acquisition unit is used to acquire two-dimensional appearance images, infrared transmission images, and three-dimensional point cloud data of the medical polymer consumables to be tested on the production line. The appearance feature extraction unit is used to input a two-dimensional appearance image into a convolutional neural network to extract appearance texture features. The transmission feature extraction unit is used to calculate the attenuation rate of transmitted light intensity relative to the background without consumables for each pixel of the infrared transmission image to generate an attenuation rate map. The horizontal gradient and vertical gradient are calculated for the attenuation rate map to generate a gradient magnitude map. The attenuation rate map and the gradient magnitude map are concatenated into a dual-channel feature map and then input into a convolutional neural network to extract the transmission physical features. The geometric feature extraction unit is used to project 3D point cloud data into a depth map, construct a Gaussian pyramid on the depth map and calculate the surface curvature based on the local neighborhood covariance matrix at each scale, and then stitch the multi-scale curvature map and the depth map into a multi-channel feature map and input it into a convolutional neural network to extract geometric contour features. The feature alignment and fusion unit is used to perform pixel-level spatial alignment of apparent texture features, transmission physical features, and geometric contour features and then input them into the multimodal fusion network. The channel attention module generates global channel weights, the spatial attention module generates a spatial attention mask, and the features are weighted layer by layer before outputting a fused feature map. The defect detection output unit is used to output the defect category and location information of the consumable under test through the detection head based on the fused feature map.

Images