Method for detecting quality of medical catheter coating based on machine vision

Abstract

The present application relates to the technical field of image processing, in particular to a medical catheter coating quality detection method based on machine vision. The method comprises: acquiring an image sequence distributed around the circumference of a medical catheter, three-dimensionally reconstructing the surface of the medical catheter and unfolding it to generate a two-dimensional texture map; generating a normal vector map registered with the two-dimensional texture map; inputting the normal vector map and the two-dimensional texture map as a dual-channel input into a preset defect detection network model for feature extraction and fusion to obtain multi-modal fusion features at different scales; inputting the multi-modal fusion features into a decoder in the defect detection network model to output the position, size and defect type of the coating defects on the medical catheter. The scheme of the present application can comprehensively, reliably and accurately automatically detect the quality of the coating of the medical catheter.

Description

Technical Field

[0001] This invention relates to the field of image processing technology. More specifically, this invention relates to a machine vision-based method for inspecting the coating quality of medical catheters. Background Technology

[0002] Medical catheters, as an important type of interventional medical device, are widely used in many clinical fields such as cardiovascular, urology, and neurology. In order to improve the biocompatibility, lubricity, antibacterial properties, or to achieve specific functions such as targeted drug release, one or more functional coatings are usually coated on the surface of the substrate of medical catheters.

[0003] The quality of the coating directly affects the clinical performance of the catheter and the patient's life safety. Any minor defects, such as scratches, bubbles, particles, peeling, or uneven coating, may lead to device failure or even serious medical accidents such as thrombosis, inflammation, or tissue damage.

[0004] Therefore, comprehensive, accurate, and efficient quality testing of medical catheter coatings is a crucial step in the production process.

[0005] Currently, the mainstream testing method is still manual visual inspection. This method is not only inefficient and costly, but the test results are also easily affected by the inspector's subjective experience and physiological state (such as fatigue), resulting in inconsistent testing standards, high rates of missed and false detections, and difficulty in meeting the needs of modern, large-scale medical device production.

[0006] To address the drawbacks of manual inspection, the industry has begun to introduce automated inspection technologies based on machine vision. Traditional two-dimensional vision inspection solutions typically acquire two-dimensional images of the surface of medical catheters using a camera, and then use image processing algorithms for defect analysis. However, medical catheters usually have a slender cylindrical shape and a smooth, highly reflective surface. Under two-dimensional imaging, it is difficult to avoid problems such as uneven lighting, specular reflection, and shadows. These interfering factors can seriously affect the stable extraction of defect features, easily misjudging reflective spots as defects, or masking true defects due to insufficient lighting. In addition, simple two-dimensional images mainly reflect the color and texture information of the surface, and are very insensitive to defects that only cause local three-dimensional morphological changes (such as slight depressions, protrusions, or uneven thickness), leading to limitations in inspection capabilities.

[0007] To address these issues, some studies have attempted to use multiple cameras or rotating mechanisms to acquire the complete circumferential surface of medical catheters, but this increases system complexity and the difficulty of image stitching. Other approaches, while attempting to introduce 3D measurement techniques, often struggle to balance detection accuracy, speed, and universality across different material coatings. In particular, they fail to effectively integrate the target's 3D geometric features with its 2D appearance features, thus failing to fully leverage the complementary advantages of multimodal information to improve the detection rate and accuracy of complex defects. Summary of the Invention

[0008] The purpose of this invention is to propose a machine vision-based method for inspecting the coating quality of medical catheters, in order to solve the problem that existing technologies cannot perform comprehensive and accurate quality inspection of medical catheters; to this end, this invention provides a solution in one aspect.

[0009] The present invention provides a machine vision-based method for detecting the coating quality of medical catheters, comprising:

[0010] A sequence of images circumferentially distributed around a medical catheter is acquired, and the surface of the medical catheter is reconstructed in three dimensions and unfolded to generate a two-dimensional texture map.

[0011] The surface images of the medical catheter under different lighting directions are acquired, and the gray values ​​of the same pixel position under different lighting directions are combined into an observation vector. Based on the lighting direction and the observation vector, the photometric stereo equations are solved to generate a normal vector map that is registered with the two-dimensional texture map.

[0012] The normal vector map and the two-dimensional texture map are used as dual-channel inputs and fed into a preset defect detection network model for feature extraction and fusion. The defect detection network model includes a normal vector branch and a texture branch to extract first feature maps and second feature maps at multiple scales corresponding to the normal vector map and the two-dimensional texture map, respectively. The first feature maps and second feature maps at the same scale are weighted and fused using an attention mechanism to obtain multimodal fusion features at different scales.

[0013] The multimodal fusion features are input into the decoder in the defect detection network model to output the location, size, and type of defects in the coating of the medical catheter.

[0014] Optionally, the step of acquiring an image sequence distributed circumferentially around the medical catheter, performing three-dimensional reconstruction of the surface of the medical catheter and unfolding its surface to generate a two-dimensional texture map includes:

[0015] The medical catheter is placed on a controllable rotating device and rotated at a preset angular velocity;

[0016] The synchronously triggered line scan camera continuously acquires images of the surface of the rotating medical catheter, forming a complete circumferential image;

[0017] Using pre-calibrated camera parameters, the pixel coordinates of the circular image are converted into three-dimensional point cloud coordinates, thus completing the reconstruction of the three-dimensional point cloud model.

[0018] Using the axis of the medical catheter as a reference, the point cloud data on the three-dimensional point cloud model are expanded using cylindrical coordinates to generate a two-dimensional texture map.

[0019] Optionally, the step of obtaining the normal vector map includes:

[0020] Multiple light sources are arranged at different preset positions around the medical catheter;

[0021] With the medical catheter remaining stationary, the light sources are activated sequentially, and a camera is used to capture surface images under the individual illumination direction of each light source.

[0022] Based on the gray values ​​of the same pixel position in different surface images under different lighting directions, the photometric solid equations are solved to calculate the surface normal vector corresponding to each pixel, thereby forming a normal vector map; the R, G, and B values ​​of each pixel in the normal vector map correspond to the X, Y, and Z components of its surface normal direction, respectively.

[0023] Optionally, both the normal vector branch and the texture branch are encoder structures based on convolutional neural networks, wherein the model parameters of the normal vector branch and the texture branch are not shared.

[0024] Optionally, the step of obtaining the multimodal fusion features includes:

[0025] The attention module is used to process the first and second feature maps at different scales to generate spatial weights.

[0026] The first and second feature maps at the corresponding scales are weighted according to the spatial weights to obtain the multimodal fusion features at the corresponding scales.

[0027] Optionally, the decoder includes at least one multi-scale detection head;

[0028] The multi-scale detection head includes a classification subnetwork and a regression subnetwork. The classification subnetwork is used to identify the defect type to which the defect in the medical catheter belongs, and the regression subnetwork is used to predict the bounding box of the defect to determine its location and size.

[0029] Optionally, the defect types include at least scratches, bubbles, particles, and peeling.

[0030] Optionally, the classification subnetwork and the regression subnetwork are both fully convolutional networks.

[0031] Optionally, the different scales are three scales.

[0032] Optionally, the process of azimuth angle arrangement of the light source is as follows:

[0033] Above and around the medical catheter, with its surface detection area as the center, four LED light sources are arranged in a ring array. The four light sources are symmetrically distributed with respect to the camera optical axis, with azimuth angles of 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively.

[0034] The beneficial effects of this invention are as follows:

[0035] Compared with existing technologies, this invention combines three-dimensional reconstruction with photometric stereo methods to obtain two-dimensional texture maps and normal vector maps of medical catheters. This not only solves the interference problem caused by reflections and shadows when traditional two-dimensional vision detects cylindrical surfaces, but also accurately captures the microscopic three-dimensional morphological features of the coating surface through the normal vector map. Furthermore, this invention uses the appearance information carried by the texture map and the geometric information carried by the normal vector map as dual-channel inputs, and performs parallel feature extraction and weighted fusion using a defect detection network model. This complementary fusion of multimodal information greatly enhances the saliency of defect features, effectively distinguishes between real defects and lighting artifacts, and significantly improves the detection capability and classification accuracy of minor defects such as scratches, pits, and particles that only manifest as local morphological changes. Thus, it achieves comprehensive, reliable, and accurate automated detection of the coating quality of medical catheters. Attached Figure Description

[0036] Figure 1 The flowchart illustrating the steps of the machine vision-based medical catheter coating quality inspection method in this embodiment is shown in the diagram. Detailed Implementation

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0038] like Figure 1 As shown, the machine vision-based medical catheter coating quality inspection method in this embodiment includes the following steps:

[0039] Step S1: Obtain an image sequence of the circumferential distribution of the medical catheter, perform three-dimensional reconstruction on the surface of the medical catheter and unfold its surface to generate a two-dimensional texture map.

[0040] Specifically, the medical catheter is placed on a high-precision rotating platform, and a fixed industrial camera captures an image frame at preset angles, such as 5 degrees, as the catheter rotates at a constant speed, thereby obtaining an image sequence covering the entire circumference of the catheter.

[0041] The image sequence is processed using the Structure of Motion (SFM) algorithm or the Multi-View Stereo (MVS) algorithm to stitch them together into a complete circular image (each row of the circular image corresponds to a cross-section on the circumference of the duct, and each column corresponds to a position on the axial direction of the duct), reconstructing a three-dimensional point cloud model of the duct surface. The three-dimensional point cloud model is then transformed into cylindrical coordinates or parametrically unfolded to map it from a three-dimensional surface to a two-dimensional plane. The pixel information from the image sequence is then filled into the two-dimensional plane according to the mapping relationship, generating a complete and seamless two-dimensional texture map.

[0042] The reconstruction process of the above 3D point cloud model is as follows:

[0043] First, before data acquisition, the external parameters such as the camera's precise position and orientation relative to the rotation axis, as well as the camera's internal parameters such as focal length and distortion, are determined through a calibration process.

[0044] Secondly, after the acquisition is completed, based on the above calibration parameters, the two-dimensional coordinates of each pixel in the circumferential image, namely the row number and column number, are combined with the rotation angle corresponding to the acquisition, and the x, y, z coordinates of each pixel in the three-dimensional Cartesian coordinate system are calculated through trigonometric geometric relationships; after all the pixels are transformed, a three-dimensional point cloud model describing the surface of the duct is formed.

[0045] In one embodiment, point cloud data is converted from a Cartesian coordinate system to a cylindrical coordinate system, using the axis of the medical catheter as a reference (Z-axis). The circumferential angle and axial height of the cylindrical coordinates are then mapped to the horizontal and vertical axes of a two-dimensional image, respectively, to generate a two-dimensional texture map.

[0046] Step S2: Images of the surface of the medical catheter are acquired under preset light sources in different directions, and a normal vector map registered with the two-dimensional texture map is calculated and generated based on the photometric stereo method.

[0047] Specifically, with the camera and medical catheter in constant positions, a ring or dome-shaped LED light source array is used to sequentially illuminate at least three light sources at different positions, and images of the surface of the medical catheter in each illuminated state are acquired.

[0048] For example, four LED light sources are arranged in a ring array above and around the medical catheter, centered on its surface detection area. These four light sources are symmetrically distributed with respect to the camera's optical axis, for example, at azimuth angles of 0 degrees, 90 degrees, 180 degrees, and 270 degrees, respectively, each making a 45-degree angle with the horizontal plane. This arrangement illuminates the surface of the medical catheter from different directions, effectively capturing minute unevenness and texture variations on the catheter's surface. During acquisition, the medical catheter remains stationary, only the first LED light source is turned on, the camera exposes once, and the first illuminated image is acquired. Then, the first LED light source is turned off, the second LED light source is turned on, the camera exposes again, and the second image is acquired. This process is repeated until all four LED light sources are individually illuminated and captured by the camera, ultimately resulting in four surface images that are perfectly aligned at the pixel level but with different illumination directions.

[0049] In this embodiment, the surface normal vector corresponding to each pixel is solved by using the incident light in a known direction and the brightness information of each pixel in the image, and the photometric stereo method (such as the Lambert photometric stereo method) is used to obtain the normal vector map.

[0050] Specifically, for any pixel in the surface image under any illumination direction, its grayscale value in the surface image under all illumination directions constitutes an observation vector; based on the known four illumination directions and observation vector, the photometric solid equation system is established and solved, and the three-dimensional unit normal vector of the medical catheter surface corresponding to any pixel can be accurately calculated, thus obtaining the normal vector map of all pixels.

[0051] Since photometric stereochemistry is an existing technology, it will not be described in detail here.

[0052] Since the camera's viewpoint used to obtain the normal vector map is exactly the same as that used to generate the 2D texture map, the generated normal vector map is precisely aligned with the 2D texture map at the pixel level. The R, G, and B values ​​of each pixel in the normal vector map correspond to the X, Y, and Z components of its surface normal direction, respectively.

[0053] Step S3: Input the normal vector map and the two-dimensional texture map into the defect detection network model, and output the location, size and type of defects on the medical catheter.

[0054] The defect detection network model includes an encoder and a decoder.

[0055] The encoder includes a two-stream convolutional neural network and a channel attention module; the decoder is a multi-scale detection head; the multi-scale detection head includes a classification sub-network and a regression sub-network, the classification sub-network is used to identify the defect type to which the defect belongs, and the regression sub-network is used to predict the bounding box of the defect to determine its location and size.

[0056] The two-stream convolutional neural network includes a normal branch for extracting geometric features of the normal vector map and a texture branch for extracting appearance features of the texture map.

[0057] In one embodiment, both the normal vector branch and the texture branch use a ResNet50 residual network as their backbone encoder. Specifically, the normal vector map is input into the normal vector branch to learn the geometric shape change features caused by surface bumps, scratches, etc., to extract the corresponding first feature map. The two-dimensional texture map is input into the texture branch to learn the appearance texture change features caused by coating color anomalies, foreign particles, etc., to obtain the corresponding second feature map.

[0058] The first and second feature maps extracted above can be feature maps at multiple different scales, or they can be feature maps at a single scale.

[0059] In this process, feature maps at multiple different scales can be extracted from the three stages in the middle of the ResNet50 network, for example, the corresponding feature maps can be extracted from the third, fourth and fifth residual blocks.

[0060] In the above, for both the normal vector branch and the texture branch, three feature maps at different scales were obtained. These feature maps at different scales contain hierarchical information from local details to global contours, providing a basis for subsequent detection of defects of different sizes.

[0061] It should be noted that the network weights and biases of the two branches mentioned above are updated independently during training, that is, the parameters are not shared, thus avoiding mutual interference in the learning process of different modal features.

[0062] The channel attention module uses an attention mechanism to perform weighted fusion of the first feature map and the second feature map at the same scale extracted by the normal vector branch and the texture branch to obtain multimodal fusion features.

[0063] Specifically, after extracting the first and second feature maps at different scales of the normal vector map and the two-dimensional texture map, the first and second feature maps at the same scale are input into a channel attention module, such as the Squeeze-and-Excitation module. The channel attention module assigns different spatial weights to the features of different channels. The weighted first and second feature maps at the same scale are fused by element-wise addition or channel concatenation to obtain multimodal fusion features, and thus a set of multi-scale multimodal fusion features is obtained.

[0064] Furthermore, based on the multimodal fusion features, the coating defects are located and classified by the decoder in the defect detection network model, and the location, size and type information of the defects are output.

[0065] The decoder decodes the fused features to predict the bounding box coordinates and confidence level of the defect target. The coordinates and dimensions of the bounding box represent the location and size of the defect. Simultaneously, the classification branch in the decoder classifies the features within each bounding box, outputting the probability that it belongs to a defect category, such as scratches, bubbles, particles, or peeling. This enables precise defect localization, size measurement, and defect type identification.

[0066] Specifically, for each scale of multimodal fusion features input to the decoder, the classification subnetwork and the regression subnetwork process them in parallel.

[0067] The classification subnetwork is a fully convolutional network that predicts a set of probability scores for each spatial location on the multimodal fusion features, corresponding to a predefined defect category, such as scratches, bubbles, particles, or peeling. The regression subnetwork is also a fully convolutional network that predicts four values ​​for each location, representing the offset of the predicted bounding box center point relative to that location, as well as the height and width of the bounding box.

[0068] Finally, by integrating the prediction results at all scales and performing nonmaximum suppression processing, the specific category of each detected defect, as well as its precise location coordinates and size on the two-dimensional texture map, can be output to achieve the detection of the coating quality of medical catheters.

[0069] The present invention improves the detection capability and classification accuracy of minor defects such as scratches, pits, and particles that only manifest as local morphological changes, thereby realizing comprehensive, reliable and accurate automated detection of the coating quality of medical catheters.

[0070] In the description of this specification, "multiple" means at least two, such as two, three or more, etc., unless otherwise expressly and specifically defined.

[0071] While various embodiments of the invention have been shown and described in this specification, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention.

Claims

1. A machine vision-based method for inspecting the coating quality of medical catheters, characterized in that, include: A sequence of images circumferentially distributed around a medical catheter is acquired, and the surface of the medical catheter is reconstructed in three dimensions and unfolded to generate a two-dimensional texture map. The surface images of the medical catheter under different lighting directions are acquired, and the gray values ​​of the same pixel position under different lighting directions are combined into an observation vector. Based on the lighting direction and the observation vector, the photometric stereo equations are solved to generate a normal vector map that is registered with the two-dimensional texture map. The normal vector map and the two-dimensional texture map are used as dual-channel inputs and fed into a preset defect detection network model for feature extraction and fusion. The defect detection network model includes a normal vector branch and a texture branch to extract first feature maps and second feature maps at multiple scales corresponding to the normal vector map and the two-dimensional texture map, respectively. The first feature maps and second feature maps at the same scale are weighted and fused using an attention mechanism to obtain multimodal fusion features at different scales. The multimodal fusion features are input into the decoder in the defect detection network model to output the location, size, and type of defects in the coating of the medical catheter.

2. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 1, characterized in that, The steps of acquiring an image sequence distributed circumferentially around the medical catheter, reconstructing the surface of the medical catheter in three dimensions and unfolding its surface to generate a two-dimensional texture map include: The medical catheter is placed on a controllable rotating device and rotated at a preset angular velocity; The synchronously triggered line scan camera continuously acquires images of the surface of the rotating medical catheter, forming a complete circumferential image; Using pre-calibrated camera parameters, the pixel coordinates of the circular image are converted into three-dimensional point cloud coordinates, thus completing the reconstruction of the three-dimensional point cloud model. Using the axis of the medical catheter as a reference, the point cloud data on the three-dimensional point cloud model are expanded using cylindrical coordinates to generate a two-dimensional texture map.

3. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 1, characterized in that, The steps for obtaining the normal vector map include: Multiple light sources are arranged at different preset positions around the medical catheter; With the medical catheter remaining stationary, the light sources are activated sequentially, and a camera is used to capture surface images under the individual illumination direction of each light source. Based on the gray values ​​of the same pixel position in different surface images under different lighting directions, the photometric solid equations are solved to calculate the surface normal vector corresponding to each pixel, thereby forming a normal vector map; the R, G, and B values ​​of each pixel in the normal vector map correspond to the X, Y, and Z components of its surface normal direction, respectively.

4. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 1, characterized in that, Both the normal vector branch and the texture branch are encoder structures based on convolutional neural networks, and the model parameters of the normal vector branch and the texture branch are not shared.

5. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 1, characterized in that, The steps for obtaining the multimodal fusion features include: The attention module is used to process the first and second feature maps at different scales to generate spatial weights. The first and second feature maps at the corresponding scales are weighted according to the spatial weights to obtain the multimodal fusion features at the corresponding scales.

6. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 1, characterized in that, The decoder includes at least one multi-scale detection head; The multi-scale detection head includes a classification subnetwork and a regression subnetwork. The classification subnetwork is used to identify the defect type to which the defect in the medical catheter belongs, and the regression subnetwork is used to predict the bounding box of the defect to determine its location and size.

7. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 6, characterized in that, The defect types include at least scratches, bubbles, particles, and peeling.

8. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 6, characterized in that, The classification subnetwork and the regression subnetwork are both fully convolutional networks.

9. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 1, characterized in that, The different scales refer to three scales.

10. The machine vision-based method for inspecting the coating quality of medical catheters according to claim 3, characterized in that, The process of setting the azimuth angle of the light source is as follows: Above and around the medical catheter, with its surface detection area as the center, four LED light sources are arranged in a ring array. The four light sources are symmetrically distributed with respect to the camera optical axis, with azimuth angles of 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively.

Images