Hernia mesh quality defect detection method based on image processing and deep learning

Abstract

This invention discloses a method for detecting quality defects in hernia meshes based on image processing and deep learning. The method comprises the following steps: collecting image data of hernia meshes as raw samples; constructing a production defect detection model for hernia meshes based on a residual network; training the production defect detection model using the raw samples; constructing a physical defect detection model for hernia meshes based on image processing; and using the production defect detection model and the physical defect detection model to detect the collected image data of the hernia mesh to be tested, and outputting a quality detection result. This invention effectively solves the problems of low accuracy, high cost, and low efficiency of manual detection, and achieves good detection results on actual data.

Description

Technical Field

[0001] The present invention belongs to the field of medical device production and testing, and specifically relates to a hernia patch quality defect detection method based on image processing and deep learning. Background Art

[0002] Modern tension-free repair surgery is currently the main method for treating hernias. Hernia patches have been widely used in tension-free hernia repair surgery, and the quality of hernia patches directly affects the effectiveness of hernia treatment. Hernia patch testing indicators include two parts: one is to detect the physical properties of the hernia patch itself, such as porosity, pore size, etc., and the other is to detect whether there are defects caused by external factors, such as missed stitches (inside the patch, on the edge of the patch, oil stains, etc.). In industrial production, defect detection of hernia patches is mainly manual inspection, which is inevitably prone to misjudgment due to visual fatigue; secondly, manual inspection is easily affected by external interference, and detection efficiency is difficult to guarantee; at the same time, it also brings higher labor costs.

[0003] Therefore, there is an urgent need to establish a quality defect detection method for hernia patches so that unqualified products can be effectively and quickly identified from the produced hernia patches. Summary of the Invention

[0004] The purpose of the present invention is to provide a hernia patch quality defect detection method based on image processing and deep learning, which effectively solves the problems of low accuracy, high labor cost and low efficiency of manual detection, and achieves good detection results on actual data.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for detecting quality defects of hernia patches based on image processing and deep learning, comprising:

[0007] Step 1: collecting image data of hernia patch as original samples;

[0008] Step 2: Build a hernia patch production defect detection model based on the residual network;

[0009] Step 3: Using original samples to train the production defect detection model;

[0010] Step 4: Construct a physical defect detection model for hernia patch based on image processing;

[0011] Step 5: Detect the collected image data of the hernia patch to be tested using the production defect detection model and the physical defect detection model, and output a quality detection result, including:

[0012] Step 5-1: input the collected image data of the hernia mesh to be tested into the production defect detection model. If the production defect detection model outputs an unqualified result, the test result of the current hernia mesh to be tested is unqualified and the test is terminated; otherwise, the current hernia mesh to be tested is regarded as a pre-qualified hernia mesh and proceeds to the next step;

[0013] Step 5-2: Input the image data of the pre-qualified hernia patch into the physical defect detection model. If the output of the physical defect detection model is unqualified, the detection result of the hernia patch to be tested is unqualified, and the detection is ended; otherwise, the detection result of the hernia patch to be tested is qualified, and the detection is ended.

[0014] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0015] Preferably, the method for constructing a hernia patch production defect detection model based on a residual network includes:

[0016] Step 2-1, constructing a feature extraction network pair to extract a feature matrix of the input hernia patch image data, wherein the feature extraction network pair includes a first feature extraction network and a second feature extraction network with dynamic weight sharing, wherein the first feature extraction network and the second feature extraction network have the same network structure, and the network structure is composed of a convolutional layer, a maximum pooling layer, multiple convolution groups and an average pooling layer, each convolution group is composed of a backbone and a residual structure, wherein the backbone is composed of two convolutional layers, and the residual structure is directly connected from the input to the output;

[0017] Step 2-2, build a classification network, and output the current detection category based on the feature matrix of the hernia patch extracted by the first feature extraction network. The classification network consists of a fully connected layer and an AM-softmax layer. The input feature matrix is converted into a one-dimensional feature vector through the fully connected layer to obtain the low-dimensional features of the hernia patch data. The feature vector is used as the input of the AM-softmax layer to output the category and probability to which the current hernia patch belongs, and the category is qualified or unqualified.

[0018] Preferably, the training of the production defect detection model using original samples includes:

[0019] The production defect detection model is first trained using a public data set, and then transfer learning is adopted to train the production defect detection model that has been trained using the public data set using original samples.

[0020] Preferably, the training of the production defect detection model using original samples includes:

[0021] The image data of qualified hernia meshes in the original sample are taken as the first training set, and the image data of unqualified hernia meshes in the original sample are taken as the second training set;

[0022] The image data in the first training set are sequentially input into the first feature extraction network, and the image data in the second training set are sequentially input into the second feature extraction network;

[0023] Calculating the Euclidean distance between the first hernia patch image feature output by the first feature extraction network and the second hernia patch image feature output by the second feature extraction network, and calculating the contrast loss function based on the Euclidean distance;

[0024] Inputting the first hernia patch image feature output by the first feature extraction network into the classification network, and calculating the AM-softmax loss function as the classification loss function according to the output result of the classification network;

[0025] The aggregation contrast loss function and the classification loss function are used as the total loss function of the production defect detection model to update the model parameters and realize the training of the production defect detection model.

[0026] Preferably, the contrast loss function is as follows:

[0027]

[0028]

[0029] Where L(W,(Y,X1,X2)) represents the contrast loss value, N represents the total number of samples, X1 and X2 are the sample pairs input for comparison, which are the first hernia patch image features and the second hernia patch image features, respectively. Y represents the label of the sample pair, the label of the positive sample pair is 1, and the label of the negative sample pair is 0. W represents the Euclidean distance between the two hernia patch image features in the sample pair, margin is the set threshold, which represents the maximum distance between the two hernia patch image features in the negative sample pair, and P is the data dimension of the hernia patch image feature. represents the rth element in the first hernia patch image feature, Represents the rth element in the second hernia patch image feature.

[0030] Preferably, the total loss function is as follows:

[0031] Loss=λ1Loss ctt +λ2Loss cls

[0032] In the formula, Loss is the total loss function, Loss ctt is the contrast loss function, Loss cls is the classification loss function, λ1 and λ2 represent two loss weights respectively.

[0033] Preferably, the method of constructing a physical defect detection model for a hernia patch based on image processing includes:

[0034] Step 4-1, image preprocessing: convert the image data of the hernia patch from BGR color space to grayscale space, and then perform binarization processing according to a preset threshold;

[0035] Step 4-2, image feature extraction: extract all contours of the hernia patch, and calculate the contour area and contour perimeter of the hernia patch based on the contours;

[0036] Step 4-3, physical feature comparison: Calculate the porosity of the hernia patch based on the contour area, and calculate the pore size of the hernia patch based on the contour circumference. If the calculated porosity and pore size of the current hernia patch meet the physical characteristics of the standard hernia patch, the current hernia patch is qualified; otherwise, it is unqualified.

[0037] Preferably, the step of calculating the contour area and contour perimeter of the hernia patch according to the contour comprises:

[0038] The specific formula for calculating the contour area is:

[0039]

[0040] Where S i is the contour area of contour i, K i is the number of pixels contained in contour i, is the area of pixel j within contour i, x j is the horizontal coordinate of pixel point j in the coordinate system, y (j+1)％(Ki+1) is the pixel point (j+1)%(K i +1) The vertical coordinate in the coordinate system, y (j-1)％(Ki+1) is pixel (j-1)%(K i +1) in the coordinate system, i∈{1,2,3,…,I}, j∈{1,2,3,…,K i}, I is the total number of contours extracted from the current hernia patch;

[0041] The specific formula for calculating the contour perimeter is:

[0042]

[0043] Where C i is the perimeter of contour i, xj is the horizontal coordinate of pixel point j in the coordinate system, y j is the vertical coordinate of pixel point j in the coordinate system, x j+1 is the horizontal coordinate of pixel j+1 in the coordinate system, y j+1 is the vertical coordinate of pixel j+1 in the coordinate system.

[0044] Preferably, the method of calculating the porosity corresponding to the hernia patch according to the contour area and calculating the pore size corresponding to the hernia patch according to the contour perimeter includes:

[0045] The specific formula for calculating porosity is:

[0046]

[0047] Where Ratio is the porosity of the hernia patch, A is the sum of the contour areas of the hernia patch, and S max is the maximum contour area of the hernia mesh;

[0048] The specific formula for calculating the aperture is:

[0049]

[0050] Where, Radius i is the aperture of hernia patch profile i, and λ is the aperture conversion weight.

[0051] The hernia patch quality defect detection method based on image processing and deep learning provided by the present invention has the following beneficial effects compared with the existing technology:

[0052] 1) A production defect detection model for hernia patches was constructed based on the residual network. By utilizing the excellent nonlinear mapping ability of deep learning between input and output data, the quality problems of hernia patches produced during the processing were successfully detected without overly complex calculations, thus achieving the first-level defect detection of hernia patches.

[0053] 2) A physical defect detection model for hernia patches was constructed using image processing technology. It has a good quality detection effect for a single sample image and successfully detects quality problems caused by the physical properties of the hernia patches encountered during the processing, thus realizing the second-level defect detection of hernia patches.

[0054] 3) The multi-level defect detection method for hernia patches that combines deep learning and image processing technology achieves high-accuracy detection of hernia patches under the combined action of production defect detection and physical defect detection.

[0055] 4) The application of transfer learning in the training of multi-level detection models effectively solved the small sample problem of supervised learning methods in hernia patch detection tasks. At the same time, auxiliary training was introduced to dynamically share weights with the main training, thereby improving the accuracy and efficiency of the training model.

[0056] 5) In the design of the loss function, the AM-softmax loss function and the contrast loss function are added to maximize the inter-class differences and minimize the intra-class differences, making the extracted features more accurate and improving the performance of the detection model.

[0057] 6) The proposed multi-level defect detection model for hernia patches has achieved high defect recognition accuracy and detection rate on the existing hernia patch automated production line. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Flowchart of the hernia patch quality defect detection method based on image processing and deep learning of the present invention;

[0059] Figure 2 Schematic diagram of the structure of the feature extraction network of the present invention;

[0060] Figure 3 This is a schematic diagram of the structure of the classification network of the present invention;

[0061] Figure 4 This is a flow chart of the execution of the physical defect detection model of the present invention;

[0062] Figure 5 This is a diagram showing the effect of the Grad-CAM algorithm of the present invention on generating attention for test cases. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0065] In order to overcome the defects of hernia mesh detection in the prior art, this embodiment proposes a hernia mesh quality defect detection method based on the combination of image processing and deep learning. The specific steps are as follows:

[0066] Step 1: Collect image data of hernia patch as original samples.

[0067] The image acquisition system was constructed using an MV-CS016-10GM / GC camera, an MVL-HF1228M-6MPE lens, and a ring light source. A light-absorbing cloth was placed on the acquisition surface to reduce noise. The captured hernia mesh images included both qualified and unqualified ones (those with substandard aperture, substandard porosity, missing stitches at the edges, missing stitches within the fabric, and oil stains). These image sets served as the raw sample for inspection and analysis.

[0068] Step 2: Establish a hernia patch production defect detection model based on the residual network.

[0069] like Figure 1 As shown in the figure, the production defect detection model is the first-level detection model. It takes the original hernia mesh image as input, extracts features through the feature extraction network, and then classifies the extracted features. The output is the quality category of the input hernia mesh, that is, qualified (qualified hernia mesh) or unqualified (unqualified hernia mesh). The specific steps for model establishment are as follows:

[0070] Step 2-1: Construct a feature extraction network pair to extract the feature matrix of the input hernia patch image data.

[0071] The feature extraction network pair of this embodiment includes a first feature extraction network (feature extraction network 1) and a second feature extraction network (feature extraction network 2) with dynamic weight sharing. The first feature extraction network and the second feature extraction network have the same network structure. The specific structure is as follows: Figure 2 As shown, the network consists of a convolutional layer, a maximum pooling layer, multiple convolution groups, and an average pooling layer. Each convolution group consists of a backbone and a residual structure. The backbone consists of two convolutional layers. The residual structure directly connects the input to the output, adding residuals and solving the degradation problem. Finally, the corresponding feature matrix is output through the average pooling layer. For ease of application, the feature extraction network in this embodiment adopts the existing resnet34 network structure, such as the network structure disclosed in "Deep Residual Learning for Image Recognition".

[0072] Step 2-2: Build a classification network and output the currently detected category based on the hernia patch feature matrix extracted by the first feature extraction network.

[0073] The structure of the classification network in this embodiment is as follows Figure 3As shown in the figure, it consists of a fully connected layer and an improved Softmax layer (i.e., AM-softmax layer). The AM-softmax layer uses the strategy of reducing the intra-class distance and increasing the inter-class distance. The input feature matrix is converted into a one-dimensional feature vector through the fully connected layer to obtain the low-dimensional features of the hernia patch data. Then, the output of the intermediate output layer is converted through the AM-softmax layer and expressed in the form of probability, that is, the probability of the different categories to which the object to be detected belongs is obtained. The AM-softmax function can be used to convert the output values of multiple categories into a probability distribution in the range of [0,1] and 1.

[0074] Step 3: Use original samples to train the production defect detection model.

[0075] Step 3-1: Build a training dataset.

[0076] The original samples collected in step 1 are split into a training set and a validation set at a ratio of 9:1. The residual network model is trained using the training set and validated using the validation set. In this embodiment, the production defect detection model is first trained using a public dataset, and then transfer learning is used to further train the production defect detection model that has been trained using the public dataset using the original samples.

[0077] Step 3-2: Take the image data of qualified hernia patches in the original sample as the first training set, and take the image data of unqualified hernia patches in the original sample as the second training set. The image data in the first training set are sequentially input into the first feature extraction network, and the image data in the second training set are sequentially input into the second feature extraction network.

[0078] The first-level detection model of this embodiment uses a group of original hernia patch images of the superior category as the input of feature extraction network 1, and a group of original hernia patch images of the inferior category as the input of feature extraction network 2 (because the images of original hernia patches of the inferior category account for the majority in the data set, they are input into feature extraction network 2 for auxiliary comparison to ensure the richness of the represented visual information, so that the model can learn more essential features to distinguish hernia patches). The model trains two hernia patch feature extraction networks with dynamically shared weights, and then sends the output hernia patch features to the classification network, and the output is the quality category of the input hernia patch, that is, qualified (qualified hernia patch) or unqualified (unqualified hernia patch).

[0079] Step 3-3, calculate the Euclidean distance between the first hernia patch image feature output by the first feature extraction network and the second hernia patch image feature output by the second feature extraction network, and calculate the contrast loss function based on the Euclidean distance; input the first hernia patch image feature output by the first feature extraction network into the classification network, and calculate the AM-softmax loss function as the classification loss function according to the output result of the classification network; aggregate the contrast loss function and the classification loss function as the total loss function of the production defect detection model to update the model parameters and realize the training of the production defect detection model.

[0080] Design the total loss function: The loss function used by the first-level detection model is as follows: Figure 1 As shown in , the total loss function is designed as follows:

[0081] Loss=λ1Loss ctt +λ2Loss cls

[0082] In the formula, Loss is the total loss function, Loss ctt is the contrast loss function, Loss cls is the classification loss function, λ1 and λ2 represent two adjustable loss weights.

[0083] Among them, the contrast loss function Loss ctt The specific definition is as follows:

[0084]

[0085]

[0086] Where L(W,(Y,X1,X2)) represents the contrast loss value, N represents the total number of samples, X1 and X2 are the sample pairs input for comparison, Y represents the label of the sample pair, the label of the positive sample pair (the sample pair contains the same substance) is 1, and the label of the negative sample pair (the sample pair contains different substances) is 0. W It represents the Euclidean distance between the two hernia patch image features in the sample pair. Margin is the set threshold, which represents the maximum distance between the two hernia patch image features in the negative sample pair. It is generally set to 1. P is the data dimension of the hernia patch image feature. represents the rth element in the first hernia patch image feature, Represents the rth element in the second hernia patch image feature.

[0087] Among them, the classification loss function Loss cls It is obtained by the conventional AM-softmax loss function, as follows:

[0088]

[0089] Where N is the total number of samples, s is the cosine scaling factor, and cosθ yi Convert the input hernia patch image feature i to category y i The cosine value of the angle, m is the angle parameter, is the preset value, c is the number of classification categories, is the weight matrix The transpose of is the weight matrix W j The transpose of i is the output of the hernia patch image feature i (i.e., the feature of the i-th sample) after the fully connected layer, i∈{1,2,3,…,N}.

[0090] Step 3-4: Use the stochastic gradient descent (SGD) method to train the network. During the training process, when the loss converges and no longer changes significantly within 5 consecutive iterations, stop the model training.

[0091] In addition, after each optimization of the model parameters, dynamic weight sharing is performed between the first feature extraction network and the second feature extraction network. This embodiment provides a dynamic weight sharing method as follows:

[0092] In the first and second iterations, the first feature extraction network and the second feature extraction network update their weight parameters respectively. From the third iteration, the initial weight parameters of the third iteration are obtained based on the weight parameters updated in the first and second iterations, and the weight parameters updated in the third iteration are applied to the initial weight parameters of the fourth iteration, and so on. For example, the initial weight parameters of the first feature extraction network and the second feature extraction network from the third time onwards are w′1(n)=w2(n-1)*0.8+w2(n-2)*0.2, w′2(n)=w1(n-1)*0.8+w1(n-2)*0.2, n≥3, wherein w′1(n) and w′2(n) are the initial weight parameters of the first feature extraction network and the second feature extraction network at the nth iteration, respectively, w1(n-1) and w2(n-1) are the updated weight parameters of the first feature extraction network and the second feature extraction network at the n-1th iteration, respectively, and w1(n-2) and w2(n-2) are the updated weight parameters of the first feature extraction network and the second feature extraction network at the n-2th iteration, respectively.

[0093] Step 4: Establish a hernia patch physical defect detection model based on image processing technology.

[0094] The physical defect detection model is a secondary model of the multi-level defect detection model, such as Figure 4As shown in the figure, the model uses pre-qualified hernia mesh images detected by the production defect detection model as input, and further image processing methods are used to achieve higher-level defect detection of hernia meshes. First, the physical features of the hernia mesh images are calculated. Next, these physical features are compared with the corresponding features of standard hernia meshes. The model outputs the hernia mesh's qualified category, i.e., qualified hernia mesh or unqualified hernia mesh.

[0095] Step 4-1: Image preprocessing.

[0096] The pre-qualified hernia patch images retained by the production defect detection model are preprocessed by converting the images from BGR color space to grayscale space, and then binarized according to a pre-set threshold to facilitate subsequent feature extraction.

[0097] The specific formula for converting BGR color space to grayscale space is:

[0098] Gray=0.1140*B+0.5870*G+0.2989*R

[0099] Where Gray is the pixel value in the converted grayscale space, B is the blue channel value, B∈[0,255], G is the green channel value, G∈[0,255], and R is the red channel value, R∈[0,255].

[0100] Step 4-2: Image feature extraction.

[0101] To detect physical defects, it is necessary to extract characteristic parameters of the hernia patch, including contour area and characteristic perimeter. This embodiment uses the cv2.findContours function in the OpenCV interface to find the contour of the hernia patch. The logic part uses a pre-processed, pre-qualified hernia patch image as input, sets the contour retrieval mode parameter to RETR_TREE, retrieves all contours, and establishes a hierarchical tree structure of contours. The corresponding contour approximation method parameter is set to CHAIN_APPROX_SIMPLE, compresses the elements in the horizontal, vertical, and diagonal directions, and only retains the end point coordinates in that direction. The return value of the logic is the corresponding contour.

[0102] The area and perimeter of the obtained contour are then calculated. Given the contour as an irregular image, the contour is divided into groups of three coordinate points, and the internal area of the connected pixels is calculated and summed to obtain the corresponding contour area. The arcLength function (arc length function) is then used to calculate the corresponding contour perimeter. The corresponding indicators are calculated using the physical properties obtained above.

[0103] The specific formula for calculating the contour area function is:

[0104]

[0105] Where S i is the contour area of contour i, K i is the number of pixels contained in contour i, is the area of pixel j within contour i, i∈{1,2,3,…,I}, j∈{1,2,3,…,K i}, I is the total number of contours extracted from the current hernia patch, x i is the horizontal coordinate of pixel i in the coordinate system, is pixel (i+1)%(K i +1) in the coordinate system, is pixel (i-1)%(K i +1) in the coordinate system.

[0106] The specific formula for calculating the contour perimeter function is:

[0107]

[0108] Where C i is the perimeter of contour i, x j is the horizontal coordinate of pixel j in the coordinate system, y j is the vertical coordinate of pixel j in the coordinate system, x j+1 is the horizontal coordinate of pixel j+1 in the coordinate system, y j+1 is the vertical coordinate of pixel j+1 in the coordinate system.

[0109] Step 4-3: Comparison of physical characteristics.

[0110] The physical characteristics of prequalified hernia meshes are compared against those of standard hernia meshes. If any characteristic deviates, the mesh is deemed unqualified. Specifically, the system uses the extracted contour area and perimeter features from the corresponding hernia mesh image as input. The porosity is calculated from the contour area, and the pore size is calculated from the contour perimeter. Based on the resulting physical characteristics, the system outputs a quality score for the current hernia mesh.

[0111] The specific formula for calculating porosity is:

[0112]

[0113] Where Ratio is the porosity of the hernia patch, A is the sum of the contour areas of the hernia patch, and S max is the maximum contour area of the hernia mesh.

[0114] The specific formula for calculating the aperture is:

[0115]

[0116] Where, Radius i is the aperture of hernia patch contour i (in mm), and λ is the aperture conversion weight, which is calculated based on the hardware distance measurement. The aperture conversion weight is determined by the distance between the industrial camera and the hernia patch in the actual production environment. When the camera used for image acquisition determines the distance, λ is usually taken as λ = distance * k, where distance is the distance between the industrial camera and the hernia patch in meters, and k is a preset coefficient with an adjustable value, for example, 0.57.

[0117] Step 5: Apply a multi-level hernia patch detection model based on deep learning and image processing.

[0118] In actual inspection applications, the two inspection models are combined to achieve multi-level inspection of hernia meshes, further improving inspection accuracy. First, production defect detection of hernia meshes is implemented based on deep learning. Next, image processing technology is used to perform physical defect inspection on pre-qualified hernia meshes retained at the previous level, ultimately determining the quality category of the hernia mesh. The specific workflow is as follows:

[0119] Step 5-1: Input the collected hernia patch image data into the multi-level defect detection model.

[0120] Step 5-2: The multi-level detection model will be executed sequentially. First, it will pass through any feature extraction network in the hernia mesh production defect detection model to obtain the corresponding low-dimensional hernia mesh features. The low-dimensional hernia mesh features are input into the classification network, and the probability of the detected substance belonging to different categories is output. If the probability is unqualified, the test result of the current hernia mesh to be tested is unqualified and the test ends; otherwise, the current hernia mesh to be tested is regarded as a pre-qualified hernia mesh and proceeds to the next step;

[0121] Step 5-3: Input the image data of the pre-qualified hernia mesh into the physical defect detection model. The input hernia mesh image is first preprocessed and its physical properties, including porosity and pore size, are extracted. Finally, the physical characteristics of the mesh are compared with those of a standard hernia mesh to determine its compliance and output the result. If the physical defect detection model outputs a failure, the current hernia mesh to be tested is considered unqualified and the test ends. Otherwise, the current hernia mesh to be tested is considered qualified and the test ends.

[0122] In the physical feature comparison, the physical features of the standard hernia patch can be set according to the actual production requirements. A hernia patch is considered qualified if it is within the allowable error of the physical features of the standard hernia patch (for example, within plus or minus 0.01 mm of the standard aperture). If one item is not met, it is an unqualified hernia patch.

[0123] The effect of the present invention is demonstrated below through a specific example.

[0124] The hernia mesh quality defect detection method based on the combination of image processing and deep learning can quickly measure unclassified hernia meshes. The process is as follows:

[0125] (1) The MV-CS016-10GM / GC camera, MVL-HF1228M-6MPE lens, ring light source and other equipment were used to build a data acquisition platform, and light-absorbing cloth was laid on the acquisition surface to reduce the impact of noise.

[0126] (2) Obtain several hernia patches, including qualified hernia patches and unqualified hernia patches (the pore size does not meet the standard, the porosity does not meet the standard, the stitches are missing at the edge of the fabric, the stitches are missing inside the fabric, and there are oil stains), and use them as original samples for data collection through the data collection platform.

[0127] (3) The collected data is preprocessed and the image is cropped to solve the problem of large background noise caused by the large field of view; image enhancement is performed by rotation, noise addition, resizing, etc., so that the sample size is increased, which greatly increases the sample data used for model training.

[0128] (4) The pre-processed image labels are set as qualified hernia mesh and unqualified hernia mesh, and divided into training set and validation set at a ratio of 9:1. The training set is input into the residual network training model for training, and the validation set is put into the trained model for verification. Figure 5 As shown in the figure, it is a heat map drawn after the hernia patch production defect detection model and verification samples are input into Grad-CAM (corresponding to the given category, which areas the network actually focuses on).

[0129] (5) The production defect detection model provided in this application was taken as the experimental group, and the existing resnet34 classification model was taken as the control group. A 10-fold cross-validation was performed on the experimental group and the control group using the training set and the validation set. The experimental comparison results are shown in Table 1 (including Table 1).

[0130] Table 1 Experimental comparison results

[0131] Number of cross validations 1 2 3 4 5 6 Accuracy of the experimental group 96.01％ 98.04％ 95.24％ 95.46％ 96.1％ 96.53％ Accuracy of the control group 94.12％ 86.27％ 92.16％ 90.2％ 92.16％ 94％

[0132] Table 1 Experimental comparison results

[0133] Number of cross validations 7 8 9 10 Mean Accuracy of the experimental group 96.32％ 96.6％ 95.56％ 96％ 96.18％ Accuracy of the control group 92％ 98％ 96％ 94％ 92.89％

[0134] As can be seen from the above table, the production defect detection model proposed in this application is superior to the traditional model in hernia patch defect detection. The average accuracy obtained by model training is 96.18%. The model is credible in the task of production defect detection based on self-built hernia patches.

[0135] (6) The test sample is input into the multi-level hernia mesh quality classification model based on the combination of deep learning and image processing technology. First, it will pass through the hernia mesh production defect detection module to detect the production defects. If the probability of the hernia mesh being unqualified is obtained, the operation output result will be immediately terminated; otherwise, the execution will continue. Then, the hernia mesh physical defect detection model will pre-process the input hernia mesh image to obtain the physical properties. After calculation by formula, the corresponding category is obtained by comparison with the standard value, and the detection accuracy of the multi-level detection model is calculated.

[0136] The detection accuracy of the model Acc is defined as follows:

[0137]

[0138] (7) The hardware configuration of the test environment is GPU: GeForce RTX 2080 Ti; CPU: Intel Xeon Gold6132@2.60GHZ. The detection model achieved an accuracy of 96.0% by detecting an untrained blind category set, and the average detection time for each hernia patch image was 60ms.

[0139] In summary, this example provides a method for detecting quality defects of hernia patches based on a combination of image processing technology and deep learning, which realizes the rapid and accurate screening of the quality of hernia patches in the field of medical device production. The method for detecting quality defects of hernia patches mainly includes primary production defect detection and secondary physical defect detection. The specific operation steps are as follows: first, based on the transfer learning method, the residual network model is trained with the hernia patch image data that has been preprocessed and image enhanced, and the test samples are input into the trained primary production defect detection model to detect the production defects of the samples. If the test finds that there are production defects, the test process is exited and the unqualified samples are eliminated. Otherwise, the secondary test is continued. Next, the physical properties of the pre-qualified hernia patches are tested using image processing technology, and finally the result of the secondary physical defect test is the final quality test result.

[0140] Under the appropriate test environment, the detection model achieved excellent results in classifying hernia mesh quality, achieving a 96.0% detection rate for the blind class set and an average processing time of 60ms per image. Furthermore, heatmap analysis confirmed that the corresponding multi-level detection model had good class attention. Therefore, this method enables rapid and accurate detection and classification of hernia mesh quality in complex production environments.

[0141] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for detecting quality defects of hernia patches based on image processing and deep learning, characterized in that: The hernia patch quality defect detection method based on image processing and deep learning includes: Step 1: collecting image data of hernia patch as original samples; Step 2: Build a hernia patch production defect detection model based on the residual network; Step 3: Using original samples to train the production defect detection model; Step 4: Construct a physical defect detection model for hernia patch based on image processing; Step 5: Detect the collected image data of the hernia patch to be tested using the production defect detection model and the physical defect detection model, and output a quality detection result, including: Step 5-1: input the collected image data of the hernia mesh to be tested into the production defect detection model. If the production defect detection model outputs an unqualified result, the test result of the current hernia mesh to be tested is unqualified and the test is terminated; otherwise, the current hernia mesh to be tested is regarded as a pre-qualified hernia mesh and proceeds to the next step; Step 5-2: Input the image data of the pre-qualified hernia patch into the physical defect detection model. If the output of the physical defect detection model is unqualified, the detection result of the hernia patch to be tested is unqualified, and the detection is ended; otherwise, the detection result of the hernia patch to be tested is qualified, and the detection is ended.

2. The hernia patch quality defect detection method based on image processing and deep learning according to claim 1, characterized in that: The hernia patch production defect detection model based on the residual network is constructed, including: Step 2-1, constructing a feature extraction network pair to extract a feature matrix of the input hernia patch image data, wherein the feature extraction network pair includes a first feature extraction network and a second feature extraction network with dynamic weight sharing, wherein the first feature extraction network and the second feature extraction network have the same network structure, and the network structure is composed of a convolutional layer, a maximum pooling layer, multiple convolution groups and an average pooling layer, each convolution group is composed of a backbone and a residual structure, wherein the backbone is composed of two convolutional layers, and the residual structure is directly connected from the input to the output; Step 2-2, build a classification network, and output the current detection category based on the feature matrix of the hernia patch extracted by the first feature extraction network. The classification network consists of a fully connected layer and an AM-softmax layer. The input feature matrix is converted into a one-dimensional feature vector through the fully connected layer to obtain the low-dimensional features of the hernia patch data. The feature vector is used as the input of the AM-softmax layer to output the category and probability to which the current hernia patch belongs, and the category is qualified or unqualified.

3. The hernia patch quality defect detection method based on image processing and deep learning according to claim 2, characterized in that: The training of the production defect detection model using original samples includes: The production defect detection model is first trained using a public data set, and then transfer learning is adopted to train the production defect detection model that has been trained using the public data set using original samples.

4. The hernia patch quality defect detection method based on image processing and deep learning according to claim 2, characterized in that: The training of the production defect detection model using original samples includes: The image data of qualified hernia meshes in the original sample are taken as the first training set, and the image data of unqualified hernia meshes in the original sample are taken as the second training set; The image data in the first training set are sequentially input into the first feature extraction network, and the image data in the second training set are sequentially input into the second feature extraction network; Calculating the Euclidean distance between the first hernia patch image feature output by the first feature extraction network and the second hernia patch image feature output by the second feature extraction network, and calculating the contrast loss function based on the Euclidean distance; Inputting the first hernia patch image feature output by the first feature extraction network into the classification network, and calculating the AM-softmax loss function as the classification loss function according to the output result of the classification network; The aggregation contrast loss function and the classification loss function are used as the total loss function of the production defect detection model to update the model parameters and realize the training of the production defect detection model.

5. The hernia patch quality defect detection method based on image processing and deep learning according to claim 4, characterized in that: The contrast loss function is as follows: ； ； Where, represents the contrast loss value, represents the total number of sample pairs, represents the sample pair number, The sample pairs for comparison are input, namely the first hernia patch image feature and the second hernia patch image feature, Represents the label of the sample pair, the label of the positive sample pair is 1, and the label of the negative sample pair is 0. represents the Euclidean distance between the two hernia patch image features in the sample pair, is the set threshold, which represents the maximum distance between the two hernia patch image features in the negative sample pair. is the data dimension of the hernia patch image feature, represents the rth element in the first hernia patch image feature, Represents the rth element in the second hernia patch image feature.

6. The hernia patch quality defect detection method based on image processing and deep learning according to claim 4, characterized in that: The total loss function is as follows: ； Where, is the total loss function, is the contrast loss function, is the classification loss function, and Represent two loss weights respectively.

7. The hernia patch quality defect detection method based on image processing and deep learning according to claim 1, characterized in that: The method of constructing a physical defect detection model for a hernia patch based on image processing includes: Step 4-1, image preprocessing: convert the image data of the hernia patch from BGR color space to grayscale space, and then perform binarization processing according to a preset threshold; Step 4-2, image feature extraction: extract all contours of the hernia patch, and calculate the contour area and contour perimeter of the hernia patch based on the contours; Step 4-3, physical feature comparison: Calculate the porosity of the hernia patch based on the contour area, and calculate the pore size of the hernia patch based on the contour circumference. If the calculated porosity and pore size of the current hernia patch meet the physical characteristics of the standard hernia patch, the current hernia patch is qualified; otherwise, it is unqualified.

8. The hernia patch quality defect detection method based on image processing and deep learning according to claim 7, characterized in that: The method of calculating the contour area and contour perimeter of the hernia patch according to the contour includes: The specific formula for calculating the contour area is: ； Where, For outline The contour area, For outline The number of pixels included, Pixel In outline The area inside, Pixel The horizontal coordinate in the coordinate system, Pixel The vertical coordinate in the coordinate system, Pixel The vertical coordinate in the coordinate system, , , The total number of contours extracted for the current hernia mesh; The specific formula for calculating the contour perimeter is: ； Where, For outline The perimeter of the contour, Pixel The horizontal coordinate in the coordinate system, Pixel The vertical coordinate in the coordinate system, Pixel The horizontal coordinate in the coordinate system, Pixel The vertical coordinate in the coordinate system.

9. The method for detecting quality defects of hernia patches based on image processing and deep learning according to claim 8, wherein: Calculating the porosity of the hernia patch according to the contour area and calculating the pore size of the hernia patch according to the contour perimeter include: The specific formula for calculating porosity is: ； Where, is the porosity of the hernia patch, represents the sum of the contour areas of the hernia mesh, is the maximum contour area of the hernia mesh; The specific formula for calculating the aperture is: ； Where, Contour for hernia mesh The aperture, The aperture conversion weight.

Images