Syringe Scale Defect Detection Method Based on Deep Learning and Scale Grouping Matching

Abstract

The present invention discloses a syringe scale defect detection method based on deep learning and scale grouping matching, which includes the following steps: Step 1, place the syringe to be tested on the conveying mechanism, and the image sensor obtains images of a number of syringes in real time within the field of view, and select the syringe image containing the complete syringe from them; Step 2, use a deep neural network to extract the syringe scale from the syringe image; Step 3, based on the extracted syringe scale, use the scale grouping allocation algorithm to detect the defects on the scale. This syringe scale defect detection method can overcome the problems of high cost, low accuracy, long development cycle, and dependence on defect samples existing in the existing syringe scale defect detection methods, and can achieve accurate detection of syringe scale defects in the case of only a small number of normal syringe samples.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical device detection devices, and particularly relates to a syringe scale defect detection method based on the combination of deep learning and a scale grouping matching algorithm. Background Art

[0002] In the production and manufacturing process of syringes, defects will inevitably occur, and syringe scale defects are a common type among them. Syringes with scale defects will seriously affect the judgment of medical staff on the dosage of injected drugs, thereby endangering the health of patients. Therefore, the detection of syringe scale defects is very necessary. Syringe scale defects can be roughly divided into two categories. One category is frequent defects, such as scale breakage, abnormal scale length, abnormal scale width, scale missing, and scale redundancy, etc. The second category is occasional defects, referring to other defects that do not belong to frequent defects, such as accidental defects caused by external interference, operation errors, occasional failures, and other problems.

[0003] Currently, syringe manufacturers mainly detect scale defects through two methods: human eyes or machine vision defect detection devices. The human eye detection method not only has low detection efficiency and high cost, but also is difficult to meet the large-scale production requirements of syringes. Machine vision defect detection devices can be further divided into two types: traditional machine vision detection devices and deep learning detection devices. Traditional machine vision detection devices construct features manually to describe product surface defects, capture defect features in syringe images, and then complete defect detection. This type of method requires a separate design of feature description methods for each type of defect, relies too much on the development experience of algorithm designers, consumes expensive human costs, and is difficult to adapt to the application scenarios of syringes with a wide variety of device types and defect types. Defect detection technology based on deep learning can automatically learn defect features from the dataset and has strong feature learning capabilities. Deep learning technology can not only avoid the cumbersome design process of manually constructing features, but also obtain a more accurate description than manually constructed features. Therefore, compared with traditional machine vision technology, the detection technology based on deep learning has better performance in the syringe scale defect detection scenario. However, currently, the performance of deep learning detection devices generally depends on the richness of the training dataset. This type of device performs well in detecting frequent defects (abundant defect samples), but has poor detection effects on occasional defects (scarce defect samples). In addition, existing deep learning defect detection devices require a large amount of manual and time costs to collect and label training data during development, and it is difficult to meet the requirements of rapid development of new products. Summary of the Invention

[0004] The object of the present invention is to provide a syringe scale defect detection method based on deep learning and scale grouping matching, which can overcome the problems of high cost, low precision, long development cycle and dependence on defect samples existing in the existing syringe scale defect detection methods, and can achieve accurate detection of syringe scale defects in the case of only a small number of normal (defect-free) syringe samples.

[0005] To achieve the above object, the solution of the present invention is as follows:

[0006] A syringe scale defect detection method based on deep learning and scale grouping matching includes the following steps:

[0007] Step 1: Place the syringe to be tested on the conveying mechanism, and the image sensor obtains images of a number of syringes in real time within the field of view, and select the syringe image containing a complete syringe from them;

[0008] Step 2: Use a deep neural network to extract the syringe scale from the syringe image;

[0009] Step 3: Based on the extracted syringe scale, use the scale grouping allocation algorithm to detect defects on the scale.

[0010] The specific content of the above Step 1 is as follows:

[0011] Step S1-1: Define the initialization as the waiting state C1, and record the average brightness value of each frame of the image collected by the image sensor;

[0012] Step S1-2: In the waiting state C1, and the average brightness values of D1 consecutive frames of the continuously acquired images rise continuously, define entering the preparatory state C2, otherwise keep the waiting state C1 unchanged;

[0013] Step S1-3: When in the preparatory state C2 and the image brightness value remains basically unchanged for D2 consecutive frames, cache the single image acquired at this time, and set the state at this time as the candidate state C3; if in the preparatory state C2, and after the image average brightness value stops rising continuously, the image average brightness value fails to remain basically unchanged within D3 consecutive frames, then restore to the waiting state C1;

[0014] Step S1-4: When in the candidate state C3, and the image brightness values of D4 consecutive frames are in a downward trend, it is regarded that the image cached in Step S1-3 is a valid image, and set the state at this time as the determined state C4, and set the state as the waiting state C1 after obtaining the image; if in the candidate state c3, but the average brightness value within D5 consecutive frames of the cumulative image fails to remain continuously decreasing, then restore to the waiting state C1.

[0015] In the above step S1-2, D1 is set to 4, in step S1-3, D2 is set to 10, D3 is set to 5, in step S1-4, D4 is set to 4, and D5 is set to 10.

[0016] In the above step S1-2, "the average brightness value continuously rises" is defined as: in two adjacent frames of images, the average brightness value of the latter frame of image is 5% greater than that of the previous frame of image;

[0017] In step S1-3, "remain basically unchanged" is defined as: in two adjacent frames of images, the change in the average brightness value of the latter frame of image compared to the previous frame of image is less than or equal to 5%;

[0018] In step S1-4, "continuously decreases" is defined as: in two adjacent frames of images, the change in the average brightness value of the latter frame of image compared to the previous frame of image is less than 5%.

[0019] The specific content of the above step two is:

[0020] Step S2-1: Collect a certain number of syringe images {TrainX1}, and label the collected syringe images to obtain the labeled images {Y1} of the syringe scale. Then, use the syringe images and the segmentation map to make the training dataset {(TrainX1, Y1)} and the validation dataset {ValidX1};

[0021] Step S2-2: Construct a syringe scale segmentation model based on a convolutional neural network. Input the training dataset {(TrainX1, Y1)} into the syringe scale segmentation model, and after iterating the model parameters, obtain the optimal syringe scale segmentation model N1 on the validation dataset {ValidX1};

[0022] Step S2-3: Use the syringe scale segmentation model N1 to process the syringe images collected on-site to obtain the syringe scale segmentation map M1 and the image coordinate information of the syringe scale.

[0023] In the above step S2-1, the labelme software is used to label the syringe images; the specific steps are: use an external rectangular box to label all the scales on the syringe to obtain a single-channel labeled image {Y1}; the scales in the labeled image are the foreground, and the rest are the background; set the foreground pixel value to 255 and the background pixel value to 0.

[0024] In the above step S2-2, the specific network structure of the syringe scale segmentation model is:

[0025] The first and second layers are convolutional layers, the convolutional kernel size is 3, the stride is 1, the padding is 1, and 32 feature maps are output; after each convolutional layer, there is a batch normalization layer and a ReLU activation layer;

[0026] The third layer is a max pooling layer, and the pooling window size is 2×2;

[0027] The fourth, fifth, and sixth layers are convolutional layers with a convolutional kernel size of 3, a stride of 1, and a padding of 1, outputting 64 feature maps; after each convolutional layer, there are a batch normalization layer and a ReLU activation layer;

[0028] The seventh layer is a max pooling layer with a pooling window size of 2×2;

[0029] The eighth, ninth, tenth, and eleventh layers are convolutional layers with a convolutional kernel size of 3, a stride of 1, and a padding of 1, outputting 64 feature maps; after each convolutional layer, there are a batch normalization layer and a ReLU activation layer;

[0030] The twelfth layer is a transposed convolutional layer that performs transposed convolution on the output of the second layer. The transposed convolutional kernel size is 1, the stride is 1, the padding is 0, and it outputs 1 feature map;

[0031] The thirteenth layer is a transposed convolutional layer that performs transposed convolution on the output of the sixth layer. The transposed convolutional kernel size is 2, the stride is 2, the padding is 0, and it outputs 1 feature map;

[0032] The fourteenth layer is a transposed convolutional layer that performs transposed convolution on the output of the eleventh layer. The transposed convolutional kernel size is 4, the stride is 4, the padding is 0, and it outputs 1 feature map;

[0033] The fifteenth layer is a processing layer that concatenates the outputs of the twelfth, thirteenth, and fourteenth layers into a three-layer feature map;

[0034] The sixteenth layer is a convolutional layer that performs convolution on the output of the fifteenth layer; the convolutional kernel size is 1, and the stride is 1;

[0035] The loss function is defined as the cross-entropy loss function, and the optimization algorithm uses the Adam optimizer.

[0036] The specific steps of the above step three are as follows:

[0037] Step S3-1: Number all the independent pixel blocks of the syringe scale extracted in step two from left to right according to their positions in the image as {b(1), b(2),..., b(N)}, where N is the number of all independent pixel blocks in the sample;

[0038] Step S3-2: Start processing from the pixel block with the smallest number;

[0039] Step S3-3: Set the currently largest numbered scale group as g(I), and the currently ungrouped and smallest numbered pixel block as b(X). Calculate the horizontal pixel distance D(I, X) between g(I) and b(X); if D(I, X) is less than T1 times the standard interval between adjacent scales, then incorporate the b(X) pixel block into the scale group g(I); otherwise, create a new scale group g(I + 1) and incorporate b(X) into g(I + 1);

[0040] Step S3-4: Repeat step S3-3 until all pixel blocks are grouped;

[0041] Step S3-5: Check the number of scale groups; if the total number of scale groups exceeds the number of scales on the standard syringe, then it is determined that there are redundant scales in the current syringe image; conversely, if the total number of scale groups is less than the number of scales on the standard syringe, then it is determined that the current syringe image lacks scales;

[0042] Step S3-6: Check the number of pixel blocks in each scale group. If there are multiple pixel blocks in any scale group, then it is determined that there is a scale break defect in the current syringe image;

[0043] Step S3-7: Check the interval between adjacent scale groups in the grouping; if the interval between any two adjacent scale groups is greater than T2 times the adjacent scale standard interval or T3 times the average distance between scale groups in the current syringe image, then it is considered that there are defective scales that have not been extracted between the two scales outside the distance range, and it is determined that the current sample has accidental scale defects;

[0044] Step S3-8: Check the ratio of the length and width of each scale group to the ratio of the length and width of the standard scale on a normal syringe. If the length ratio of any scale group exceeds the interval T4 or the width ratio exceeds the interval T5, then the current syringe image is determined to have abnormal scale dimensions;

[0045] Step S3-9: Check the ratio of the length and width of each scale group to the average length ratio and average width ratio of all scale groups in the current syringe image. If the length ratio of any scale group exceeds the interval T6 or the width ratio exceeds the interval T7, then the current syringe image is determined to have abnormal scale dimensions.

[0046] In the above step S3-2, if the currently processed pixel block is the pixel block with the smallest number, then establish a scale group to store the current pixel block, number the scale groups in the order of establishment, and the number of the first established scale group is g(1).

[0047] T1 is set to 1 / 3, T2 is set to 1.3, T3 is set to 1.15, T4 is set to [0.8, 1.2], T5 is set to [0.8, 1.2], T6 is set to [0.9, 1.1], and T7 is set to [0.9, 1.1].

[0048] After adopting the above solution, the present invention can accurately detect the scale defects of syringes by using only a small number of normal (defect-free) syringe samples. Through the combination of deep learning and scale grouping, the device can effectively reduce the enterprise's human resource cost, improve the detection accuracy, and effectively shorten the product development cycle of syringe defect detection. Specifically, in step one, the brightness value change law of the image is used to obtain a syringe image containing a complete syringe. This method enables the detection device to obtain a complete syringe image by using only a high-speed image sensor, and can overcome the dependence on devices such as distance sensors and laser sensors in traditional methods. Therefore, the present invention can effectively reduce the cost of the detection device and avoid the cumbersome detection device debugging process; in step two, the neural network is trained only by using normal (defect-free) syringe samples, which can greatly improve the neural network training efficiency and save labor costs. At the same time, step two transforms the scale defect detection problem into a scale segmentation problem, which can reduce the influence of the variability of scale defects on the defect detection algorithm and reduce the difficulty of designing the subsequent defect detection algorithm (step three). Therefore, the present invention can accurately locate various scale defects, even newly emerging accidental defects; in step three, a scale grouping matching algorithm is used to detect the defects of the scales extracted in step two. This matching algorithm has low requirements for computing resources and can meet the needs of rapid detection of syringe scale defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 FIG. is a schematic perspective view of the detection device used by the present invention to collect syringe images;

[0050] Figure 2 is Figure 1 a top view schematic diagram of;

[0051] Figure 3 is Figure 1 a partially enlarged view from a top view angle;

[0052] Figure 4 FIG. is a schematic structural diagram of the syringe to be detected.

[0053] Figure 5 FIG. is a flow chart of the syringe scale defect detection of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0054] Hereinafter, the technical solutions and beneficial effects of the present invention will be described in detail with reference to the accompanying drawings.

[0055] The present invention provides a method for detecting syringe scale defects based on deep learning and scale grouping matching. For the image of the syringe to be detected collected on the production line, the deep learning algorithm and the scale grouping matching method are used to accurately detect and locate the scale defects of the syringe.

[0056] First, as shown in Figure 1 and Figure 2As shown in the figure, the device for obtaining the image studied in the present invention includes a host computer 1, an image sensor 2, and a conveying mechanism 3. Among them, several syringes 4 to be detected are placed on the conveyor belt of the conveying mechanism 3, and the conveying mechanism 3 is used to convey the syringes 4; the image sensor 2 is arranged above one side of the conveying mechanism 3, the image sensor 2 is perpendicular to the conveyor belt of the conveying mechanism 3, and the image sensor 2 faces the syringes 4 on the conveying mechanism 3. The image sensor 2 is also electrically connected to the host computer 1. The image sensor 2 collects the images of the syringes 4 on the conveying mechanism 3 and transmits them to the host computer 1 for image processing. This structure is an existing structure and will not be elaborated here.

[0057] The syringe scale defect detection method of the present invention includes three steps. First, select a suitable syringe image from several images obtained in real time within the field of view of the image sensor; then, use a deep neural network to extract the syringe scale from the syringe image; finally, use a scale grouping and distribution algorithm to detect defects on the scale based on the scale extraction result; as Figure 5 shown, the specific steps are as follows:

[0058] Step 1, select a syringe image

[0059] This step mainly locates and captures the syringe image according to the movement law of the syringe on the conveyor belt and uses the brightness change within the field of view of the image sensor. After analysis, it is found that, in cooperation with Figure 3 , on the production line, the brightness value of the syringe 4 is relatively high compared to the background 5. This characteristic makes the pixel brightness value in the area where the syringe 4 is located in the image obtained by the image sensor 2 high while the brightness value of the background 5 pixels is low. Therefore, when the image obtained by the image sensor 2 contains the syringe 4, the average brightness value of this image will be greater than the average brightness value of the image background pixels. When the syringe 4 enters the field of view of the image sensor 2 along with the conveying mechanism 3, the average brightness value of the image obtained by the image sensor 2 will gradually increase; when the syringe 4 leaves the field of view of the image sensor 2 along with the conveying mechanism 3, the average brightness value of the image obtained by the image sensor 2 will gradually decrease. In addition, when the syringe 4 is completely included in the field of view of the image sensor 2, the average brightness value of the image obtained by the image sensor 2 remains basically unchanged. Therefore, according to the change characteristics of the brightness value obtained by the image sensor 2, it can be judged whether the field of view of the image sensor 2 contains a complete syringe.

[0060] During operation, first, the conveying mechanism 3 conveys the syringes 4 to be detected to the shooting area of the image sensor 2, and the image sensor 2 obtains an image containing the complete syringe 4. Then, an image acquisition module can be defined to obtain the image containing the syringe 4 based on the brightness value difference between the syringe 4 and the background 5 in the image. The specific steps for the image acquisition module to work and obtain the syringe image are as follows:

[0061] Step S1-1: When the image acquisition module starts running, it is initialized to the waiting state C1; then, it continuously acquires images from the image sensor in real time, calculates and records the average brightness value of each frame of the image.

[0062] Step S1-2: When the image acquisition module is in the waiting state C1 and the average brightness values of D1 consecutive frames of acquired images continuously increase, set the state of the image acquisition module to the preparatory state C2; otherwise, keep the waiting state C1 unchanged.

[0063] In this embodiment, D1 is set to 4.

[0064] "The average brightness value continuously increases" is defined as: in two adjacent frames of images, the average brightness value s of the latter frame of the image i is larger than that of the previous frame of the image s i-1 by 5%, that is

[0065] Step S1-3: When the image acquisition module is in the preparatory state C2 and the image brightness value remains basically unchanged for D2 consecutive frames, the image acquisition module caches the single image acquired at this time, and sets the state of the image acquisition module to the candidate state C3. If the image acquisition module is in the preparatory state C2, and after the average brightness value of the image stops continuously increasing, the average brightness value of the image fails to remain basically unchanged within D3 consecutive frames, the state of the image acquisition module is restored to the waiting state C1.

[0066] In this embodiment, D2 is set to 10; D3 is set to 5.

[0067] "Remain basically unchanged" is defined as: in two adjacent frames of images, the average brightness value s of the latter frame of the image i changes less than or equal to 5% compared with the previous frame of the image s i-1 that is

[0068] Step S1-4: When the image acquisition module is in the candidate state C3 and the brightness values of D4 consecutive frames of images are in a downward trend, it can be determined that the image cached in Step S1-3 is a valid image, that is, an image containing a single complete syringe, then set the state of the image acquisition module to the determined state C4. At the same time, the image acquisition module hands over the image cached in Step S1-3 to the scale extraction module for processing, and then sets the state of the image acquisition module to the waiting state C1. If the image acquisition module is in the candidate state C3, but the average brightness value within D5 frames of accumulated images fails to remain continuously decreasing, the image acquisition module is restored to the waiting state C1.

[0069] In this embodiment, D4 is set to 4; D5 is set to 10.

[0070] "Continuously decreasing" is defined as: in two adjacent frames of images, the average brightness value s of the latter frame of the imagei Compared with the previous frame image s i-1 The change is less than 5%, that is

[0071] Step 2, syringe scale extraction

[0072] This step can be defined as a scale extraction module, whose purpose is to use a syringe scale segmentation model based on deep learning (hereinafter referred to as the deep syringe scale segmentation model) to extract the scale from the syringe image, that is Figure 4 The shown syringe scale 41 is used as the detection object, and the training data of the deep syringe scale segmentation model only includes normal syringe images.

[0073] After obtaining the image containing the syringe based on Step 1, the host computer hands the obtained image to the scale extraction module for processing. The content of the scale extraction module for extracting the syringe scale includes the following specific steps:

[0074] Step S2-1: Collect a certain number of syringe images {TrainX1}, and use software to label the collected syringe images to obtain the labeled images {Y1} of the syringe scale. Then, use the syringe images and segmentation maps to make a training data set {(TrainX1, Y1)} and a validation data set {ValidX1};

[0075] During the process of labeling the images, the scale extraction module sets the pixel values of the normal scales in the syringe images to 255 and the background part to 0. Since the goal is to extract normal scales, and there are often multiple scales on a single syringe, only a small number of normal samples need to be collected when collecting samples for algorithm training. This strategy can effectively reduce the enterprise's human resource cost, improve the detection accuracy, and effectively shorten the development cycle of syringe defect detection products;

[0076] In this embodiment, the labelme software is used to label the syringe images; all the scales on the syringe are labeled with an external rectangular frame to obtain a single-channel labeled image {Y1}; the scales in the labeled image are the foreground, and the rest are the background; the foreground pixel values are set to 255, and the background pixel values are set to 0.

[0077] Step S2-2: Build a syringe scale segmentation model based on a convolutional neural network, input the training data set {(TrainX1, Y1)} into the syringe scale segmentation model, and after iterating the model parameters, obtain the optimal syringe scale segmentation model N1 on the validation data set {ValidX1};

[0078] Step S2-3: Use the syringe scale segmentation model N1 to process the syringe images collected on-site to obtain the syringe scale segmentation map M1 and the image coordinate information of the syringe scale;

[0079] In this embodiment, in step S2-2, a syringe scale segmentation model based on a convolutional neural network is adopted. The specific structure of the syringe scale segmentation model is as follows:

[0080] The first and second layers are convolutional layers. The convolutional kernel size is 3, the stride is 1, the padding is 1, and 32 feature maps are output. After each convolutional layer, there is a batch normalization layer and a ReLU activation layer;

[0081] The third layer is a max pooling layer, and the pooling window size is 2×2;

[0082] The fourth, fifth, and sixth layers are convolutional layers. The convolutional kernel size is 3, the stride is 1, the padding is 1, and 64 feature maps are output. After each convolutional layer, there is a batch normalization layer and a ReLU activation layer;

[0083] The seventh layer is a max pooling layer, and the pooling window size is 2×2;

[0084] The eighth, ninth, tenth, and eleventh layers are convolutional layers. The convolutional kernel size is 3, the stride is 1, the padding is 1, and 64 feature maps are output. After each convolutional layer, there is a batch normalization layer and a ReLU activation layer;

[0085] The twelfth layer is a deconvolution layer, which deconvolves the output of the second layer. The deconvolutional kernel size is 1, the stride is 1, the padding is 0, and 1 feature map is output;

[0086] The thirteenth layer is a deconvolution layer, which deconvolves the output of the sixth layer. The deconvolutional kernel size is 2, the stride is 2, the padding is 0, and 1 feature map is output;

[0087] The fourteenth layer is a deconvolution layer, which deconvolves the output of the eleventh layer. The deconvolutional kernel size is 4, the stride is 4, the padding is 0, and 1 feature map is output;

[0088] The fifteenth layer is a processing layer, which concatenates the twelfth, thirteenth, and fourteenth layers into a three-layer feature map;

[0089] The sixteenth layer is a convolutional layer, which convolves the output result of the fifteenth layer. The convolutional kernel size is 1, the stride is 1.

[0090] The loss function is defined as a cross-entropy loss function, and the optimization algorithm adopts an Adam optimizer.

[0091] Step three, detect scale defects

[0092] This step can be defined as a defect detection module. After the syringe scale is extracted in Step 2, the host computer sends the extracted scale segmentation map M1 and the coordinate information of the syringe scale to the defect detection module for processing. There may be noise and mis-extracted defective scales in the scale segmentation map M1. To avoid the influence of these two factors on the detection result, the defect detection module first numbers the extracted scales, then further groups the scales based on the numbers, and finally, based on the scale grouping matching algorithm, matches the scale grouping result with the standard normal scale. The scales with poor matching degree are judged as defective scales, so as to determine whether there are scale defects in the syringe image.

[0093] The specific detection steps for the defect detection module to detect scale defects are as follows:

[0094] Step S3-1: Number all the independent pixel blocks in the output result of the syringe scale extraction module from left to right according to their positions in the image as {b(1), b(2),..., b(N)}, where N is the number of all independent pixel blocks in the sample;

[0095] Step S3-2: Start processing from the pixel block with the smallest number. If the currently processed pixel block is the one with the smallest number, establish a scale group to store the current pixel block, number the scale group according to the establishment order, and the number of the first established scale group is g(1);

[0096] Step S3-3: Set the currently numbered largest scale group as g(I), and the currently ungrouped and smallest-numbered pixel block as b(X), and calculate the horizontal pixel distance D(I, X) between g(I) and b(X); if D(I, X) is less than T1 times the standard interval between adjacent scales, then incorporate the pixel block b(X) into the scale group g(I); otherwise, establish a new scale group g(I + 1) and incorporate b(X) into g(I + 1);

[0097] In this embodiment, T1 is set to 1 / 3;

[0098] Step S3-4: Repeat Step S3-3 until all pixel blocks are grouped;

[0099] Step S3-5: Check the number of scale groups. If the total number of scale groups exceeds the number of scales on the standard syringe, then the defect detection module determines that there are extra scales in the current syringe image. On the contrary, if the total number of scale groups is less than the number of scales on the standard syringe, then the defect detection module determines that the current syringe image lacks scales;

[0100] Step S3-6: Check the number of pixel blocks in each scale group. If there are multiple pixel blocks in any scale group, then the defect detection module determines that there is a scale break defect in the current syringe image;

[0101] Step S3-7: Check the intervals between adjacent scale groups in the grouping. If the interval between any two adjacent scale groups is greater than T2 times the adjacent scale standard interval or T3 times the average distance between scale groups in the current syringe image, the defect detection module considers that there are defective scales that have not been extracted between the two scales beyond the distance range, and determines that the current sample has accidental scale defects;

[0102] In this embodiment, T2 is set to 1.3; T3 is set to 1.15;

[0103] Step S3-8: Check the ratios of the length and width of each scale group to the ratio of the length and the ratio of the width of the standard scale on a normal syringe. If the length ratio of any scale group exceeds the interval T4 or the width ratio exceeds the interval T5, the current syringe image is determined to have abnormal scale dimensions;

[0104] In this embodiment, T4 is set to [0.8, 1.2]; T5 is set to [0.8, 1.2];

[0105] Step S3-9: Check the ratios of the length and width of each scale group to the average length ratio and the average width ratio of all scale groups in the current syringe image. If the length ratio of any scale group exceeds the interval T6 or the width ratio exceeds the interval T7, the current syringe image is determined to have abnormal scale dimensions.

[0106] In this embodiment, T6 is set to [0.9, 1.1]; T7 is set to [0.9, 1.1].

[0107] The above embodiments are only used to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention fall within the protection scope of the present invention.

Claims

1. A syringe scale defect detection method based on deep learning and scale group matching, characterized in that It includes the following steps: Step 1: Place the syringe to be measured on the conveying mechanism. The image sensor obtains images of a number of syringes in real time within the field of view, and selects the syringe image containing a complete syringe from them. Step 2: Use a deep neural network to extract the syringe scale from the syringe image. Step 3: Based on the extracted syringe scale, use a scale grouping and assignment algorithm to detect defects on the scale. The specific steps of Step 3 are as follows: Step S3-1: Number all independent pixel blocks of the syringe scale extracted in Step 2 from left to right according to their positions in the image as {b(1), b(2), …, b(N)}, where N is the number of all independent pixel blocks in the sample. Step S3-2: Start processing from the pixel block with the smallest number. Step S3-3: Set the currently numbered largest scale group as g(I), and the currently ungrouped and smallest numbered pixel block as b(X). Calculate the horizontal pixel distance D(I, X) between g(I) and b(X). If D(I, X) is less than T1 times the standard interval between adjacent scales, then incorporate the b(X) pixel block into the scale group g(I); otherwise, establish a new scale group g(I + 1) and incorporate b(X) into g(I + 1). Step S3-4: Repeat Step S3-3 until all pixel blocks are grouped. Step S3-5: Check the number of scale groups. If the total number of scale groups exceeds the number of scales on a standard syringe, then it is determined that the current syringe image has extra scales; conversely, if the total number of scale groups is less than the number of scales on a standard syringe, then it is determined that the current syringe image lacks scales. Step S3-6: Check the number of pixel blocks in each scale group. If there are multiple pixel blocks in any scale group, then it is determined that the current syringe image has a scale break defect. Step S3-7: Check the interval between adjacent scale groups in the grouping. If the interval between any two adjacent scale groups is greater than T2 times the standard interval between adjacent scales or T3 times the average distance between scale groups in the current syringe image, then it is considered that there are defective scales that cannot be extracted between the two scales beyond the distance range, and it is determined that the current sample has accidental scale defects. Step S3-8: Check the ratios of the length and width of each scale group to the length and width of the standard scale on a normal syringe. If the length ratio of any scale group exceeds the interval T4 or the width ratio exceeds the interval T5, then the current syringe image is determined to have abnormal scale dimensions. Step S3-9: Check the ratios of the length and width of each scale group to the average length and average width of all scale groups in the current syringe image. If the length ratio of any scale group exceeds the interval T6 or the width ratio exceeds the interval T7, then the current syringe image is determined to have abnormal scale dimensions.

2. The method according to claim 1, wherein: The specific content of Step 1 is as follows: Step S1-1: Define the initialization as the waiting state C1, and record the average brightness value of each frame of image collected by the image sensor. Step S1-2: In the waiting state C1, and if the average brightness values of D1 consecutive frames of images continuously increase, define entering the preparatory state C2; otherwise, keep the waiting state C1 unchanged. Step S1-3: When in the preparatory state C2 and the image brightness value remains basically unchanged for D2 consecutive frames, cache the single image obtained at this time and set the state at this time to the candidate state C3; if in the preparatory state C2 and the average image brightness value fails to remain basically unchanged within D3 consecutive frames after stopping continuous increase, restore to the waiting state C1; Step S1-4: When in the candidate state C3 and the image brightness values of D4 consecutive frames are in a downward trend, consider the image cached in Step S1-3 as a valid image, set the state at this time to the determined state C4, and set the state to the waiting state C1 after obtaining the image; if in the candidate state C3 but the average brightness value within D5 accumulated frames fails to maintain continuous decline, restore to the waiting state C1.

3. The method according to claim 2, wherein: In the said Step S1-2, D1 is set to 4, in Step S1-3, D2 is set to 10, D3 is set to 5, in Step S1-4, D4 is set to 4, and D5 is set to 10.

4. The method according to claim 2, wherein: In the said Step S1-2, "continuous increase in average brightness value" is defined as: in two adjacent frames of images, the average brightness value of the latter frame of image is 5% larger than that of the previous frame of image; In Step S1-3, "remain basically unchanged" is defined as: in two adjacent frames of images, the change in the average brightness value of the latter frame of image compared to the previous frame of image is less than or equal to 5%; In Step S1-4, "continuous decline" is defined as: in two adjacent frames of images, the change in the average brightness value of the latter frame of image compared to the previous frame of image is less than 5%.

5. The method according to claim 1, characterized in that: The specific content of the said Step Two is: Step S2-1: Collect a certain number of syringe images {TrainX1}, annotate the collected syringe images to obtain the annotated images {Y1} of syringe scales, and then use the syringe images and the segmentation maps to make the training dataset {(TrainX1, Y1)} and the validation dataset {ValidX1}; Step S2-2: Build a syringe scale segmentation model based on a convolutional neural network, input the training dataset {(TrainX1, Y1)} into the syringe scale segmentation model, and obtain the optimal syringe scale segmentation model N1 on the validation dataset {ValidX1} after iterating the model parameters; Step S2-3: Use the syringe scale segmentation model N1 to process the syringe images collected on-site to obtain the syringe scale segmentation map M1 and the image coordinate information of the syringe scales.

6. The method according to claim 5, wherein: In the said Step S2-1, the labelme software is used to annotate the syringe images; the specific steps are: use an external rectangular box to annotate all the scales on the syringe to obtain a single-channel annotated image {Y1}; the scales in the annotated image are the foreground and the rest are the background; set the foreground pixel value to 255 and the background pixel value to 0.

7. The method according to claim 5, characterized in that: In the said Step S2-2, the specific network structure of the syringe scale segmentation model is: The first and second layers are convolutional layers, the convolutional kernel size is 3, the stride is 1, the padding is 1, and 32 feature maps are output; after each convolutional layer, there is a batch normalization layer and a ReLU activation layer; The third layer is a max pooling layer, and the pooling window size is 2×2; The fourth, fifth, and sixth layers are convolutional layers with a convolutional kernel size of 3, a stride of 1, and a padding of 1, outputting 64 feature maps; after each convolutional layer, there is a batch normalization layer and a ReLU activation layer; The seventh layer is a max pooling layer with a pooling window size of 2×2; The eighth, ninth, tenth, and eleventh layers are convolutional layers with a convolutional kernel size of 3, a stride of 1, and a padding of 1, outputting 64 feature maps; after each convolutional layer, there is a batch normalization layer and a ReLU activation layer; The twelfth layer is a transposed convolutional layer that performs transposed convolution on the output of the second layer. The transposed convolutional kernel size is 1, the stride is 1, the padding is 0, and it outputs 1 feature map; The thirteenth layer is a transposed convolutional layer that performs transposed convolution on the output of the sixth layer. The transposed convolutional kernel size is 2, the stride is 2, the padding is 0, and it outputs 1 feature map; The fourteenth layer is a transposed convolutional layer that performs transposed convolution on the output of the eleventh layer. The transposed convolutional kernel size is 4, the stride is 4, the padding is 0, and it outputs 1 feature map; The fifteenth layer is a processing layer that concatenates the twelfth, thirteenth, and fourteenth layers into a three-layer feature map; The sixteenth layer is a convolutional layer that performs convolution on the output result of the fifteenth layer. The convolutional kernel size is 1 and the stride is 1; The loss function is defined as the cross-entropy loss function, and the optimization algorithm uses the Adam optimizer.

8. The method according to claim 1, wherein: In the step S3-2, if the currently processed pixel block is the pixel block with the smallest number, a scale group is established to store the current pixel block, and the scale group is numbered according to the establishment order. The scale group with the earliest establishment is numbered g(1).

9. The method according to claim 1, characterized in that: T1 is set to 1 / 3, T2 is set to 1.3, T3 is set to 1.15, T4 is set to [0.8, 1.2], T5 is set to [0.8, 1.2], T6 is set to [0.9, 1.1], and T7 is set to [0.9, 1.1].

Images