Metal particle detection method and system under complex background and medium

Abstract

The invention relates to the technical field of computer vision, in particular to a metal particle detection method and system under a complex background and a medium. The method comprises the following steps: adopting a multi-sample fusion data enhancement method to perform data enhancement on a metal particle image under an actual detection working condition and a metal particle image under a single-color background to obtain a new sample and an original sample set to form a data set; inserting a SimAM attention module into each YOLOv8 model, carrying out lightweight design on a detection head structure, and constructing a metal particle detection model; training a metal particle detection model based on the data set; acquiring a to-be-detected metal particle image in real time; and inputting a to-be-detected metal particle image into the trained metal particle detection model, and outputting metal particle detection frame information in the image. The detection accuracy and real-time performance of the metal particles under the complex background are improved, and the flexibility of the model in processing the metal particle detection under the new and unseen complex background is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system and medium for detecting metal particles under complex backgrounds. Background Art

[0002] In the industrial manufacturing process, the detection of metal particles is crucial to ensure product quality. For example, in automobile manufacturing, metal particles may have an adverse effect on the performance of the engine; in electronic manufacturing, metal particles may cause circuit failures, etc. With the development of modern deep learning technology, especially the application of convolutional neural networks (CNN) in metal particle detection, the detection of metal particles has become more accurate and efficient. Specifically, the specific application process of metal particle detection as target detection is: given an input image or video frame to be detected, determine whether there are metal particles in the image, and if so, frame the size and position of the metal particles.

[0003] For example, Chinese patent CN117893795A, a method for detecting metal particles on the inner surface of GIS equipment based on improved YOLOv5, obtains and screens pictures of particles and no particles and annotates them as a data set, and introduces the CBAM attention mechanism module and (ATM) adaptive threshold module into the feature extraction network of the original YOLOv5 algorithm to improve the YOLOv5 model, and then puts the data set image into the improved YOLOv5 for feature extraction to obtain feature maps of three different scales, and then fuses the feature maps of the three different scales, and uses the non-maximum suppression (NMS) algorithm to screen the candidate frame to obtain the final particle detection frame, and finally uses the pre-trained YOLOv5 model to identify the metal particles on the inner surface, thereby realizing the effective detection of metal particles on the inner surface, and does not require special design of the network, and the use of a general network structure can still achieve good experimental results.

[0004] However, the background of the metal particle image under the actual working condition is non-uniform color, usually a complex background composed of white stripes, stains, and wrinkles caused by reflective components. However, the existing technology usually directly uses the metal particle image under a single color background or regards the background of the metal particle image as a single color background for training, without considering the actual situation of metal particle detection under complex background. Specifically, on the one hand, for the metal particle image input into the network, the target particles only occupy a small part of the overall image area, while most of the area is the background and impurity information of the image, which leads to the image features in the backbone network layer of the general network. When the large amount of background and impurity information gradually evolves into a large amount of redundant information, which will cover up the characteristic information of some target particles, thereby causing the overall detection accuracy of the model to decrease and the model calculation complexity to increase, and thus it is difficult to meet the accuracy and real-time requirements of metal particle detection; on the other hand, the general network structure of the prior art and the introduction of the CBAM attention mechanism module cannot adapt to the complex background changes of metal particle detection under actual working conditions, because the weight of CBAM is fixed through training, which means that once the training is completed, the weight will no longer change, thereby limiting the flexibility of the model in dealing with new and unseen metal particle detection under complex backgrounds. Summary of the invention

[0005] In response to the above technical problems, the present invention proposes a method, system and medium for detecting metal particles in complex backgrounds, which is designed to meet the requirements of metal particle detection accuracy and real-time performance in complex backgrounds, and to improve the flexibility of the model in handling metal particle detection in new and unseen complex backgrounds.

[0006] In a first aspect, the present application provides a method for detecting metal particles in a complex background, comprising the following steps: Acquire an image of metal particles under actual detection conditions, recorded as a first original sample; Acquire an image of metal particles under a single color background, which is recorded as the second original sample; Performing data enhancement on the first original sample and the second original sample using a data enhancement method of multi-sample fusion to obtain a new sample after data enhancement, wherein the set of the new sample, the first original sample and the second original sample constitutes a data set; The improved YOLOv8 model is used to build a metal particle detection model. The metal particle detection model inserts the SimAM attention mechanism module into the backbone network and the neck network of the YOLOv8 model, and performs a lightweight design on the detection head structure. The metal particle detection model is trained based on the data set. The input of the trained metal particle detection model is the metal particle detection image, and the output is the metal particle detection frame information. Acquire the image of metal particles to be detected in real time; The metal particle image to be detected is input into the trained metal particle detection model, and the metal particle detection frame information in the image is output.

[0007] In some embodiments, a data enhancement method of multi-sample fusion is used to perform data enhancement on the first original sample and the second original sample to obtain a new sample after data enhancement, and the set of new samples constitutes a data set, including: The Cutmix data enhancement method is used to enhance the data of the first original sample, and the metal particle image under the actual detection condition after data enhancement is obtained, which is recorded as the first new sample; The second original sample is enhanced by using the Mosaic data enhancement method to obtain a metal particle image under a single color background after data enhancement, which is recorded as the second new sample; The set of the first new sample, the second new sample, the first original sample and the second original sample constitutes a data set.

[0008] In some embodiments, the Cutmix data enhancement method is used to perform data enhancement on the first original sample to obtain a metal particle image under actual detection conditions after data enhancement, including: Annotate the images in the first original sample to obtain labels corresponding to each image; Randomly select two images from the first original sample and randomly select a rectangular area; The two images are mixed based on the selected rectangular area, and the obtained mixed image is the metal particle image under the actual detection condition after data enhancement; Calculate the mixing ratio of the two images, and perform weighted average on the labels corresponding to the two images based on the mixing ratio to obtain the label of the mixed image.

[0009] In some embodiments, the second original sample is enhanced using a Mosaic data enhancement method to obtain a data-enhanced metal particle image under a single color background, including: Annotate the images in the second original sample to obtain labels corresponding to each image; Randomly select a number of images from the second original sample, and randomly crop each selected image to determine the position of the cropped area in the selected image; The cropped images are stitched together, and the resulting stitched image is the metal particle image under the single color background after data enhancement; Based on the position of the cropped area in the selected image and the label corresponding to the selected image, a label of the stitched image is determined.

[0010] In some embodiments, the metal particle detection model inserts a SimAM attention mechanism module into both the backbone network and the neck network of the YOLOv8 model, including: The metal particle detection model inserts a SimAM attention mechanism module after the spatial pyramid pooling layer of the backbone network of the YOLOv8 model, and inserts a SimAM attention mechanism module after each deep feature fusion layer in the neck network.

[0011] In some embodiments, the lightweight design of the detection head structure includes: A shared parameter module is added after the input of the detection head of the YOLOv8 model to postpone the branch task, and a multi-scale information fusion module is designed in the shared parameter module to replace the 3×3 convolution module in the detection head of the YOLOv8 model. The multi-scale information fusion module is used to perform the following steps: Perform convolution operation on the feature map of the input detection head to obtain the initial feature map; Divide the initial feature map into several groups, and perform linear operations on each group of feature maps to obtain several new feature maps; Concatenate the new feature map with the initial feature map to obtain the initial output; Perform a 1×1 convolution operation on the initial output to obtain the fused feature map.

[0012] In some embodiments, the use of the improved YOLOv8 model to construct a metal particle detection model further includes: The Wise-IoU loss function is fused with the Inner-IoU loss function to obtain the fused loss function, which is recorded as the IWIoU loss function; The metal particle detection model uses the IWIoU loss function to replace the CIoU loss function used by the YOLOv8 model.

[0013] In some embodiments, the expression of the IWIoU loss function is: in, represents the IWIoU loss function, Represents the adjustment factor in the Wise-IoU loss function, Represents the value of the Inner-IoU loss function.

[0014] In a second aspect, the present application provides a metal particle detection system in a complex background, comprising: An image acquisition module is used to acquire and annotate images of metal particles under actual detection conditions, which are recorded as the first original sample; acquire and annotate images of metal particles under a single color background, which are recorded as the second original sample; and acquire images of metal particles to be detected in real time; A data enhancement module is used to perform data enhancement on the first original sample and the second original sample by adopting a data enhancement method of multi-sample fusion to obtain a new sample after data enhancement, and the set of the new sample, the first original sample and the second original sample constitutes a data set; A model building module is used to build a metal particle detection model using an improved YOLOv8 model. The metal particle detection model inserts SimAM attention mechanism modules into the backbone network and the neck network of the YOLOv8 model, and performs a lightweight design on the detection head structure. A model training module is used to train a metal particle detection model based on a data set. The input of the trained metal particle detection model is a metal particle detection image, and the output is metal particle detection frame information. The detection result output module is used to input the metal particle image to be detected into the trained metal particle detection model and output the metal particle detection frame information in the image.

[0015] In a third aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the metal particle detection method in a complex background as described above.

[0016] The beneficial technical effects of the present invention include at least: 1. The metal particle detection method, system and medium under complex background are designed to meet the requirements of metal particle detection accuracy and real-time performance under complex background. First, the data enhancement method of multi-sample fusion is used to enhance the metal particle images under actual detection control and the metal particle images under a single color background with different data enhancement strategies, and the data set is combined with the original samples to form a data set, which can more effectively improve the diversity of samples, help the subsequent metal particle detection model to learn the feature representation of metal particle images under complex backgrounds during training, reduce the dependence on single color backgrounds, thereby improving the generalization ability of the model under unseen complex backgrounds, and thus improving the metal particle detection accuracy under complex backgrounds to a certain extent; then, for metal particles In order to solve the technical problem of complex background in particle images, SimAM attention mechanism modules are inserted into the backbone network and neck network of the YOLO model. Without introducing additional parameters, the network's attention to the target metal particles is improved, thereby improving the efficiency of the metal particle detection model in extracting target particle-related feature information and improving the detection accuracy. At the same time, in response to the real-time requirements of metal particle detection, a lightweight detection head design based on a multi-scale information fusion convolution module is proposed. The multi-scale information fusion convolution module replaces the 3×3 ordinary convolution module in the original YOLO model detection head, and combined with the idea of ​​parameter sharing, the branch task is postponed to achieve the lightweight of the YOLOv8 network model, thereby greatly improving the real-time processing capability of the metal particle detection model.

[0017] 2. In order to complementarily enhance the generalization ability of the subsequent metal particle detection model in a complex background, the present application uses the Cutmix data enhancement method to mix a partial area of ​​an image in the first original sample with a corresponding area of ​​another image, which means that the mixed image still retains the complex background features of the original image and introduces new targets, thereby being able to effectively simulate the complex background changes under actual working conditions. When the generated first new sample is subsequently used to train the metal particle detection model, it helps the model learn how to identify metal particle detection targets in a changing environment, thereby improving the model's adaptability to detecting metal particle images under actual detection conditions; 3. Under a single color background, the metal particle detection target may be easier to distinguish from the background, but it may also cause the model to rely too much on this simple distinction, resulting in overfitting of the model and inability to effectively handle background changes that may occur in practical applications, such as changes in lighting conditions, slight changes in background color, etc. To this end, the present application uses the Mosaic data enhancement method to splice and combine multiple different images in the second original sample, and each image has its own metal particle detection target and background, thereby introducing a variety of different background and detection target combinations, and then simulating the appearance of the detection target under a complex background composed of a single color, increasing the diversity of samples, and helping the metal particle detection model to learn a more robust feature representation, reduce dependence on a single color background, reduce the risk of model overfitting, make the model more robust and generalized, and effectively handle background changes that may occur in practical applications; 4. The prior art generally adopts the method of introducing the CBAM attention mechanism module into the YOLO model to improve the influence of the detection target on the prediction. However, the CBAM attention mechanism module is a two-stage attention module, which increases the computational complexity of the model and causes the overall detection accuracy of the model to decrease, limiting the flexibility of the model in processing metal particle detection in new and unseen complex backgrounds. To this end, the present application inserts the SimAM attention mechanism module after the spatial pyramid pooling layer of the backbone network of the YOLOv8 model, and inserts the SimAM attention mechanism module after each deep feature fusion layer in the neck network, so as to measure the similarity between neurons according to the energy function, so as to allocate attention weights instead of relying on fixed weights, thereby improving the flexibility of the metal particle detection model in processing metal particle detection in new and unseen complex backgrounds. Moreover, the addition of the SimAM attention mechanism module does not introduce additional parameters, so it does not cause the network model to generate additional computational complexity, thereby improving the extraction efficiency of the metal particle detection model for the target particle related feature information, and enhancing the model's attention to the metal particle detection target, thereby helping the model to more accurately locate metal particles in complex backgrounds and enhancing the overall detection accuracy of the model; 5. By adding a shared parameter module after the input of the detection head of the YOLOv8 model, the design idea of ​​the shared parameter module is to add a shared pre-processing module before these branches, and the shared parameter module uniformly processes the input of all branches, and then distributes the uniformly processed feature maps to each branch. At the same time, this application designs a multi-scale information fusion module to replace the 3×3 convolution module in the detection head of the YOLOv8 model. The multi-scale information fusion module is computationally efficient and only needs to perform convolution, grouping, linear operation, splicing and 1x1 convolution to realize the fusion of feature information, without a large number of matrix multiplications like 3x3 ordinary convolutions, thereby greatly reducing the computational complexity of the model, realizing the lightweight of the metal particle detection model, thereby improving the real-time processing capability of the metal particle detection model, and meeting the real-time requirements of the metal particle target detection task; 6. In the training process of the metal particle detection model, since the training data set will inevitably contain low-quality examples, the existing technology adds geometric metrics such as distance and aspect ratio to the IoU loss function, which will aggravate the penalty for low-quality examples, thereby reducing the generalization performance of the model. To this end, this application proposes an innovative IoU loss function calculation criterion by combining the dynamic processing idea of ​​Wise-IoU and the auxiliary border idea of ​​Inner-IoU, named IWIoU. The IWIoU proposed in this application is calculated by introducing auxiliary borders. and compare it with the adjustment factor in Wise-IoU Multiply to balance the weights between different categories. Compared with the CIoU loss function in the existing YOLOv8 model, IWIoU can not only dynamically adjust the bounding box regression loss like Wise-IoU to improve the performance of the metal particle detection model at different training stages, but also introduce auxiliary bounding boxes in IWIoU and select different scale factors to play a positive role in the regression of low IoU samples or the convergence of high IoU samples, thereby improving the generalization ability of the metal particle detection model.

[0018] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described below in conjunction with the accompanying drawings: Figure 1 The figure is a flow chart of a method for detecting metal particles in a complex background according to an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the original network architecture of the YOLOv8 model in the prior art.

[0021] Figure 3Schematic diagram of the network architecture of the metal particle detection model according to an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the structure of a lightweight detection head of a metal particle detection model according to an embodiment of the present invention.

[0023] Figure 5 Schematic diagram of the principle of a multi-scale information fusion module according to an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of the original detection head structure of the YOLOv8 model in the prior art.

[0025] Figure 7 Schematic diagram of the structure of the metal particle detection system in a complex background according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0027] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the referred device or element must have a specific direction, be constructed and operate in a specific direction. Therefore, it should not be understood as a limitation of the present invention.

[0028] Please see attached Figure 1 , Figure 1 A schematic flow chart of a method for detecting metal particles in a complex background provided by an embodiment of the present specification is shown.

[0029] like Figure 1 As shown, the metal particle detection method under complex background may at least include the following steps: Step 101 : obtaining an image of metal particles under actual detection conditions, which is recorded as a first original sample.

[0030] Among them, the background of the metal particle image under actual detection conditions is a non-uniform color, usually a complex background composed of white stripes, stains, and wrinkles caused by reflective components.

[0031] Step 102: Acquire an image of metal particles under a single color background, which is recorded as a second original sample.

[0032] Among them, the metal particle image under a single color background usually has four background colors: red, green, blue and white.

[0033] Step 103, using a multi-sample fusion data enhancement method to perform data enhancement on the first original sample and the second original sample to obtain a new sample after data enhancement, and the set of the new sample, the first original sample and the second original sample constitutes a data set.

[0034] It can be understood that the data enhancement method for multi-sample fusion in the present embodiment specifically refers to adopting different data enhancement strategies for the metal particle images under actual detection industrial control and the metal particle images under a single color background, respectively, and then taking the set of the metal particle images under actual detection industrial control and the metal particle images under a single color background after data enhancement as new samples after data enhancement.

[0035] By adopting a multi-sample fusion data enhancement method to perform different data enhancement strategies on the metal particle images under actual detection control and the metal particle images under a single color background, the diversity of samples can be more effectively improved, which will help the subsequent metal particle detection model to learn the feature representation of metal particle images under complex backgrounds during the training process, reduce dependence on single color backgrounds, thereby improving the generalization ability of the model in unseen complex backgrounds, thereby improving the metal particle detection accuracy under complex backgrounds to a certain extent.

[0036] Furthermore, in order to complementarily enhance the generalization ability of the subsequent metal particle detection model in complex backgrounds and reduce the dependence on a single color background, in this embodiment, a multi-sample fusion data enhancement method is used to perform data enhancement on the first original sample and the second original sample to obtain a new sample after data enhancement. The set of new samples constitutes a data set, including: The Cutmix data enhancement method is used to enhance the data of the first original sample, and the metal particle image under the actual detection condition after data enhancement is obtained, which is recorded as the first new sample; The second original sample is enhanced by using the Mosaic data enhancement method to obtain a metal particle image under a single color background after data enhancement, which is recorded as the second new sample; The set of the first new sample, the second new sample, the first original sample and the second original sample constitutes a data set.

[0037] On the one hand, in this embodiment, the Cutmix data enhancement method is used to perform data enhancement on the first original sample to obtain a metal particle image under actual detection conditions after data enhancement, including: Step 201 : annotate the images in the first original sample to obtain a label corresponding to each image.

[0038] Among them, labeling the image in the first original sample is to label the size, shape and position of the metal particles in the image, and the labels corresponding to each image include the size, shape and position of the metal particles framed in the image.

[0039] Step 202, randomly selecting two images from the first original sample, and randomly selecting a rectangular area; Step 203 , the two images are mixed based on the selected rectangular area, and the obtained mixed image is the metal particle image under the actual detection condition after data enhancement.

[0040] Specifically, the method for blending the two images based on the selected rectangular area is as follows: according to the selected rectangular area M, the corresponding area position in the B image is first cropped, and then the corresponding area position in the A image is placed in the corresponding cropped area in the B image to fill the pixel value of the corresponding cropped area in the B image to achieve blending of the two images.

[0041] Step 204 , calculating the mixing ratio of the two images, and performing weighted averaging on the labels corresponding to the two images based on the mixing ratio to obtain a label of the mixed image.

[0042] Specifically, the data and labels of the mixed image in this embodiment can be expressed as: in, represents the data of the resulting mixed image, represents the label of the resulting mixed image, and Respectively represent the data of the two selected images, is a binary mask, M represents the selected rectangular area, and the operator Indicates that the elements at corresponding positions are multiplied one by one. and Respectively represent the labels corresponding to the two selected images, It represents the mixing ratio of the two images, that is, the area ratio of the selected rectangular area in the two images.

[0043] It can be understood that the above steps 202 to 204 are implementation steps for using the Cutmix data enhancement method to generate a new sample by mixing two images in the first original sample. The method of using the Cutmix data enhancement method to perform data enhancement on other images in the first original sample to generate more new samples can refer to steps 202 to 204.

[0044] This embodiment uses the Cutmix data enhancement method to mix a partial area of ​​an image in the first original sample with a corresponding area of ​​another image. This means that the mixed image still retains the complex background features of the original image while introducing new targets, thereby effectively simulating the complex background changes under actual working conditions. When the generated first new sample is subsequently used to train the metal particle detection model, the model is helped to learn how to identify metal particle detection targets in a changing environment, thereby improving the model's adaptability to detecting metal particle images under actual detection conditions.

[0045] Under a single color background, the metal particle detection target may be easier to distinguish from the background, but it may also cause the model to rely too much on this simple distinction, resulting in overfitting of the model and failure to effectively handle background changes that may occur in practical applications, such as changes in lighting conditions and slight changes in background color. Therefore, on the other hand, in this embodiment, the Mosaic data enhancement method is used to enhance the data of the second original sample to obtain a metal particle image under a single color background after data enhancement, including: Step 301, annotating the images in the second original sample to obtain labels corresponding to each image; Step 302: randomly select a number of images from the second original sample, and randomly crop each selected image to determine the position of the cropping area in the selected image.

[0046] For example, the position of the cropping area in the selected image may be determined by the coordinates of the upper left corner and the lower right corner of the cropping area.

[0047] Step 303 , stitching the cropped images, and the obtained stitched image is the metal particle image under the single color background after data enhancement.

[0048] The cropped images may be spliced ​​in a random manner, which is not limited in this embodiment.

[0049] Step 304 : determining a label of the stitched image based on the position of the cropped area in the selected image and the label corresponding to the selected image.

[0050] Specifically, based on the position of the cropped area in the selected image and the label corresponding to the selected image, the label of the stitched image is determined as follows: 1. Initialize the label list: Create an empty list to store the labels of the newly generated stitched images.

[0051] 2. Traverse each cropping area: For each cropping area, determine its corresponding position in the new stitched image based on its position information in the selected image.

[0052] 3. Generate new labels: Generate corresponding labels for the new stitched image based on the labels corresponding to the selected image and the corresponding position information of the cropped area in the stitched image. For example, if the position of a cropped area in the selected image contains a label of a metal particle, then in the newly generated stitched image, the position of the metal particle should correspond to the corresponding position information of the cropped area in the stitched image, and the size and shape of the metal particle remain unchanged.

[0053] 4. Merge labels: Merge the labels in all cropped areas into a list and use this list as the label of the newly generated stitched image.

[0054] It can be understood that the above steps 302 to 304 are implementation steps for generating a new sample by splicing multiple images in the second original sample using the Mosaic data enhancement method. The method of using the Mosaic data enhancement method to perform data enhancement on other images in the second original sample to generate more new samples can refer to steps 302 to 304.

[0055] This embodiment uses the Mosaic data enhancement method to splice and combine multiple different images in the second original sample. Each image has its own metal particle detection target and background, thereby introducing a variety of different combinations of backgrounds and detection targets, and then simulating the appearance of the detection target in a complex background composed of a single color, increasing the diversity of samples, and helping the metal particle detection model to learn a more robust feature representation, reduce dependence on a single color background, reduce the risk of model overfitting, make the model more robust and generalized, and effectively handle background changes that may occur in practical applications.

[0056] Step 104, using the improved YOLOv8 model to build a metal particle detection model. The metal particle detection model inserts the SimAM attention mechanism module into the backbone network and the neck network of the YOLOv8 model, and performs a lightweight design on the detection head structure.

[0057] Among them, the YOLOv8 model provides five network models of different scales: YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l and YOLOv8x. The difference between these network models is that the larger the scale, the deeper the depth of the network model, the more channels, and the better the performance, but the memory and computational complexity of the model are also larger, which places higher requirements on the hardware platform for running the model. In addition, in view of the strong portability and modularity between the models, the small model can be converted into a large model by increasing the number of network layers and channels, which makes the small model have higher flexibility. Therefore, preferably, according to the real-time requirements of the metal particle target detection task, this embodiment selects the smallest and fastest running YOLOv8n in the YOLOv8 model as the basic model for improvement to improve the comprehensive performance of the model.

[0058] The original network architecture of the YOLOv8 model can be found in the attached Figure 2 ,The original network structure of the YOLOv8 model consists of four parts: backbone network (i.e., Backbone network), neck network (i.e., Neck network), detection head (i.e., Head network) and loss function.

[0059] The existing technology usually adopts the method of introducing the CBAM attention mechanism module into the YOLO model to improve the influence of the detection target on the prediction. However, the CBAM attention mechanism module is a two-stage attention module, which first applies attention in the channel dimension and then applies attention in the spatial dimension. This requires learning the attention weight through an additional convolutional layer, and since the CBAM attention mechanism module needs to learn two sets of weights (channel weights and spatial weights), this increases the computational complexity of the model. For the metal particle image input into the network, the target particles only occupy a small part of the overall image area, and most of the area is the background and impurity information of the image. Therefore, when the image features are iteratively calculated in the backbone network layer of the network and the CBAM attention mechanism module is used to learn the attention weights, a large amount of background information will gradually evolve into a large amount of irrelevant information, which will cover up the characteristic information of some target particles, thereby causing the overall detection accuracy of the model to decrease. At the same time, since the weights of the CBAM attention mechanism module are fixed through training, this means that once the training is completed, the weights will no longer change, thereby limiting the flexibility of the model in dealing with metal particle detection in new and unseen complex backgrounds.

[0060] Therefore, in order to improve the metal particle detection model's extraction efficiency of target particle-related feature information and enhance the overall detection accuracy and flexibility of the model, further, in this embodiment, please refer to the attached Figure 3 The metal particle detection model inserts the SimAM attention mechanism module into the backbone network and the neck network of the YOLOv8 model (i.e. Figure 3 The simplified attention module in , Simplified Attention Module, referred to as SimAM), includes: The metal particle detection model inserts a SimAM attention mechanism module after the spatial pyramid pooling layer of the backbone network of the YOLOv8 model, and inserts a SimAM attention mechanism module after each deep feature fusion layer in the neck network.

[0061] It should be noted that the type of attention mechanism module and the selection of the adding position in this embodiment are mainly based on the following considerations: (1) Adaptability and flexibility: The SimAM attention mechanism module can measure the similarity between neurons based on the energy function to assign attention weights instead of relying on fixed weights, thereby improving the flexibility of the metal particle detection model in dealing with new and unseen complex backgrounds; (2) Computational efficiency: Adding the SimAM attention mechanism module does not introduce additional parameters, so it does not cause the network model to generate additional computational workload. This improves the metal particle detection model’s efficiency in extracting feature information related to the target particles while enhancing the model’s attention to the metal particle detection target, thereby improving the network’s recognition and positioning accuracy and enhancing the model’s overall detection accuracy. (3) In the backbone network of the YOLOv8 model, the spatial pyramid pooling layer is located at the end of the Backbone network, which serves to connect the Neck network. The special bottom-up and top-down structure of the Neck network enables features of different scales to be fully integrated in the deep feature fusion layer. Therefore, this embodiment inserts a SimAM attention mechanism module after the spatial pyramid pooling layer and inserts a SimAM attention mechanism module after each deep feature fusion layer in the Neck network. This can guide the model to pay more attention to features related to the target metal particle detection task without increasing the number of parameters, thereby helping the model to more accurately locate metal particles in complex backgrounds.

[0062] Further, in view of the real-time requirements for the detection of metal particles in activated sludge, in this embodiment, please refer to the attached Figure 4 and attached Figure 5 , lightweight design of the detection head structure, including: A shared parameter module is added after the input of the detection head of the YOLOv8 model to postpone the branch task, and a multi-scale information fusion module is designed in the shared parameter module to replace the 3×3 convolution module in the detection head of the YOLOv8 model. The multi-scale information fusion module is used to perform the following steps: Perform convolution operation on the feature map of the input detection head to obtain the initial feature map; Divide the initial feature map into n groups, and perform a linear operation T on each group of feature maps to obtain a number of new feature maps, wherein the size of the kernel of the linear operation T is usually 1×1, 3×3, 5×5, or 7×7, etc., which is not limited in this embodiment; Concatenate the new feature map with the initial feature map to obtain the initial output; Perform a 1×1 convolution operation on the initial output to obtain the fused feature map.

[0063] Please see attached Figure 6 In the existing YOLOv8 model, the detection head usually contains multiple branches, each of which is responsible for different tasks, such as bounding box prediction, category classification, and confidence estimation. In the existing YOLOv8 model, the entire Head network contains 12 3×3 convolutions and 6 1×1 convolutions, which significantly increases the computational complexity of the model and makes it difficult to meet the real-time requirements of activated sludge metal particle detection.

[0064] To this end, this embodiment adds a shared parameter module after the input of the detection head of the YOLOv8 model. The design idea of ​​the shared parameter module is to add a shared pre-processing module before these branches. The shared parameter module uniformly processes the inputs of all branches, and then distributes the uniformly processed feature maps to each branch. At the same time, this embodiment designs a multi-scale information fusion module to replace the 3×3 convolution module in the detection head of the YOLOv8 model. The multi-scale information fusion module is computationally efficient and only needs to perform convolution, grouping, linear operation, splicing and 1×1 convolution to achieve the fusion of feature information, without a large number of matrix multiplications like 3×3 ordinary convolutions, thereby greatly reducing the computational complexity of the model, achieving lightweight metal particle detection model, thereby improving the real-time processing capability of the metal particle detection model, and meeting the real-time requirements of metal particle target detection tasks.

[0065] Step 105 , training a metal particle detection model based on the data set, wherein the input of the trained metal particle detection model is a metal particle detection image, and the output is metal particle detection frame information.

[0066] Among them, the implementation method of training the metal particle detection model based on the data set in this embodiment is similar to the implementation method of training the metal particle detection model based on the data set in the prior art, and this embodiment will not be repeated here.

[0067] Step 106: Acquire the image of the metal particles to be detected in real time.

[0068] Step 107 , inputting the metal particle image to be detected into the trained metal particle detection model, outputting the metal particle detection frame information in the image, and realizing metal particle detection under complex background.

[0069] IoU (Intersection over Union) is a method for calculating the degree of overlap between two regions in an image. It is used to measure the similarity between the predicted bounding box and the true bounding box in the target detection task. Specifically, IoU first calculates the intersection area of ​​the two bounding boxes, then calculates the union area of ​​the two bounding boxes, and finally divides the above intersection area by the union area to get the IoU value, which ranges from 0 to 1. The larger the IoU value, the higher the degree of overlap between the predicted bounding box given by the model and the true bounding box, and the higher the accuracy of the model in locating the object. In practical applications, an IoU threshold is usually set according to the task requirements. If the obtained IoU value is greater than the IoU threshold, the detection is considered correct. During the training process, IoU is often used as a component of the loss function to improve the learning effect of the network for target detection and segmentation tasks. However, when there is no overlap between the bounding boxes, the value of IoU cannot be calculated, resulting in its gradient being 0 and unable to be optimized, which is also a major disadvantage of the IoU loss function.

[0070] To this end, the existing YOLOv8 model usually adopts the idea of ​​aspect ratio consistency in the CIoU loss function to solve the technical problem of gradient vanishing in IoU. However, whether it is the CIoU loss function commonly used by the YOLOv8 model or the currently popular DIoU (Distance-IoU) and EIoU (Efficient IoU), they all accelerate convergence and improve detection performance by adding new geometric penalty terms to the IoU loss function, while ignoring the limitations of the IoU loss function itself. Specifically, in the training process of the metal particle detection model, since the training data set inevitably contains low-quality examples, adding geometric metrics such as distance and aspect ratio to the IoU loss function will intensify the penalty for low-quality examples, thereby reducing the generalization performance of the model. In the training process of the metal particle detection model under complex background, a good loss function should weaken the penalty for geometric metrics when the predicted candidate box and the target box overlap well, so as to avoid excessive intervention in training, so that the model has better generalization ability. To this end, in another embodiment of the present specification, the improved YOLOv8 model is used to construct a metal particle detection model, which also includes: The Wise-IoU loss function is fused with the Inner-IoU loss function to obtain the fused loss function, which is recorded as the IWIoU loss function; The metal particle detection model uses the IWIoU loss function to replace the CIoU loss function used by the YOLOv8 model.

[0071] Among them, Inner-IoU proposes to calculate IoU with auxiliary bounding boxes to improve the generalization ability of the model without adding any new penalty terms. Wise-IoU uses a dynamic method to calculate IoU in category prediction loss. The expression of Wise-IoU loss function is: in, Represents the adjustment factor in the Wise-IoU loss function, in order to prevent Produces gradients that hinder convergence, which can be and Separate from the calculation graph, the superscript * in the above formula indicates this operation. The meanings of other characters can refer to the meanings of characters in Wise-IoU in the prior art, which are similar and will not be repeated in this embodiment.

[0072] Furthermore, in this embodiment, the expression of the IWIoU loss function is: in, represents the IWIoU loss function proposed in this embodiment, Represents the adjustment factor in the Wise-IoU loss function, Represents the value of the Inner-IoU loss function, which can be expressed as: Among them, inter is the area of ​​the internal intersection, which represents the area of ​​the overlapping part of the predicted bounding box and the true bounding box, and union is the area of ​​the internal union, which represents the total area of ​​the predicted bounding box and the true bounding box.

[0073] This embodiment combines the dynamic processing idea of ​​Wise-IoU and the auxiliary border idea of ​​Inner-IoU to propose an innovative IoU loss function calculation criterion, named IWIoU. The IWIoU proposed in this embodiment is calculated by introducing an auxiliary border. and compare it with the adjustment factor in Wise-IoU Multiply to balance the weights between different categories. Compared with the CIoU loss function in the YOLOv8 model, the IWIoU proposed in this embodiment can not only dynamically adjust the border regression loss like Wise-IoU to improve the performance of the metal particle detection model at different training stages, but also introduce auxiliary borders in IWIoU and select different scale factors to play a positive role in the regression of low IoU samples or the convergence of high IoU samples, thereby improving the generalization ability of the metal particle detection model.

[0074] Please see attached Figure 7 , Figure 7 A schematic diagram of the structure of a metal particle detection system in a complex background provided in one embodiment of this specification.

[0075] like Figure 7 As shown, the metal particle detection system under complex background may at least include an image acquisition module 1, a data enhancement module 2, a model construction module 3, a model training module 4 and a detection result output module 5, wherein: Image acquisition module 1, used to acquire and annotate the metal particle image under the actual detection condition, recorded as the first original sample, acquire and annotate the metal particle image under the single color background, recorded as the second original sample, and acquire the metal particle image to be detected in real time; Data enhancement module 2, used for performing data enhancement on the first original sample and the second original sample by adopting a data enhancement method of multi-sample fusion to obtain a new sample after data enhancement, and the set of the new sample, the first original sample and the second original sample constitutes a data set; Model building module 3 is used to build a metal particle detection model using the improved YOLOv8 model. The metal particle detection model inserts the SimAM attention mechanism module into the backbone network and neck network of the YOLOv8 model, and performs a lightweight design on the detection head structure; Model training module 4, used for training the metal particle detection model based on the data set, the input of the trained metal particle detection model is the metal particle detection image, and the output is the metal particle detection frame information; The detection result output module 5 is used to input the metal particle image to be detected into the trained metal particle detection model, and output the metal particle detection frame information in the image.

[0076] It can be understood that the technical concept of the metal particle detection system under complex background provided by this embodiment is similar to the technical concept of the metal particle detection method under complex background described above, and this embodiment will not be repeated here.

[0077] Another embodiment of the present specification provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in the above-mentioned metal particle detection method in a complex background. If the component modules of the above-mentioned electronic device are implemented in the form of software functional units and predicted or used as independent downstream tasks, they can be stored in a computer-readable storage medium.

[0078] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of this specification is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains one or more available media integrations. Available media may be magnetic media (eg, floppy disks, hard disks, tapes), optical media (eg, digital versatile discs (DVD)), or semiconductor media (eg, solid state drives (SSD)).

[0079] The above description is only the preferred embodiment disclosed in this application and the description of the technical principle used. Those skilled in the art should understand that the scope of protection involved in this disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but also should cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this disclosure (but not limited to) to form a technical solution.

[0080] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Claims

1. A method for detecting metal particles under complex backgrounds, characterized in that: The following steps are involved: Acquire an image of metal particles under actual detection conditions, recorded as a first original sample; Acquire an image of metal particles under a single color background, which is recorded as the second original sample; Performing data enhancement on the first original sample and the second original sample using a data enhancement method of multi-sample fusion to obtain a new sample after data enhancement, wherein the set of the new sample, the first original sample and the second original sample constitutes a data set; The improved YOLOv8 model is used to build a metal particle detection model. The metal particle detection model inserts the SimAM attention mechanism module into the backbone network and the neck network of the YOLOv8 model, and performs a lightweight design on the detection head structure. The metal particle detection model is trained based on the data set. The input of the trained metal particle detection model is the metal particle detection image, and the output is the metal particle detection frame information. Acquire the image of metal particles to be detected in real time; The metal particle image to be detected is input into the trained metal particle detection model, and the metal particle detection frame information in the image is output.

2. The method for detecting metal particles under complex background as claimed in claim 1, characterized in that: The first original sample and the second original sample are data enhanced by using a multi-sample fusion data enhancement method to obtain a new sample after data enhancement. The set of the new sample, the first original sample and the second original sample constitutes a data set, including: The Cutmix data enhancement method is used to enhance the data of the first original sample, and the metal particle image under the actual detection condition after data enhancement is obtained, which is recorded as the first new sample; The second original sample is enhanced by using the Mosaic data enhancement method to obtain a metal particle image under a single color background after data enhancement, which is recorded as the second new sample; The set of the first new sample, the second new sample, the first original sample and the second original sample constitutes a data set.

3. The method for detecting metal particles under complex background as claimed in claim 2, characterized in that: The Cutmix data enhancement method is used to enhance the data of the first original sample to obtain the metal particle image under the actual detection condition after data enhancement, including: Annotate the images in the first original sample to obtain labels corresponding to each image; Randomly select two images from the first original sample and randomly select a rectangular area; The two images are mixed based on the selected rectangular area, and the obtained mixed image is the metal particle image under the actual detection condition after data enhancement; Calculate the mixing ratio of the two images, and perform weighted average on the labels corresponding to the two images based on the mixing ratio to obtain the label of the mixed image.

4. The method for detecting metal particles under complex background as claimed in claim 2, characterized in that: The Mosaic data enhancement method is used to enhance the data of the second original sample to obtain a metal particle image under a single color background after data enhancement, including: Annotate the images in the second original sample to obtain labels corresponding to each image; Randomly select a number of images from the second original sample, and randomly crop each selected image to determine the position of the cropped area in the selected image; The cropped images are stitched together, and the resulting stitched image is the metal particle image under the single color background after data enhancement; Based on the position of the cropped area in the selected image and the label corresponding to the selected image, a label of the stitched image is determined.

5. The method for detecting metal particles under complex background as claimed in claim 1, characterized in that: The metal particle detection model inserts the SimAM attention mechanism module into the backbone network and the neck network of the YOLOv8 model, including: The metal particle detection model inserts a SimAM attention mechanism module after the spatial pyramid pooling layer of the backbone network of the YOLOv8 model, and inserts a SimAM attention mechanism module after each deep feature fusion layer in the neck network.

6. The method for detecting metal particles under complex background as claimed in claim 1, characterized in that: The lightweight design of the detection head structure includes: A shared parameter module is added after the input of the detection head of the YOLOv8 model to postpone the branch task, and a multi-scale information fusion module is designed in the shared parameter module to replace the 3×3 convolution module in the detection head of the YOLOv8 model. The multi-scale information fusion module is used to perform the following steps: Perform convolution operation on the feature map of the input detection head to obtain the initial feature map; Divide the initial feature map into several groups, and perform linear operations on each group of feature maps to obtain several new feature maps; Concatenate the new feature map with the initial feature map to obtain the initial output; Perform a 1×1 convolution operation on the initial output to obtain the fused feature map.

7. The method for detecting metal particles under complex background as claimed in claim 1, characterized in that: The metal particle detection model constructed by using the improved YOLOv8 model also includes: The Wise-IoU loss function is fused with the Inner-IoU loss function to obtain the fused loss function, which is recorded as the IWIoU loss function; The metal particle detection model uses the IWIoU loss function to replace the CIoU loss function used by the YOLOv8 model.

8. The method for detecting metal particles under complex background as claimed in claim 7, characterized in that: The expression of the IWIoU loss function is: in, represents the IWIoU loss function, Represents the adjustment factor in the Wise-IoU loss function, Represents the value of the Inner-IoU loss function.

9. The metal particle detection system under complex background is characterized by: include: An image acquisition module is used to acquire and annotate images of metal particles under actual detection conditions, which are recorded as the first original sample; acquire and annotate images of metal particles under a single color background, which are recorded as the second original sample; and acquire images of metal particles to be detected in real time; A data enhancement module is used to perform data enhancement on the first original sample and the second original sample by adopting a data enhancement method of multi-sample fusion to obtain a new sample after data enhancement, and the set of the new sample, the first original sample and the second original sample constitutes a data set; A model building module is used to build a metal particle detection model using an improved YOLOv8 model. The metal particle detection model inserts SimAM attention mechanism modules into the backbone network and the neck network of the YOLOv8 model, and performs a lightweight design on the detection head structure. A model training module is used to train a metal particle detection model based on a data set. The input of the trained metal particle detection model is a metal particle detection image, and the output is metal particle detection frame information. The detection result output module is used to input the metal particle image to be detected into the trained metal particle detection model and output the metal particle detection frame information in the image.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for detecting metal particles in a complex background according to any one of claims 1 to 8 is implemented.

Images