Visual inspection method and device for small target weld surface defects and medium thereof

Abstract

The invention provides a visual detection method and device for small target weld surface defects and a medium thereof.The method comprises the steps that a target weld surface picture is obtained, then defect positioning and detection are conducted on the target weld surface picture through a weld defect recognition model, and a defect analysis result is obtained; the construction process of the welding seam defect identification model comprises the following steps: acquiring a welding seam defect sample set and dividing the welding seam defect sample set into a welding seam defect training set and a welding seam defect verification set; the method comprises the steps that a YOLOv8 improved network model is constructed, the YOLOv8 improved network model comprises a backbone network and a neck network, the backbone network is provided with a ResBlockCBAM module and a Ghost ghost module, and the neck network is provided with an SPPELAN module; performing iterative training on the YOLOv8 improved network model according to the weld defect training set to obtain a weld defect training model; and according to the weld defect verification set, optimizing the weld defect training model to obtain the weld defect identification model, thereby improving the detection capability of the weld surface defects, and reducing the parameter quantity of the model at the same time.

Description

Technical Field

[0001] The present application relates to the field of rail transit, and in particular to a method, device and medium for visually detecting small target weld surface defects. Background Art

[0002] The body of a rail transit vehicle is mainly formed by welding a large number of metal materials such as aluminum alloy, carbon steel and stainless steel. The final quality of the weld is crucial to the safe operation of rail transit vehicles. During the body manufacturing process, the final quality of the weld is affected by factors such as the welder's ability, process method, and working environment. At present, the welds in key parts of the body are inspected by surface flaw detection or volume flaw detection, focusing on the quality of welds in key parts; other welds in the body are inspected manually by visual inspection to identify surface defects of the welds. However, due to the huge number of body welds, the workload of manual visual inspection is large, and the inspection results vary from person to person. At this stage, we are mainly exploring the use of artificial intelligence visual recognition technology to replace manual identification of weld surface defects.

[0003] Among the existing technologies, the machine vision detection method for rail transit vehicle body weld defect detection mainly uses the improved YOLOv5 algorithm to detect weld defects. However, due to the complex background of the weld on the body surface and the overlap between defects, most existing methods have problems such as low detection accuracy of small targets in complex environments, difficulty in detection, missed detection and false detection, and difficulty in deployment due to the large model size. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a visual detection method, device and medium based on small target weld surface defects. By introducing the ResBlock_CBAM module, Ghost module and SPPELAN module into the original YOLOv8 model, the detection capability of small target weld surface defects is improved, while the number of model parameters is reduced, thereby achieving lightweight deployment of the model.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a visual detection method for small target weld surface defects, comprising: Obtain target weld surface image; Through the weld defect recognition model built based on the YOLOv8 algorithm, the defect location and detection of the target weld surface image are carried out to obtain the defect analysis result of the target weld surface image; Among them, the construction process of the weld defect recognition model includes the following steps: Obtain a labeled weld defect sample set, and divide the weld defect sample set into a weld defect training set and a weld defect verification set; Construct a YOLOv8 improved network model. The YOLOv8 improved network model includes a backbone network and a neck network. The convolution module of the backbone network is equipped with a ResBlock_CBAM module, the bottleneck module of the backbone network is equipped with a Ghost module, and the neck network is equipped with a SPPELAN module. According to the weld defect training set, supervised iterative training is performed on the YOLOv8 improved network model to obtain the weld defect training model; According to the weld defect verification set, the weld defect training model is verified and optimized to obtain the weld defect recognition model.

[0006] Furthermore, in some embodiments, the YOLOv8 improved network model also includes a head network, and each iterative training process of the YOLOv8 improved network model includes the following steps: Input the weld defect training set into the backbone network for feature extraction to obtain the first weld defect feature set; The first weld defect feature set is input into the neck network, and based on the SPPELAN module in the neck network, the first weld defect feature set is subjected to feature enhancement processing to obtain the second weld defect feature set; Inputting the second weld defect feature set into the head network for defect prediction to obtain a weld defect prediction set; Based on the anchor frame matching method, the weld defect prediction set and the weld defect training set are paired in position to obtain the difference position data set between the weld defect prediction set and the weld defect training set. Based on the loss function, the gradient difference is calculated for the difference position data set to obtain the gradient loss value of the current iteration; According to the gradient loss value, the multi-layer network parameters of the YOLOv8 improved network model are back-propagated and updated through the iterative optimizer; If the multi-layer network parameters of the YOLOv8 improved network model in the current iterative training meet the preset training requirements, the current iterative training of the YOLOv8 improved network model is terminated to obtain the weld defect training model.

[0007] Further, in some embodiments, the SPPELAN module includes a preprocessing layer, a spatial pyramid pooling layer, an enhanced local attention network layer, a spatial splicing layer and a fused convolution layer, the preprocessing layer is connected to the upsampling layer of the neck network, the spatial pyramid pooling layer and the enhanced local attention network layer are respectively connected to the preprocessing layer, the spatial splicing layer is respectively connected to the spatial pyramid pooling layer and the enhanced local attention network layer, and the fused convolution layer is connected to the spatial splicing layer; Among them, based on the SPPELAN module in the neck network, the first weld defect feature set is subjected to feature enhancement processing to obtain the second weld defect feature set, including the following steps: Based on the preprocessing layer, the first weld defect feature set output by the upsampling layer of the neck network is preprocessed to obtain a third weld defect feature set; Based on the enhanced local attention network layer, the third weld defect feature set is reduced in dimension to obtain the fourth weld defect feature set; Based on the spatial pyramid pooling layer, multi-scale feature extraction is performed on the feature map output by the upsampling layer to obtain the fifth weld defect feature set of different spatial scales; Based on the spatial stitching layer, the fourth weld defect feature set is stitched with each fifth weld defect feature set to obtain a sixth weld defect feature set; Based on the fused convolutional layer, the channel dimension of the sixth weld defect feature set is adjusted to obtain the second weld defect feature set.

[0008] Furthermore, in some embodiments, the weld defect training set is input into the backbone network for feature extraction to obtain a first weld defect feature set, including: The weld defect training set is input into the convolution module of the backbone network, and based on the ResBlock_CBAM module in the convolution module, the weld defect training set is subjected to deep feature extraction to obtain the third weld defect feature set; The third weld defect feature set is input into the bottleneck module of the backbone network, and based on the Ghost module in the bottleneck module, the third weld defect feature set is lightweight-fused to obtain the first weld defect feature set.

[0009] Further, in some embodiments, the ResBlock_CBAM module includes a residual network layer and a convolutional block attention layer, the convolutional block attention layer is connected to the middle convolutional layer of the convolutional module, and the residual network layer is connected between the initial convolutional layer and the last convolutional layer of the convolutional module in a skip connection manner; Among them, based on the ResBlock_CBAM module in the convolution module, deep feature extraction is performed on the weld defect training set to obtain the third weld defect feature set, including the following steps: Based on the convolutional block attention layer, the weld defect training set processed by the intermediate convolutional layer is weighted to obtain the seventh weld defect feature set; Based on the residual network layer, residual convolution is performed on the weld defect training set output by the initial convolution layer to obtain the eighth weld defect feature set, and the seventh weld defect feature set and the eighth weld defect feature set are spatially superimposed to obtain the third weld defect feature set.

[0010] Further, in some embodiments, the convolutional block attention layer includes a channel attention layer and a spatial attention layer, and the convolutional block attention layer is connected to the residual network layer; Among them, based on the convolution block attention layer, the weld defect training set processed by the intermediate convolution layer is weighted to obtain the seventh weld defect feature set, including the following steps: Based on the channel attention layer, the weld defect training set processed by the intermediate convolution layer is subjected to channel importance weighting processing to obtain a ninth weld defect feature set, and the weld defect training set processed by the intermediate convolution layer is element-wise multiplied with the ninth weld defect feature set to obtain a tenth weld defect feature set; Based on the spatial attention layer, the tenth weld defect feature set is spatially weighted to obtain the eleventh weld defect feature set, and the tenth weld defect feature set is element-wise multiplied with the eleventh weld defect feature set to obtain the seventh weld defect feature set.

[0011] Further, in some embodiments, the Ghost module includes a first Ghost convolutional layer, a second Ghost convolutional layer and a depth-separable network layer, the first Ghost convolutional layer is connected to the bottleneck input layer of the bottleneck module, the second Ghost convolutional layer is connected to the first Ghost convolutional layer, the bottleneck output layer of the bottleneck module is connected to the second Ghost convolutional layer, and the depth-separable network layer is connected between the bottleneck input layer and the bottleneck output layer in a skip connection manner; Among them, based on the Ghost module in the bottleneck module, the third weld defect feature set is lightweight-fused to obtain the first weld defect feature set, including the following steps: Based on the first Ghost convolutional layer, the third weld defect feature set of the bottleneck input layer is spatially expanded to obtain the twelfth weld defect feature set; Based on the second Ghost convolutional layer, the spatial feature dimension reduction of the twelfth weld defect feature set is performed to obtain the thirteenth weld defect feature set; Based on the depthwise separable convolutional layer, spatial feature extraction is performed on the third weld defect feature set of the bottleneck input layer to obtain the fourteenth weld defect feature set, and the thirteenth weld defect feature set and the fourteenth weld defect feature set are spatially fused to obtain the third weld defect feature set.

[0012] Furthermore, in some embodiments, before obtaining the labeled weld defect sample set, the visual inspection method further includes: Obtain the original weld defect image set; According to the appearance of weld surface defects of rail transit vehicle bodies, the original weld defect image set is classified to obtain multiple weld defect type image sets; The image annotation tool is used to annotate each weld defect type image set to obtain a labeled weld defect sample set.

[0013] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the visual detection method of the above-mentioned first aspect embodiment when executing the computer program.

[0014] To achieve the above-mentioned purpose, the third aspect of the embodiment of the present application proposes a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the visual inspection method of the above-mentioned first aspect embodiment is implemented.

[0015] The embodiments of the present application have the following beneficial effects: the present application obtains a target weld surface image, and then uses a weld defect recognition model constructed based on the YOLOv8 algorithm to locate and detect defects in the target weld surface image, thereby obtaining a defect analysis result of the target weld surface image; wherein, in the process of constructing the weld defect recognition model, the following steps are included: obtaining a labeled weld defect sample set, and dividing the weld defect sample set into a weld defect training set and a weld defect verification set; constructing a YOLOv8 improved network model, the YOLOv8 improved network model includes a backbone network and a neck network, the convolution module of the backbone network is provided with a ResBlock_CBAM module, and the bottleneck module of the backbone network is provided with a There is a Ghost module, and the neck network is equipped with a SPPELAN module; according to the weld defect training set, the YOLOv8 improved network model is supervised iterative training to obtain the weld defect training model; according to the weld defect verification set, the weld defect training model is verified and optimized to obtain the weld defect recognition model, and then based on the introduction of ResBlock_CBAM module, Ghost module and SPPELAN module in the original YOLOv8 model, the detection capability of small target weld surface defects is improved, and the parameter amount of the model is reduced, and the lightweight deployment of the model is realized, that is, the model can be lightweight deployed in mobile devices, which further facilitates the weld defect detection work. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a method for visually detecting small target weld surface defects provided by some embodiments of the present application; Figure 2 is a flow chart of a visual detection method for small target weld surface defects provided by other embodiments of the present application; Figure 3 is a schematic diagram of a target weld surface provided by some embodiments of the present application; Figure 4 is a schematic diagram of defect analysis results of a target weld surface image provided by some embodiments of the present application; Figure 5is a flow chart of constructing a weld defect recognition model provided by some embodiments of the present application; Figure 6 It is a flowchart of each iterative training of the YOLOv8 improved network model provided in some embodiments of the present application; Figure 7 is a structural diagram of the SPPELAN module provided in some embodiments of the present application; Figure 8 Some embodiments of the present application provide Figure 6 Flow chart of step S602; Fig. 9 Some embodiments of the present application provide Figure 6 Flow chart of step S601 in FIG. Fig.10 is a structural diagram of a ResBlock_CBAM module provided in some embodiments of the present application; Fig.11 Some embodiments of the present application provide Fig. 9 Flow chart of step S901 in FIG. Fig.12 Some embodiments of the present application provide Fig.11 Flow chart of step S1101 in FIG. Fig.13 is a structural diagram of a bottleneck module provided in some embodiments of the present application; Fig.14 is a structural diagram of a Ghost module provided in some embodiments of the present application; Fig.15 Some embodiments of the present application provide Fig. 9 Flowchart of step S902 in FIG. Fig.16 It is a flow chart before obtaining a labeled weld defect sample set provided by some embodiments of the present application; Fig.17 It is a schematic diagram of the accuracy of detecting small target defects of pores by weld defect recognition models constructed by different YOLOv8 algorithms provided in some embodiments of the present application; Fig.18 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] In the description of the present application, it should be understood that descriptions involving orientation, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0019] It should also be noted that in the description of this application, "several" means more than one, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used to distinguish the technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0021] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0022] Among the existing technologies, the machine vision detection method for rail transit vehicle body weld defect detection mainly uses the improved YOLOv5 algorithm to detect weld defects. However, due to the complex background of the weld on the body surface and the overlap between defects, most existing methods have problems such as low detection accuracy of small targets in complex environments, difficulty in detection, missed detection and false detection, and difficulty in deployment due to the large model size.

[0023] Based on this, the present application obtains a target weld surface image, and then uses a weld defect recognition model constructed based on the YOLOv8 algorithm to locate and detect defects in the target weld surface image, thereby obtaining a defect analysis result of the target weld surface image; wherein, in the process of constructing the weld defect recognition model, the following steps are included: obtaining a labeled weld defect sample set, and dividing the weld defect sample set into a weld defect training set and a weld defect verification set; constructing a YOLOv8 improved network model, the YOLOv8 improved network model includes a backbone network and a neck network, the convolution module of the backbone network is provided with a ResBlock_CBAM module, and the bottleneck module of the backbone network is provided with a Ghost The neck network is equipped with a SPPELAN module; according to the weld defect training set, the YOLOv8 improved network model is supervised iteratively trained to obtain the weld defect training model; according to the weld defect verification set, the weld defect training model is verified and optimized to obtain the weld defect recognition model, and then based on the introduction of ResBlock_CBAM module, Ghost module and SPPELAN module in the original YOLOv8 model, the detection capability of small target weld surface defects is improved, and the parameter amount of the model is reduced, and the lightweight deployment of the model is realized, that is, the model can be lightweight deployed in mobile devices, which further facilitates the weld defect detection work.

[0024] The embodiments of the present application provide a method, device and medium, system and storage medium for visual detection of small target weld surface defects, which are specifically described through the following embodiments.

[0025] In the first aspect, a marker screening system based on spatial omics in an embodiment of the present application is first described.

[0026] Reference Figure 1 and Figure 2 As shown, Figure 1 is a flow chart of a visual detection method for small target weld surface defects provided by some embodiments of the present application, Figure 2 It is a flow chart of a visual detection method for small target weld surface defects provided in some other embodiments of the present application. The visual detection method may include but is not limited to steps S101 to S102.

[0027] Step S101: Acquire a target weld surface image.

[0028] In an optional embodiment, referring to Figure 3 As shown, Figure 3 is a schematic diagram of a target weld surface provided in some embodiments of the present application.

[0029] Step S102: Using a weld defect recognition model constructed based on the YOLOv8 algorithm, defects are located and detected on the target weld surface image to obtain a defect analysis result of the target weld surface image.

[0030] In an optional embodiment, referring to Figure 4 As shown, Figure 4 It is a schematic diagram of the defect analysis results of the target weld surface image provided in some embodiments of the present application. By using a weld defect recognition model constructed based on the YOLOv8 algorithm, defects in the target weld surface image are located and detected to obtain the defect analysis results of the target weld surface image.

[0031] Among them, refer to Figure 5 As shown, Figure 5 It is a flowchart of constructing a weld defect recognition model provided by some embodiments of the present application. The method for constructing a weld defect recognition model may include but is not limited to steps S501 to S504.

[0032] Step S501: Obtain a labeled weld defect sample set, and divide the weld defect sample set into a weld defect training set and a weld defect verification set.

[0033] Specifically, a labeled weld defect sample set is obtained, and then data preprocessing is performed on the weld defect sample set to obtain a preprocessed weld defect sample set, and then the weld defect sample set is divided into a weld defect training set and a weld defect verification set by a retention method.

[0034] Step S502: construct a YOLOv8 improved network model.

[0035] Among them, the YOLOv8 improved network model includes Backbone network, Neck network and Head network. The Backbone network is connected to the Neck network, the Neck network is connected to the Head network, the convolution module of the backbone network is equipped with a ResBlock_CBAM module, the bottleneck module of the backbone network is equipped with a Ghost module, and the neck network is equipped with a SPPELAN module.

[0036] Step S503: According to the weld defect training set, supervised iterative training is performed on the YOLOv8 improved network model to obtain a weld defect training model.

[0037] Specifically, the weld defect training set is input into the YOLOv8 improved network model so that the YOLOv8 improved network model can perform supervised iterative training to obtain the weld defect training model.

[0038] Step S504: verify and optimize the weld defect training model according to the weld defect verification set to obtain a weld defect recognition model.

[0039] In an optional embodiment, the weld defect verification set is input into the weld defect training model for forward propagation to obtain defect prediction results, the defect prediction results are compared with the annotation information in the weld defect verification set, and various evaluation indicators, such as accuracy, recall rate, F1 score, etc., are calculated. Then, according to the results of the evaluation indicators, the performance of the weld defect training model on various weld defects is analyzed, and the structure or parameters of the model are adjusted to improve the performance of the model. For example, the number of convolutional layers can be increased to improve the model's ability to detect tiny defects; or hyperparameters such as learning rate and batch size can be adjusted to optimize the model's training process. Finally, the adjusted weld defect training model and parameters are used to retrain the model, and the evaluation is verified again through the weld defect verification set until the performance of the weld defect training model on various weld defects meets the expected requirements, thereby obtaining a weld defect recognition model.

[0040] In an optional embodiment, referring to Figure 6 As shown, Figure 6 It is a flowchart of each iterative training of the YOLOv8 improved network model provided in some embodiments of the present application. The iterative training method of the YOLOv8 improved network model may include but is not limited to steps S601 to S606.

[0041] Step S601: input the weld defect training set into the backbone network for feature extraction to obtain a first weld defect feature set.

[0042] In an optional embodiment, the image data in the weld defect training set is input into the Backbone network in batches. The backbone network extracts features of the input images through a series of convolutional layers, pooling layers and other structures to obtain a first weld defect feature set.

[0043] It should be noted that during the feature extraction process, the network will gradually learn the low-level features (such as edges, textures) and high-level features (such as semantic information) in the image.

[0044] Step S602: Input the first weld defect feature set into the neck network, and based on the SPPELAN module in the neck network, perform feature enhancement processing on the first weld defect feature set to obtain a second weld defect feature set.

[0045] In an optional embodiment, the image data in the first weld defect feature set is input into the Neck network in batches, and based on the SPPELAN module in the Neck network, the first weld defect feature set is subjected to feature enhancement processing to obtain the second weld defect feature set.

[0046] Step S603: Based on the anchor frame matching method, the weld defect prediction set and the weld defect training set are positionally matched to obtain a difference position data set between the weld defect prediction set and the weld defect training set.

[0047] In an optional embodiment, based on the anchor frame matching method, the weld defect prediction set and the weld defect training set are positionally paired, that is, for each prediction frame in the weld defect prediction set, its IoU value with each real frame in the weld defect training set is calculated to obtain the difference position data set between the weld defect prediction set and the weld defect training set.

[0048] It should be noted that IoU (Intersection over Union) is a commonly used matching metric in target detection, which is used to calculate the degree of overlap between the predicted box and the real box.

[0049] Step S604: Based on the loss function, a gradient difference calculation is performed on the difference position data set to obtain a gradient loss value of the current iteration.

[0050] In an optional embodiment, based on the mean squared error loss function, the loss value between the model prediction result and the true label of the weld defect training set is calculated, and then, using the chain rule, starting from the loss function, the gradient difference between the model prediction result and the weld defect training set is calculated layer by layer according to the loss value to obtain the gradient loss value of the current iteration.

[0051] Step S605: According to the gradient loss value, the multi-layer network parameters of the YOLOv8 improved network model are back-propagated and updated through the iterative optimizer.

[0052] In an optional embodiment, the multi-layer network parameters of the YOLOv8 improved network model are back-propagated and updated through the Adam iterative optimizer according to the gradient loss value.

[0053] Step S606: If the multi-layer network parameters of the YOLOv8 improved network model in the current iterative training meet the preset training requirements, the current iterative training of the YOLOv8 improved network model is terminated to obtain a weld defect training model.

[0054] In an optional embodiment, if the multi-layer network parameters of the YOLOv8 improved network model in the current iterative training meet the preset training requirements (for example: the gradient loss value reaches a certain threshold, the recall rate of the model reaches a certain threshold), the current iterative training of the YOLOv8 improved network model is terminated to obtain the weld defect training model.

[0055] It should be noted that, refer to Figure 7 As shown, Figure 7 It is a structural diagram of the SPPELAN module provided by some embodiments of the present application. The SPPELAN module includes a Transition preprocessing layer, a spatial pyramid pooling layer, a Conv enhanced local attention network layer, a Concat spatial splicing layer and a Conv fusion convolution layer. The spatial pyramid pooling layer includes multiple MaxPool2d layers of different spatial scales. The Transition preprocessing layer is connected to the upsampling layer of the Neck network, the spatial pyramid pooling layer and the Conv enhanced local attention network layer are respectively connected to the Transition preprocessing layer, the Concat spatial splicing layer is respectively connected to the spatial pyramid pooling layer and the Conv enhanced local attention network layer, and the Conv fusion convolution layer is connected to the Concat spatial splicing layer.

[0056] Reference Figure 8 As shown, Figure 8 Some embodiments of the present application provide Figure 6 The method may include but is not limited to steps S801 to S805.

[0057] Step S801: Based on the preprocessing layer, the first weld defect feature set output by the upsampling layer of the neck network is preprocessed to obtain a third weld defect feature set.

[0058] In an optional embodiment, based on the Transition preprocessing layer, the first weld defect feature set output by the upsampling layer of the neck network is preprocessed to obtain a third weld defect feature set.

[0059] Step S802: Based on the enhanced local attention network layer, the third weld defect feature set is reduced in dimension to obtain a fourth weld defect feature set.

[0060] Among them, the Conv enhanced local attention network layer is a 1x1 convolution layer.

[0061] In an optional embodiment, based on the Conv enhanced local attention network layer, the third weld defect feature set is reduced in dimension to obtain a fourth weld defect feature set.

[0062] Step S803: performing multi-scale feature extraction on the feature map output by the upsampling layer based on the spatial pyramid pooling layer to obtain a fifth weld defect feature set of different spatial scales.

[0063] In an optional embodiment, based on the spatial pyramid pooling layer, multiple MaxPool2d layers of different spatial scales perform multi-scale feature extraction on the feature map output by the upsampling layer to obtain a fifth weld defect feature set of different spatial scales.

[0064] Step S804: Based on the spatial stitching layer, the fourth weld defect feature set is stitched with each fifth weld defect feature set to obtain a sixth weld defect feature set.

[0065] In an optional embodiment, based on the Concat spatial splicing layer, the fourth weld defect feature set and each fifth weld defect feature set are spatially channel spliced ​​to obtain a sixth weld defect feature set.

[0066] Step S805: Based on the fused convolutional layer, the channel dimension of the sixth weld defect feature set is adjusted to obtain a second weld defect feature set.

[0067] It should be noted that the Conv fusion convolution layer is a 1x1 convolution layer.

[0068] In an optional embodiment, based on the Conv fusion convolution layer, the channel dimension of the sixth weld defect feature set is adjusted to obtain the second weld defect feature set. It should be noted that in the process from step S801 to step S805, the SPPELAN module can be used to effectively capture the global information and local information of the first weld defect feature set, thereby enhancing the detection capability of the YOLOv8 network for multi-scale targets.

[0069] Reference Fig. 9 As shown, Fig. 9 Some embodiments of the present application provide Figure 6 The method may include but is not limited to steps S901 to S902.

[0070] Step S901: input the weld defect training set into the convolution module of the backbone network, and based on the ResBlock_CBAM module in the convolution module, perform deep feature extraction on the weld defect training set to obtain a third weld defect feature set.

[0071] In a possible embodiment, the weld defect training set is input into the ConvModule convolution module of the backbone network, and based on the ResBlock_CBAM module in the ConvModule convolution module, deep feature extraction is performed on the weld defect training set to obtain a third weld defect feature set.

[0072] Step S902: input the third weld defect feature set into the bottleneck module of the backbone network, and based on the Ghost module in the bottleneck module, perform lightweight feature fusion on the third weld defect feature set to obtain the first weld defect feature set.

[0073] Specifically, the third weld defect feature set is input into the GhostBottleNeck bottleneck module of the backbone network, and based on the Ghost module in the GhostBottleNeck bottleneck module, the third weld defect feature set is lightweight-fused to obtain the first weld defect feature set.

[0074] Further, refer to Fig.10 As shown, Fig.10 This is a structural diagram of the ResBlock_CBAM module provided in some embodiments of the present application. The ResBlock_CBAM module includes a ResBlock residual network layer and a CBAM convolutional block attention layer. The CBAM convolutional block attention layer is connected to the middle convolutional layer of the convolutional module. The ResBlock residual network layer is connected between the initial convolutional layer and the last convolutional layer of the convolutional module in a jump connection manner. The CBAM convolutional block attention layer includes a channel attention layer and a spatial attention layer. The convolutional block attention layer is connected to the residual network layer.

[0075] It should be noted that the middle convolution layer of the convolution module includes two ordinary convolution layers.

[0076] Reference Fig.11 As shown, Fig.11 Some embodiments of the present application provide Fig. 9 In the flowchart of step S901, the method may include but is not limited to steps S1101 to S1102.

[0077] Step S1101: Based on the convolutional block attention layer, the weld defect training set processed by the intermediate convolutional layer is weighted to obtain the seventh weld defect feature set.

[0078] Specifically, based on the CBAM convolutional block attention layer, the weld defect training set processed by the intermediate convolutional layer is weighted to obtain the seventh weld defect feature set.

[0079] Step S1102: Based on the residual network layer, residual convolution is performed on the weld defect training set output by the initial convolution layer to obtain an eighth weld defect feature set, and the seventh weld defect feature set and the eighth weld defect feature set are spatially superimposed to obtain a third weld defect feature set.

[0080] Specifically, based on the ResBlock residual network layer, residual convolution is performed on the weld defect training set output by the initial convolution layer to obtain the eighth weld defect feature set, and the seventh weld defect feature set and the eighth weld defect feature set are spatially superimposed to obtain the third weld defect feature set.

[0081] Reference Fig.12 As shown, Fig.12 Some embodiments of the present application provide Fig.11 The method may include but is not limited to steps S1201 to S1202.

[0082] Step S1201: Based on the channel attention layer, the weld defect training set processed by the intermediate convolution layer is subjected to channel importance weighting processing to obtain a ninth weld defect feature set, and the weld defect training set processed by the intermediate convolution layer is element-wise multiplied with the ninth weld defect feature set to obtain a tenth weld defect feature set.

[0083] Specifically, in the channel attention layer, the weight of each channel is dynamically adjusted based on the importance of each channel in the channel attention layer, and then the weld defect training set after processing by the intermediate convolution layer is weighted by channel importance to obtain the ninth weld defect feature set, and the weld defect training set after processing by the intermediate convolution layer is multiplied element by element with the ninth weld defect feature set to obtain the tenth weld defect feature set.

[0084] Step S1202: Based on the spatial attention layer, perform spatial position weighted processing on the tenth weld defect feature set to obtain the eleventh weld defect feature set, and multiply the tenth weld defect feature set by the eleventh weld defect feature set element by element to obtain the seventh weld defect feature set.

[0085] Specifically, the weight of each spatial position is dynamically adjusted in the spatial attention layer, and then the tenth weld defect feature set is spatially weighted to obtain the eleventh weld defect feature set, and the tenth weld defect feature set is element-by-element multiplied with the eleventh weld defect feature set to obtain the seventh weld defect feature set.

[0086] Further, refer to Fig.13 and Fig.14 As shown, Fig.13 is a structural diagram of a bottleneck module provided in some embodiments of the present application, Fig.14It is a structural diagram of the Ghost module provided in some embodiments of the present application. The GhostBottleNeck bottleneck module is connected to the ConvBNSiLU activation layer of the Backbone backbone network, and the Concat splicing layer of the Backbone backbone network is connected to the GhostBottleNeck bottleneck module.

[0087] Among them, the GhostBottleNeck bottleneck module includes an Input bottleneck input layer and an Input bottleneck output layer, the Ghost module includes a first Ghost convolutional layer, a second Ghost convolutional layer and a DWConv deeply separable network layer, the first Ghost convolutional layer is connected to the Input bottleneck input layer of the bottleneck module, the second Ghost convolutional layer is connected to the first Ghost convolutional layer, the Input bottleneck output layer of the bottleneck module is connected to the second Ghost convolutional layer, and the DWConv deeply separable network layer is connected between the Input bottleneck input layer and the Input bottleneck output layer in a jump connection manner.

[0088] Reference Fig.15 As shown, Fig.15 Some embodiments of the present application provide Fig. 9 In the flowchart of step S902, the method may include but is not limited to steps S1501 to S1503.

[0089] Step S1501: Based on the first Ghost convolutional layer, the third weld defect feature set of the bottleneck input layer is spatially expanded to obtain a twelfth weld defect feature set.

[0090] Specifically, based on the first Ghost convolutional layer, the third weld defect feature set of the bottleneck input layer is spatially expanded to obtain the twelfth weld defect feature set.

[0091] Step S1502: Based on the second Ghost convolutional layer, perform spatial feature dimensionality reduction on the twelfth weld defect feature set to obtain the thirteenth weld defect feature set.

[0092] Specifically, based on the second Ghost convolution layer, the spatial feature dimension reduction is performed on the twelfth weld defect feature set output by the first Ghost convolution layer to obtain the thirteenth weld defect feature set.

[0093] Step S1503: Based on the depth-separable convolutional layer, perform spatial feature extraction on the third weld defect feature set of the bottleneck input layer to obtain a fourteenth weld defect feature set, and perform spatial fusion processing on the thirteenth weld defect feature set and the fourteenth weld defect feature set to obtain a third weld defect feature set.

[0094] Specifically, based on the depthwise separable convolutional layer, the spatial features of the third weld defect feature set of the bottleneck input layer are extracted to obtain the fourteenth weld defect feature set, and then the thirteenth weld defect feature set output by the second Ghost convolutional layer is spatially fused with the fourteenth weld defect feature set to obtain the third weld defect feature set.

[0095] Reference Fig.16 As shown, Fig.16 This is a flowchart provided by some embodiments of the present application before obtaining a labeled weld defect sample set. The visual inspection method may include but is not limited to steps S1601 to S1603.

[0096] Step S1601: Obtain the original weld defect image set.

[0097] Specifically, an original weld defect image set is obtained.

[0098] Step S1602: Classify the original weld defect image set according to the weld surface defect appearance of the rail transit vehicle body to obtain multiple weld defect type image sets.

[0099] In an optional embodiment, according to the morphology of weld surface defects of a rail transit vehicle body, the weld defect type image collection can be divided into four types: (a) weld bead; (b) undercut; (c) pores; and (d) depressions.

[0100] Step S1603: Use an image annotation tool to perform type annotation on each weld defect type image set to obtain a labeled weld defect sample set.

[0101] In an optional embodiment, each weld defect type image set is labeled by using the image labeling tool labelimg to obtain a labeled weld defect sample set.

[0102] Furthermore, it should be noted that the weld defect recognition models constructed by different YOLOv8 algorithms perform defect detection on the target weld surface images to obtain comparative experimental results, as described in Table 1 below.

[0103] Table 1

[0104] Reference Fig.17 As shown, Fig.17 Schematic diagram of the accuracy of detecting small target defects of pores by weld defect recognition models constructed by different YOLOv8 algorithms provided in some embodiments of the present application. In summary, according to Table 1 and Fig.17According to data analysis, the average accuracy of this application in the self-built data set is improved by 5.7% compared with YOLOv8, reaching 94.6%. The improved weld defect recognition model significantly improves the detection capability of small targets, and provides a more efficient and accurate method for weld surface defect detection of rail transit vehicle bodies. At the same time, it also reduces the number of model parameters and realizes lightweight deployment of the model, that is, the model can be lightweight deployed in mobile devices, which further facilitates weld defect detection, and provides strong technical support for real-time surface defect detection algorithms and the deployability of models.

[0105] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned visual detection method based on small target weld surface defects when executing the computer program. The electronic device can be any smart terminal including a mobile phone, a tablet computer, a car computer, etc.

[0106] See also Fig.18 , Fig.18 : is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application, and the electronic device includes: The processor 1801 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the QSPI serial port transfer method and / or cache data reading method provided in the embodiments of the present application; The memory 1802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1802 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1802, and the processor 1801 calls and executes the QSPI serial port transfer method and / or cache data reading method provided in the embodiments of this application; Input / output interface 1803, used to implement information input and output; Communication interface 1804, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 1805 that transmits information between the various components of the device (e.g., the processor 1801 , the memory 1802 , the input / output interface 1803 , and the communication interface 1804 ); The processor 1801 , the memory 1802 , the input / output interface 1803 and the communication interface 1804 are connected to each other in communication within the device via the bus 1805 .

[0107] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the visual detection method based on small target weld surface defects provided in the embodiment of the present application is implemented.

[0108] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0109] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0110] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0111] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0112] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0113] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0114] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0116] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0119] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A visual inspection method for small target weld surface defects, characterized in that: include: Obtain target weld surface image; By using a weld defect recognition model built based on the YOLOv8 algorithm, defects are located and detected on the target weld surface image to obtain a defect analysis result of the target weld surface image; The process of constructing the weld defect recognition model includes the following steps: Obtaining a labeled weld defect sample set, and dividing the weld defect sample set into a weld defect training set and a weld defect verification set; Constructing a YOLOv8 improved network model, the YOLOv8 improved network model includes a backbone network and a neck network, the convolution module of the backbone network is provided with a ResBlock_CBAM module, the bottleneck module of the backbone network is provided with a Ghost module, and the neck network is provided with a SPPELAN module; According to the weld defect training set, supervised iterative training is performed on the YOLOv8 improved network model to obtain a weld defect training model; The weld defect training model is verified and optimized according to the weld defect verification set to obtain the weld defect recognition model.

2. The visual inspection method according to claim 1, characterized in that: The YOLOv8 improved network model also includes a head network. In each iterative training process of the YOLOv8 improved network model, the following steps are included: Inputting the weld defect training set into the backbone network for feature extraction to obtain a first weld defect feature set; Inputting the first weld defect feature set into the neck network, and performing feature enhancement processing on the first weld defect feature set based on the SPPELAN module in the neck network to obtain a second weld defect feature set; Inputting the second weld defect feature set into the head network for defect prediction to obtain a weld defect prediction set; Based on the anchor frame matching method, the weld defect prediction set and the weld defect training set are positionally paired to obtain a difference position data set between the weld defect prediction set and the weld defect training set; Based on the loss function, a gradient difference calculation is performed on the difference position data set to obtain a gradient loss value of the current iteration; According to the gradient loss value, back-propagation update is performed on the multi-layer network parameters of the YOLOv8 improved network model through an iterative optimizer; If the multi-layer network parameters of the YOLOv8 improved network model in the current iterative training meet the preset training requirements, the current iterative training of the YOLOv8 improved network model is terminated to obtain the weld defect training model.

3. The visual inspection method according to claim 2, characterized in that: The SPPELAN module includes a preprocessing layer, a spatial pyramid pooling layer, an enhanced local attention network layer, a spatial splicing layer and a fused convolution layer, wherein the preprocessing layer is connected to the upsampling layer of the neck network, the spatial pyramid pooling layer and the enhanced local attention network layer are respectively connected to the preprocessing layer, the spatial splicing layer is respectively connected to the spatial pyramid pooling layer and the enhanced local attention network layer, and the fused convolution layer is connected to the spatial splicing layer; Wherein, based on the SPPELAN module in the neck network, the first weld defect feature set is subjected to feature enhancement processing to obtain a second weld defect feature set, comprising the following steps: Based on the preprocessing layer, preprocessing the first weld defect feature set output by the upsampling layer of the neck network to obtain a third weld defect feature set; Based on the enhanced local attention network layer, reducing the dimension of the third weld defect feature set to obtain a fourth weld defect feature set; Based on the spatial pyramid pooling layer, multi-scale feature extraction is performed on the feature map output by the upsampling layer to obtain a fifth weld defect feature set of different spatial scales; Based on the spatial stitching layer, the fourth weld defect feature set is stitched with each of the fifth weld defect feature sets to obtain a sixth weld defect feature set; Based on the fused convolutional layer, the channel dimension of the sixth weld defect feature set is adjusted to obtain the second weld defect feature set.

4. The visual inspection method according to claim 2, characterized in that: The weld defect training set is input into the backbone network for feature extraction to obtain a first weld defect feature set, including: Inputting the weld defect training set into the convolution module of the backbone network, and performing deep feature extraction on the weld defect training set based on the ResBlock_CBAM module in the convolution module to obtain a third weld defect feature set; The third weld defect feature set is input into the bottleneck module of the backbone network, and based on the Ghost module in the bottleneck module, the third weld defect feature set is subjected to lightweight feature fusion to obtain the first weld defect feature set.

5. The visual inspection method according to claim 4, characterized in that: The ResBlock_CBAM module includes a residual network layer and a convolutional block attention layer, the convolutional block attention layer is connected to the middle convolutional layer of the convolutional module, and the residual network layer is connected between the initial convolutional layer and the last convolutional layer of the convolutional module in a skip connection manner; Among them, based on the ResBlock_CBAM module in the convolution module, deep feature extraction is performed on the weld defect training set to obtain a third weld defect feature set, including the following steps: Based on the convolution block attention layer, weighted processing is performed on the weld defect training set processed by the intermediate convolution layer to obtain a seventh weld defect feature set; Based on the residual network layer, residual convolution is performed on the weld defect training set output by the initial convolution layer to obtain an eighth weld defect feature set, and the seventh weld defect feature set and the eighth weld defect feature set are spatially superimposed to obtain the third weld defect feature set.

6. The visual inspection method according to claim 5, characterized in that: The convolutional block attention layer includes a channel attention layer and a spatial attention layer, and the convolutional block attention layer is connected to the residual network layer; Wherein, based on the convolutional block attention layer, weighted processing is performed on the weld defect training set processed by the intermediate convolutional layer to obtain the seventh weld defect feature set, comprising the following steps: Based on the channel attention layer, performing channel importance weighted processing on the weld defect training set processed by the intermediate convolution layer to obtain a ninth weld defect feature set, and performing element-by-element multiplication of the weld defect training set processed by the intermediate convolution layer and the ninth weld defect feature set to obtain a tenth weld defect feature set; Based on the spatial attention layer, the tenth weld defect feature set is spatially weighted to obtain the eleventh weld defect feature set, and the tenth weld defect feature set is element-wise multiplied with the eleventh weld defect feature set to obtain the seventh weld defect feature set.

7. The visual inspection method according to claim 4, characterized in that: The Ghost module includes a first Ghost convolutional layer, a second Ghost convolutional layer and a depth-separable network layer, the first Ghost convolutional layer is connected to the bottleneck input layer of the bottleneck module, the second Ghost convolutional layer is connected to the first Ghost convolutional layer, the bottleneck output layer of the bottleneck module is connected to the second Ghost convolutional layer, and the depth-separable network layer is connected between the bottleneck input layer and the bottleneck output layer in a skip connection manner; The step of performing lightweight feature fusion on the third weld defect feature set based on the Ghost module in the bottleneck module to obtain the first weld defect feature set includes the following steps: Based on the first Ghost convolutional layer, spatial feature expansion is performed on the third weld defect feature set of the bottleneck input layer to obtain a twelfth weld defect feature set; Based on the second Ghost convolution layer, performing spatial feature dimensionality reduction on the twelfth weld defect feature set to obtain a thirteenth weld defect feature set; Based on the depthwise separable convolutional layer, spatial feature extraction is performed on the third weld defect feature set of the bottleneck input layer to obtain a fourteenth weld defect feature set, and the thirteenth weld defect feature set and the fourteenth weld defect feature set are spatially fused to obtain the third weld defect feature set.

8. The visual inspection method according to claim 1, characterized in that: Before obtaining the labeled weld defect sample set, the visual inspection method further includes: Obtain the original weld defect image set; According to the morphology of weld surface defects of the rail transit vehicle body, the original weld defect image set is classified to obtain multiple weld defect type image sets; Each of the weld defect type image sets is labeled by an image annotation tool to obtain the labeled weld defect sample set.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the visual detection method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the visual inspection method according to any one of claims 1 to 8 is implemented.

Images