Nonferrous metal target detection method for scrapped vehicles based on attention twin network and multi-source image fusion

Abstract

The present invention belongs to the field of image processing technology, and discloses a method for detecting nonferrous metal targets of scrapped vehicles based on an attention twin network and multi-source image fusion. The present invention builds a nonferrous metal target detection model for scrapped vehicles based on an improved yolov8 model, builds a backbone network with a twin network structure in the model, and introduces an attention mechanism into the backbone network to form an attention twin network. At the same time, it integrates multi-source image inputs of two channels, visible light and infrared thermal imaging. When detecting nonferrous metals in a mixture of scrapped vehicles, it can more accurately focus on areas with nonferrous metal characteristics. For nonferrous metal parts with irregular shapes or mixed with other materials, the attention twin network is beneficial to improving the recognition accuracy of the target by learning its unique feature pattern. The present invention is beneficial to improving the accuracy and efficiency of recycling nonferrous metal fragments from scrapped vehicles.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and target detection, and specifically relates to a method for detecting non-ferrous metal targets of scrapped vehicles based on an attention twin network and multi-source image fusion. Background Art

[0002] Nonferrous metal target detection of scrapped vehicles refers to the process of identifying and distinguishing nonferrous metals (such as copper, aluminum, zinc, magnesium, etc.) from metal fragments or parts obtained after the dismantling of scrapped vehicles. Accurately identifying and separating these metals through target detection technology is of great significance for improving resource recycling rate, reducing environmental pollution and reducing raw material costs.

[0003] With the rapid development of the automobile industry, the number of scrapped cars has increased year by year. How to efficiently and accurately recycle and process the non-ferrous metals in these scrapped cars has become an important research topic. Non-ferrous metals such as copper and aluminum have high recycling value, but due to the differences in their physical and chemical properties, their sorting and detection become complicated. At present, the common methods for sorting non-ferrous metals from scrapped cars on the market mainly focus on eddy current, X-ray, spectrum, and machine vision.

[0004] However, the existing methods for sorting nonferrous metals from scrapped cars have many problems in practical applications. For example, the eddy current sorting method is not effective for sorting weakly magnetic metals; gas chromatography-mass spectrometry is a destructive test that requires sample preparation and is not suitable for all samples; in spectral detection, the recognition rate of X-ray fluorescence spectroscopy is low and is only suitable for preliminary screening and quantitative analysis, and ultraviolet-visible spectrophotometry is limited to the detection of specific elements; machine vision detection methods have defects in data processing speed, light and environmental adaptability, defect recognition ability, anti-interference ability, and algorithm scalability and adaptability. Summary of the invention

[0005] In view of the shortcomings of traditional methods in sorting non-ferrous metals from scrapped vehicles, the present invention proposes a scrapped vehicle non-ferrous metal target detection method based on attention twin network and multi-source image fusion to improve the recognition accuracy and efficiency of non-ferrous metal targets of scrapped vehicles, so as to accurately identify and separate these metals.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] The scrapped car nonferrous metal target detection method based on attention twin network and multi-source image fusion includes the following steps:

[0008] Step 1. First, obtain visible light images and infrared thermal images of scrapped nonferrous metal fragments using a visible light camera and an infrared thermal imager. When collecting, the visible light images and infrared thermal images are one-to-one in time and space to form corresponding image pairs, and perform image preprocessing and annotation operations to obtain a training data set;

[0009] Step 2. Build a scrapped car non-ferrous metal target detection model based on the improved YOLOV8 model architecture, where the scrapped car non-ferrous metal target detection model includes two backbone networks, a neck network, and a detection head network;

[0010] Define two backbone networks, backbone network 1 and backbone network 2. Backbone network 1 and backbone network 2 adopt twin network structure. At the same time, introduce attention mechanism SEAttention into the two backbone networks to form attention twin network.

[0011] Among them, the backbone network 1 is used to receive visible light image input and perform feature extraction on the input visible light image; the backbone network 2 is used to receive infrared thermal imaging image and perform feature extraction on the input infrared thermal imaging image;

[0012] The neck network is used to receive the output features of the backbone network 1 and the backbone network 2, fuse the output features of the two backbone networks, and realize the detection of non-ferrous metal targets of scrapped vehicles through the detection head network;

[0013] The improved YOLOV8 model architecture is obtained by making the following improvements on the original YOLOV8 model structure:

[0014] The backbone network is set to two, and is used to extract features of visible light images and infrared thermal imaging images respectively, so as to realize multi-source image input and feature extraction, and then realize multi-source image fusion;

[0015] The RepNCSPELAN4 module in the YOLOV9 model is used to replace the c2f module in the shallow network of the original YOLOV8 model, and feature extraction and fusion are performed in the shallow network to obtain programmable gradient information;

[0016] Add SEAttention to the c2f module in the deep network of the original YOLOV8 model, and enhance the ability of target detection by introducing a lightweight and efficient attention module to strengthen feature representation and introduce an attention mechanism;

[0017] Replace some convolution blocks in the original YOLOV8 model with Adown downsampling modules to reduce the spatial dimension of the feature map;

[0018] Replace the sppf module of the original YOLOV8 model with the SPPELAN module;

[0019] ASFF is used to improve the detection head network and adaptively fuse features of different scales;

[0020] Step 3. Train the improved YOLOV8 model based on the training data set to obtain a trained scrapped car non-ferrous metal target detection model, and use the model to perform target detection on scrapped car non-ferrous metals to obtain detection results.

[0021] In addition, based on the above-mentioned scrapped car non-ferrous metal target detection method based on attention twin network and multi-source image fusion, the present invention also proposes a corresponding scrapped car non-ferrous metal target detection system based on attention twin network and multi-source image fusion, and the scrapped car non-ferrous metal target detection system adopts the following technical solutions:

[0022] The scrapped car nonferrous metal target detection system based on the attention twin network and multi-source image fusion includes a visible light camera, an infrared thermal imager and a computer device, wherein the visible light camera and the infrared thermal imager synchronously collect images of scrapped car nonferrous metal fragments to obtain visible light images and infrared thermal imaging images respectively. The computer device includes a memory and at least one processor. The memory stores executable code. When the processor executes the executable code, it is used to implement the scrapped car nonferrous metal target detection method based on the attention twin network and multi-source image fusion as described above.

[0023] The present invention has the following advantages:

[0024] As described above, the present invention relates to a method and system for detecting nonferrous metal targets of scrapped vehicles based on attention twin network and multi-source image fusion. The present invention combines two different types of images, namely visible light images and infrared thermal imaging images, realizes the input of multi-source images, and can obtain more comprehensive target information. Visible light images provide color and texture details, and infrared thermal imaging images can detect targets that are prominent due to temperature differences in complex environments. By fusing these multi-source images, the limitations of a single image can be compensated, and the detection accuracy of nonferrous metal targets can be improved. In addition, based on the original YOLOV8 model, the present invention defines two backbone networks, namely backbone network 1 and backbone network 2. Backbone network 1 is used to receive visible light image input and perform feature extraction on the input visible light image; backbone network 2 is used to receive infrared thermal imaging images and perform feature extraction on the input infrared thermal imaging images. Backbone network 1 and backbone network 2 adopt twin network structure, and introduce attention mechanism SEAttention in the two backbone networks to form attention twin network. The attention twin network structure is used to extract and compare the features of the two input samples, and the features are adaptively weighted. The attention mechanism SEAttention is introduced on the c2f module in the deep network. The attention mechanism can automatically learn and highlight the key areas and features in the image, reduce the interference of irrelevant information, and more accurately focus on the areas with non-ferrous metal characteristics when detecting non-ferrous metals in scrapped car mixtures. For non-ferrous metal parts with irregular shapes or mixed with other materials, the attention twin network can improve the recognition accuracy of the target by learning its unique feature patterns. The method of the present invention is conducive to improving the accuracy and efficiency of recycling non-ferrous metal fragments from scrapped cars. In addition, the effective detection and recovery of non-ferrous metals in scrapped car mixtures can reduce dependence on natural resources and improve the recycling rate of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a network structure diagram of a scrapped car nonferrous metal target detection method based on an attention twin network and multi-source image fusion in an embodiment of the present invention;

[0026] Figure 2 A network structure diagram of the RepNCSPELAN4 module in an embodiment of the present invention;

[0027] Figure 3 A network structure diagram of a RepNCSP module in an embodiment of the present invention;

[0028] Figure 4 This is a structural diagram of the RepNBottleneck module in an embodiment of the present invention;

[0029] Figure 5: is a network structure diagram of the RepConvN module in an embodiment of the present invention;

[0030] Figure 6 : is a network structure diagram of the Adown downsampling module in an embodiment of the present invention;

[0031] Figure 7 This is a network structure diagram of Concat in an embodiment of the present invention;

[0032] Figure 8 This is a network structure diagram of the attention module SEAttention in an embodiment of the present invention;

[0033] Fig. 9 SE in the embodiment of the present invention - Bottleneck's network structure diagram;

[0034] Fig.10 A schematic diagram of compression operation in an embodiment of the present invention;

[0035] Fig.11 A schematic diagram of an excitation operation in an embodiment of the present invention;

[0036] Fig.12 4 is a network structure diagram of the SPPELAN module in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0038] Example 1

[0039] This embodiment describes a method for detecting non-ferrous metal targets of scrapped vehicles based on an attention twin network and multi-source image fusion. The method builds a scrapped vehicle non-ferrous metal target detection model based on an improved yolov8 model. A backbone network with a twin network structure is adopted in the model, and an attention mechanism is introduced into the backbone network to form an attention twin network. At the same time, multi-source image fusion technology is introduced, thereby enabling efficient and accurate target detection of scrapped vehicle non-ferrous metal fragments.

[0040] like Figure 1 As shown, the scrapped car nonferrous metal target detection method based on the attention twin network and multi-source image fusion in this embodiment includes the following steps:

[0041] Step 1. First, obtain the visible light image and infrared thermal image obtained by using a visible light camera and an infrared thermal imager to collect images of scrapped non-ferrous metal fragments of scrapped cars. When collecting, the visible light image and the infrared thermal image are one-to-one in time and space to form corresponding image pairs, and perform image preprocessing and annotation operations to obtain a training data set.

[0042] Here, the one-to-one correspondence in time and space specifically refers to a pair of images consisting of two corresponding images obtained by correspondingly shooting with a visible light camera and an infrared thermal imager at the same time and in the same spatial position.

[0043] The present invention combines two different types of images, visible light and infrared thermal imaging. The RGB spectrum of visible light can provide rich color and texture information, while the infrared spectrum of infrared thermal imaging can detect the temperature and thermal characteristics of objects. The combination of the two can provide a more comprehensive understanding of the target object, which is conducive to improving the accuracy of non-ferrous metal target identification and analysis of scrapped vehicles.

[0044] Specifically, firstly, images of scrapped automobile nonferrous metal fragments are collected.

[0045] In this embodiment, high-quality visible light cameras and infrared thermal imagers are selected to ensure that they have appropriate resolution, frame rate, and sensitivity to meet the needs of specific applications. Geometric correction is performed on the equipment to ensure the spatial accuracy and consistency of visible light images and infrared thermal imaging images, providing a basis for subsequent image fusion.

[0046] Consider the lighting conditions of the acquisition environment. Avoid strong or weak visible light that interferes with the infrared thermal imaging effect. At the same time, ensure that the ambient temperature is within the effective measurement range of the infrared thermal imager. Try to maintain the stability of the acquisition scene and reduce the movement and vibration of objects to obtain clearer images. In addition, it is necessary to ensure that the acquisition of visible light images and infrared thermal imaging images is synchronized. Use AnalyzlR software to achieve this so that the two images correspond in time.

[0047] The main purpose of image preprocessing is to remove secondary information from the image, restore useful real information, simplify the information associated with it, and thus simplify the data as much as possible to improve the reliability of image extraction, separation, comparison and characterization.

[0048] In this embodiment, the preprocessing operations of the visible light image and the thermal imaging image are the same, including image denoising (Gaussian filtering), image enhancement (grayscale processing), geometric transformation (scaling, rotation), image annotation, and image normalization.

[0049] First, select a 3×3 Gaussian kernel and set the standard deviation to 0.5 to retain more image details and prevent the loss of important details. Call the Gaussian filter function and pass the image as input to the function. The function will return the image after Gaussian filtering.

[0050] Grayscale processing. In the RGB model, if R=G=B, the color represents a grayscale color, where the value of R=G=B is called the grayscale value. Therefore, each pixel of the grayscale image only needs one byte to store the grayscale value, and the grayscale range is 0-255.

[0051] Compared with color visible light images, grayscale images occupy less space and can visually increase image contrast, reduce noise, and enhance image details, thereby highlighting the target area during the marking process.

[0052] Image geometric transformation, also known as image space transformation, processes the collected images through geometric transformations such as translation, transposition, mirroring, rotation, and scaling to correct the systematic errors of the image acquisition system and the random errors of the instrument position (imaging angle, perspective relationship), enrich the diversity of the data set, and reduce overfitting caused by too small a data set during training and verification.

[0053] In this embodiment, the geometric transformation is mainly realized through the PhotoShop platform. On the basis of the original data set, the visible light and infrared thermal imaging images of the scrapped non-ferrous metal fragments are processed by rotation, cropping, and rotation + cropping.

[0054] Next, perform the LetterBox operation, use proportional scaling, find the long side and scale it to the target size (640), and then adjust the short side according to the scaling ratio of the long side. If the short side is less than the target size after scaling, fill the gray edges on both sides of the short side to the target size, and scale the image to the 640× 640 size supported by the model.

[0055] Then, the collected images are labeled to establish a data set. In this embodiment, the labelImg labeling format YOLO is used to improve the labeling quality through manual labeling and cross-validation.

[0056] The image annotation operation uses the LabelImg tool to draw a bounding box for each object and assign a category label.

[0057] In labelImg, define the data of visible light and infrared thermal imaging respectively. Prepare the data set that needs to be labeled. Create a new folder named Data, create a folder named "images" to store the image files that need to be labeled, and create a folder named "labels" to store the labeled label files. Create a "text" test folder, a "train" training folder, and a "val" verification folder in the "images" folder and the "labels" folder respectively.

[0058] The established image data set is input through RGB color mode input and infrared spectrum mode, and multi-source image input is used to provide a basis for subsequent image fusion, thereby improving the recognition rate and versatility of the model from the perspective of multiple views.

[0059] The RGB color mode obtains a variety of colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other, providing the visible light color information of objects.

[0060] The infrared spectral mode can capture the unique characteristics of objects in the infrared band.

[0061] Relying solely on the RGB color mode may be affected by factors such as lighting conditions and surface color interference of objects, while the infrared spectrum mode is not affected by the color of visible light, can penetrate some surface coverings, and detect the material characteristics inside the object, making up for the shortcomings of the RGB mode. For non-ferrous metal crushed objects, their surface state may change due to factors such as the crushing process and storage time. The RGB color mode is more sensitive to changes in surface color, while the infrared spectrum mode pays more attention to the material nature of the object. The combined use can better adapt to changes in the state of the object and ensure the reliability of classification. This embodiment combines the two modes to provide a richer information dimension for classification and identification, thereby greatly improving the accuracy of classification of non-ferrous metal crushed objects.

[0062] The scaled pixel values ​​are further converted from integers to floating point numbers to support gradient calculations in model training. The pixel values ​​of the image are normalized to the [0.0, 1.0] interval using the Sobel operator.

[0063] Specifically, we first calculate the horizontal and vertical gradients, and then calculate the gradient histogram. We directly use the Sobel operator in OpenCV with a kernel size of 1 to calculate. The Sobel operator consists of two 3×3 templates, one for calculating the gradient in the X direction and the other for calculating the gradient in the Y direction.

[0064] The present invention provides richer information for target detection and enhances the target feature representation by fusing visible light and infrared thermal imaging images, and can comprehensively utilize the advantages of both to improve the accuracy of target detection. Fusion of these two types of information can improve the accuracy and reliability of target detection. Target detection algorithms are usually sensitive to image quality and environmental conditions.

[0065] Multi-source image fusion can improve the robustness of target detection by integrating information from multiple data sources. Even if one data source is disturbed or its quality degrades, the fused image can still provide sufficient information for target detection.

[0066] Step 2. Build a scrapped car nonferrous metal target detection model based on the improved YOLOV8 model architecture, such as Figure 1 As shown, the model includes two backbone networks, a neck network and a detection head network.

[0067] Two backbone networks are defined as backbone network one and backbone network two. Backbone network one and backbone network two adopt a twin network structure. At the same time, the attention mechanism SEAttention is introduced into the two backbone networks to form an attention twin network.

[0068] The first backbone network is used to receive visible light image input and perform feature extraction on the input visible light image; the second backbone network is used to receive infrared thermal imaging image and perform feature extraction on the input infrared thermal imaging image.

[0069] The neck network is used to receive the output features of backbone network one and backbone network two, fuse the output features of the two backbone networks, and realize the detection of non-ferrous metal targets of scrapped vehicles through the detection head network.

[0070] The following improvements are made based on the original YOLOV8 model structure to obtain an improved YOLOV8 model.

[0071] The backbone network is set to two, and is used to extract features of visible light images and infrared thermal imaging images respectively, so as to realize multi-source image input and feature extraction, and then realize multi-source image fusion.

[0072] The RepNCSPELAN4 module in the YOLOV9 model is used to replace the c2f module in the shallow network of the original YOLOV8 model, and feature extraction and fusion are performed in the shallow network to obtain programmable gradient information.

[0073] SEAttention is added to the c2f module in the deep network of the original YOLOV8 model to obtain the C2f_SE module, which strengthens the feature representation and introduces the attention mechanism to enhance the ability of target detection.

[0074] Some convolution blocks in the original YOLOV8 model are replaced with Adown downsampling modules to reduce the spatial dimension of the feature map.

[0075] Replace the sppf module of the original YOLOV8 model with the SPPELAN module.

[0076] ASFF is used to improve the detection head network and adaptively fuse features of different scales.

[0077] The present invention forms a larger feature tensor by splicing two feature tensors, RGB color mode input and infrared spectrum mode input, along a certain dimension. Image fusion is performed by inputting multi-source images, thereby improving the recognition rate and versatility of the model. In addition, the present invention introduces the RepNCSPELAN4 module of yolov9 to replace the c2f module to perform feature extraction-fusion in the shallow network and obtain programmable gradient information. Part of the conv is replaced by the Adown downsampling module to reduce the spatial dimension of the feature map, which helps the model capture the features of the image at a higher level and reduces the amount of calculation. The sppf module is replaced by the SPPELAN module, which can enhance the model's ability to locate the boundary of the target and effectively improve the accuracy of target detection by extracting semantic features at different scales. In the deep network, SEAttention is added to the c2f module, and the ability of target detection is enhanced by introducing a lightweight and efficient attention module to strengthen feature representation and introduce an attention mechanism. The output of the convolutional layer does not consider the dependence on each channel. The goal is to allow the network to selectively enhance features with large amounts of information so that subsequent processing can make full use of these features and suppress useless features. ASFF is used to improve the detection head. It improves the model performance by adaptively fusing features of different scales, especially in small object detection. ASFF can effectively improve the YOLOv8 object detection model and achieve significant improvements in indicators such as mAP and recall.

[0078] In combination with the above improvements, the following describes in detail each network in the improved YOLOV8 model.

[0079] The backbone network 1 includes a convolution module, a RepNCSPELAN4 module, a downsampling module, a C2f_SE module and a first SPPELAN module. There are two convolution modules, namely the first and second convolution modules; there are two RepNCSPELAN4 modules, namely the first and second RepNCSPELAN4 modules; there are two C2f_SE modules, namely the first and second C2f_SE modules; there are three downsampling modules, namely the first, second and third downsampling modules. The input visible light image first passes through the first and second convolution modules, then the first RepNCSPELAN4 module, then the first downsampling module, then the second RepNCSPELAN4 module, then the second downsampling module, then the first C2f_SE module, then the third downsampling module, then the second C2f_SE module, and finally outputs through the first SPPELAN module.

[0080] Backbone network 2 includes convolution module, RepNCSPELAN4 module, downsampling module, C2f_SE module, Concat module and second SPPELAN module. There are two convolution modules, which are the third and fourth convolution modules respectively; there are two RepNCSPELAN4 modules, which are the third and fourth RepNCSPELAN4 modules respectively; there are four C2f_SE modules, which are the third, fourth, fifth and sixth C2f_SE modules respectively; there are three downsampling modules, which are the fourth, fifth and sixth downsampling modules respectively; there are three Concat modules, which are the first, second and third Concat modules respectively.

[0081] The input infrared thermal imaging image first passes through the third and fourth convolution modules, and then passes through the third RepNCSPELAN4 module, the fourth downsampling module and the fourth RepNCSPELAN4 module in sequence; the output features of the fourth RepNCSPELAN4 module are spliced ​​with the output features of the second RepNCSPELAN4 module in the first Concat module; the output features of the first Concat module pass through the third C2f_SE module, the fifth downsampling module and the fourth C2f_SE module in sequence; the output features of the fourth C2f_SE module are spliced ​​with the output features of the first C2f_SE module in the second Concat module; the output features of the second Concat module pass through the fifth C2f_SE module and the sixth downsampling module in sequence; the output features of the sixth downsampling module are spliced ​​with the output features of the second C2f_SE module in the third Concat module; the output features of the third Concat module are output after passing through the sixth C2f_SE module and the second SPPELAN module in sequence.

[0082] The neck network includes a splicing and fusion module, a convolution module, an upsampling module, a C2f_SE module, and a Concat module. There are two splicing and fusion modules, which are defined as the first and second splicing and fusion modules respectively; there are four convolution modules, which are the fifth, sixth, seventh, and eighth convolution modules respectively; there are three upsampling modules, which are defined as the first, second, and third upsampling modules respectively; there are four C2f_SE modules, which are defined as the seventh, eighth, ninth, and tenth C2f_SE modules respectively; there are four Concat modules, which are the fourth, fifth, sixth, and seventh Concat modules respectively.

[0083] The output features of the first SPPELAN module and the second SPPELAN module enter the first splicing and fusion module for splicing and fusion; the output features of the first splicing and fusion module pass through the fifth convolution module and the first upsampling module in sequence; the output features of the first upsampling module and the output features of the fifth C2f_SE module are spliced ​​in the fourth Concat module; the output features of the fourth Concat module pass through the seventh C2f_SE module, the sixth convolution module and the second upsampling module in sequence; the output features of the second upsampling module and the output features of the third C2f_SE module are spliced ​​in the fifth Concat module; the output features of the fifth Concat module pass through the eighth C2f_SE module, the seventh convolution module and the third upsampling module in sequence; the output features of the third upsampling module and the output features of the seventh C2f_SE module are spliced ​​in the sixth Concat module; the output features of the sixth Concat module pass through the ninth C2f_SE module and the eighth convolution module in sequence; the output features of the eighth convolution module and the output features of the second SPPELAN module are spliced ​​in the seventh Concat module.

[0084] The output of the seventh Concat module is output to the detection head network after passing through the tenth C2f_SE module; the output features of the seventh, eighth and ninth C2f_SE modules are spliced ​​and fused in the second splicing and fusion module and then output to the detection head network.

[0085] First, the visible light image input and infrared thermal imaging image are input into backbone network 1 and backbone network 2 respectively, which are used to process the features of visible light image and infrared thermal imaging image. Then, by introducing the SEAttention mechanism module in the c2f module, a C2f-SE module is formed to automatically learn the importance weights of different channel features. For visible light images, more attention is paid to the features related to non-ferrous metals in the color channel; for infrared thermal imaging images, more attention is paid to the channels with significant temperature features. Finally, the visible light and infrared thermal imaging features processed by the attention mechanism are fused through the concat module to obtain the fused feature vector, which integrates the information of the two modal images and contains richer feature representations, which is conducive to subsequent target detection. The above backbone network 1 and backbone network 2 structures together form the attention twin network structure.

[0086] like Figure 1It shows that backbone network 1 and backbone network 2 adopt partial weight sharing for two different input channels. By sharing weights in the underlying feature extraction layer, the two backbone networks can learn common features of images from each other, such as edges, textures, etc., which helps the model to better integrate the information of the two modalities. Weights are not shared in the high-level feature extraction layer so that the two backbone networks can better capture their respective unique features, reduce excessive dependence on specific modalities, and increase the generalization ability of the model.

[0087] like Figure 2 The module structure of RepNCSPELAN4 is shown. The RepNCSPELAN4 module is a generalized efficient layer aggregation network (GELAN) for feature extraction and fusion. Compared with the original c2f module, it is a commonly used feature processing component, but when processing complex scenes or small targets, the performance of the c2f module may be lacking. Its processing flow is as follows:

[0088] First enter the features Figure X , perform a convolution Conv operation, the convolution kernel size of Conv is k=1, the step size is s=1, and the padding is p=0; then split it through Split and split it into two parts X1 and X2, where X2 performs two convolution Conv and RepNCSP operations, the convolution kernel size of Conv is k=3, the step size is s=1, and the padding is p=1. Finally, the two split parts are fused through Concat and convolved again through Conv, the convolution kernel size of Conv is k=1, the step size is s=1, and the padding is p=0.

[0089] like Figure 3 The structure of the RepNCSP module of RepNCSPELAN4 is shown, and its processing flow is as follows:

[0090] First enter the features Figure X , and two branches are processed respectively. One branch performs a convolution Conv operation, the convolution kernel size of Conv is k=1, the step size is s=1, and p=0 is padded, and then N RepNBottleneck operations are performed; the other branch only performs a convolution Conv operation once, the convolution kernel size of Conv is k=1, the step size is s=1, and p=0 is padded. Finally, the features output by the two branches are fused through Concat and convolved again through a Conv, the convolution kernel size of Conv is k=1, the step size is s=1, and p=0 is padded.

[0091] like Figure 4 The structure of the RepNBottleneck module in the RepNCSP module is shown, and the processing flow is as follows:

[0092] First enter the input features Figure X, by performing 1 RepConvN operation and then a convolution Conv operation, where the convolution kernel size k=1, step size s=1, and padding p=0.

[0093] like Figure 5 The structure of the submodule RepConvN in the RepNBottleneck module is shown, and the processing flow is as follows:

[0094] First, input X, whose number of channels is c1. There are two convolution operations here, one is a 3×3 convolution kernel, and the other is a 1×1 convolution kernel. For the 3×3 convolution kernel, its stride s is 1 and the padding p is 0. This means that when performing the convolution operation, one pixel is moved at a time and no edge padding is performed. For the 1×1 convolution kernel, the stride s is also 1 and the padding p is 0. Then, the results of the two convolution Conv operations are combined together through the Concat operation. Finally, the output Y is obtained, whose number of channels is c2.

[0095] The present invention introduces the RepNCSPELAN4 module of yolov9 to replace the c2f module to perform feature extraction and fusion in a shallow network to obtain programmable gradient information. A shallow network usually has a lower computational complexity and a faster running speed. In target detection, the basic features of the target can be quickly obtained by performing feature extraction using a shallow network. By fusing the features extracted by the shallow network, feature information at different levels can be integrated to enhance the ability to describe the target. The target's detailed information can be captured, and the target's overall semantics can be understood, thereby improving the accuracy of target detection. Obtaining programmable gradient information can provide more flexibility and controllability for target detection. By adjusting the calculation method and parameters of the gradient, the feature extraction and target detection process can be optimized according to specific task requirements and data characteristics.

[0096] In addition, this embodiment replaces some convolution modules with Adown downsampling modules to reduce the spatial dimension of the feature map, helping the model to capture image features at a higher level while reducing the amount of calculation.

[0097] like Figure 6 As shown, the processing flow of the Adown downsampling module is as follows:

[0098] First, the feature map C is input; then, the feature map undergoes an average pooling operation to average pool the input feature map to reduce the spatial dimension of the feature; then, the input is split into two parts by Split; one part first undergoes a maximum pooling operation to extract the significant features in the input, and then a CBS operation is performed; the other part directly undergoes a CBS operation; then, the two parts that have undergone maximum pooling and CBS operations are concatenated together through a Concat operation.

[0099] In addition, some convolution conv are replaced with Upsample upsampling modules to reduce the amount of calculation and the number of parameters, improve the feature resolution, and enhance the flexibility and scalability of the model. The nearest neighbor difference method is used to make the target image Pixel value Equal to the inverse transformation to the original image The pixel value of the nearest point. The formula is as follows:

[0100] ;

[0101] ( x , y ) = a r g m i n ( x , y ) [ ( x ^ − x ) 2 + ( y ^ − y ) 2 ] ;

[0102] in, Represents the pixel value of the point in the original image that is closest to the target image point; Represents the coordinates of a pixel in the target image. Indicates the original image The nearest point, The function represents the value of the independent variable when the minimum value is achieved.

[0103] The present invention replaces part of conv with Adown downsampling module, reduces the spatial dimension of feature map, helps the model capture the features of the image at a higher level, and reduces the amount of calculation. Reducing the spatial dimension of feature map can significantly reduce the amount of calculation. In target detection, a large amount of image data usually needs to be processed, and the amount of calculation is very large. By reducing the spatial dimension of feature map, the computational complexity of the model can be reduced, the reasoning speed can be accelerated, and target detection can be run in real time on devices with limited resources. After the spatial dimension of feature map is reduced, the model can capture the features of the image at a higher level. In target detection, high-level features usually contain more abstract and semantically informative content, which is very critical for accurately identifying the category and position of the target. Reducing the spatial dimension of feature map also helps to improve the generalization ability of the model. In target detection, the model needs to be able to adapt to different image data and scene changes. By reducing the spatial dimension of feature map, the model can learn more universal feature representations and reduce dependence on specific data distribution.

[0104] This embodiment uses the cocat module to fuse the image feature set, directly fuses the features, connects different features, and inputs them into a model for training (first fuse the features of multiple layers, then train the predictor on the fused features, and only after complete fusion, perform detection).

[0105] The Concat operation can concatenate the two feature tensors of RGB color mode input and infrared spectrum mode input along a certain dimension to form a larger feature tensor. It can increase the dimension and diversity of features. By concatenating features from different sources, the feature space is expanded, allowing the model to see more information and changes, thereby improving recognition accuracy. In addition, the concat operation can also promote interaction and integration between features. Since features at different levels or sources have different semantics and representations, they complement each other. By concatenating them together, communication and integration between them can be promoted, further improving the performance of the model. Concat is an increase in the number of channels, that is, the number of features (channels) describing the image itself has increased, while the information under each feature has not increased.

[0106] like Figure 7 As shown in the figure, for the two inputs, the number of channels is the same and there is a convolution behind it. Suppose the channels of the visible light and infrared thermal imaging inputs are X1, X2, ..., X i and Y1, Y2, ..., Y j , then the concat output channel is:

[0107] .

[0108] Where c1 represents the number of visible light input channels, c2 represents the number of infrared thermal imaging input channels, and K i and K j Respectively represent the convolution kernels corresponding to the visible light and infrared thermal imaging input channels.

[0109] In addition, in the deep network, the attention module SEAttention is added to the c2f module to form the C2f_SE module. The attention twin network structure is used to extract and compare the features of the two input samples. The feature representation is strengthened by introducing a lightweight and efficient attention module and the attention mechanism is introduced to enhance the target detection capability.

[0110] The network structure with the attention module SEAttention is as follows Figure 8 shown.

[0111] First, perform a convolution operation to extract the features of the input data. Then, use the Split operation to split the data in order to achieve different subsequent processing. SE_Bottleneck1 and SE_Bottleneckn are bottleneck structures with a Squeeze-and-Excitation (SE) mechanism, which can effectively improve the performance and efficiency of the model. Then, use the Concat operation to reassemble and splice the multiple split parts together. Finally, use the convolution layer to perform a convolution operation again to further extract features or process the spliced ​​results. Fig. 9 Shows SE - Bottleneck's network structure.

[0112] The output of the convolutional layer does not consider the dependence on each channel. The goal is to allow the network to selectively enhance features with large amounts of information so that subsequent processing can make full use of these features and suppress useless features.

[0113] SEAttention mainly includes two steps: Squeeze and Excitation.

[0114] Squeeze: The feature map is compressed from the spatial dimension through global average pooling to obtain a one-dimensional vector with the same number of channels. The squeeze operation is as follows Fig.10 As shown in Figure 2. This operation is a global average pooling. After the compression operation, the feature map is compressed into a 1×1×C vector.

[0115] Excitation: Perform two fully connected layer operations on this one-dimensional vector, and then pass it through the ReLU activation function to obtain the weight of each channel. Then multiply these weights with the original feature map to adjust the importance of different channels. The Excitation operation is as follows Fig.11 As shown. Excitation consists of two fully connected layers, where SERatio is a scaling parameter. The purpose of this parameter is to reduce the number of channels and thus reduce the amount of computation. The first fully connected layer has C*SERatio neurons, the input is 1×1×C, and the output is 1×1×C×SERadio. The second fully connected layer has C neurons, the input is 1×1×C×SERadio, and the output is 1×1×C.

[0116] The C2F module has a clear feature extraction and processing flow. SEAttention is inserted in the intermediate stage after feature extraction to adjust the channel importance of the extracted features. For insertion in the intermediate stage: Assume that the C2F module outputs a feature at a certain position Figure X, with a shape of (batchsize, channels, height, width). First, global average pooling is performed on the data to obtain a compressed feature vector with a shape of (batchsize, channels, 1,1). Then, after two fully connected layers and a ReLU activation function, a channel weight vector is obtained. Finally, the weight vector is combined with the original feature vector. Figure X Multiply to get the adjusted feature map, and continue the subsequent C2F module processing flow. Add SEAttention at the output of the C2F module to adjust the final output features. For adding at the output: After the C2F module outputs the feature map, directly apply the SEAttention operation flow to it, that is, perform global average pooling, fully connected layer processing, and multiplication with the original feature map.

[0117] The present invention adds SEAttention to the c2f module in a deep network, and enhances the ability of target detection by introducing a lightweight and efficient attention module to strengthen feature representation and introduce an attention mechanism. In a twin network, target detection usually involves measuring the similarity of two inputs (template image and search image). The attention mechanism can optimize this similarity measurement process, so that the network can more accurately judge the similarity of targets in two images. The attention mechanism can automatically learn and focus on key areas and features in the image, suppressing irrelevant information. In target detection, this helps to highlight the important features of the target and improve the expressiveness of the features. By analyzing and optimizing the attention weights, some unimportant connections and parameters can be removed, the size of the model can be reduced, and the deployment and application of the model can be facilitated.

[0118] In this embodiment, the sppf module is replaced by the SPPELAN module to solve the fusion difficulty. The SPPELAN module can enhance the model's ability to locate the boundary of the target and effectively improve the accuracy of target detection by extracting semantic features at different scales.

[0119] like Fig.12 As shown, the input data has a shape of [B, C, H, W], where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width.

[0120] CBS is used for preliminary feature processing. Then features are gradually extracted and optimized through CV1 to CV5. Two-dimensional maximum pooling (MaxPool2d) is used for downsampling to reduce the amount of data and highlight the main features. Concat splices data along the dimension (dim=1) to integrate the features of different parts. Spatial pyramid pooling (spp) is incorporated into the ELAN structure for layer aggregation. It starts with a convolutional layer that adjusts the channel dimension, followed by a series of spatial pooling operations to capture multi-scale contextual information, and then these outputs are connected and consolidated through another convolutional layer. This structure optimizes the network's ability to extract detailed features from various spatial hierarchies. It solves the information bottleneck problem in deep neural networks and improves the performance of the model, helping the model to process and learn data more efficiently to improve the accuracy of target detection.

[0121] The present invention replaces the sppf module with the SPPELAN module, which can enhance the model's ability to locate the boundaries of the target, and effectively improves the accuracy of target detection by extracting semantic features at different scales. Accurate boundary positioning enables the model to more accurately determine the scope of the target and reduce false detection and missed detection. For targets with irregular shapes or similar to the background, good boundary positioning capabilities are particularly important. Enhancing the boundary positioning capability can make the segmentation results more refined and provide a more accurate basis for subsequent image processing and analysis. By extracting semantic features at different scales, the model can better adapt to targets of various scales, and both small and large targets can be effectively detected. Features of different scales contain information at different levels. Small-scale features may pay more attention to details and local structures, while large-scale features focus more on overall shape and contextual information. Fusion of these different-scale features can enrich the feature representation of the model and improve the accuracy of target detection. Multi-scale feature extraction can make the model more robust to changes and interference in the image.

[0122] This embodiment uses ASFF to improve the detection head network. ASFF (Adaptive Spatial Feature Fusion) is an adaptive spatial feature fusion head that improves model performance by adaptively fusing features of different scales, especially in small object detection. ASFF can effectively improve the YOLOv8 object detection model and achieve significant improvements in indicators such as mAP and recall. Specifically, ASFF includes a feature pyramid, an adaptive fusion module, and a prediction module.

[0123] ASFF first constructs the input feature map into a feature pyramid, which contains features of different scales; then uses an adaptive fusion module to fuse features of different scales and assigns fusion weights according to the importance of each feature map; finally, a prediction module is used to predict the fused features to generate the final detection result.

[0124] Step 3. Train the improved YOLOV8 model based on the training data set to obtain a trained scrapped car non-ferrous metal target detection model, and use the model to perform target detection on scrapped car non-ferrous metals to obtain detection results.

[0125] The model outputs the coordinates (A, B, C, D), category probability, and confidence score of each scrap metal image bounding box. A and B represent the horizontal and vertical coordinates of the upper left corner of the bounding box, and C and D represent the horizontal and vertical coordinates of the lower right corner of the bounding box. These four coordinates can clearly define the scope of the target object in the image.

[0126] In model training, a self-collected database is used. The characteristics of this dataset are as follows:

[0127] This model uses multi-source image fusion technology. When collecting data, the non-ferrous metal fragments in the scrapped car mixture are collected through visible light and infrared thermal imaging. To perform image fusion, the infrared and visible light images must first be strictly registered to ensure their spatial correspondence. This is achieved by comparing image features or using geometric correction. Among them, the non-ferrous metals under infrared thermal imaging have been analyzed by response surface tests. When the heating table temperature is set to 322°C and the infrared thermal imager emissivity is set to 0.89, the infrared thermal imaging image recognition rate is higher. Large amount of data, diverse data: The database contains more than 6,500 images, covering a large number of scrapped non-ferrous metal fragments such as stainless steel, aluminum, and copper.

[0128] Accurate data labeling: The images in the database have been manually labeled using the LabelImg tool to draw bounding boxes for each object and assign category labels, with high labeling quality. At the same time, a data set is established.

[0129] In the training of scrapped car non-ferrous metal target detection model, a comprehensive loss function is defined, which includes classification loss, positioning loss and target confidence loss.

[0130] Among them, the classification loss is calculated using the cross entropy (CEL) loss, and the formula is as follows:

[0131] .

[0132] Where N is the number of samples, is the true category label. If the true category of the sample is i, then , otherwise 0, is the class probability predicted by the model for sample i.

[0133] The positioning loss is calculated using the DloU (Distance-loU) loss formula, which is as follows:

[0134] ;

[0135] ;

[0136] Among them, b and bt represent the center points of the predicted box and the true box respectively, P represents the Euclidean distance between the two points, and c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box.

[0137] It indicates the similarity between the predicted box and the true box. The IoU value is between 0 and 1. The larger the value, the higher the overlap between the two areas and the more accurate the prediction result.

[0138] The target confidence loss is calculated using the binary cross-entropy loss function, and the formula is as follows:

[0139] L C r o s s − E n t r o p y = − 1 N ∑ i = 1 N [ y i l o g ( p i ) + ( 1 − y i ) l o g ( 1 − p i ) ] ;

[0140] Where N represents the number of samples, is the true category label. If the true category of the sample is i, then , otherwise 0, is the class probability predicted by the model for sample i.

[0141] Comprehensive loss function The above loss functions are weighted and summed up. The specific formula is as follows:

[0142] ;

[0143] in, , , is a weight coefficient used to balance the importance of different losses.

[0144] The learning rate is an important hyperparameter in deep learning, which controls the update speed of the model during the training process. Since this model is used for image classification and recognition, after many experiments, it was found that when the learning rate is higher than 0.1, the model often has the problem of overfitting; when the learning rate is lower than 0.001, the model training takes a long time, so the learning rate of this model is set to 0.01.

[0145] Adaptive Moment Estimation (Adam) combines the advantages of the momentum method and the RMSprop (Root Mean Square Propagation) algorithm, while adaptively adjusting the learning rate of each parameter. It calculates the first-order moment estimate (i.e., the mean of the gradient) and the second-order moment estimate (i.e., the variance of the gradient) of each parameter, and then adjusts the learning rate based on these estimates. It has a fast convergence speed and good adaptability to different problems. There is no need to manually adjust the learning rate, and the learning rate can be automatically adjusted to adapt to different parameters and data features. By adding L1 and L2 regularization terms to prevent the model from overfitting caused by adaptive moment estimation optimization, after multiple debugging, the number of model iterations is set to 350, at which time the model performance is better and overfitting does not occur. At the same time, regularization can constrain the complexity of the model, making it smoother and more generalized.

[0146] After the model is trained, a trained scrapped car non-ferrous metal target detection model is obtained, and the model is used to perform target detection on the scrapped car non-ferrous metal to obtain the detection result.

[0147] Specifically, firstly, a visible light camera and an infrared thermal imager are used to synchronously capture images of scrapped non-ferrous metal fragments of scrapped vehicles to obtain visible light images and infrared thermal images respectively. Then, the visible light images and infrared thermal images are preprocessed respectively and input into the trained scrapped non-ferrous metal target detection model to output the detection results, which include the coordinates of the bounding box of each scrap metal image, the category probability and confidence score as well as the performance index.

[0148] The method of the present invention is based on multi-source image fusion technology and builds a model based on the attention twin network structure, which can greatly improve the recognition accuracy of non-ferrous metal targets of scrapped vehicles, thereby accurately identifying and separating these metals.

[0149] Example 2

[0150] This embodiment 2 describes a scrapped car nonferrous metal target detection system based on attention twin network and multi-source image fusion, which includes a visible light camera, an infrared thermal imager and a computer device,

[0151] The visible light camera and the infrared thermal imager simultaneously collect images of the scrapped non-ferrous metal fragments of the scrapped vehicles, and obtain visible light images and infrared thermal imaging images respectively. The computer device includes a memory and at least one processor, and the memory stores executable code. When the processor executes the executable code, it is used to implement the scrapped non-ferrous metal target detection method of the scrapped vehicles based on the attention twin network and multi-source image fusion in the above-mentioned embodiment 1.

[0152] The present invention provides an advanced technical means for the scrapped car recycling industry, which is helpful to promote the development of the environmental protection industry. Recycling enterprises can improve their competitiveness by adopting this technology, and also contribute to environmental protection.

[0153] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A scrapped car nonferrous metal target detection method based on attention twin network and multi-source image fusion, characterized in that: The steps include: Step 1. First, obtain visible light images and infrared thermal imaging images obtained by using a visible light camera and an infrared thermal imager to collect images of scrapped non-ferrous metal fragments of scrapped cars. When collecting, the visible light images and infrared thermal imaging images are one-to-one in time and space to form corresponding image pairs, and image preprocessing and annotation operations are performed to obtain a training data set; Step 2. Build a scrapped car non-ferrous metal target detection model based on the improved YOLOV8 model architecture, where the scrapped car non-ferrous metal target detection model includes two backbone networks, a neck network, and a detection head network; Define two backbone networks, backbone network 1 and backbone network 2. Backbone network 1 and backbone network 2 adopt twin network structure. At the same time, introduce attention mechanism SEAttention into the two backbone networks to form attention twin network. Among them, the backbone network 1 is used to receive visible light image input and perform feature extraction on the input visible light image; the backbone network 2 is used to receive infrared thermal imaging image and perform feature extraction on the input infrared thermal imaging image; The neck network is used to receive the output features of the backbone network 1 and the backbone network 2, fuse the output features of the two backbone networks, and realize the detection of non-ferrous metal targets of scrapped vehicles through the detection head network; The following improvements were made on the basis of the original YOLOV8 model to obtain the improved YOLOV8 model; The RepNCSPELAN4 module in the YOLOV9 model is used to replace the c2f module in the shallow network of the original YOLOV8 model, and feature extraction and fusion are performed in the shallow network to obtain programmable gradient information; Add SEAttention to the c2f module in the deep network of the original YOLOV8 model to obtain the C2f_SE module to strengthen the feature representation and introduce the attention mechanism to enhance the ability of target detection; Replace some convolution blocks in the original YOLOV8 model with Adown downsampling modules to reduce the spatial dimension of the feature map; Replace the sppf module of the original YOLOV8 model with the SPPELAN module; ASFF is used to improve the detection head network and adaptively fuse features of different scales; Step 3. Train the improved YOLOV8 model based on the training data set to obtain a trained scrapped car non-ferrous metal target detection model, and use the model to perform target detection on scrapped car non-ferrous metals to obtain detection results; The backbone network 1 includes a convolution module, a RepNCSPELAN4 module, a downsampling module, a C2f_SE module and a first SPPELAN module; There are two convolution modules, namely the first and second convolution modules; there are two RepNCSPELAN4 modules, namely the first and second RepNCSPELAN4 modules, there are two C2f_SE modules, namely the first and second C2f_SE modules; there are three downsampling modules, namely the first, second and third downsampling modules; The input visible light image first passes through the first and second convolution modules, then the first RepNCSPELAN4 module, then the first downsampling module, then the second RepNCSPELAN4 module, then the second downsampling module, then the first C2f_SE module, then the third downsampling module, then the second C2f_SE module, and finally output by the first SPPELAN module; The backbone network 2 includes a convolution module, a RepNCSPELAN4 module, a downsampling module, a C2f_SE module, a Concat module and a second SPPELAN module; There are two convolution modules, which are the third and fourth convolution modules respectively; there are two RepNCSPELAN4 modules, which are the third and fourth RepNCSPELAN4 modules respectively; there are four C2f_SE modules, which are the third, fourth, fifth and sixth C2f_SE modules respectively; there are three downsampling modules, which are the fourth, fifth and sixth downsampling modules respectively; there are three Concat modules, which are the first, second and third Concat modules respectively; The input infrared thermal imaging image first passes through the third and fourth convolution modules, and then passes through the third RepNCSPELAN4 module, the fourth downsampling module and the fourth RepNCSPELAN4 module in sequence; the output features of the fourth RepNCSPELAN4 module are spliced ​​with the output features of the second RepNCSPELAN4 module in the first Concat module; the output features of the first Concat module pass through the third C2f_SE module, the fifth downsampling module and the fourth C2f_SE module in sequence; the output features of the fourth C2f_SE module are spliced ​​with the output features of the first C2f_SE module in the second Concat module; the output features of the second Concat module pass through the fifth C2f_SE module and the sixth downsampling module in sequence; the output features of the sixth downsampling module are spliced ​​with the output features of the second C2f_SE module in the third Concat module; the output features of the third Concat module are output after passing through the sixth C2f_SE module and the second SPPELAN module in sequence.

2. The method for detecting non-ferrous metal targets of scrapped vehicles based on attention twin network and multi-source image fusion according to claim 1 is characterized in that: The neck network includes a splicing and fusion module, a convolution module, an upsampling module, a C2f_SE module and a Concat module; There are two splicing and fusion modules, which are defined as the first and second splicing and fusion modules respectively; there are four convolution modules, which are the fifth, sixth, seventh and eighth convolution modules respectively; there are three upsampling modules, which are defined as the first, second and third upsampling modules respectively; there are four C2f_SE modules, which are defined as the seventh, eighth, ninth and tenth C2f_SE modules respectively; there are four Concat modules, which are the fourth, fifth, sixth and seventh Concat modules respectively; The output features of the first SPPELAN module and the second SPPELAN module enter the first splicing and fusion module for splicing and fusion; the output features of the first splicing and fusion module pass through the fifth convolution module and the first upsampling module in sequence; the output features of the first upsampling module and the output features of the fifth C2f_SE module are spliced ​​in the fourth Concat module; the output features of the fourth Concat module pass through the seventh C2f_SE module, the sixth convolution module and the second upsampling module in sequence; the output features of the second upsampling module and the output features of the third C2f_SE module are spliced ​​in the fifth Concat module; the output features of the fifth Concat module pass through the eighth C2f_SE module, the seventh convolution module and the third upsampling module in sequence; the output features of the third upsampling module and the output features of the seventh C2f_SE module are spliced ​​in the sixth Concat module; the output features of the sixth Concat module pass through the ninth C2f_SE module and the eighth convolution module in sequence; the output features of the eighth convolution module and the output features of the second SPPELAN module are spliced ​​in the seventh Concat module; The output of the seventh Concat module is output to the detection head network after passing through the tenth C2f_SE module; the output features of the seventh, eighth and ninth C2f_SE modules are spliced ​​and fused in the second splicing and fusion module and then output to the detection head network.

3. The method for detecting nonferrous metal targets of scrapped vehicles based on attention twin network and multi-source image fusion according to claim 1 is characterized in that: The processing flow of the RepNCSPELAN4 module is as follows: First, input the feature map X and perform a convolution Conv operation. The convolution kernel size of Conv is k=1, the step size is s=1, and the padding is p=0. Then split it into two parts X1 and X2 through Split, where X2 undergoes two convolution Conv and RepNCSP operations. The convolution kernel size of Conv is k=3, the step size is s=1, and the padding is p=1. Finally, the two split parts are fused through Concat and convolved again through Conv. The convolution kernel size of Conv is k=1, the step size is s=1, and the padding is p=0.

4. The method for detecting nonferrous metal targets of scrapped vehicles based on attention twin network and multi-source image fusion according to claim 1 is characterized in that: The processing flow of the Adown downsampling module is as follows: First, the feature map is input; then, the feature map undergoes an average pooling operation to average pool the input feature map to reduce the spatial dimension of the feature; then, the input is split into two parts by Split; one part first undergoes a maximum pooling operation to extract the significant features in the input, and then a CBS operation is performed; the other part directly undergoes a CBS operation; then, the two parts that have undergone maximum pooling and CBS operations are concatenated together through a Concat operation.

5. The method for detecting nonferrous metal targets of scrapped vehicles based on attention twin network and multi-source image fusion according to claim 1 is characterized in that: Add SEAttention to the c2f module of the deep network of the original YOLOV8 model; enhance the ability of target detection by introducing a lightweight and efficient attention module to strengthen feature representation and introduce an attention mechanism.

6. The method for detecting nonferrous metal targets of scrapped vehicles based on attention twin network and multi-source image fusion according to claim 1 is characterized in that: The ASFF includes a feature pyramid, an adaptive fusion module and a prediction module; ASFF first constructs the input feature map into a feature pyramid, which contains features of different scales; Then, an adaptive fusion module is used to fuse features of different scales, and the fusion weight is assigned according to the importance of each feature map; finally, a prediction module is used to predict the fused features to generate the final detection result.

7. The method for detecting nonferrous metal targets of scrapped vehicles based on attention twin network and multi-source image fusion according to claim 1 is characterized in that: In step 1, the preprocessing operations of the visible light image and the infrared thermal imaging image are the same, and include image denoising, image enhancement, geometric transformation, image annotation, and image normalization operations; The image annotation operation uses the LabelImg tool to draw a bounding box for each object and assign a category label.

8. The method for detecting nonferrous metal targets of scrapped vehicles based on attention twin network and multi-source image fusion according to claim 1 is characterized in that: In the training of the scrapped car nonferrous metal target detection model, a comprehensive loss function is defined, which includes classification loss, positioning loss and target confidence loss; Among them, the classification loss is calculated using the cross entropy loss, and the formula is as follows: Where N is the number of samples, y i is the true category label. If the true category of the sample is i, then y i =1, otherwise 0, p i is the category probability predicted by the model for sample i; The positioning loss is calculated using the DloU loss formula, which is as follows: L DIoU =1-DIoU; Where b and bt represent the center points of the predicted box and the true box respectively, P represents the Euclidean distance between the two points, and c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the true box; IoU represents the similarity between the predicted box and the true box. The IoU value is between 0 and 1. The larger the value, the higher the overlap between the two areas and the more accurate the prediction result. The target confidence loss is calculated using the binary cross entropy loss function, and the formula is as follows: Where N represents the number of samples, y i is the true category label. If the true category of the sample is i, then y i =1, otherwise 0, p i is the category probability predicted by the model for sample i; Comprehensive loss function L total The above loss functions are weighted and summed up. The specific formula is as follows: Among them, λ1, λ2, and λ3 are weight coefficients used to balance the importance of different losses.

Images