A small target detection method based on multi-scale fusion of pixel-level enhanced features

Abstract

The present invention proposes a small target detection method based on multi-scale fusion of pixel-level enhanced features. The method firstly labels each pixel point and the entire image of the original input image, then extracts and enhances the features of the original input image, extracts and fuses the multi-scale feature layers and assigns weights to the fused feature layers, and finally obtains the category of the small target. The present invention can better meet the small target detection task, is efficient and accurate, improves the accuracy and efficiency of small target detection, has high application value and economic benefits, and has been proved through practical verification that this method can be well applied to the small target detection task on the surface of objects.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection on any object surface, and in particular to tiny target detection on the object surface. Background Art

[0002] The detection of tiny targets is applied in many fields, including machinery manufacturing, medicine, construction, electronics, etc. However, the detection of tiny targets on the surface of objects has always been a major challenge in terms of accuracy, and its low-precision detection results are far from meeting the detection requirements.

[0003] With the improvement of modern intelligence, people have more and more stringent requirements for target detection in machine vision. Not only do they need high accuracy and efficiency, but the targets to be detected are also getting smaller and smaller. Traditional small target detection uses manual detection methods, which have no unified standards, high missed detection and false detection rates, and low efficiency, and cannot meet the requirements of accuracy and efficiency. Therefore, it is urgent to break through the current small target detection method based on manual detection, improve the level of automation and intelligence of small target detection, and provide people with an intelligent method to improve the efficiency and accuracy of small target detection. Summary of the invention

[0004] The technical problem to be solved by the present invention is: how to accurately and efficiently detect tiny targets on the surface of an object.

[0005] In order to solve the above technical problems, the technical solution of the present invention is to provide a small target detection method based on multi-scale fusion of pixel-level enhanced features, which is characterized by comprising the following steps:

[0006] Step 1: Establish and train a small target detection network model, which includes a first-stage network model and a second-stage network model. The first-stage network model is a feature enhancement model guided by a visual target. Through the mapping relationship between the target and each pixel, the detailed features of the small target are enhanced at the pixel level. The second-stage network model derives and fuses the multi-scale feature layer in the first-stage network model, designs a feature weight assignment module, and assigns weights to the fused multi-scale feature layer as input data for the target classification network. Finally, the category of the small target is obtained. The training of the small target detection network model includes the following steps:

[0007] Step 101: A certain amount of training image data is collected by a small target detection device, and the training image data used by the first-stage network model and the training image data used by the second-stage network model are labeled to obtain a training image data set, wherein:

[0008] The following method is used to annotate the training image data used by the first stage network model:

[0009] Use pixel-level annotation methods to annotate each pixel in the training image. When there are N types of tiny targets, each pixel is annotated with a corresponding label value from N+1 labels according to whether it belongs to a certain target. One label value represents that the current pixel belongs to the image background, and the remaining N label values ​​represent the type of tiny target to which the current pixel belongs.

[0010] The training image data used by the second-stage network model is annotated in the following way:

[0011] According to whether each training image contains a small target and the type of the small target, each training image is labeled with a corresponding label value; when there are N types of small targets, each training image is labeled with a corresponding label value among N+1 labels according to whether it belongs to a certain target, where 1 label value represents that the current training image does not contain a small target, and the remaining N label values ​​represent the types of small targets contained in the current training image;

[0012] Step 102: sending the training image data set for training the first-stage network model obtained in step 101 to the first-stage network model, and training the first-stage network model through forward propagation and back propagation, so as to establish a correct mapping relationship for each pixel in the image;

[0013] Step 103: Through continuous iteration and optimization, when the loss value of the first-stage network model is less than a specified threshold, the training of the first-stage network model is stopped and the training of the second-stage network model is switched;

[0014] Step 104: sending the training image data set for training the second-stage network model obtained in step 101 to the trained first-stage network model, extracting and fusing the multi-scale features in the first-stage network model, and performing weight distribution on the fused features by the weight distribution network; back-propagating the loss value calculated by the second-stage network model, thereby optimizing the parameter variables of the second-stage network model;

[0015] Step 105: by continuously iterating and optimizing the parameters of the second-stage network model, when the second-stage network model completes the specified number of training times, the training of the second-stage network model is stopped, at which point the training of the entire small target detection network model is completed, and finally the complete small target detection network model is saved;

[0016] Step 2: Send the image data of the tiny target to be detected into the trained tiny target detection network model, and finally obtain the category of the tiny target.

[0017] Preferably, in step 1, the feature enhancement model is composed of 12 layers of full convolutional layers of the same dimension. There is no downsampling operation in the convolution process, that is, each convolution stride is 1 pixel, and all the intermediate feature layers are consistent with the width and height of the original input image. When the feature layers remain of the same dimension, the channel dimension shows an increasing trend, that is, the channel dimensions of each layer are: 16, 16, 32, 32, 32, 64, 64, 64, 128, 128, 128, N; at the same time, in order to better capture the features of small targets, all convolution kernels are 3×3 pixels in size, and the enhancement process is shown as follows:

[0018] Y FA =f′(f 11 (…f2(f1(X))…))

[0019] In the formula, represents the input image; Represents the output result after the input image is enhanced; f i represents the first 11 convolution operations of the pixel-level enhanced feature module, where i = 1, 2, ..., 11; f′ represents the 12th convolution operation of the pixel-level enhanced feature module;

[0020] f i It is expressed as: In the formula, w i represents the learning parameters of the i-th convolutional layer, i = 1, 2, 3, ..., 11; represents the i-th feature layer, whose width and height are both 160; C i Indicates the number of channels of the i-th feature layer; represents a convolution operation with a kernel size of 3×3 and a stride of 1; BN(·) represents a batch normalization operation, and δ(·) represents a ReLU activation function

[0021] f′ is expressed as:

[0022] Preferably, in step 104, extracting and fusing the multi-scale features in the first-stage network model refers to deriving the previous convolution layer and the final output feature layer for each channel dimension change in the first-stage network model, that is, deriving the 2nd, 5th, 8th, 11th, and 12th layers, and each feature layer after the export is compressed and fused in the channel dimension through a convolution operation with a corresponding convolution kernel size of 5×5 pixels and a channel dimension of 4, and all compressed and fused features and the last layer of the pixel-level enhanced feature module are spliced ​​along the channel direction, and finally a multi-scale feature layer is obtained, and the expression formula is as follows:

[0023]

[0024]

[0025]

[0026]

[0027] E=Cat([Γ1;Γ2;Γ3;Γ4;F 12 ])

[0028] Where: represents the result of fusion of the i-th scale feature layer, i = 1, 2, 3, 4; Cat(·) represents the splicing operation along the channel direction; Represents the multi-scale feature layer obtained after splicing.

[0029] Preferably, in step 104, the weight distribution network distributes weights to the fused features by squeezing the width and height of the feature layer in a stepwise compression manner to obtain the feature value of each feature layer, and then makes all the feature values ​​interact once. The specific operation is as follows:

[0030] Step 1041: performing two convolution operations on a feature layer, and the channel dimensions of the feature layer after convolution are 16 and 32 respectively;

[0031] Step 1042: After performing a global average pooling operation, a 32-dimensional feature vector is obtained, and then a convolution operation is performed to obtain the feature value of the feature layer. The convolution kernel size of this convolution operation is 1×1 pixel, the convolution stride is 1 pixel, and the channel dimension of the feature layer after convolution is 1;

[0032] The operations of step 1041 and step 1042 are performed on all feature layers that need to participate in weight allocation to obtain feature values ​​of each layer, and then all the obtained feature values ​​are interacted through a full connection operation. This full connection operation is a full connection layer with consistent input and output dimensions, and finally the weight value of each feature layer is obtained;

[0033] Multiply the feature layers by their respective weight values ​​to finally achieve the weight distribution of each feature layer using the following formula:

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] Where: represents the k-th channel feature layer in the multi-scale feature layer E, where k = 1, 2, ..., 17; and Represent the results after the first and second convolution respectively; GAP(·) represents the global average pooling operator; FC(·) represents the fully connected operation; σ(·) represents the sigmoid activation function; Represents the weight after all representative values ​​interact; Represents element-wise multiplication.

[0040] In view of the characteristics of unclear targets, large interference and small feature quantity, the present invention designs a small target detection method based on multi-scale fusion of pixel-level enhanced features, which can better meet the detection task, is accurate and efficient, has high application value and high economic benefits. Practical verification proves that the method provided by the present invention can be well applied to the small target detection task on the surface of objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the algorithm flow chart of the present invention;

[0042] Figure 2 It is a complete network model diagram of small target detection of the present invention;

[0043] Figure 3 This is a first-stage network model diagram of small target detection of the present invention;

[0044] Figure 4 It is a second-stage network model diagram of small target detection of the present invention;

[0045] Figure 5 This is the weight distribution network model diagram in the second stage of small target detection of the present invention DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0047] like Figure 1 As shown, the present invention provides a small target detection method based on multi-scale fusion of pixel-level enhanced features, which specifically includes the following steps:

[0048] Step 1: Figure 2As shown in the figure, the small target detection model consists of two stages: the first-stage network model and the second-stage network model. Use a suitable environment to build a complete small target detection model, and configure related parameters such as learning rate, optimizer, and number of iterations.

[0049] Step 2: A certain amount of training image data is collected by the small target detection device, and label data corresponding to the training image data used by the first-stage network model and the training image data used by the second-stage network model of the small target detection model are marked.

[0050] For the training image data used by the first-stage network model, the present invention adopts the method of labeling each pixel. For example, when there are N types of small targets, the labeling possibilities of each pixel are N+1. In this embodiment, 0 represents that the current pixel belongs to the image background, and 1, 2, ..., n, ..., N represent that the current pixel belongs to the nth type of small target.

[0051] For the label data corresponding to the training image data used by the second-stage network model, the present invention adopts the method of labeling the entire image. For example, when there are N types of small targets, the labeling possibilities for each image are N+1, where 0 represents that the current image does not contain a small target, and 1, 2, ..., n, ..., N represent that the current image contains the nth type of small target.

[0052] Step 3: The first stage network model of the small target detection model is as follows Figure 3 As shown, the original training image data is fed into the first-stage network model, and the first-stage network model is trained through forward propagation and back propagation to establish the correct mapping relationship for each pixel in the image. The main steps include:

[0053] Step 301: Establish a first-stage network model based on a deep neural network model as a mapping relationship at the image pixel level. The deep neural network model consists of 12 convolutional layers and does not include any downsampling operations. All feature layers in the middle are consistent with the width and height of the original input image. Through multiple convolution operations in the deep neural network model, an N+1-dimensional output layer is finally obtained.

[0054] The forward propagation process of the deep neural network model is expressed as follows:

[0055] Y FA =f′(f 11 (…f2(f1(X))…)) (1)

[0056] In formula (1), represents the input image; Represents the output result after the input image is enhanced; f irepresents the first 11 convolution operations of the pixel-level enhanced feature module, where i = 1, 2, ..., 11; f′ represents the 12th convolution operation of the pixel-level enhanced feature module.

[0057] f i It is expressed as follows:

[0058]

[0059] In formula (2), w i represents the learning parameters of the i-th convolutional layer, i = 1, 2, 3, ..., 11; represents the i-th feature layer, whose width and height are both 160; C i Indicates the number of channels of the i-th feature layer; represents a convolution operation with a kernel size of 3×3 and a stride of 1; BN(·) represents a batch normalization operation, and δ(·) represents a ReLU activation function.

[0060] f′ is expressed as follows:

[0061]

[0062] Step 302: Back propagation is performed on the deep neural network model to optimize the parameter variables, thereby establishing a correct mapping relationship at the pixel level. The output layer and the labeled training image data are subjected to a loss function calculation to obtain the loss value of each iteration, and finally the first-stage network model parameters are back propagated to optimize.

[0063] The loss function of the first-stage network model is the mean square error at the pixel level, expressed as follows:

[0064]

[0065] In formula (4), n represents the number of images in a training sample set, h represents the height of the image, and w represents the width of the image; label (k，i，j ) represents the kth training image X in the training sample set k The true label value of the pixel point coordinate (i, j); pre_label (k，i，j) Denotes the training image X k The predicted image Y obtained after passing the first stage network model FA The predicted value at pixel coordinates (i, j).

[0066] Step 4: By continuously iterating and optimizing the parameters of the first-stage network model, when the loss value Loss1 of the first-stage network model is less than the specified threshold, the training parameters of the first-stage network model are frozen, that is, the training of the first-stage network model parameters is completed at this time, and then the training of the second-stage network model is started.

[0067] Step 5: The second stage network model of the small target detection model is as follows Figure 4 As shown in , the original training image data is fed into the trained first-stage network model, the multi-scale features in the first-stage network model are extracted and fused, and the weight distribution network performs weight distribution on the fused features, as shown in Figure 5 As shown. The loss value calculated by the second-stage network model is back-propagated to optimize the parameter variables of the second-stage network model. It mainly includes the following steps:

[0068] Step 501: Take out the previous convolution layer and the last single-channel output layer of each channel dimension change in the first-stage network model, compress the extracted feature layer in the channel dimension through the convolution operation, compress its channel number to 4, and splice all compressed features and the last layer of the pixel-level enhanced feature module along the channel direction, and finally obtain a multi-scale feature layer, which is expressed as follows:

[0069]

[0070]

[0071]

[0072]

[0073] E=Cat([Γ1;Γ2;Γ3;Γ4;F 12 ]) (9)

[0074] In formula (5) to formula (9): represents the result of fusion of the i-th scale feature layer, i = 1, 2, 3, 4; Cat(·) represents the splicing operation along the channel direction; Represents the multi-scale feature layer obtained after splicing.

[0075] Step 502: Use the weight allocation network to allocate weights to the obtained multi-scale feature layer. Each weight allocated by the weight allocation network represents the importance of each feature layer. The operation on each feature layer is as follows: first, perform two large-stride convolution operations on the feature layer, then perform a global average pooling operation, and then perform another convolution operation to obtain a feature value of the feature layer. Perform the above operations on all feature layers that need to participate in weight allocation to obtain the feature values ​​of each layer, and then perform a full connection operation on the group of feature values ​​to finally obtain the weight value of each feature layer. Multiply the feature layers by their respective weight values ​​to finally achieve weight allocation for each feature layer, which is expressed as follows:

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] In formula (10) to formula (14): represents the k-th channel feature layer in the multi-scale feature layer E, where k = 1, 2, ..., 17; and Represent the results after the first and second convolution respectively; GAP(·) represents the global average pooling operator; FC(·) represents the fully connected operation; σ(·) represents the sigmoid activation function; Represents the weight after all representative values ​​interact; Represents element-wise multiplication.

[0082] Step 503: Through continuous convolution, pooling and full connection, each training image finally obtains N+1 classification results, and the classification results and the corresponding labels are cross-entropy loss calculated to obtain the loss value, and the loss value is back-propagated to optimize the parameter variables of the second-stage network model. Among them, the loss function of the second-stage network model is the cross-entropy loss function of the entire image label, which is expressed as follows:

[0083]

[0084] In formula (15): C i represents the true label value of the i-th training image in the training sample set; p i Represents the predicted value of the i-th training image.

[0085] Step 6: By continuously iterating and optimizing the parameters of the second-stage network model, when the second-stage network model completes the specified number of training times, the training of the second-stage network model is stopped. At this time, the training of the entire small target detection model is completed, and finally the complete small target detection model is saved.

[0086] Step 7: Load the trained small target detection model, send the small target image data to be detected into the loaded small target detection model, and finally obtain the category of the small target.

Claims

1. A small target detection method based on multi-scale fusion of pixel-level enhanced features, characterized in that: The following steps are involved: Step 1: Establish and train a small target detection network model, which includes a first-stage network model and a second-stage network model. The first-stage network model is a feature enhancement model guided by a visual target. Through the mapping relationship between the target and each pixel, the detailed features of the small target are enhanced at the pixel level. The second-stage network model derives and fuses the multi-scale feature layer in the first-stage network model, designs a feature weight assignment module, and assigns weights to the fused multi-scale feature layer as input data for the target classification network. Finally, the category of the small target is obtained. The training of the small target detection network model includes the following steps: Step 101: A certain amount of training image data is collected by a small target detection device, and the training image data used by the first-stage network model and the training image data used by the second-stage network model are labeled to obtain a training image data set, wherein: The following method is used to annotate the training image data used by the first stage network model: Use pixel-level annotation methods to annotate each pixel in the training image. When there are N types of tiny targets, each pixel is annotated with a corresponding label value from N+1 labels according to whether it belongs to a certain target. One label value represents that the current pixel belongs to the image background, and the remaining N label values ​​represent the type of tiny target to which the current pixel belongs. The training image data used by the second-stage network model is annotated in the following way: According to whether each training image contains a small target and the type of the small target, each training image is labeled with a corresponding label value; when there are N types of small targets, each training image is labeled with a corresponding label value among N+1 labels according to whether it belongs to a certain target, where 1 label value represents that the current training image does not contain a small target, and the remaining N label values ​​represent the types of small targets contained in the current training image; Step 102: sending the training image data set for training the first-stage network model obtained in step 101 to the first-stage network model, and training the first-stage network model through forward propagation and back propagation, so as to establish a correct mapping relationship for each pixel in the image; Step 103: Through continuous iteration and optimization, when the loss value of the first-stage network model is less than a specified threshold, the training of the first-stage network model is stopped and the training of the second-stage network model is switched; Step 104: sending the training image data set for training the second-stage network model obtained in step 101 to the trained first-stage network model, extracting and fusing the multi-scale features in the first-stage network model, and performing weight distribution on the fused features by the weight distribution network; back-propagating the loss value calculated by the second-stage network model, thereby optimizing the parameter variables of the second-stage network model; Step 105: by continuously iterating and optimizing the parameters of the second-stage network model, when the second-stage network model completes the specified number of training times, the training of the second-stage network model is stopped, at which point the training of the entire small target detection network model is completed, and finally the complete small target detection network model is saved; Step 2: Send the image data of the tiny target to be detected into the trained tiny target detection network model, and finally obtain the category of the tiny target.

2. A method for detecting small targets based on multi-scale fusion of pixel-level enhanced features as claimed in claim 1, characterized in that: In step 1, the feature enhancement model consists of 12 layers of full convolutional layers of the same dimension. There is no downsampling operation in the convolution process, that is, each convolution stride is 1 pixel, and all the intermediate feature layers are consistent with the width and height of the original input image. When the feature layers keep the same dimension, the channel dimension shows an increasing trend, that is, the channel dimensions of each layer are: 16, 16, 32, 32, 32, 64, 64, 64, 128, 128, 128, N; at the same time, in order to better capture the features of small targets, all convolution kernels are 3×3 pixels in size, and the enhancement process is shown as follows: Y FA =f′(f 11 (…f2(f1(X))…)) In the formula, represents the input image; Represents the output result after the input image is enhanced; f i represents the first 11 convolution operations of the pixel-level enhanced feature module, where i = 1, 2, ..., 11; f′ represents the 12th convolution operation of the pixel-level enhanced feature module; f i It is expressed as: In the formula, w i represents the learning parameters of the i-th convolutional layer, i = 1, 2, 3, ..., 11; represents the i-th feature layer, whose width and height are both 160; C i Indicates the number of channels of the i-th feature layer; represents a convolution operation with a kernel size of 3×3 and a stride of 1; BN(·) represents a batch normalization operation, and δ(·) represents a ReLU activation function f′ is expressed as:

3. A method for detecting small targets based on multi-scale fusion of pixel-level enhanced features as claimed in claim 2, characterized in that: In step 104, extracting and fusing the multi-scale features in the first-stage network model refers to deriving the previous convolution layer and the final output feature layer of each channel dimension change in the first-stage network model, that is, deriving the 2nd, 5th, 8th, 11th, and 12th layers. Each feature layer after deriving is compressed and fused in the channel dimension through a convolution operation with a corresponding convolution kernel size of 5×5 pixels and a channel dimension of 4, and all compressed and fused features and the last layer of the pixel-level enhanced feature module are spliced ​​along the channel direction to finally obtain a multi-scale feature layer. The expression formula is as follows: E=Cat([Γ1;Γ2;Γ3;Γ4;F 12 ]) Where: represents the result of fusion of the i-th scale feature layer, i = 1, 2, 3, 4; Cat(·) represents the splicing operation along the channel direction; Represents the multi-scale feature layer obtained after splicing.

4. A method for detecting small targets based on multi-scale fusion of pixel-level enhanced features as claimed in claim 3, characterized in that: In step 104, the weight distribution network distributes weights to the fused features by squeezing the width and height of the feature layer in a stepwise manner to obtain the feature values ​​of each feature layer, and then makes all the feature values ​​interact once. The specific operation is as follows: Step 1041: performing two convolution operations on a feature layer, and the channel dimensions of the feature layer after convolution are 16 and 32 respectively; Step 1042: After performing a global average pooling operation, a 32-dimensional feature vector is obtained, and then a convolution operation is performed to obtain the feature value of the feature layer. The convolution kernel size of this convolution operation is 1×1 pixel, the convolution stride is 1 pixel, and the channel dimension of the feature layer after convolution is 1; The operations of step 1041 and step 1042 are performed on all feature layers that need to participate in weight allocation to obtain feature values ​​of each layer, and then all the obtained feature values ​​are interacted through a full connection operation. This full connection operation is a full connection layer with consistent input and output dimensions, and finally the weight value of each feature layer is obtained; Multiply the feature layers by their respective weight values ​​to finally achieve the weight distribution of each feature layer using the following formula: Where: represents the k-th channel feature layer in the multi-scale feature layer E, where k = 1, 2, ..., 17; and Represent the results after the first and second convolution respectively; GAP(·) represents the global average pooling operator; FC(·) represents the fully connected operation; σ(·) represents the sigmoid activation function; Represents the weight after all representative values ​​interact; Represents element-wise multiplication.

Images