A highway small target detection method

Abstract

The application discloses a highway small target detection method, comprising the following steps: acquiring an unlabeled data set and performing data enhancement processing on each input image in the unlabeled data set to form a corresponding reconstructed image; establishing a target detection network model and detecting the reconstructed image to obtain a corresponding target detection result. The method is different from a traditional target detection model, has high precision in identifying small pixel targets, especially has strong adaptability to abnormal weather scenes of highways, can more accurately detect abnormal objects on highways, and uses a masking reconstruction data enhancement method to obtain more accurate frames for small target object detection. In view of the characteristics of small target objects, such as few features and unbalanced samples, the loss function is improved, and a balanced focal loss function is adopted to alleviate the class imbalance problem, thereby improving the precision of small target detection and better applying to highways.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image recognition and computer vision, and particularly relates to a highway small target detection method. BACKGROUND

[0002] The highway is a modern symbol and a manifestation of a country's comprehensive national strength. The significance and role of the highway for a country mainly lie in that its construction and operation involve various aspects of the country's economy and social life. However, objects other than cars may appear on the highway, such as goods spilled by trucks, animals, garbage and the like, which have great safety hazards. Through computer vision technology, real-time images are collected by using a camera to detect foreign objects appearing on the highway, and timely measures are taken to handle the foreign objects, so as to maintain the smoothness of the highway.

[0003] The existing target detection method is a detection method based on deep learning. Usually, several categories of target data sets are first collected, then a general target detection model is used for training, and finally the trained model is detected. Although the detection method based on deep learning has high detection accuracy, for the pictures collected on the highway, the pixels of the foreign objects are small, the available features are less, and the positioning accuracy is high, which leads to difficulty in detection and even the foreign objects are ignored. Obviously, the existing target detection model does not have ideal actual effect when applied to the highway. SUMMARY

[0004] The application aims at the above problems and provides a highway small target detection method to overcome the problem that the traditional target detection model is difficult to achieve good detection effect.

[0005] To achieve the above object, the technical scheme adopted by the application is as follows:

[0006] The highway small target detection method provided by the application comprises the following steps:

[0007] S1, acquiring an unlabeled data set X={x1, x2, …, xN}, and performing data enhancement processing on each input image in the unlabeled data set to form corresponding reconstructed images l ,…,x N} represent the lth input image, l=1, 2, …, N; l

[0008] S2, establishing a target detection network model and detecting the reconstructed images ​​The detection is performed to obtain a corresponding target detection result. The target detection network model comprises a feature extraction module, a dynamic instance interaction head, and a classification and regression branch unit. The feature extraction module adopts an FPN network. The dynamic instance interaction head comprises N feature extraction units. Each feature extraction unit comprises a self-attention module, a full connection layer, a first convolutional layer, a second convolutional layer, a ReLu function, and a view operation. The target detection network model performs the following operations:

[0009] S21, reconstructing images The input feature extraction module is obtained. The corresponding multi-scale feature map is obtained.

[0010] S22, setting N proposal boxes and corresponding proposal features. The proposal box is represented as a four-dimensional vector formed by normalized center coordinates, height, and width. The proposal feature has the same dimension as the output feature of the feature extraction module.

[0011] S23, inputting the proposal box and the multi-scale feature Figure One A corresponding ROI feature is obtained through the RoIAlign operation.

[0012] S24, inputting the proposal feature and the ROI feature of each proposal box into the feature extraction unit of the dynamic instance interaction head one by one to obtain the corresponding target box and target feature. The feature extraction unit performs the following operations:

[0013] The self-attention module is used to perform self-attention operation on the proposal feature to obtain a first feature.

[0014] The first feature is converted into a one-dimensional vector through the full connection layer to form a second feature.

[0015] The ROI feature and the second feature are input into the first convolutional layer, and are sequentially input into the second convolutional layer, the ReLu function, and then the view operation is used to adjust the dimension to obtain the corresponding target feature.

[0016] S25, updating the proposal box and the proposal feature to the target box and the target feature. Step S23 is returned to be executed until the iteration number is completed to obtain the interaction feature.

[0017] S26, inputting the interaction feature into the classification and regression branch unit to obtain the target detection result.

[0018] Preferably, the data augmentation processing is performed on each input image in the unlabeled data set to form a corresponding reconstructed image The data augmentation module is adopted to implement the data augmentation module. The data augmentation module comprises a first encoder, a second encoder, and a decoder, and performs the following operations:

[0019] S11, adopting an unlabeled data set X={x1, x2, …, x l ,…,xN training the second encoder and the decoder, wherein the second encoder E θ satisfies decoder satisfies denotes a block-wise binary mask with image block size of WxH pixels, W denotes the pixel width of the input image x, and H denotes the pixel height of the input image x;

[0020] S12, dividing each input image into S image blocks;

[0021] S13, performing the following operations on each divided input image:

[0022] S131, converting the divided input image into a vector by using the first encoder;

[0023] S132, obtaining the attention map Attn i of the i-th image block based on an attention strategy:

[0024] Attn i = q cls · k i , i∈{0,1,…,p 2 -1}

[0025] wherein q cls denotes the query of the image block sequence, k i denotes the key embedding of the i-th image block, and p denotes the size of the image block;

[0026] S133, obtaining the top K index set Ω by sorting each attention map:

[0027] Ω = top-rank(Attn, K)

[0028] wherein top-rank(·, K) denotes returning the index of the top K largest elements, and Attn denotes the set of Attn i ;

[0029] S134, obtaining the binary mask M * :

[0030]

[0031] wherein denotes a down-round operation, mod(·) denotes a modulo operation, and Ω i denotes the i-th element in the index set Ω;

[0032] S135, obtaining the masked image M * according to the binary mask M *⊙x, divide the masked image into non-overlapping image blocks and discard the image blocks blocked by the binary mask, the remaining visible image blocks are sent into the pre-trained second encoder and decoder to generate the corresponding reconstructed image

[0033] Preferably, the loss function of the target detection network model is calculated as follows:

[0034]

[0035] wherein,

[0036]

[0037] wherein, is the focal loss of the balanced focal points of the predicted classification and the real classification, is the L1 loss of the predicted box and the real box, is the distance intersection over union loss of the predicted box and the real box, λ cls , λ L1 , λ diou correspond to, in turn, the coefficient of, α t is the weight factor for balancing the number of positive and negative samples, p t is the probability that the prediction is a positive sample, γ j is the focusing coefficient of the jth class, j = 1, 2, …, T, T is the total number of classes, γ j is decoupled into a first component γ b and a second component The first component γ b is used to control the basic behavior of the classifier, is a variable parameter, and is selected using a gradient guide mechanism g j represents the cumulative gradient ratio of the jth class of positive and negative samples, and the value range is [0, 1], s is a proportion factor for determining the upper limit of γ j , y pz represents the predicted value, y gz represents the true value, z = 1, 2, …, n, n represents the number of target objects, ρ 2 (b p , b g ) represents the Euclidean distance between the center point b p of the predicted box and the center point b g of the real box, c represents the diagonal distance of the smallest rectangle covering the predicted box and the real box at the same time, represents the penalty term, and IOU represents the intersection over union.

[0038] Preferably, the number of iterations E = 6, and the number of proposal boxes and proposal features N = 100.

[0039] Compared with the prior art, the application has the following beneficial effects:

[0040] The method is different from the traditional target detection model, has high precision in identifying small pixel targets, has strong adaptability to abnormal weather scenes of highways, and can more accurately detect abnormal objects on highways, and uses a masking reconstruction data enhancement method to obtain more accurate frames for small target object detection, and improves the loss function for the characteristics of small target objects and sample imbalance, uses a balanced focal loss function to alleviate the class imbalance problem, balances the loss contribution of positive and negative samples and difficult samples, thereby improving the precision of small target detection and better applying to highways. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The flowchart of the highway small target detection method of the application is shown in the figure.

[0042] Figure 2 The structural diagram of the target detection network model of the application is shown in the figure.

[0043] Figure 3 The structural diagram of the data enhancement module of the application is shown in the figure.

[0044] Figure 4 The structural diagram of the feature extraction module of the application is shown in the figure.

[0045] Figure 5 The interaction process diagram of the dynamic instance interaction head of the application is shown in the figure. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0047] It should be noted that when a component is referred to as being "connected" to another component, it can be directly connected to the other component or there can be a middle component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application belongs. The terms used in the specification of the application are only for the purpose of describing the specific embodiments of the application and are not intended to limit the application.

[0048] As shown in Figures 1-5 A highway small target detection method, comprising the following steps:

[0049] S1, obtain an unlabeled dataset X = {x1, x2, …, x l ,…,x N}, and perform data augmentation processing on each input image in the unlabeled dataset to form a corresponding reconstructed image x l , l = 1, 2, …, N.

[0050] In an embodiment, data augmentation processing is performed on each input image in the unlabeled dataset to form a corresponding reconstructed image is implemented using a data augmentation module, which includes a first encoder, a second encoder, and a decoder, and performs the following operations:

[0051] S11, train the second encoder and the decoder using the unlabeled dataset X = {x1, x2, …, x l ,…,x N}, where the learnable parameters θ of the second encoder E θ satisfy the decoder satisfy represents a block-by-block binary mask with a block size of W x H pixels, W represents the pixel width of the input image x, and H represents the pixel height of the input image x;

[0052] S12, divide each input image into S image blocks;

[0053] S13, perform the following operations on each divided input image:

[0054] S131, convert the divided input image into a vector using the first encoder;

[0055] S132, obtain the attention map Attn i of the i-th image block based on an attention strategy:

[0056] Attn i = q cls ·k i , i ∈ {0, 1, …, p 2 -1}

[0057] where q cls represents the query of the image block sequence, k i represents the key embedding of the i-th image block, and p represents the size of the image block;

[0058] S133, obtain the top K index set Ω by sorting the attention maps:

[0059] Ω = top-rank(Attn, K)

[0060] where top-rank(·, K) denotes the index of the top K largest elements, Attn denotes the Attn i set;

[0061] S134, obtaining a binary mask M * :

[0062]

[0063] where denotes a down-round operation, mod(·) denotes a modulo operation, and i denotes the i-th element in the index set

[0064] S135, obtaining a masked image M * from the binary mask M * and discarding the image blocks occluded by the binary mask, and sending the remaining visible image blocks to the pre-trained second encoder and decoder to generate the corresponding reconstructed image

[0065] As shown in Figure 3 , the data augmentation module includes a first encoder (Encoder), a second encoder (encoder), and a decoder (decoder), and through a heat map (Heat map), the relationship between the data is visually represented after the attention operation is performed, and Top-k represents the ranking of each attention map.

[0066] S2, establishing a target detection network model and detecting the reconstructed image to obtain the corresponding target detection result, the target detection network model includes a feature extraction module, a dynamic instance interaction head, and a classification and regression branch unit, the feature extraction module adopts an FPN network, the dynamic instance interaction head includes N feature extraction units, each feature extraction unit includes a self-attention module, a full connection layer, a first convolutional layer, a second convolutional layer, a ReLu function, and a view operation, and the target detection network model performs the following operations:

[0067] S21, inputting the reconstructed image into the feature extraction module to obtain the corresponding multi-scale feature map;

[0068] S22, setting N proposal boxes and corresponding proposal features, the proposal box is represented as a four-dimensional vector formed by normalized center coordinates, height, and width, and the proposal feature has the same dimension as the output feature of the feature extraction module;

[0069] S23, inputting the proposal box and the multi-scale feature Figure OneA corresponding ROI feature is obtained through a RoIAlign operation;

[0070] S24, the proposal feature and the ROI feature of each proposal box are input into a feature extraction unit of the dynamic instance interaction head one by one, and a corresponding target box and target feature are obtained, and the feature extraction unit performs the following operations:

[0071] The self-attention module is used for self-attention operation on the proposal feature to obtain a first feature;

[0072] The first feature is converted into a one-dimensional vector through a full connection layer to form a second feature;

[0073] The ROI feature and the second feature are input into a first convolutional layer, and are sequentially input into a second convolutional layer and a ReLu function, and then a view operation is used to adjust the dimension to obtain a corresponding target feature;

[0074] S25, the proposal box and the proposal feature are updated to be the target box and the target feature, and step S23 is returned to be executed until the iteration number is completed, and an interaction feature is obtained;

[0075] S26, the interaction feature is input into a classification and regression branch unit to obtain a target detection result.

[0076] As shown in Figure 5 , Proposal Feat represents the proposal feature, Roi Feat represents the ROI feature, Self-Attention represents the self-attention module, and Parmas represents the second feature. Figure 2 In the formula, the feature vector represents the proposal feature and the ROI feature of each proposal box.

[0077] In an embodiment, the loss function of the target detection network model is calculated as follows:

[0078]

[0079] Wherein,

[0080]

[0081] In the formula, is the focal loss of the predicted classification and the real classification, is the L1 loss of the predicted box and the real box, is the distance intersection over union loss of the predicted box and the real box, λ cls , λ L1 , λ diou correspond to respectively. t is the weight factor for balancing the number of positive and negative samples, p​t γ j is the focusing coefficient of the j-th class, j = 1, 2, …, T, T is the total number of classes, γ j is decoupled into a first component γ b and a second component The first component γ b is used to control the basic behavior of the classifier, is a variable parameter, which is selected using a gradient-guided mechanism g j represents the cumulative gradient ratio of positive and negative samples of the j-th class, and its value range is [0, 1], s is a proportion factor to determine the upper limit of γ j , y pz represents the predicted value, y gz represents the true value, z = 1, 2, …, n, n represents the number of target objects, ρ 2 (b p ,b g ) represents the Euclidean distance between the center point b p of the predicted box and the center point b g of the real box, c represents the diagonal distance of the minimum rectangle covering the predicted box and the real box at the same time, represents the penalty term, and IOU represents the intersection over union.

[0082] In an embodiment, the number of iterations E = 6, and the number of proposal boxes and proposal features N = 100.

[0083] Specifically, in this embodiment, the feature extraction module adopts a FPN network based on ResNet. The FPN network is a feature pyramid, which is a prior art, and its structure is as shown in Figure 4 The FPN network is obtained by the following steps: (1) a bottom-up path, passing through the backbone, using the feature activation output of the last residual structure of each stage, the outputs of the residual modules conv2, conv3, conv4, and conv5 are represented as {C2, C3, C4, C5}; (2) a top-down path and a lateral connection, the deep feature map is up-sampled to obtain a higher resolution map, and then the feature map after up-sampling is spliced with the bottom-up feature map through the lateral connection, as shown in Figure 4 The feature pyramids of P2 to P5 are constructed. The number of layers of the pyramid is represented by l, the resolution of each layer of feature map is 2 l lower than that of the input image, and all pyramid layers have 256 channels. Let the size of the reconstructed image be h x w, h is the height of the reconstructed image, and w is the width of the reconstructed image. The outputs of each stage of the FPN network are shown in Table 1:

[0084] Table 1

[0085]

[0086] The proposal boxes and proposal features are both learnable, and one-to-one correspondence. A set of learnable proposal boxes are used as region proposals, which are represented by four-dimensional parameters ranging from 0 to 1, which are normalized center coordinates, height and width. The parameters of the proposal boxes are updated during training by backpropagation algorithm. Backpropagation is the most commonly used method to train artificial neural networks, which propagates the error of the output layer back layer by layer, and updates the network parameters by calculating the partial derivative to minimize the error loss function. These learnable proposal boxes are the statistics of potential target positions in the training set, which can be regarded as an initial guess of the most likely target region in the image without considering the input. However, these boxes only provide rough positioning information, and the pose and shape of the object are lost, which is not conducive to subsequent classification and regression, so the learnable proposal features are used to represent the features of each instance. The feature is a high-dimensional latent vector, which is used to encode rich instance characteristics.

[0087] The classification and regression branch unit is prior art, and the regression prediction is performed by a three-layer perception calculation, and the classification prediction is realized by a linear mapping layer, which will not be described here. The target detection network model uses a set of prediction losses for fixed size classification and box coordinate prediction. The set-based loss generates the best bipartite matching between the prediction and the real object. For example, the target detection network model detects 100 target boxes, and the real box is also expanded into 100 detection boxes. In this way, the prediction and the real are both two 100-element sets. By using the Hungarian algorithm to perform bipartite matching, that is, one-to-one correspondence between the elements of the prediction set and the real set, the matching loss is minimized, and the loss is calculated by positive and negative sample pairs.

[0088] The loss function of the target detection network model is γ j balances the difficult and easy samples, and γ b is used to control the basic behavior of the classifier, and does not act on the class imbalance problem. is a variable parameter that determines the attention degree of the jth class learning in the positive and negative imbalance problem, and the gradient guide mechanism is used to select In order to better meet the needs, g j is controlled in the range of [0, 1]. As a weight coefficient, it is used to balance the loss contribution of different classes, so that the rare samples make more loss contribution than the common samples. For rare class data, the weight coefficient is set to a larger value to increase its loss contribution, and for frequent class data, the weight coefficient remains around 1. The final loss is the sum of all pairs normalized by the number of objects in the training batch.

[0089] Since the types of the spilled objects, animals and garbage on the highway are not many, and the number of different categories also presents an extremely unbalanced state. In order to solve the extreme imbalance of categories, a focus coefficient and a weight coefficient are added on the basis of the traditional focal loss function. The embodiment adopts distance intersection over union (DIOU). Since the spilled objects on the highway are small targets under the real-time camera shooting, the prediction box is often larger than the real box, forming an inclusion relationship. The loss of the target box is the same when the target box is in the center and the corner of the prediction box. By adding a penalty term for measuring the distance between the center points of the target box and the prediction box.

[0090] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the description.

[0091] The above embodiments only express the more specific and detailed embodiments described in the present application, but should not be construed as limiting the scope of the patent application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for detecting small targets on highways, characterized in that: The method for detecting small targets on highways includes the following steps: S1. Obtain the unlabeled dataset X = {x1, x2, ..., x...} l ,…,x N The system then performs data augmentation on each input image in the unlabeled dataset to generate the corresponding reconstructed image. x l Let l represent the l-th input image, where l = 1, 2, ..., N; The process involves performing data augmentation on each input image in the unlabeled dataset to generate the corresponding reconstructed image. A data augmentation module is used, which includes a first encoder, a second encoder, and a decoder, and performs the following operations: S11. Using an unlabeled dataset X = {x1, x2, ..., x...} l ,…,x N Train the second encoder and decoder, where the second encoder E θ The learnable parameter θ satisfies The decoder satisfy M∈{0,1} W×H This indicates the use of a block-by-block binary mask with an image block size of W×H pixels, where W represents the pixel width of the input image x and H represents the pixel height of the input image x. S12. Divide each input image into S image blocks; S13. Perform the following operation on each segmented input image: S131. Use the first encoder to convert the segmented input image into a vector; S132. Obtain the attention map Attn of the i-th image patch based on an attention strategy. i : Attn i =q cls ·k i ,i∈{0,1,…,p 2 -1} In the formula, q cls Represents a query for an image patch sequence, k i Let p represent the key embedding of the i-th image patch, and p represent the size of the image patch; S133. Sort each attention map and obtain the top K index sets Ω: Ω = top-rank(Attn, K) In the formula, top-rank(·,K) represents the index of the top K largest elements, and Attn represents Attn. i A set; S134. Obtain the binary mask M * : In the formula, Ω represents rounding down, mod(·) represents modulo operation, and Ω represents the rounding down operation. i This represents the i-th element in the index set Ω; S135, According to the binary mask M * Obtain the masking image M * The masked image is divided into non-overlapping image blocks, and the image blocks obscured by the binary mask are discarded. The remaining visible image blocks are fed into the pre-trained second encoder and decoder to generate the corresponding reconstructed image. S2. Establish an object detection network model and process the reconstructed image. The target detection network model performs detection to obtain corresponding target detection results. It includes a feature extraction module, a dynamic instance interaction head, and classification and regression branch units. The feature extraction module uses an FPN network. The dynamic instance interaction head includes N feature extraction units, each of which includes a self-attention module, a fully connected layer, a first convolutional layer, a second convolutional layer, a ReLU function, and a view operation. The target detection network model performs the following operations: S21, Reconstruct the image Input the feature extraction module to obtain the corresponding multi-scale feature map; S22. Set N suggestion boxes and corresponding suggestion features. The suggestion box is represented as a four-dimensional vector formed by the normalized center coordinates, height and width. The suggestion feature has the same dimension as the output feature of the feature extraction module. S23. By mapping the proposal boxes and multi-scale feature maps one by one, the corresponding ROI features are obtained through the RoIAlign operation. S24. Input the suggestion features and ROI features of each suggestion box into the feature extraction unit of the dynamic instance interaction header to obtain the corresponding target box and target features. The feature extraction unit performs the following operations: The proposed features are subjected to self-attention operation using a self-attention module to obtain the first feature; The first feature is transformed into a one-dimensional vector through a fully connected layer to form the second feature; The ROI features and the second feature are input into the first convolutional layer, and then passed through the second convolutional layer and the ReLU function in sequence. The dimensions are then adjusted using the view operation to obtain the corresponding target features. S25. Update the suggestion box and suggestion feature to correspond to the target box and target feature, and return to execute step S23 until the number of iterations is completed and the interaction features are obtained. S26. Input the interactive features into the classification and regression branch unit to obtain the target detection results.

2. The method for detecting small targets on highways as described in claim 1, characterized in that: The loss function of the object detection network model The calculation is as follows: in, In the formula, To predict the equilibrium focus loss between the classification and the true classification, The L1 loss is calculated between the predicted bounding box and the ground truth bounding box. The intersection-over-union (IoU) loss is the distance between the predicted bounding box and the ground truth bounding box. cls , λ L1 , λ diou Corresponding to in turn The coefficient, α t To balance the weighting factors for the number of positive and negative samples, p t To predict the probability that a sample is positive, γ j γ is the focus coefficient for the j-th class, j = 1, 2, ..., T, where T is the total number of classes. j Decoupling to the first component γ b Second component First component γ b Used to control the basic behavior of the classifier For variable parameters, a gradient-guided mechanism is used for selection. g j γ represents the cumulative gradient ratio between positive and negative samples of class j, with a value range of [0,1]. j The upper limit scaling factor, y pz Represents the predicted value, y gz Let z = 1, 2, ..., n, where n represents the number of target objects, and ρ represents the true value. 2 (b p ,b g ) represents the center point b of the prediction box. p and the center point b of the real frame g The Euclidean distance is given by c, where c represents the diagonal distance of the smallest rectangle that simultaneously covers both the predicted and ground truth bounding boxes. The term "penalty" is represented by "IOU".

3. The method for detecting small targets on highways as described in claim 1, characterized in that: The number of iterations E = 6, and the number of suggestion boxes and suggestion features N = 100.

Images