Image target detection method and system based on improved LW-DETR algorithm

Abstract

The invention discloses an image target detection method and system based on an improved LW-DETR algorithm, and the method comprises the following steps: S1, collecting an image data set, and labeling the type and position of each to-be-detected object in an image; obtaining a target detection data set, and dividing the target detection data set according to a proportion to obtain a training set, a verification set and a test set; s2, an LW-DETR target detection network model is constructed, Deformable multi-head attention is adopted for multi-head attention in an encoder of the LW-DETR target detection network model, and a DN-DETR denoising strategy is applied to a decoder; s3, using the target detection data set to pre-train the improved LW-DETR target detection network model, and then using the image data set collected in the S1 to perform training adjustment; and S4, using the model trained in the step S3 to obtain a target detection frame and a classification prediction result corresponding to the test set in the step S1 or the image in the practical application, drawing the target detection frame on the corresponding image according to the target detection frame and the classification prediction result, and performing classification marking.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection in computer vision, and specifically is an image target detection method and system improved based on the current LW-DETR algorithm. Background Art

[0002] In recent years, with the development of deep learning technology, target detection algorithms have achieved significant improvements in accuracy and efficiency. Different fields have different requirements for target detection technology, which has promoted the diversified development of technology. However, in some scenarios with high safety requirements, such as autonomous driving, medical diagnosis and other fields that have strict requirements on detection speed and accuracy, target detection technology still faces great challenges. In the field of target detection, the current mainstream model series are mainly divided into models based on convolutional neural networks and models based on Transformer architecture. In practical applications, the appropriate model should be selected according to the requirements of specific tasks and the limitations of computing resources. For example, when processing high-resolution images or tasks that need to capture long-distance dependencies, you can consider using the Transformer model; while in scenarios that process small and medium-sized data sets or have high requirements for computing efficiency, you can choose to use the convolutional neural network model.

[0003] In the early days of the development of the field of object detection, convolutional neural networks represented by RCNN, YOLO and other series dominated. Convolutional neural networks can effectively extract local features in images through convolution operations, and gradually build higher-level feature representations through hierarchical structures, which is crucial for object detection tasks. At the same time, convolutional neural networks are generally low in computational complexity, especially when dealing with small and medium-sized data sets, and can complete the training and reasoning process faster. The design of the convolution kernel also makes it highly sensitive to the local information of the image, which helps the model locate the target more accurately in the object detection task. However, its long-distance dependency modeling ability is weak, and the local receptive field limits its ability to capture long-distance dependencies, resulting in certain limitations in the model's understanding of image context information. And as the number of convolutional neural network layers increases, the model structure becomes more and more complex, which not only increases the difficulty of model training, but also may lead to problems such as overfitting.

[0004] The Transformer-based object detection model has a strong ability to model long-distance dependencies. It can effectively capture long-distance dependencies in the input sequence through the self-attention mechanism, which is particularly important in object detection tasks. It helps the model better understand the contextual information in the image and improve the accuracy of object detection. The multi-head attention mechanism of the Transformer model supports parallel computing, which enables the model to process large-scale data sets more efficiently during training and reasoning. However, the high computational cost of the Transformer architecture requires more computing resources during training and reasoning, especially when processing high-resolution images. In order to give full play to the performance of the Transformer model, a large amount of labeled data is usually required for training, which is extremely difficult in some application scenarios. At the same time, the Transformer model makes up for its shortcomings in capturing position information through methods such as position encoding, but compared with convolutional neural networks, it still has a certain gap in extracting local detail information.

[0005] DETR is the first model to apply the Transformer model to target detection. It uses an end-to-end target detection method to directly map image features to the bounding box and category of the target. The DETR model uses a self-attention mechanism to capture the relationship between targets and provides position information for each pixel position through a position encoder. The DETR model simplifies the target detection process and improves the efficiency of target detection. At the same time, its self-attention mechanism makes the model advantageous in global context modeling. However, the DETR model also has problems such as long training time and difficulty in accurately detecting small targets. In response to the above problems, LW-DETR proposes a method of feature fusion of the output results of the multi-layer encoder in the Transformer encoder to obtain a stronger feature map to improve the detection performance of small target objects; staggered windows and global attention are used to reduce the amount of model calculation. LW-DETR has higher accuracy at the same inference speed, but the accuracy of the LW-DETR model is poorer than that of the YOLO series at the same parameter amount. Summary of the invention

[0006] In view of the above problems existing in the prior art, the present invention provides a sparse attention image target detection method and system based on the improved LW-DETR algorithm with high computational efficiency, good detection performance and fast convergence. The present invention combines the encoder feature fusion method proposed by LW-DETR with the encoder part in Deformable DETR, improves the encoder part to reduce the amount of model calculation; and then combines the denoising strategy in DN-DETR with the encoder of LW-DETR to accelerate the convergence of the model.

[0007] The present invention adopts the following technical scheme:

[0008] An image target detection method based on an improved LW-DETR algorithm comprises the following steps:

[0009] S1. Collect image data sets, and mark the category and position of each object to be detected in the image; obtain target detection data sets, and divide the target detection data sets according to the proportion to obtain training sets, verification sets, and test sets;

[0010] S2. Build the LW-DETR target detection network model, use Deformable multi-head attention in the encoder of the LW-DETR target detection network model, and use the DN-DETR denoising strategy in the decoder;

[0011] S3, pre-training the improved LW-DETR target detection network model using the target detection dataset, and then using the image dataset collected in step S1 for training and adjustment;

[0012] S4. Use the model trained in step S3 to obtain the target detection box and classification prediction results corresponding to the test set or the image in the actual application in step S1, and draw the target detection box on the corresponding image and perform classification marking according to the target detection box and classification prediction results.

[0013] Preferably, in step S1, the Object365 target detection dataset is downloaded, and the image size in the image dataset is processed to be consistent with the Object365 target detection dataset; each image corresponds to an annotation file in XML format, in which each object to be detected in the image is annotated with its category and position.

[0014] Preferably, in step S2, the specific implementation steps of constructing the LW-DETR target detection network model are as follows:

[0015] S201, select a convolutional neural network as the feature extraction network Backbone, after the image passes through the Backbone, the deep convolution feature will be extracted, and the feature fusion of the N-layer output results in the Backbone is used as the input of the encoder;

[0016] S202, using the ViT model as the detection encoder, the original ViT includes a block layer and a Transformer encoding layer; the block layer is used to divide the input image into patches; the Transformer encoding layer in the original ViT includes a global self-attention layer and an FFN layer, using window self-attention, and aggregating multi-level feature maps, encoder intermediate layers and final feature maps to form an encoded feature map;

[0017] S203, use the projection layer Project to connect the encoder and the decoder, the projection layer uses the C2f module, and the projection layer uses the encoded feature map obtained in step S202 as input for feature fusion;

[0018] S204, the decoder uses a stacked Transformer decoding layer, where each layer contains a self-attention sublayer, a cross-attention sublayer, and a FFN sublayer;

[0019] S205, use Deformable multi-head attention;

[0020] S206. Introduce the denoising strategy proposed by DN-DETR into the LW-DETR decoder. During the training process, add noise to the input data, and then require the model to recover the original data from the disturbed data. Add a denoising module. The addition of the denoising module requires that the decoder input format be adjusted to two parts: category label embedding and learnable anchor frame. The category label embedding part contains the category of the object to be detected and a vector of identifiers used to distinguish between the denoising and matching parts, and the learnable anchor frame part contains the coordinates of the center point of the object to be detected (x, y) and the width and height (w, h) of the frame.

[0021] S207, the classification function calculation formula is as follows:

[0022]

[0023] Among them, BCE (Binary Cross-Entropy) represents the binary cross entropy loss function, N pos and N neg Represent positive and negative samples respectively, s i Represents the classification prediction score of the i-th image, t i The fraction representing the true value of the i-th image: t = s α u 1-α , s is the classification prediction value, u is the IoU calculation value, and α is a hyperparameter;

[0024] The overall loss function calculation formula is as follows:

[0025] L=L cls +λ iou L iou +λ1L1

[0026] Among them, L iou is the GIoU (Generalized Intersection over Union) loss function, L1 is the L1 loss function, and λ iou and λ1 are hyper parameters respectively.

[0027] Preferably, in step S3, the Object365 target detection data set is used to pre-train the improved LW-DETR target detection network model, and then the model is adjusted; the training set is used to train and adjust the model, and the validation set is used to evaluate the model performance to prevent overfitting, and finally the test set is used to perform final performance testing and evaluation on the improved LW-DETR target detection network model.

[0028] Preferably, an image target detection method based on an improved LW-DETR algorithm comprises the following steps:

[0029] Step 1: Collect image datasets. Each image corresponds to an XML annotation file, in which the category and location of each object to be detected in the image are annotated; download the Object365 target detection dataset to prepare for pre-training of the subsequent target detection model;

[0030] Step 2: Divide the data set obtained in step 1 according to the preset ratio to obtain the corresponding training set, validation set and test set respectively;

[0031] Step 3: Build an improved LW-DETR target detection network, replace the multi-head attention in the LW-DETR encoder with Deformable multi-head attention to reduce the amount of parameter calculation; combine the denoising task idea in DN-DETR to transform the encoder part of LW-DETR to accelerate convergence;

[0032] Step 4: Randomly select a preset single batch of images in the training set as the input of the target detection network. After the backbone network feature extraction, N effective feature maps of different scales are obtained. Then, the feature aggregation module further fuses the obtained effective feature maps to obtain a stronger feature map.

[0033] Step 5: Send the output in step 4 to the encoder of the target detection model. The feature fusion strategy of LW-DETR is applied in the encoder to concatenate the outputs of the 2nd and 4th layers with the output feature map of the last layer to obtain the encoder output.

[0034] Step 6: Send the output obtained in step 5 to the projection layer Project of the target detection model for feature fusion, and then obtain a stronger feature fusion map to improve the detection performance of small target objects;

[0035] Step 7: Send the feature map obtained in step 6 to the encoder of the target detection model to obtain the final detection result.

[0036] Step 8: Draw a target detection frame on the corresponding image and perform classification marking based on the above target detection frame prediction and classification prediction results.

[0037] The present invention also provides an image target detection system based on an improved LW-DETR algorithm, which is used to execute the above method and includes the following modules:

[0038] Dataset acquisition module: collect image datasets, annotate the category and position of each object to be detected in the image; obtain target detection datasets, and divide the target detection datasets according to the proportion to obtain training sets, validation sets, and test sets;

[0039] LW-DETR target detection network model construction module: The multi-head attention in the encoder of the LW-DETR target detection network model adopts Deformable multi-head attention, and the decoder applies the denoising strategy of DN-DETR;

[0040] Model training module: Use the target detection dataset to pre-train the improved LW-DETR target detection network model, and then use the collected image dataset for training and adjustment;

[0041] Target detection module: Use the trained model to obtain the target detection box and classification prediction results corresponding to the test set or the image in the actual application. Draw the target detection box on the corresponding image and mark the classification according to the target detection box and classification prediction results.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention effectively improves the accuracy and efficiency of target detection by optimizing the LW-DETR model structure and introducing the Deformable multi-head attention and DN-DETR denoising training strategies. At the same time, the present invention can also realize fast and accurate detection of targets in images, and has broad application prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of an image target detection method based on an improved LW-DETR algorithm in a preferred embodiment of the present invention;

[0045] Figure 2 It is a structural diagram of the LW-DETR model in the present invention;

[0046] Figure 3 It is a schematic diagram of the improved LW-DETR model encoder;

[0047] Figure 4 It is a schematic diagram of the decoder of the improved LW-DETR model;

[0048] Figure 5 It is a block diagram of an image target detection system based on an improved LW-DETR algorithm according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0050] like Figure 1-4 As shown, this embodiment provides an image target detection method based on an improved LW-DETR algorithm, comprising the following steps:

[0051] S1. Collect image datasets and label the categories and locations of each object to be detected in the image; download the Object365 target detection dataset to prepare for pre-training of subsequent target detection models;

[0052] S2. Build the LW-DETR target detection network model, replace the multi-head attention in the LW-DETR encoder with Deformable multi-head attention, and apply the DN-DETR denoising strategy in the decoder;

[0053] S3. Use the Object365 target detection dataset to pre-train the LW-DETR target detection network model, and use the training set to train and adjust the model to improve model performance; use the validation set to evaluate the model performance, and finally use the test set to perform performance testing and evaluation on the improved LW-DETR target detection network model;

[0054] S4. Use the model trained in step S3 to obtain the target detection box and classification prediction results corresponding to the test set or the image in the actual application in step S1, and draw the target detection box on the corresponding image and perform classification marking according to the target detection box and classification prediction results.

[0055] Specifically, in step S1, the processing of the data set includes collection and division. The size of the collected data set images must be consistent with the Object365 data set to better play the role of pre-training. Each image corresponds to an XML-formatted annotation file, in which the category and position of each object to be detected in the image are annotated. Then the obtained data set is divided according to the preset ratio to obtain the corresponding training set, verification set and test set, respectively, to ensure the effectiveness evaluation of the model at different stages.

[0056] In step S2, the target detection model is based on the improvement of LW-DETR algorithm, such as Figure 2 As shown in the figure, the specific implementation steps for building the LW-DETR target detection network model are as follows:

[0057] S201, select a suitable convolutional neural network as the feature extraction network Backbone (such as ResNet50, DenseNet, etc.), the image will be extracted after passing through the Backbone to obtain deep convolution features, and the output results of a total of N layers in the Backbone are feature fused as the input of the encoder;

[0058] S202. Use the ViT model as the detection encoder. The original ViT contains a block layer and a Transformer encoding layer. The block layer is used to divide the input image into block patches to adapt to the Transformer serialized input. The Transformer encoding layer in the original ViT contains a global self-attention layer and an FFN layer. However, due to the high computational cost of global self-attention, its time complexity is proportional to the square of the number of tokens (patches). It is necessary to reduce the computational complexity by using window self-attention, and aggregate the multi-level feature maps, the encoder intermediate layer and the final feature map to form a stronger encoding feature map.

[0059] S203, use the projection layer Project to connect the encoder and decoder. The projection layer uses the encoder aggregate feature map obtained in step S202 as input for feature fusion. The projection layer uses the C2f module to obtain richer gradient flow information while ensuring lightweight, playing a role in feature fusion;

[0060] S204, the decoder uses a stacked Transformer decoding layer. Each layer contains a self-attention sublayer, a cross-attention sublayer, and an FFN sublayer. In order to improve computational efficiency, the decoder uses deformable cross-attention. DETR and its variants usually use 6 decoding layers, but LW-DETR uses 3 Transformer decoding layers, which reduces the reasoning speed of forward inference;

[0061] The specific implementation steps for improving the LW-DETR model are as follows:

[0062] S205, such as Figure 3As shown, the multi-head attention in S202 is replaced with Deformable multi-head attention. Deformable multi-head attention is an attention mechanism proposed in Deformable DETR, which allows the model to focus only on a small part of the key area (sampling points) in the image when processing each query, rather than the entire feature map, thereby reducing the amount of calculation and improving efficiency. This mechanism refines the position of the sampling points by learning offsets, so that the model can locate the target area more accurately. The Deformable multi-head attention mechanism involves multiple parameters, such as the number of sampling points, the learning rate of the offset, etc. These parameters need to be adjusted according to the specific task and data set to achieve the best performance. The adjustment of parameters needs to be carried out through experiments, observing the convergence speed, detection accuracy and other indicators of the model under different parameters, so as to select the most appropriate parameter settings.

[0063] S206, such as Figure 4 As shown in the figure, the denoising training proposed by DN-DETR is introduced into the LW-DETR decoder. During the training process, noise (such as center point position offset noise, detection box size scaling offset noise, etc.) is added to the input data, and then the model is required to recover the original and clean data from the perturbed data to improve the convergence speed and performance of the model. On the basis of maintaining the original structure of the LW-DETR decoder, the improved model adds a denoising module, and the original decoder part is classified as a matching module. The denoising module adds noise in the attention layer of the decoder and learns to remove it, and the matching part is used to detect objects. The addition of the denoising module requires that the decoder input format be adjusted to two parts: class label embedding (Class Label Embedding) and learnable anchor boxes (learnable Anchors). The class label embedding part is a vector containing the category of the image object to be detected and an identifier used to distinguish between the denoising and matching parts, while the learnable anchor box part is a vector containing the coordinates of the center point of the object to be detected (x, y) and the width and height (w, h) of the box.

[0064] S207, the classification loss function calculation formula is as follows:

[0065]

[0066] Among them, N pos and N neg Represent positive and negative samples respectively, s i Represents the classification prediction score of the i-th image, t i The fraction representing the true value of the i-th image: t = s α u 1-α , s is the classification prediction value, u is the IoU calculation value, and α is a hyperparameter that needs to be adjusted according to actual training.

[0067] The overall loss function calculation formula is as follows:

[0068] L=L cls +λ iou L iou +λ1L1

[0069] Among them, the calculation uses GIoU, the calculation uses L1 loss, λ iou and λ1 are hyperparameters and need to be adjusted according to actual training.

[0070] In step S3, the training set images in the Object365 dataset are first combined with the images in the validation set (except the first 5,000 images) for detection pre-training to improve the model's feature extraction and detection performance. Then the dataset collected in step S1 is used to train the target detection model in order to achieve the best results. During the training process, the training set is used to optimize the model performance; the validation set is used to evaluate the model performance to prevent overfitting; the test set is used to evaluate the model's performance and calculate indicators such as precision, recall, F1 score, average precision (AP) and mean average precision (mAP).

[0071] Adjust the model architecture or training parameters based on the evaluation results and perform iterative optimization.

[0072] In step S4, the trained model is used for actual application, and the objects detected by the model and the classification predictions are marked according to the application requirements. The specific deployment process is as follows:

[0073] S401, Environment Preparation: Select appropriate hardware devices according to the complexity of the model and the requirements for inference speed. For target detection models, high computing performance is usually required, so high-performance GPUs or TPUs are ideal choices. It is also necessary to install a suitable operating system and necessary software frameworks, such as TensorFlow, PyTorch, etc.

[0074] S402, Model conversion: If the trained model is based on a specific framework or format, and the deployment platform requires a different format or framework, model conversion may be required. For example, convert a PyTorch model to ONNX format for deployment on more platforms. Use the corresponding conversion tools or scripts to convert the model, and ensure that the converted model is consistent with the original model in terms of accuracy and performance.

[0075] S403, model deployment: including deployment to servers and embedded devices. For deployment to servers, you need to upload the model files and related codes to the server, and configure the environment on the server, including installing necessary software and libraries. Write or adjust deployment scripts to load models, process input data, run predictions, and output results. Ensure that the server has sufficient resources to handle concurrent requests and real-time data processing. For deployment to embedded devices, model compression and quantization may be required to reduce model size and speed up inference. Use appropriate tools and techniques to deploy models to embedded devices, such as using frameworks such as TensorFlow Lite or PyTorch Mobile.

[0076] like Figure 5 As shown, this embodiment discloses an image target detection system based on improved LW-DETR, which is used to execute the above method embodiment and includes the following modules:

[0077] Dataset acquisition module: collect image datasets, annotate the category and position of each object to be detected in the image; obtain target detection datasets, and divide the target detection datasets according to the proportion to obtain training sets, validation sets, and test sets;

[0078] LW-DETR target detection network model construction module: The multi-head attention in the encoder of the LW-DETR target detection network model adopts Deformable multi-head attention, and the decoder applies the denoising strategy of DN-DETR;

[0079] Model training module: Use the target detection dataset to pre-train the improved LW-DETR target detection network model, and then use the collected image dataset for training and adjustment;

[0080] Target detection module: Use the trained model to obtain the target detection box and classification prediction results corresponding to the test set or the image in the actual application. Draw the target detection box on the corresponding image and mark the classification according to the target detection box and classification prediction results.

[0081] For other contents of this embodiment, please refer to the above method embodiment.

[0082] It should be understood that the specific embodiments described above are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described above can be combined with each other as long as they do not conflict with each other.

Claims

1. An image target detection method based on an improved LW-DETR algorithm, characterized in that: The following steps are involved: S1. Collect image data sets and label the category and position of each object to be detected in the image; Obtain the target detection data set, and divide the target detection data set according to the proportion to obtain the training set, verification set and test set; S2. Build the LW-DETR target detection network model, use Deformable multi-head attention in the encoder of the LW-DETR target detection network model, and use the DN-DETR denoising strategy in the decoder; S3, pre-training the improved LW-DETR target detection network model using the target detection dataset, and then using the image dataset collected in step S1 for training and adjustment; S4. Use the model trained in step S3 to obtain the target detection box and classification prediction results corresponding to the test set or the image in the actual application in step S1, and draw the target detection box on the corresponding image and perform classification marking according to the target detection box and classification prediction results.

2. The image target detection method according to claim 1, characterized in that: In step S1, the Object365 target detection dataset is downloaded, and the image size in the image dataset is processed to be consistent with the Object365 target detection dataset; each image corresponds to an annotation file in XML format, in which the category and position of each object to be detected in the image are annotated.

3. The image target detection method according to claim 1, characterized in that: In step S2, the specific implementation steps of building the LW-DETR target detection network model are as follows: S201, select a convolutional neural network as the feature extraction network Backbone, after the image passes through the Backbone, the deep convolution feature will be extracted, and the feature fusion of the N-layer output results in the Backbone is used as the input of the encoder; S202, using the ViT model as the detection encoder, the original ViT includes a block layer and a Transformer encoding layer; the block layer is used to divide the input image into patches; the Transformer encoding layer in the original ViT includes a global self-attention layer and an FFN layer, using window self-attention, and aggregating multi-level feature maps, encoder intermediate layers and final feature maps to form an encoded feature map; S203, use the projection layer Project to connect the encoder and the decoder, the projection layer uses the C2f module, and the projection layer uses the encoded feature map obtained in step S202 as input for feature fusion; S204, the decoder uses a stacked Transformer decoding layer, where each layer contains a self-attention sublayer, a cross-attention sublayer, and a FFN sublayer; S205, use Deformable multi-head attention; S206. Introduce the denoising strategy proposed by DN-DETR into the LW-DETR decoder. During the training process, add noise to the input data, and then require the model to recover the original data from the disturbed data. Add a denoising module. The addition of the denoising module requires that the decoder input format be adjusted to two parts: category label embedding and learnable anchor frame. The category label embedding part contains the category of the object to be detected and a vector of identifiers used to distinguish between the denoising and matching parts, and the learnable anchor frame part contains the coordinates of the center point of the object to be detected (x, y) and the width and height (w, h) of the frame. S207, the classification function calculation formula is as follows: Among them, BCE represents the binary cross entropy loss function, N pos and N neg Represent positive and negative samples respectively, s i Represents the classification prediction score of the i-th image, t i The fraction representing the true value of the i-th image: t = s α u 1-α , s is the classification prediction value, u is the IoU calculation value, and α is a hyperparameter; The overall loss function calculation formula is as follows: L=L cls +λ iou L iou +λ1L1 Among them, L iou is the GIoU loss function, L1 is the L1 loss function, λ iou and λ1 are hyper parameters respectively.

4. The image target detection method according to claim 1, characterized in that: In step S3, the training set images and validation set images in the Object365 target detection dataset are first merged for detection pre-training, then the training set is used to train and adjust the model, the validation set is used to evaluate the model performance, and finally the test set is used to perform performance testing and evaluation on the improved LW-DETR target detection network model.

5. An image target detection system based on an improved LW-DETR algorithm, used to execute the method according to any one of claims 1 to 4, characterized in that: Contains the following modules: Dataset acquisition module: collect image datasets, annotate the category and position of each object to be detected in the image; obtain target detection datasets, and divide the target detection datasets according to the proportion to obtain training sets, validation sets, and test sets; LW-DETR target detection network model construction module: The multi-head attention in the encoder of the LW-DETR target detection network model adopts Deformable multi-head attention, and the decoder applies the denoising strategy of DN-DETR; Model training module: Use the target detection dataset to pre-train the improved LW-DETR target detection network model, and then use the collected image dataset for training and adjustment; Target detection module: Use the trained model to obtain the target detection box and classification prediction results corresponding to the test set or the image in the actual application. Draw the target detection box on the corresponding image and mark the classification according to the target detection box and classification prediction results.

Images