YOLOv7-based few-shot small-scale training and file deployment method

Abstract

This invention relates to a method for few-shot, small-scale training and file deployment based on YOLOv7, comprising: collecting small-scale, small-sample data of the object to be detected; processing the small-sample data to obtain training data; constructing a small-scale object detection model based on YOLOv7; improving the small-scale object detection model; training the improved small-scale object detection model using the training data to obtain a final model; storing the model file of the final model in a designated folder, compiling the model to generate an executable file, completing file deployment, and realizing object detection. This invention summarizes the deployment steps of YOLOv7 on Jetson Nano, enabling rapid deployment and use of the model on this device.

Description

Technical Field

[0001] This invention relates to the field of computer vision target detection technology, and in particular to a method for few-sample, small-scale training and file deployment based on YOLOv7. Background Technology

[0002] Object detection, as a crucial component of computer vision, has received extensive research attention over the past decade. The goal of object detection is to find a specified object with accurate localization information in a given image or video and assign a corresponding label to each identified object. In recent years, with the rapid development of deep neural networks, object detection has made significant progress in accuracy and speed, finding wide applications in robot navigation, aerospace, and industrial inspection. However, low-pixel, small-scale object detection still faces considerable difficulties and challenges. Due to the limited number of pixels, it is difficult to extract effective information from object detection models, often leading to missed detections and false positives. Meanwhile, research on small-scale object detection has significant value in many application scenarios. For example, in autonomous driving, if a car can accurately predict traffic lights, pedestrians, and other vehicles at a considerable distance, it can allow sufficient time for action response, greatly increasing the reliability and safety of autonomous driving. Therefore, small-scale object detection is currently a very important and urgent research area in the field of object detection.

[0003] The goal of small-scale object detection is to improve the ability to detect low-pixel or relatively small-scale objects in images or videos. Multi-scale feature fusion is a primary method to improve the accuracy of models for small-scale object detection. For example, images can be scaled up to multiple scales, and dedicated detectors can be trained for each scale. Feature pyramid networks can also be used to perform small-scale object detection by leveraging multi-scale feature fusion within convolutional layers. Additionally, using shorter connections to improve the information flow in the pyramid network and introducing simple attention mechanisms can also address the problem of small-scale object detection. Current small-scale object detection methods still have room for improvement in accuracy; therefore, it is necessary to develop higher-performance models for training to obtain more accurate small-scale object detection results. Summary of the Invention

[0004] The purpose of this invention is to provide a method for training and deploying YOLOv7 with few samples and small scales. This method utilizes the SPD-Conv module and the FRM module to enhance the detection capability of the YOLOv7 model for small targets. First, feature extraction is performed by introducing the backbone network of the SPD-Conv module. Then, feature maps of different sizes are obtained through multi-scale fusion. Each feature map is then input into the FRM to obtain a prediction head for detection. After pre-training and retraining on data of specified small-scale objects, the final target detection model used for inference is obtained. Finally, the model is deployed on Jetson Nano to achieve real-time detection of small-scale targets.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] Methods for few-shot, small-scale training and file deployment based on YOLOv7 include:

[0007] Collect small-scale, small-sample data of the items to be detected, process the small-sample data, and obtain training data;

[0008] A small-scale object detection model is constructed based on YOLOv7, the small-scale object detection model is improved, and the improved small-scale object detection model is trained using the training data to obtain the final model;

[0009] The final model file is stored in a designated folder, and the model is compiled to generate an executable file. The file is then deployed to achieve target detection.

[0010] Optionally, processing the small sample data to obtain training data includes:

[0011] Mosaic data augmentation is performed on the small sample data. Several photos are selected, and the photos are randomly scaled, cropped, and arranged to obtain the training data.

[0012] Optionally, improving the small-scale target detection model includes:

[0013] An SPD-Conv module and a feature refinement network (FRM) module are added to the network of the small-scale target detection model. The SPD-Conv module is used to replace the convolutional module with a stride of 2 in YOLOv7, and achieves downsampling of the feature map without using convolution stride or pooling. The FRM module is added between the YOLOv7 feature pyramid and the prediction head to suppress the conflict in the multi-scale feature fusion of the YOLOv7 feature pyramid and prevent small targets from being submerged in conflicting information.

[0014] Optionally, downsampling the feature map includes:

[0015] The SPD-Conv module includes: an SPD layer and a non-strided convolution layer;

[0016] The SPD layer performs downsampling on the internal feature map of the small-scale target detection model based on the original image conversion technique, and transforms the feature map x into the feature map X′. Based on the non-strided convolution layer with added C2 filter, the feature map X′ is judged, and based on the judgment result, the feature map X′ is transformed into the feature map X″.

[0017] Optionally, the FRM module added between the YOLOv7 feature pyramid and the prediction head includes:

[0018] The feature refinement network includes: a channel purification module and a space purification module;

[0019] The output of the channel purification module is:

[0020]

[0021] in, Let represent the output vector of the m-th layer at position (x, y), where a, b, c, and d are channel adaptive weights;

[0022] The output of the space purification module is:

[0023]

[0024] Where x and y are the spatial locations of the feature map, and k is the number of channels in the input feature map. Let (x, y) be the output feature vector at position (x, y). and Here, c represents the spatial attention weights relative to the m-th layer, and c is the number of channels. It is represented as the value at position (x, y) of the m-th feature map in the k-th channel.

[0025] Optionally, training the improved small-scale target detection model includes:

[0026] Obtain the COCO dataset, pre-train the improved small-scale target detection model based on the COCO dataset, obtain the pre-trained model, and train the pre-trained model using the pre-trained model;

[0027] In both pre-training and training, the coordinate loss uses the CIoU loss function, while the target confidence loss and classification loss both use the BCEWithLoigitsLoss loss function.

[0028] Optionally, obtaining the final model includes: transforming the trained pre-trained model into the final model through structural reparameterization.

[0029] Optionally, storing the model file of the final model in a designated folder includes:

[0030] The specified folder is configured with TensorRT, CUDA, CUDNN, OpenCV, and Protobuf software environments. The model files of the final model will be exported via ONNX and stored in the configured specified folder.

[0031] Optionally, obtaining the final model includes:

[0032] Obtain evaluation metrics, and evaluate the final model based on the evaluation metrics;

[0033] The method for obtaining the evaluation indicators is as follows:

[0034]

[0035] Where AP is the evaluation metric, N is the number of images, P(k) is the Precision value when k images can be identified, and Δr(k) is the change in Recall value when the number of identified images changes from k-1 to k.

[0036] The beneficial effects of this invention are as follows:

[0037] The method of this invention can effectively solve the problem of fine-grained information loss during training by introducing SPD-Conv and FRM modules, thereby improving the detection accuracy of small-scale targets.

[0038] The method of this invention summarizes and generalizes the deployment steps of YOLOv7 on Jetson Nano, enabling the model to be quickly deployed and used on this device. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a method for few-shot, small-scale training and file deployment based on YOLOv7 according to an embodiment of the present invention.

[0041] Figure 2This is a schematic diagram of the FRM module in a YOLOv7-based few-shot small-scale training model according to an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the SPDConv module in a YOLOv7-based few-shot small-scale training model according to an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of a small-scale training model based on YOLOv7 according to an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] like Figure 1 As shown, this invention discloses a method for few-shot, small-scale training and file deployment based on YOLOv7, including:

[0047] Small-scale sample data of the objects to be detected is collected, processed, and used to obtain training data; a small-scale object detection model is built based on YOLOv7, such as... Figure 4 The improved small-scale target detection model is trained using training data to obtain the final model. The model file of the final model is stored in a specified folder, and the model is compiled to generate an executable file. The file is then deployed to achieve target detection.

[0048] Processing small sample data to obtain training data includes: performing Mosaic data augmentation on small sample data, selecting several photos, randomly scaling, cropping and arranging the photos to obtain training data.

[0049] Improvements to the small-scale object detection model include adding an SPD-Conv module and a feature refinement network (FRM) module to the network. The SPD-Conv module replaces the convolutional module with a stride of 2 in YOLOv7, achieving feature map downsampling without using convolution stride or pooling. The FRM module is added between the YOLOv7 feature pyramid and the prediction head to suppress conflicts in the multi-scale feature fusion of the YOLOv7 feature pyramid, preventing small targets from being submerged in conflicting information.

[0050] Feature map downsampling includes: Figure 3 As shown, the SPD-Conv module includes an SPD layer and a non-strided convolution layer. The SPD layer performs downsampling operations on the internal feature maps of the small-scale object detection model based on the original image transformation technique, and transforms the feature map X(S, S, C1) into the feature map X′(S / scale, S / scale, scale). 2 C1), and scale based on a non-strided convolution layer with added C2 filter. 2 C1 is used to determine if C2 < scale. 2 When C1, the feature map X′(S / scale, S / scale, scale) is... 2 C1) is transformed into feature map X″(S / scale, S / scale, C2).

[0051] The FRM module, added between the YOLOv7 feature pyramid and prediction head, includes: a feature refinement network comprising: a channel cleansing module and a spatial cleansing module;

[0052] Training the improved small-scale object detection model includes: acquiring the COCO dataset, pre-training the improved small-scale object detection model based on the COCO dataset, acquiring the pre-trained model, and training the pre-trained model using the pre-trained model. In both the pre-training and training processes, the coordinate loss uses the CIoU loss function, and the target confidence loss and classification loss both use the BCEWithLoigitsLoss loss function.

[0053] Obtaining the final model involves transforming the pre-trained model into the final model through structural reparameterization.

[0054] The final model file is stored in the specified folder. This includes configuring the TensorRT, CUDA, CUDNN, OpenCV and Protobuf software environment in the specified folder, exporting the final model file through ONNX, and storing the model file in the configured specified folder.

[0055] In object detection, there are two ways to define small scale: the first is based on relative scale, where the square root of the ratio of the object's bounding box area to the image area is less than a certain value (a commonly used value is 0.03). The second is based on absolute scale, where objects with an absolute size less than 32×32 are considered small objects. This definition is based on object classification in the COCO dataset, and this invention adopts the second definition.

[0056] The COCO dataset is a public dataset for object detection, provided by the Microsoft team. The COCO dataset (which includes training, validation, and test sets) is available via the internet.

[0057] like Figure 1 As shown, this invention discloses a method for few-shot, small-scale training and file deployment based on YOLOv7, including:

[0058] Step 1: Perform data augmentation on the dataset of small-scale objects to be detected, such as keys, medicine bottles, and mobile phones in indoor camera video data;

[0059] Mosaic data augmentation is used to augment small-scale, small-sample data of the objects to be detected. Mosaic data augmentation can select four images and randomly scale, crop, and arrange them to obtain new training data.

[0060] Step 2: Train the model using few samples and small scales based on YOLOv7, such as... Figure 3 ;

[0061] Design a YOLOv7-based model capable of training on small datasets and at small scales. This model builds upon YOLOv7 by introducing the SPD-Conv module into the backbone and reconstructing the neck network structure. It also introduces a Feature Refinement Network (FRM), the structure of which is shown below. Figure 2 As shown, four detection heads are generated for inference after fusing features from four scales.

[0062] The SPD-Conv module consists of an SPD layer and a non-strided conversion layer. The SPD layer extends the original image transformation technique to downsampling operations on feature maps both within and throughout the CNN. Figure 2 For any intermediate feature map X of size S*S*C1, the series of sub-feature maps cut out are as follows:

[0063]

[0064] The sliced ​​sub-feature maps are concatenated along the channel dimension to obtain feature map X′. The SPD layer transforms feature map X(S, S, C1) into feature map X′(S / scale, S / scale, scale). 2 C1). After the SPD layer operation, add a non-strided convolution layer with a C2 filter, where C2 < scale. 2 C1 further transforms X′(S / scale, S / scale, scale) 2 C1) is X″(S / scale, S / scale, C2).

[0065] The Feature Refinement Network (FRM) consists of two parallel branches: a channel cleanup module and a spatial cleanup module. The channel cleanup module is as follows: Figure 1 As shown, X m Defined as the input of the m-th layer (m = {1, 2, 3}) of the FRM. x(n, m) is defined as the result of adjusting the size of the n-th layer to the m-th layer. This represents the value at position (x, y) of the m-th feature map in the k-th channel. Therefore, the output of the upper branch channel purification module is:

[0066]

[0067] in Let represent the output vector of the m-th layer at position (x, y). a, b, c, and d are channel adaptive weights, defined as:

[0068] [a m b m c m d m ]=σ[AP(F)+MP(F)] (3)

[0069] F represents the feature generated by the Concat operation, such as... Figure 1 As shown. σ represents the sigmoid operation. AP and MP are the average pooling and max pooling operations, respectively. The two weights are then added spatially, and channel-based adaptive weights are generated after the sigmoid operation. The spatial cleansing module generates the relative weights for all positions of the channel using softmax, so the output of the next branch spatial cleansing module is:

[0070]

[0071] In equation (4), x and y represent the spatial location of the feature map, and k represents the channel of the input feature map. This represents the output feature vector at position (x, y). and Let μ represent the spatial attention weights relative to the m-th layer, where c is the number of channels. μ, ν, η, and ξ are derived from equation (5):

[0072] [μ m v mm η m ξ m ]=Sofimax(F) (5)

[0073] Where F has the same meaning as in equation (3), it represents the feature generated by the Concat operation. Softmax is used to normalize the feature map along the channel direction to obtain the relative weights of the same position on different channels. The output of the entire module is:

[0074]

[0075] Step 3: Train the model from Step 2 on the COCO training set to obtain a pre-trained model. During training, the coordinate loss uses the CIoU loss function, and the target confidence loss and classification loss use the BCEWithLoigitsLoss loss function.

[0076] Step 4: Use the new training data obtained in Step 1 to continue training the pre-trained model in Step 3 to obtain a model capable of detecting specified small-scale objects. The loss function used in this training step is the same as in Step 3.

[0077] Step 5: Through structural reparameterization, the model obtained in Step 4 is transformed into the final model for inference. During the inference phase, the 3×3 convolutional blocks at specified locations in the model are replaced with RepConvN without identity connections to increase the inference speed of the model.

[0078] Step 6: Configure the TensorRT, CUDA, CUDNN, OpenCV, and Protobuf software environments on Jetson Nano.

[0079] Step 7: Export the final model trained in step (5) using ONNX and store it in the specified folder of jetsonnano.

[0080] Step 8: Compile the model to generate a .pro executable file to complete the deployment. Executing this file will build and infer the model, enabling the detection of the target.

[0081] To test the performance of step 5 for small-scale target detection, this invention uses Average Precision (AP) as the evaluation metric, where:

[0082] AP is defined as the mean of the Precision value under different Recalls, and the calculation formula is as follows:

[0083]

[0084] Where N represents the number of images, P(k) represents the Precision value when k images can be recognized, and Δr(k) represents the change in Recall value as the number of recognized images changes from k-1 to k. The formula for calculating Precision is as follows:

[0085]

[0086] Where TP represents the number of correctly detected targets, and FP represents the number of targets incorrectly identified as targets.

[0087] The formula for calculating recall is as follows:

[0088]

[0089] Where TP represents the number of correctly detected targets, and FN represents the number of targets that were not correctly identified.

[0090] AP can quantitatively reflect the performance of a classifier and is a performance metric for measuring the quality of a model. The larger the AP, the better the model's object detection performance.

[0091] As an improvement to YOLOv7 in small-scale object detection, we introduced SPD-Conv and FRM modules into the original model. SPD-Conv can solve the problems of fine-grained information loss and low feature learning efficiency caused by cross-stretch convolution or pooling layers in existing CNNs. The FRM module fully fuses multi-scale features from both channel and spatial perspectives, while avoiding the conflict problems generated in the traditional multi-scale fusion process.

[0092] Finally, the improved model was trained and deployed in Jetson Nano to achieve effective target detection.

[0093] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for few-shot small-scale training and file deployment based on YOLOv7, characterized in that, include: Collect small-scale, small-sample data of the items to be detected, process the small-sample data, and obtain training data; A small-scale object detection model is constructed based on YOLOv7, the small-scale object detection model is improved, and the improved small-scale object detection model is trained using the training data to obtain the final model; The improvements to the small-scale target detection model include: An SPD-Conv module and a feature refinement network (FRM) module are added to the network of the small-scale target detection model. The SPD-Conv module is used to replace the convolutional module with a stride of 2 in YOLOv7, and achieves downsampling of the feature map without using convolution stride or pooling. The FRM module is added between the YOLOv7 feature pyramid and the prediction head to suppress the conflict in the multi-scale feature fusion of the YOLOv7 feature pyramid and prevent small targets from being submerged in conflicting information. Implementing downsampling of the feature map includes: The SPD-Conv module includes: an SPD layer and a non-strided convolution layer; The SPD layer performs downsampling on the internal feature map of the small-scale target detection model based on the original image conversion technology, and transforms the feature map x into the feature map X′. The non-strided convolution layer with added C2 filter judges the feature map X′, and transforms the feature map X′ into the feature map X″ based on the judgment result. The FRM module added between the YOLOv7 feature pyramid and the prediction head includes: The feature refinement network includes: a channel purification module and a space purification module; The output of the channel purification module is: wherein, represents the output vector of the mth layer at the (x, y) position, a, b, c, d are channel adaptive weights; The output of the space purification module is: Where x and y are the spatial locations of the feature map, and k is the number of channels in the input feature map. Let (x, y) be the output feature vector at the position (x, y). and Here, c represents the spatial attention weights relative to the m-th layer, and c is the number of channels. This is represented as the value at position (x, y) of the m-th feature map in the k-th channel; The final model file is stored in a designated folder, and the model is compiled to generate an executable file. The file is then deployed to achieve target detection.

2. The YOLOv7-based few-shot small-scale training and file deployment method according to claim 1, characterized in that, Processing the small sample data to obtain training data includes: Mosaic data augmentation is performed on the small sample data. Several photos are selected, and the photos are randomly scaled, cropped, and arranged to obtain the training data.

3. The YOLOv7-based few-shot small-scale training and file deployment method according to claim 1, characterized in that, Training the improved small-scale target detection model includes: Obtain the COCO dataset, pre-train the improved small-scale target detection model based on the COCO dataset, obtain the pre-trained model, and train the pre-trained model using the pre-trained model; In both pre-training and training, the coordinate loss uses the CIoU loss function, while the target confidence loss and classification loss both use the BCEWithLoigitsLoss loss function.

4. The YOLOv7-based few-shot small-scale training and file deployment method according to claim 3, characterized in that, Obtaining the final model includes: transforming the trained pre-trained model into the final model through structural reparameterization.

5. The YOLOv7-based few-shot small-scale training and file deployment method according to claim 1, characterized in that, The model files of the final model are stored in the specified folder, including: The specified folder is configured with TensorRT, CUDA, CUDNN, OpenCV, and Protobuf software environments. The model files of the final model will be exported via ONNX and stored in the configured specified folder.

6. The YOLOv7-based few-shot small-scale training and file deployment method according to claim 1, characterized in that, After obtaining the final model, the following steps are included: Obtain evaluation metrics, and evaluate the final model based on the evaluation metrics; The method for obtaining the evaluation indicators is as follows: Where AP is the evaluation metric, N is the number of images, P(k) is the Precision value when k images can be identified, and Δr(k) is the change in Recall value when the number of identified images changes from k-1 to k.

Images