Weapon equipment identification method based on YOLOv7

Abstract

The invention discloses a weapon equipment identification method based on YOLOv7, and the method comprises the following steps: collecting corresponding weapon equipment images according to types, and selecting clear images with obvious features; marking the selected image, and converting the image into a YOLO format; dividing the pictures into a training set and a verification set for training to obtain an optimal model; during detection, a YOLOv7 backbone network and ViT are used to extract features of an input picture, the features obtained by the two are fused and filtered by an FSM module, then a feature map is processed, and a target category and coordinates are regressed. The method supports 11 types of artillery, large water surface warships, fixed-wing aircrafts, radars, wheeled armored cars, tanks and the like. Compared with the prior art, the detection capability is greatly enhanced, more specific weapon equipment information can be obtained, and the method has very important effect and significance in a specific scene.

Description

Technical Field

[0001] The present invention relates to the field of computer vision target detection, and in particular to a weapon equipment recognition method based on YOLOv7. Background Art

[0002] Weapon and equipment recognition is a method that uses algorithms to automatically locate and identify weapons and equipment in images. It plays an important role in the victory of future information warfare. It can help search for battlefield targets and distinguish target categories, help obtain more battlefield information, and provide guidance for various strike methods. For unmanned equipment, its recognition capability is the key to unmanned operation, reconnaissance and early warning, and autonomous attack.

[0003] In recent years, with the advancement of machine learning and deep learning technologies and their wide application in various fields, some methods have emerged to use deep learning methods to identify tanks, armored vehicles, ships and other equipment, and have achieved good results. However, due to the lack of richness of data sets and insufficient model capabilities, these methods have a common problem of single detection type, that is, the model for identifying tanks can only identify tanks, and cannot simultaneously identify tanks, aircraft, ships and other targets, and cannot identify specific models for a single type.

[0004] In view of the above shortcomings, the present invention provides a weapon equipment recognition method based on YOLOv7, which integrates the features extracted by ViT and the deep features of YOLOv7, enriches the feature information, and strengthens the model detection capability. At the same time, a weapon equipment data set with rich categories is produced to train the model, so that the model has the recognition capabilities of 11 major categories, including artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, satellites, tanks, and armed rotorcraft, as well as the recognition capabilities of important models under some equipment, which greatly enhances the recognition range and fine-grained recognition capabilities of the weapon equipment recognition algorithm, is conducive to building a more concise and efficient recognition system, and provides important technical support for modern warfare. Summary of the invention

[0005] Aiming at the shortcomings of the current weapon and equipment recognition algorithm, the present invention proposes a weapon and equipment recognition method based on YOLOv7, uses ViT to enrich the model features, constructs an FSM (Feature Select Module) module to select appropriate fused features to improve the detection effect, and expands the recognition range and recognition ability of weapon and equipment. It can recognize certain specific models of equipment in 11 major categories, including artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, artificial satellites, tanks and armed rotorcraft.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme: a weapon equipment identification method based on YOLOv7, comprising the following steps:

[0007] S1. Collect image data of various types of weapons and equipment, clean and annotate the image data, and convert them into YOLO format;

[0008] S2. Build a YOLOv7-VIT weapon and equipment recognition model. The input image is extracted through the YOLOv7 skeleton network and ViT respectively, and then deep feature fusion is performed. After filtering through the FSM module, the target category and position are regressed.

[0009] S3. Perform model training: Calculate the loss function based on the label and loss function, and use back propagation to update the model parameters according to the gradient and learning rate until the maximum number of iterations is reached;

[0010] S4, adjust the color of the output detection box, the language and font of the label;

[0011] S5. Input the image to be identified into the adjusted model to realize weapon and equipment identification.

[0012] Furthermore, the collected various types of weapon equipment image data include specific models of weapon equipment image data under eleven categories, namely, artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, artificial satellites, tanks and armed rotorcraft.

[0013] Furthermore, the collecting of various types of weapon equipment image data and the labeling of the image data after cleaning specifically include:

[0014] Use crawlers to obtain corresponding images on the Internet according to each major category of weapons, perform manual cleaning, and keep the images corresponding to the models that need to be detected, while removing unclear or difficult-to-recognize images;

[0015] Use the open source annotation tool Labelimg to annotate the cleaned image, manually frame the target location and annotate the target category name, generate an annotation file in PASCAL VOC format and convert it to YOLO format.

[0016] Furthermore, the YOLOv7-VIT weapon equipment recognition model includes YOLOv7 and VIT, and the image is respectively subjected to feature extraction by the YOLOv7 backbone network and ViT, and the last three layers of features of YOLOv7 and the last layer of features of ViT are taken out;

[0017] The sequence features output by ViT are reshaped into a matrix form corresponding to the image, and after upsampling, they are concatenated with the last layer features of YOLOv7 to enrich their semantic information. The features of each layer of YOLOv7 are input into the subsequent network structure of YOLOv7 to regress the target category and position.

[0018] Furthermore, the FSM module takes out each channel of the fused features, calculates the corresponding loss, and then selects the k channels with the lowest loss, where k is a hyperparameter. During training, after the selected k channels are determined by the first ten images, the subsequent input is no longer recalculated, and the corresponding channels are directly selected.

[0019] Furthermore, the loss function is obtained by comparing the regressed target category and position with the label category and position, and by the difference between the category, confidence, and position keys. Specifically:

[0020]

[0021] Among them, S 2 is the number of grids that the image is segmented into during the YOLOv7 target detection process, B is the number of predicted boxes for each grid detection, and λ noobj is the weight parameter for balancing the boxes without objects, λ coord is the weight parameter of the coordinate loss. The subscript i indicates that each grid cell is iterated, and the subscript j indicates that each prediction box bbox is iterated. To determine whether the jth prediction box in the i-th grid is responsible for the prediction of the object, and On the contrary; x i ,y i , w i , j i , C i , p i (c) represents the center coordinates, width, height, confidence, and category of the prediction box. Represents the center coordinates, width, height, confidence, and category of the ground truth box, and the rest are weighting coefficients.

[0022] On the other hand, this specification also provides a weapon and equipment identification device based on YOLOv7, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, a weapon and equipment identification method based on YOLOv7 as described is implemented.

[0023] On the other hand, this specification also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for identifying weapons and equipment based on YOLOv7 is implemented.

[0024] Beneficial effects of the present invention:

[0025] (1) The present invention realizes cross-category weapon equipment detection by constructing a large weapon equipment training set, and uses one model to detect targets in the three major battlefields of sea, land and air, including artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, artificial satellites, tanks and armed rotorcraft;

[0026] (2) The present invention supports the detection of specific weapon equipment models under each major category, mainly some important models, such as PHL-03 rocket launcher, Nimitz-class aircraft carrier, B-2 strategic bomber, etc.;

[0027] (3) The present invention uses a fusion of deep features extracted by ViT and YOLOv7, and uses the FSM module for screening, which reduces the number of channels, retains more effective features, and enhances the detection capability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a weapon and equipment identification method based on YOLOv7 of the present invention;

[0029] Figure 2 It is a network framework diagram of the weapon equipment identification method based on YOLOv7 of the present invention;

[0030] Figure 3 It is a CBS module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0031] Figure 4 It is the MP1 module structure diagram of the weapon equipment identification method based on YOLOv7 of the present invention;

[0032] Figure 5 It is the MP2 module structure diagram of the weapon equipment identification method based on YOLOv7 of the present invention;

[0033] Figure 6 It is the up module structure diagram of the weapon equipment identification method based on YOLOv7 of the present invention;

[0034] Figure 7 It is the ELAN module structure diagram of the weapon equipment identification method based on YOLOv7 of the present invention;

[0035] Figure 8 It is the ELAN-H module structure diagram of the weapon equipment identification method based on YOLOv7 of the present invention;

[0036] Fig. 9 It is the SPPCSPC module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0037] Fig.10 It is a VIT module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0038] Fig.11 It is a FSM module structure diagram of the weapon equipment identification method based on YOLOv7 of the present invention;

[0039] Fig.12 It is a schematic diagram of a weapon and equipment identification device based on YOLOv7 of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and effective improvement points of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] like Figure 1 As shown, the steps of the present invention include:

[0042] Step S1, collecting image data of specific types of weapons and equipment under eleven categories, including artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, artificial satellites, tanks, and armed rotorcraft, cleaning the image data, filtering out invalid images, and then annotating the filtered image data;

[0043] Step S2, converting the labeled VOC format label file into YOLO format, and then dividing the labeled weapons and equipment data set into training set, verification set, and test set;

[0044] Step S3, set the model training parameters for training. During training, the input image is extracted by YOLOv7 and ViT respectively, and then the features extracted by ViT are fused with the deep features extracted by YOLOv7 to obtain features with richer semantic information, which are filtered by the FSM (Feature Select Module) module. This module will take out each channel of the fused features and calculate its corresponding loss, and then select the k channels with the lowest loss. K is a manually set hyperparameter. After the selected k channels are determined by the first ten images during training, the subsequent input is no longer recalculated, and the corresponding channels are directly selected to retain more effective and streamlined features;

[0045] The overall structure of the model is as follows Figure 2 As shown, in YOLOv7, the specific structure of the CBS module is as follows Figure 3 As shown, the specific structure of MP1 module is as follows Figure 4 As shown in the MP2 module structure diagram, the specific structure is as follows Figure 5 As shown, the specific structure of the up module is as follows Figure 6 As shown, the specific structure of the ELAN module is as follows Figure 7 As shown, the specific structure of the ELAN-H module is as follows Figure 8 As shown, the specific structure of the SPPCSPC module is as follows Fig. 9 As shown in Fig.10 During training, the input image is extracted by YOLOv7 and ViT respectively, and then the features extracted by ViT are fused with the deep features extracted by YOLOv7 to obtain features with richer semantic information.

[0046] Step S4, using the trained model, setting detection parameters, using the test set to evaluate the model performance, and adjusting the model output to have a better visualization effect.

[0047] Furthermore, the specific process of step S1 is as follows:

[0048] Step S11, using a crawler to obtain corresponding images on the Internet according to major categories such as tanks and artillery, and then manually cleaning them, leaving only a few pictures corresponding to the models that need to be detected, while eliminating unclear or difficult to recognize pictures;

[0049] Step S12: Use the open source annotation tool Labelimg to annotate the cleaned image, manually frame the target position and annotate the target category name, and generate an annotation file in PASCAL VOC format.

[0050] Furthermore, step S2 converts the annotation file format and divides the data set. The specific process is as follows:

[0051] Step S21, use a python script to convert the PASCAL VOC format annotation file into a YOLO format txt file required for YOLOv7 training. Each picture corresponds to a txt file. Each line in the txt file corresponds to an object in the picture. The first number represents the category, and the following four numbers are the normalized coordinates of the center point and the width and height of the target box.

[0052] Step S22, using a python script to randomly select 80% of the images according to categories as a training set, 10% of the images as a validation set, and the remaining 10% of the images as a test set.

[0053] Furthermore, step S3 trains the model, and the specific process is as follows:

[0054] Step S31, setting model training parameters, including the number of target categories, learning rate, image enhancement parameters, data set corresponding path, etc.;

[0055] Step S32, loading the initial model and training data, preprocessing the training data, and then sending it to the network for forward propagation. The image passes through the YOLOv7 backbone network and ViT to extract features, and the last three layers of features of YOLOv7 and the last layer of features of ViT are taken out;

[0056] Step S33, reshape the sequence features output by ViT into a matrix form corresponding to the image, and concatenate them with the last layer features of YOLOv7 after upsampling to enrich their semantic information;

[0057] Step S34, input the features obtained by fusion of ViT and the last layer of YOLOv7 into the FSM module for screening to retain more effective features;

[0058] Step S35, after fusing the above features with the features of other layers of YOLOv7, regress the target category and position, compare them with the category and position of the label, calculate the loss function according to the difference between the category, confidence and position, as shown in formula (1), use the back propagation algorithm to update the network parameters according to the gradient and learning rate, and then repeat this process until the specified training generation is completed;

[0059]

[0060] Among them, S 2 is the number of grids that divide the image into during YOLOv7 target detection, B is the number of bounding boxes detected for each grid, and λ noobj is the weight parameter for balancing the boxes without objects, λ coord is the weight parameter of the coordinate loss. The subscript i indicates that each grid cell is iterated, and the subscript j indicates that each prediction box bbox is iterated. To determine whether the jth bbox in the i-th grid is responsible for the prediction of this obj, and on the contrary

[0061] x i ,y i , w i ,h i , C i , p i (c) represents the center coordinates, width, height, confidence, and category of the prediction box. Represents the center coordinates, width, height, confidence, and category of the ground truth box, and the rest are weighting coefficients.

[0062] Furthermore, step S4 tests the model, and the specific process is as follows:

[0063] Step S41, setting model test parameters, including batch size, model path to be loaded, test set path, IOU threshold, etc.;

[0064] Step S42, load the test set data and send it to the network for forward propagation. Similar to the training, feature extraction is performed through the YOLOv7 backbone network and ViT, and the final features are obtained through feature fusion and screening. Then, the target category and position are regressed, and invalid targets are screened out through non-maximum suppression. Then, the IOU is calculated according to the label, and compared with the set threshold to determine whether the detection is correct, and indicators such as mAP are calculated to determine whether the model performance meets the requirements;

[0065] Step S43, adjusting the color of the output detection frame, the language and font of the label, etc. as needed.

[0066] Specifically, the precision, recall rate, F1 index and average precision mAP of each category are as follows: Formula (2), Formula (3), Formula (4), Formula (5).

[0067] (2)Precision:

[0068]

[0069] (2) Recall:

[0070]

[0071] (4) F1 index:

[0072]

[0073] (5) Average Average Precision (mAP):

[0074]

[0075] True Positive (TP): a sample marked as positive and predicted as positive; False Positive (FP): a sample marked as negative and predicted as positive; True Negative (TN): a sample marked as negative and predicted as negative; False Negative (FN): a sample marked as positive and predicted as negative.

[0076] Step S43, adjusting the color of the output detection frame, the language and font of the label, etc. as needed.

[0077] Step S5, input the image to be identified into the adjusted model to perform weapon and equipment identification.

[0078] Corresponding to the aforementioned embodiment of a weapon and equipment identification method based on YOLOV7, the present invention also provides an embodiment of a weapon and equipment identification device based on YOLOV7.

[0079] See also Fig.12 A weapon and equipment identification device based on YOLOV7 provided in an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a weapon and equipment identification method based on YOLOV7 in the above embodiment.

[0080] The embodiment of a weapon and equipment identification device based on YOLOV7 provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Fig.11 As shown, a hardware structure diagram of a weapon equipment identification device based on YOLOV7 provided by the present invention is provided for any device with data processing capability, except Fig.11 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0081] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0082] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0083] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a weapon and equipment identification method based on YOLOV7 in the above embodiment is implemented.

[0084] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0085] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the weapon and equipment identification method based on YOLOV7.

[0086] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0087] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A weapon and equipment identification method based on YOLOv7, characterized in that: The method comprises the following steps: S1. Collect image data of various types of weapons and equipment, clean and annotate the image data, and convert them into YOLO format; S2. Build a YOLOv7-VIT weapon and equipment recognition model. The input image is extracted through the YOLOv7 skeleton network and ViT respectively, and then deep feature fusion is performed. After filtering through the FSM module, the target category and position are regressed. S3. Perform model training: Calculate the loss function based on the label and loss function, and use back propagation to update the model parameters according to the gradient and learning rate until the maximum number of iterations is reached; S4, adjust the color of the output detection box, the language and font of the label; S5. Input the image to be identified into the adjusted model to realize weapon and equipment identification.

2. A weapon and equipment identification method based on YOLOv7 according to claim 1, characterized in that, The collected various types of weapon equipment image data include specific models of weapon equipment image data under eleven categories: artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, artificial satellites, tanks, and armed rotorcraft.

3. A weapon and equipment identification method based on YOLOv7 according to claim 1, characterized in that, The collecting of various types of weapon equipment image data and labeling of the image data after cleaning specifically include: Use crawlers to obtain corresponding images on the Internet according to each major category of weapons, perform manual cleaning, and keep the images corresponding to the models that need to be detected, while removing unclear or difficult-to-recognize images; Use the open source annotation tool Labelimg to annotate the cleaned image, manually frame the target location and annotate the target category name, generate an annotation file in PASCAL VOC format and convert it to YOLO format.

4. A weapon and equipment identification method based on YOLOv7 according to claim 1, characterized in that, The YOLOv7-VIT weapon equipment recognition model includes YOLOv7 and VIT. The image is extracted through the YOLOv7 backbone network and ViT respectively, and the last three layers of features of YOLOv7 and the last layer of features of ViT are taken out; The sequence features output by ViT are reshaped into a matrix form corresponding to the image, and after upsampling, they are concatenated with the last layer features of YOLOv7 to enrich their semantic information. The features of each layer of YOLOv7 are input into the subsequent network structure of YOLOv7 to regress the target category and position.

5. A weapon and equipment identification method based on YOLOv7 according to claim 1, characterized in that, The FSM module takes out each channel of the fused features, calculates the corresponding loss, and then selects the k channels with the lowest loss, where k is a hyperparameter. During training, after the selected k channels are determined by the first ten images, the subsequent input is no longer recalculated, and the corresponding channels are directly selected.

6. A weapon and equipment identification method based on YOLOv7 according to claim 1, characterized in that, The loss function is obtained by comparing the regressed target category and position with the label category and position, and is obtained based on the difference between the category, confidence, and position keys. Specifically, it is: Among them, S 2 is the number of grids that the image is segmented into during the YOLOv7 target detection process, B is the number of predicted boxes for each grid detection, and λ noobj is the weight parameter for balancing the boxes without objects, λ coord is the weight parameter of the coordinate loss. The subscript i indicates that each grid cell is iterated, and the subscript j indicates that each prediction box bbox is iterated. To determine whether the jth prediction box in the i-th grid is responsible for the prediction of the object, and On the contrary; x i ,y i , w i ,h i , C i , p i (c) represents the center coordinates, width, height, confidence, and category of the prediction box. Represents the center coordinates, width, height, confidence, and category of the ground truth box, and the rest are weighting coefficients.

7. A weapon and equipment identification device based on YOLOv7, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a weapon and equipment identification method based on YOLOv7 as described in any one of claims 1-6.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by the processor, a weapon and equipment identification method based on YOLOv7 as described in any one of claims 1 to 6 is implemented.

Images