Weapon and equipment recognition method based on YOLOv7

Abstract

The present invention discloses a weapon and equipment recognition method based on YOLOv7, comprising the following steps: collecting corresponding weapon and equipment images according to category and selecting clear images with distinct features; annotating the selected images and converting them into YOLO format; dividing the images into a training set and a validation set for training to obtain an optimal model; and during detection, using the YOLOv7 backbone network and ViT to extract features from the input image respectively, fusing the features obtained by the two, filtering them through the FSM module, and then processing the feature map to regress the target category and coordinates. This method supports 11 categories, including artillery, large surface warships, fixed-wing aircraft, radar, wheeled armored vehicles, and tanks. Compared with existing technologies, the detection capability is greatly enhanced, and more specific weapon and equipment information can be obtained, which is of great significance and importance in specific scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer vision target detection, and in particular to a weapon and equipment recognition method based on YOLOv7. Background Art

[0002] Weapon and equipment recognition, a method that uses algorithms to automatically locate and identify weapons and equipment in images, will play a vital role in future information warfare. It helps locate battlefield targets and distinguish between target categories, facilitating the acquisition of more battlefield information and guiding various strike strategies. For unmanned equipment, this recognition capability is crucial for unmanned operations, reconnaissance and early warning, and autonomous attacks.

[0003] In recent years, with the advancement of machine learning and deep learning technologies and their widespread application across various fields, several deep learning methods have emerged to identify equipment such as tanks, armored vehicles, and ships, achieving promising results. However, due to insufficient dataset diversity and model capabilities, these methods suffer from a common limitation: they can only detect a single type of target. For example, a model designed to identify tanks can only recognize tanks, and cannot simultaneously identify tanks, aircraft, ships, and other targets. Furthermore, they cannot identify the specific model of a single type.

[0004] To address the above shortcomings, the present invention provides a weapon and equipment recognition method based on YOLOv7, which fuses the features extracted by ViT with the deep features of YOLOv7 to enrich the feature information and enhance the model detection capability. At the same time, a weapon and equipment dataset with rich categories is produced to train the model, so that the model has the ability to recognize 11 major categories: artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, satellites, tanks, and armed rotorcraft, as well as the ability to recognize important models of some equipment. This greatly enhances the recognition range and fine-grained recognition capability of the weapon and equipment recognition algorithm, facilitates the construction of a more concise and efficient recognition system, and provides important technical support for modern warfare. Summary of the Invention

[0005] Aiming at the shortcomings of the current weapon and equipment recognition algorithm, the present invention proposes a weapon and equipment recognition method based on YOLOv7, uses ViT to enrich the model features, constructs an FSM (Feature Select Module) module to select appropriate fused features, thereby improving the detection effect and expanding the recognition range and recognition capability of weapon and equipment. It can identify certain specific models of equipment in 11 major categories, including artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, artificial satellites, tanks and armed rotorcraft.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a weapon and equipment identification method based on YOLOv7, comprising the following steps:

[0007] S1. Collect image data of various types of weapons and equipment, clean and annotate the image data, and convert it into YOLO format;

[0008] S2. Build a YOLOv7-VIT weapon and equipment recognition model. The input image is passed through the YOLOv7 skeleton network and ViT feature extraction, and then deep feature fusion is performed. After filtering through the FSM module, the target category and position are regressed.

[0009] S3. Perform model training: Calculate the loss function based on the label and loss function, and use backpropagation to update the model parameters according to the gradient and learning rate until the maximum number of iterations is reached;

[0010] S4. Adjust the color of the output detection box, the language and font of the label;

[0011] S5. Input the image to be identified into the adjusted model to realize weapon and equipment identification.

[0012] Furthermore, the collected image data of various types of weapons and equipment include image data of specific models of weapons and equipment under eleven categories, namely, artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, artificial satellites, tanks and armed rotorcraft.

[0013] Furthermore, the collecting of various types of weapon and equipment image data and the labeling of the image data after cleaning specifically include:

[0014] We use crawlers to obtain images from the internet according to each weapon category, perform manual cleaning, and retain images corresponding to the models to be tested, while removing unclear or difficult-to-identify images.

[0015] Use the open source annotation tool Labelimg to annotate the cleaned image, manually frame the target location and mark the target category name, generate the annotation file in PASCAL VOC format and convert it to YOLO format.

[0016] Furthermore, the YOLOv7-VIT weapon and equipment recognition model includes YOLOv7 and VIT. The image is respectively subjected to the YOLOv7 backbone network and ViT feature extraction, and the last three layers of features of YOLOv7 and the last layer of features of ViT are extracted;

[0017] The sequence features output by ViT are reshaped into a matrix form corresponding to the image. After upsampling, they are concatenated with the last layer features of YOLOv7 to enrich their semantic information. The features of each layer of YOLOv7 are input into the subsequent network structure of YOLOv7 to regress the target category and position.

[0018] Furthermore, the FSM module extracts each channel of the fused features, calculates its corresponding loss, and then selects the k channels with the lowest loss, where k is a hyperparameter. During training, after the selected k channels are determined by the first ten images, the subsequent input is no longer recalculated, and the corresponding channels are directly selected.

[0019] Furthermore, the loss function is obtained by comparing the regressed target category and position with the label category and position, and is obtained based on the difference between the category, confidence, and position keys. Specifically:

[0020]

[0021] Among them, S 2 is the number of grids that the image is divided into during the YOLOv7 target detection process, B is the number of predicted boxes detected for each grid, and λ noobj is the weight parameter for balancing the boxes without objects, λ coord is the weight parameter of the coordinate loss, the subscript i indicates that each grid cell is iterated, and the subscript j indicates that each prediction box bbox is iterated. To determine whether the j-th prediction box in the i-th grid is responsible for the prediction of the object, and On the contrary; x i ,y i , w i ,j i , C i , p i (c) represents the center coordinates, width, height, confidence, and category of the prediction box. Represents the center coordinates, width, height, confidence, and category of the ground truth box, and the rest are weighting coefficients.

[0022] On the other hand, this specification also provides a weapon and equipment identification device based on YOLOv7, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements a weapon and equipment identification method based on YOLOv7 as described.

[0023] On the other hand, this specification also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it implements the weapon and equipment identification method based on YOLOv7.

[0024] Beneficial effects of the present invention:

[0025] (1) The present invention realizes cross-category weapon equipment detection by constructing a large weapon equipment training set, and uses a single model to detect targets in the three major battlefields of sea, land and air, including artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, satellites, tanks and armed rotorcraft;

[0026] (2) The present invention supports the detection of specific weapon equipment models under each major category, mainly some important models, such as the PHL-03 rocket launcher, the Nimitz-class aircraft carrier, the B-2 strategic bomber, etc.

[0027] (3) The present invention uses the deep features extracted by integrating ViT and YOLOv7, and uses the FSM module for screening to reduce the number of channels, retain more effective features, and enhance the detection ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0029] Figure 2 It is a network framework diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0030] Figure 3 It is a CBS module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0031] Figure 4 It is the MP1 module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0032] Figure 5 It is the MP2 module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0033] Figure 6 It is a block diagram of the up module of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0034] Figure 7 It is the ELAN module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0035] Figure 8 It is the ELAN-H module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0036] Figure 9 It is the SPPCSPC module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0037] Figure 10 It is a VIT module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0038] Figure 11 It is an FSM module structure diagram of the weapon and equipment identification method based on YOLOv7 of the present invention;

[0039] Figure 12 Schematic diagram of the weapon and equipment identification device based on YOLOv7 of the present invention. DETAILED DESCRIPTION

[0040] To make the purpose, technical solutions, and effective improvements of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0041] like Figure 1 As shown, the steps of the present invention include:

[0042] Step S1: collecting image data of specific models of weapons and equipment under eleven categories, including artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, satellites, tanks, and armed rotorcraft, cleaning the image data, filtering out invalid images, and then annotating the filtered image data;

[0043] Step S2: Convert the labeled VOC format file into YOLO format, and then divide the labeled weapons and equipment dataset into training set, validation set, and test set;

[0044] Step S3: Set the model training parameters for training. During training, the input image is extracted through YOLOv7 and ViT respectively. The features extracted by ViT are then fused with the deep features extracted by YOLOv7 to obtain features with richer semantic information. These features are then filtered through the FSM (Feature Select Module). This module extracts each channel of the fused features and calculates the corresponding loss. The k channels with the lowest loss are then selected, where k is a manually set hyperparameter. After the first ten images are used to determine the selected k channels during training, the corresponding channels are directly selected without recalculation for subsequent inputs, retaining more effective and streamlined features.

[0045] The overall structure of the model is as follows Figure 2 As shown, in YOLOv7, the specific structure of the CBS module is as follows Figure 3 As shown, the specific structure of MP1 module is as follows Figure 4 As shown in the MP2 module structure diagram, the specific structure is as follows Figure 5 As shown, the specific structure of the up module is as follows Figure 6 As shown, the specific structure of the ELAN module is as follows Figure 7 As shown, the specific structure of the ELAN-H module is as follows Figure 8 As shown, the specific structure of the SPPCSPC module is as follows Figure 9 As shown, the structure of ViT is Figure 10 During training, the input image is extracted by YOLOv7 and ViT respectively, and then the features extracted by ViT are fused with the deep features extracted by YOLOv7 to obtain features with richer semantic information.

[0046] In step S4, the trained model is used to set the detection parameters, the model performance is evaluated using the test set, and the model output is adjusted to have a better visualization effect.

[0047] Furthermore, the specific process of step S1 is as follows:

[0048] Step S11: Use a crawler to obtain corresponding images from the Internet according to major categories such as tanks and artillery, and then manually clean them to keep only the images corresponding to the models to be tested, while eliminating unclear or difficult-to-identify images;

[0049] In step S12, the cleaned image is annotated using the open source annotation tool Labelimg. The target location is manually framed and the target category name is annotated to generate an annotation file in the PASCAL VOC format.

[0050] Furthermore, step S2 converts the annotation file format and divides the data set. The specific process is as follows:

[0051] In step S21, a Python script is used to convert the PASCAL VOC annotation file into a YOLO formatted txt file required for YOLOv7 training. Each image corresponds to a txt file, and each line in the txt file corresponds to an object in the image. The first number represents the category, and the following four numbers are the normalized coordinates of the center point and the width and height of the target box.

[0052] In step S22, a python script is used to randomly select 80% of the images according to categories as a training set, 10% of the images as a validation set, and the remaining 10% of the images as a test set.

[0053] Furthermore, step S3 trains the model. The specific process is as follows:

[0054] Step S31, setting model training parameters, including the number of target categories, learning rate, image enhancement parameters, dataset corresponding path, etc.;

[0055] Step S32: Load the initial model and training data, pre-process the training data, and then feed it into the network for forward propagation. The image passes through the YOLOv7 backbone network and ViT feature extraction, extracting the last three layers of YOLOv7 features and the last layer of ViT features.

[0056] Step S33: reshape the sequence features output by ViT into a matrix form corresponding to the image, upsample and concatenate them with the last layer features of YOLOv7 to enrich their semantic information;

[0057] Step S34: Input the features obtained by fusing ViT and the last layer of YOLOv7 into the FSM module for screening to retain more effective features;

[0058] In step S35, after fusing the above features with the features of other layers of YOLOv7, the target category and position are regressed, and compared with the category and position of the label. The loss function is calculated based on the difference between the category, confidence, and position, as shown in formula (1). The network parameters are updated according to the gradient and learning rate using the backpropagation algorithm, and this process is repeated until the specified training generation is completed.

[0059]

[0060] Among them, S 2 is the number of grids that the image is divided into during the YOLOv7 target detection process, B is the number of bounding boxes detected for each grid, and λ noobj is the weight parameter for balancing the boxes without objects, λ coord is the weight parameter of the coordinate loss, the subscript i indicates that each grid cell is iterated, and the subscript j indicates that each prediction box bbox is iterated. To determine whether the jth bbox in the i-th grid is responsible for the prediction of this obj, and on the contrary

[0061] x i ,y i , w i , h i , C i , p i (c) represents the center coordinates, width, height, confidence, and category of the prediction box. Represents the center coordinates, width, height, confidence, and category of the ground truth box, and the rest are weighting coefficients.

[0062] Furthermore, step S4 tests the model, and the specific process is as follows:

[0063] Step S41: Set the model test parameters, including batch size, model path to be loaded, test set path, IOU threshold, etc.

[0064] Step S42: Load the test set data and feed it into the network for forward propagation. Similar to training, feature extraction is performed through the YOLOv7 backbone network and ViT, and the final features are obtained through feature fusion and screening. The target category and position are then regressed, and invalid targets are filtered out through non-maximum suppression. The IOU is then calculated based on the label and compared with the set threshold to determine whether the detection is correct. Indicators such as mAP are also calculated to determine whether the model performance meets the requirements.

[0065] Step S43: Adjust the color of the output detection frame, the language and font of the label, etc. as needed.

[0066] Specifically, the precision, recall, F1 index and average precision mAP of each category are as follows: Formula (2), Formula (3), Formula (4) and Formula (5).

[0067] (2)Precision:

[0068]

[0069] (2) Recall:

[0070]

[0071] (4) F1 index:

[0072]

[0073] (5) Average Mean Average Precision (mAP):

[0074]

[0075] True Positive (TP): a sample marked as positive and predicted as positive; False Positive (FP): a sample marked as negative and predicted as positive; True Negative (TN): a sample marked as negative and predicted as negative; False Negative (FN): a sample marked as positive and predicted as negative.

[0076] Step S43: Adjust the color of the output detection frame, the language and font of the label, etc. as needed.

[0077] Step S5: input the image to be identified into the adjusted model to perform weapon and equipment identification.

[0078] Corresponding to the aforementioned embodiment of a weapon and equipment identification method based on YOLOV7, the present invention also provides an embodiment of a weapon and equipment identification device based on YOLOV7.

[0079] See also Figure 12 An embodiment of the present invention provides a weapon and equipment identification device based on YOLOV7, including a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement a weapon and equipment identification method based on YOLOV7 in the above embodiment.

[0080] The embodiment of the weapon and equipment identification device based on YOLOV7 provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 11 As shown in the figure, a hardware structure diagram of a weapon equipment identification device based on YOLOV7 provided by the present invention is provided for any device with data processing capability, except Figure 11 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0081] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0082] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0083] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a weapon and equipment identification method based on YOLOV7 in the above embodiment is implemented.

[0084] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0085] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the weapon and equipment identification method based on YOLOV7.

[0086] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0087] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A weapon and equipment identification method based on YOLOv7, characterized in that: The method comprises the following steps: S1. Collect image data of various types of weapons and equipment, clean and annotate the image data, and convert it into YOLO format; S2. Build a YOLOv7-VIT weapon and equipment recognition model. The input image is passed through the YOLOv7 skeleton network and VIT feature extraction, and then deep feature fusion is performed. After filtering through the FSM module, the target category and position are regressed. The YOLOv7-VIT weapon and equipment recognition model includes YOLOv7 and VIT. The image is extracted through the YOLOv7 backbone network and VIT respectively, and the last three layers of features of YOLOv7 and the last layer of features of VIT are extracted; The sequence features output by VIT are reshaped into a matrix form corresponding to the image. After upsampling, they are concatenated with the last layer features of YOLOv7 to enrich their semantic information. The features of each layer of YOLOv7 are input into the subsequent network structure of YOLOv7 to regress the target category and position. The FSM module extracts each channel of the fused features and calculates its corresponding loss, then selects the k channels with the lowest loss, where k is a hyperparameter. During training, after the first ten images are used to determine the selected k channels, the corresponding channels are directly selected without recalculating the subsequent inputs. S3. Perform model training: Calculate the loss function based on the label and loss function, and use backpropagation to update the model parameters according to the gradient and learning rate until the maximum number of iterations is reached; The loss function is obtained by comparing the regressed target category and position with the label category and position, and is obtained based on the difference between the category, confidence, and position keys. Specifically, it is: Among them, S 2 is the number of grids that the image is divided into during the YOLOv7 target detection process, B is the number of predicted boxes detected for each grid, and λ noobj is the weight parameter for balancing the boxes without objects, λ coord is the weight parameter of the coordinate loss, the subscript i indicates that each grid cell is iterated, and the subscript j indicates that each prediction box bbox is iterated. To determine whether the j-th prediction box in the i-th grid is responsible for the prediction of the object, and On the contrary; x i ,y i , w i , h i , C i , p i (c) represents the center coordinates, width, height, confidence, and category of the prediction box. Represents the center coordinates, width, height, confidence, and category of the ground truth box, and the rest are weighting coefficients; S4. Adjust the color of the output detection box, the language and font of the label; S5. Input the image to be identified into the adjusted model to realize weapon and equipment identification.

2. A weapon and equipment identification method based on YOLOv7 according to claim 1, characterized in that, The collected image data of various types of weapons and equipment include image data of specific models of weapons and equipment under eleven categories: artillery, large surface warships, missiles, fixed-wing aircraft, radars, wheeled armored vehicles, submarines, guns, artificial satellites, tanks and armed rotorcraft.

3. A weapon and equipment identification method based on YOLOv7 according to claim 1, characterized in that, The collecting of various types of weapon and equipment image data and the labeling of the image data after cleaning specifically include: We use crawlers to obtain images from the internet according to each weapon category, perform manual cleaning, and retain images corresponding to the models to be tested, while removing unclear or difficult-to-identify images. Use the open source annotation tool Labelimg to annotate the cleaned image, manually frame the target location and mark the target category name, generate the annotation file in PASCAL VOC format and convert it to YOLO format.

4. A weapon and equipment identification device based on YOLOv7, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a weapon and equipment identification method based on YOLOv7 as described in any one of claims 1 to 3.

5. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a weapon and equipment identification method based on YOLOv7 as described in any one of claims 1 to 3 is implemented.

Images