A method, device, equipment and medium for detecting single target of visible light rotor in the air

Abstract

The present invention relates to a method, device, equipment and medium for detecting a single target of an aerial visible light rotor. The method comprises: obtaining a video to be detected, wherein the video to be detected includes a visible light rotor target; inputting the video to be detected into a pre-trained target detection model to obtain a detection result of the visible light rotor target in the video to be detected, wherein the target detection model is obtained by training a YOLOv5 network model with a small target detection head added. The method of the present invention can improve the detection accuracy of the visible light rotor target.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, target detection, intelligent military detection, deep learning and artificial intelligence application technology. Specifically, the present invention relates to an aerial visible light rotor single target detection method, device, equipment and medium. Background Art

[0002] At present, the target detection methods based on deep learning are widely used in civil and military fields. Existing target detection methods are mainly divided into two categories: one-stage target detection method and two-stage target detection method. The one-stage detection method does not have the process of extracting candidate boxes in the entire target detection process. Instead, the candidate boxes with fixed positions and fixed sizes are pre-set, and the category probability and position coordinate values ​​are directly regressed to the real target box (ground truth) through each candidate box. The representative methods of this type are mainly YOLO, SSD, etc. The two-stage detection method first uses the algorithm of extracting candidate boxes such as selective search to generate a series of candidate boxes as samples on the video to be detected, and then uses the target detection network to complete the target detection. The representative methods of this type are mainly R-CNN, Fast R-CNN, Faster R-CNN, etc. In comparison, the one-stage method has the advantage of speed but is often slightly inferior to the two-stage method in accuracy.

[0003] With the development of UAV technology, rotor target detection has gradually become a research hotspot in the field of machine vision, and it has broad application prospects in both military and civilian fields. Especially in the military field, monitoring and preventing enemy aerial rotor targets is an indispensable task. Rotor targets have the characteristics of strong flexibility and small size. Therefore, how to accurately detect visible light rotor targets is a problem that needs to be solved urgently. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an aerial visible light rotor single target detection method, device, equipment and medium, aiming to solve at least one of the above technical problems.

[0005] In the first aspect, the technical solution of the present invention to solve the above technical problem is as follows: a method for detecting a single target of an aerial visible light rotor, the method comprising:

[0006] Acquire a video to be detected, where the video to be detected includes a visible light rotor target;

[0007] Input the video to be detected into a pre-trained target detection model to obtain the detection result of the visible light rotor target in the video to be detected;

[0008] Among them, the target detection model is trained in the following way:

[0009] Acquire a training sample set, the training sample set including a first training sample and a second training sample, the first training sample including a first sample image collected at night, the second training sample including a second sample image collected during the day, each sample image including a visible light rotor target, each sample image corresponding to a labeling result, and for each sample image, the labeling result characterizes a position of the visible light rotor target in the sample image;

[0010] Input each sample image in the training sample set into the YOLOv5 network model. For each sample image, output the first feature map of the sample image through the feedforward network in the YOLOv5 network model, and output the second feature map corresponding to the first sample image through the neck network in the YOLOv5 network model.

[0011] For each sample image, upsample the second feature map to obtain a third feature map; fuse the first feature map and the third feature map to obtain a fourth feature map; perform convolution processing on the fourth feature map to obtain a fifth feature map, and determine the predicted detection result of the sample image according to the fifth feature map;

[0012] According to the predicted detection results and annotation results of each sample image, the total loss value of the YOLOv5 network model is obtained;

[0013] If the total loss value meets the preset training end condition, the YOLOv5 network model that meets the training end condition will be used as the target detection model. If the total loss value does not meet the preset training end condition, the model parameters of the YOLOv5 network model will be adjusted, and the YOLOv5 network model will be trained according to the adjusted model parameters until the total loss value meets the preset training end condition.

[0014] The beneficial effects of the present invention are as follows: for a video to be detected that contains a visible light rotor target, the detection result of the visible light rotor target in the video to be detected can be determined based on a pre-trained target detection model. Since the target detection model is obtained by training based on the YOLOv5 network model, the second feature map output by the neck network is upsampled in the YOLOv5 network model to obtain a third feature map; the first feature map and the third feature map output by the feedforward network are fused to obtain a fourth feature map; the fourth feature map is convolved to obtain a fifth feature map. The size of the obtained fifth feature map is much smaller than that of the image input to the YOLOv5 network model, that is, the fifth feature map can enhance the model's ability to capture small target features, so that the trained target detection model can more accurately detect the detection result of the visible light rotor target in the video to be detected.

[0015] Based on the above technical solution, the present invention can also be improved as follows.

[0016] Furthermore, the above training sample set is determined in the following way:

[0017] Acquire multiple sample videos, the multiple sample videos include night sample videos and day sample videos, and the multiple sample videos correspond to multiple scenes;

[0018] Perform frame extraction on each sample video to obtain multiple frames of initial sample images;

[0019] According to the set format, multiple frames of initial sample images are stored as a data set in a set format. The set format is a txt annotation format. The data set includes multiple lines of data. Each line of data includes the category and annotation results of a frame of initial sample image. The data set is used as a training sample set.

[0020] The beneficial effect of adopting the above further scheme is to perform frame extraction processing on each sample video, extract visible light rotor target image data, reduce the amount of data processing, and determine the training sample set with reference to the YOLO data set format (txt annotation format) for subsequent training and testing.

[0021] Furthermore, the above dataset is used as a training sample set, including:

[0022] According to the brightness of each frame of the initial sample image in the data set, a first training sample and a second training sample are determined from the multiple frames of the initial sample image.

[0023] The beneficial effect of adopting the above further scheme is that according to the brightness of the initial sample image of each frame in the data set, the data set is divided into two parts, the first training sample and the second training sample, and trained as two categories, so that the trained target detection model can adapt to more scenarios.

[0024] Furthermore, the above-mentioned determining the first training sample and the second training sample from the multiple frames of initial sample images according to the brightness of each frame of the initial sample image in the data set includes:

[0025] For each frame of the initial sample image, preprocessing the initial sample image to obtain a preprocessed image, wherein the preprocessing includes at least one of scaling, cropping, arranging, splicing, brightness adjustment, hue adjustment, contrast adjustment, mirroring and rotation;

[0026] According to the brightness of each preprocessed image in each frame of the preprocessed image, a first training sample and a second training sample are determined from the plurality of frames of the preprocessed image.

[0027] The beneficial effect of adopting the above further scheme is that preprocessing each frame of the initial sample image can increase the robustness and data diversity of the data, so that the trained target detection model is more robust and adaptable to more different scenarios.

[0028] Furthermore, the feedforward network is connected to a channel space attention module, and the method further includes:

[0029] For each sample image, the first feature map of the sample image is input into the channel space attention module to obtain the sixth feature map;

[0030] For each sample image, the first feature map and the third feature map are fused to obtain a fourth feature map, including:

[0031] The sixth feature map is fused with the third feature map to obtain a fourth feature map.

[0032] The beneficial effect of adopting the above further scheme is that, through the channel space attention module, the model can pay more attention to a specific area (the characteristics of the visible light rotor target), so that the detection accuracy of the trained target detection model is higher.

[0033] Furthermore, the feedforward network includes a BottleneckCsp module, and the method further includes:

[0034] Replace the BottleneckCsp module with the GhostBottleneck module;

[0035] For each sample image, the first feature map of the sample image outputted by the feedforward network in the YOLOv5 network model includes:

[0036] The first feature map of the sample image is output through the GhostBottleneck module.

[0037] The beneficial effect of adopting the above further solution is that by replacing the BottleneckCsp module in the original YOLOv5 network model with the GhostBottleneck module, the reasoning speed of the model can be improved, that is, the training speed of the model can be improved.

[0038] Furthermore, the method further comprises:

[0039] Divide the training sample set into training set and test set;

[0040] Input each sample image in the training sample set into the YOLOv5 network model, including:

[0041] Input each sample image in the training set into the YOLOv5 network model;

[0042] The method further includes:

[0043] Test the object detection model based on the test set.

[0044] The beneficial effect of adopting the above further scheme is that the accuracy of the model can be further improved by completing the training of the target detection model through the training set and the test set.

[0045] In a second aspect, in order to solve the above technical problems, the present invention also provides an aerial visible light rotor single target detection device, which comprises:

[0046] A video acquisition module is used to acquire a video to be detected, wherein the video to be detected includes a visible light rotor target;

[0047] The target detection module is used to input the video to be detected into a pre-trained target detection model to obtain the detection result of the visible light rotor target in the video to be detected;

[0048] Among them, the target detection model is trained in the following way:

[0049] Acquire a training sample set, the training sample set including a first training sample and a second training sample, the first training sample including a first sample image collected at night, the second training sample including a second sample image collected during the day, each sample image including a visible light rotor target, each sample image corresponding to a labeling result, and for each sample image, the labeling result characterizes a position of the visible light rotor target in the sample image;

[0050] Input each sample image in the training sample set into the YOLOv5 network model. For each sample image, output the first feature map of the sample image through the feedforward network in the YOLOv5 network model, and output the second feature map corresponding to the first sample image through the neck network in the YOLOv5 network model.

[0051] For each sample image, upsample the second feature map to obtain a third feature map; fuse the first feature map and the third feature map to obtain a fourth feature map; perform convolution processing on the fourth feature map to obtain a fifth feature map, and determine the predicted detection result of the sample image according to the fifth feature map;

[0052] According to the predicted detection results and annotation results of each sample image, the total loss value of the YOLOv5 network model is obtained;

[0053] If the total loss value meets the preset training end condition, the YOLOv5 network model that meets the training end condition will be used as the target detection model. If the total loss value does not meet the preset training end condition, the model parameters of the YOLOv5 network model will be adjusted, and the YOLOv5 network model will be trained according to the adjusted model parameters until the total loss value meets the preset training end condition.

[0054] In the third aspect, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aerial visible light rotor single target detection method of the present application is implemented.

[0055] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the aerial visible light rotor single target detection method of the present application is implemented.

[0056] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments of the present invention are briefly introduced below.

[0058] Figure 1 A schematic diagram of a flow chart of a method for detecting a single target of an aerial visible light rotor provided by an embodiment of the present invention;

[0059] Figure 2 A schematic flow chart of another method for detecting a single target in an aerial visible light rotor provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of an aerial visible light rotor single target detection device provided by one embodiment of the present invention;

[0061] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0063] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0064] The solution provided by the embodiment of the present invention can be applied to any application scenario where target detection of visible light rotor targets in the air is required. The embodiment of the present invention provides a possible implementation method, such as Figure 1 As shown, a flowchart of a method for detecting a single target of a rotor in the air using visible light is provided. The method can be executed by any electronic device, for example, a terminal device, or a terminal device and a server. For the convenience of description, the method provided by the embodiment of the present invention will be described below by taking the server as the execution subject as an example. Figure 1 The method may include the following steps:

[0065] Step S110, obtaining a video to be detected, where the video to be detected includes a visible light rotor target;

[0066] Step S120, inputting the video to be detected into a pre-trained target detection model to obtain a detection result of the visible light rotor target in the video to be detected;

[0067] Among them, the target detection model is trained through the following steps:

[0068] Step S1301, obtaining a training sample set, the training sample set including a first training sample and a second training sample, the first training sample including a first sample image collected at night, the second training sample including a second sample image collected during the day, each sample image including a visible light rotor target, each sample image corresponding to a labeling result, for each sample image, the labeling result characterizes the position of the visible light rotor target in the sample image;

[0069] Step S1302, input each sample image in the training sample set into the YOLOv5 network model, and for each sample image, output a first feature map of the sample image through the feedforward network in the YOLOv5 network model, and output a second feature map corresponding to the first sample image through the neck network in the YOLOv5 network model;

[0070] Step S1303: for each sample image, upsample the second feature map to obtain a third feature map; fuse the first feature map and the third feature map to obtain a fourth feature map; perform convolution processing on the fourth feature map to obtain a fifth feature map, and determine the predicted detection result of the sample image according to the fifth feature map;

[0071] Step S1304, obtaining the total loss value of the YOLOv5 network model according to the predicted detection results and annotation results of each sample image;

[0072] Step S1305: If the total loss value meets the preset training end condition, the YOLOv5 network model that meets the training end condition is used as the target detection model; if the total loss value does not meet the preset training end condition, the model parameters of the YOLOv5 network model are adjusted, and the YOLOv5 network model is trained according to the adjusted model parameters until the total loss value meets the preset training end condition.

[0073] By means of the method of the present invention, for a video to be detected that contains a visible light rotor target, the detection result of the visible light rotor target in the video to be detected can be determined based on a pre-trained target detection model. Since the target detection model is obtained through training based on a YOLOv5 network model, the second feature map output by the neck network is upsampled in the YOLOv5 network model to obtain a third feature map; the first feature map and the third feature map output by the feedforward network are fused to obtain a fourth feature map; the fourth feature map is convolved to obtain a fifth feature map. The size of the obtained fifth feature map is much smaller than that of the image input to the YOLOv5 network model, that is, the fifth feature map can enhance the ability of the model to capture small target features, so that the trained target detection model can more accurately detect the detection result of the visible light rotor target in the video to be detected.

[0074] The solution of the present invention is further described below in conjunction with the following specific embodiments. In this embodiment, the method for detecting a single target of an aerial visible light rotor may include the following steps:

[0075] Step S110, obtaining a video to be detected, where the video to be detected includes a visible light rotor target;

[0076] The video to be detected is a video shot of a visible light rotor target in the air, and the video to be detected can be a video captured by an image acquisition device (such as a camera). The video to be detected can be a video shot at night or during the day. The video to be detected can be a video shot in any scene such as the sky, buildings, trees, and towers.

[0077] Step S120, inputting the video to be detected into a pre-trained target detection model to obtain a detection result of the visible light rotor target in the video to be detected.

[0078] The target detection model is pre-trained based on the YOLOv5 network model. The target detection model is trained through the following steps:

[0079] Step S1301, obtaining a training sample set, the training sample set including a first training sample and a second training sample, the first training sample including a first sample image collected at night, the second training sample including a second sample image collected during the day, each sample image including a visible light rotor target, each sample image corresponding to a labeling result, for each sample image, the labeling result characterizes the position of the visible light rotor target in the sample image;

[0080] Step S1302, input each sample image (including the first sample image and the second sample image) in the training sample set into the YOLOv5 network model, for each sample image (which may be the first sample image or the second sample image), output the first feature map of the sample image through the feedforward network in the YOLOv5 network model, and output the second feature map corresponding to the first sample image through the neck network in the YOLOv5 network model;

[0081] Step S1303: for each sample image, upsample the second feature map to obtain a third feature map; fuse the first feature map and the third feature map to obtain a fourth feature map; perform convolution processing on the fourth feature map to obtain a fifth feature map, and determine the predicted detection result of the sample image according to the fifth feature map;

[0082] Step S1304, obtaining the total loss value of the YOLOv5 network model according to the predicted detection results and annotation results of each sample image;

[0083] Step S1305: If the total loss value meets the preset training end condition, the YOLOv5 network model that meets the training end condition is used as the target detection model; if the total loss value does not meet the preset training end condition, the model parameters of the YOLOv5 network model are adjusted, and the YOLOv5 network model is trained according to the adjusted model parameters until the total loss value meets the preset training end condition.

[0084] In the present application, the YOLOv5 network model is improved. The YOLOv5 network model includes four parts connected in sequence, namely, the input network, Backbone (feedforward network), Neck (neck network) and Prediction (prediction network). For each sample image, the first feature map of the sample image is output through the feedforward network in the YOLOv5 network model, and the second feature map corresponding to the first sample map is output through the neck network in the YOLOv5 network model, and then the predicted detection result of the sample image is determined by the following small target detection head method, wherein the specific implementation method of the small target detection head is: upsampling the second feature map to obtain a third feature map; fusing the first feature map and the third feature map to obtain a fourth feature map; performing convolution processing on the fourth feature map to obtain a fifth feature map, and determining the predicted detection result of the sample image according to the fifth feature map.

[0085] For the original YOLOv5 structure, if the input image size is 640*640*3, the final feature map output sizes are 80*80, 40*40 and 20*20 respectively, so the minimum receptive field for each grid in the feature map is 8x8 (640 / 8). For the drone rotor target detection task, the entire image size is 1920*1680, and the drone target is in many cases about 10*10 in size, which is often not enough for such a target receptive field. Therefore, through the above-mentioned small target detection head, the 80*80 feature map (second feature map) is upsampled to obtain the third feature map; the first feature map and the third feature map are fused to obtain the fourth feature map, and then the fourth feature map is convolved to obtain the fifth feature map of 160*160. In this way, the receptive field corresponding to each grid becomes a 4*4 grid, thereby enhancing the network's ability to capture small target features. Through the small target detection head, the features of the low-level layer can be used more deeply, because the low-level features represent more local features of the image, which is suitable for small targets such as rotor targets, making the detection of rotor targets more accurate.

[0086] Optionally, the training sample set is determined by:

[0087] Acquire multiple sample videos, the multiple sample videos include night sample videos and day sample videos, and the multiple sample videos correspond to multiple scenes;

[0088] Perform frame extraction on each sample video to obtain multiple frames of initial sample images;

[0089] According to the set format, multiple frames of initial sample images are stored as a data set in a set format. The set format is a txt annotation format. The data set includes multiple lines of data. Each line of data includes the category and annotation results of a frame of initial sample image. The data set is used as a training sample set.

[0090] Among them, the frame extraction processing for each sample video refers to the frame extraction processing for each night sample video and each day sample video. The purpose of frame extraction is to extract the image containing the visible light rotor target in the sample video as the sample image for subsequent training, reduce the amount of data processing, and speed up the model training speed. Each sample video can be a video of the rotor target in different scenes, including the sky, buildings, trees, towers, etc., to meet the recognition of rotor targets in different application scenarios.

[0091] After the frame extraction process, multiple frames of initial sample images can be stored as a data set in a set format according to a set format. The data set includes multiple rows of data, each row of data includes the category and annotation results of a frame of initial sample images. Taking a row of data as an example, the category corresponding to the row of data is used to indicate whether the initial sample image of the row is a rotor target image taken at night or a rotor target image taken during the day. The annotation result is used to characterize the position of the rotor target in the image of the initial sample image of the row. An optional method is to annotate the position of the rotor target in the image through a target box, and the annotation result may include the coordinates of the center point of the rotor target, as well as the height and width of the target box.

[0092] Among them, the center point coordinates of the rotor target can be the center point coordinates after normalization processing, and the height and width of the target frame can also be the height and width of the target frame after normalization processing. Normalization processing refers to the position before processing divided by the width and height of the image where the rotor target is located.

[0093] After the data set is obtained, the data set can be directly used as a training sample set, or the first training sample and the second training sample can be determined from multiple frames of initial sample images according to the brightness of each frame of the initial sample image in the data set.

[0094] Since there is a difference in brightness between videos shot at night and videos shot during the day, the first training sample and the second training sample can be determined according to the brightness of each frame of the initial sample image; wherein, according to the brightness of each frame of the initial sample image in the data set, one feasible way to determine the first training sample and the second training sample from multiple frames of initial sample images is: determine the brightness mean of each frame of the initial sample image in each frame of the initial sample image, and divide the initial sample images of each frame in the data set according to a preset brightness threshold to obtain the first training sample and the second training sample, wherein the brightness threshold is pre-set according to the brightness of the image shot at night and the brightness of the image shot during the day, and is used to distinguish the brightness of the image shot during the day and the brightness of the image shot at night.

[0095] In the present application, in order to expand the number and diversity of sample images in the training sample set, each frame of the initial sample image may be preprocessed. Then, according to the brightness of each frame of the initial sample image in the data set, the first training sample and the second training sample are determined from multiple frames of the initial sample image, including:

[0096] For each frame of the initial sample image, preprocessing the initial sample image to obtain a preprocessed image, wherein the preprocessing includes at least one of scaling, cropping, arranging, splicing, brightness adjustment, hue adjustment, contrast adjustment, mirroring and rotation;

[0097] According to the brightness of each preprocessed image in each frame of the preprocessed image, a first training sample and a second training sample are determined from the plurality of frames of the preprocessed image.

[0098] Through the above preprocessing, not only can the number and diversity of sample images be expanded, but the scaling process can also increase many small targets, making the network more robust and improving the detection effect of small targets such as visible light rotor targets. By adjusting the brightness, chromaticity and contrast in the image, the robustness of network detection can be increased. By randomly cropping, mirroring and rotating, the deficiencies of the training data set can be compensated to achieve the purpose of expanding the training data.

[0099] After determining the first training sample and the second training sample, the method further includes: dividing the training sample set into a training set and a test set, that is, dividing the first training sample and the second training sample into a training set and a test set, and taking one frame of sample image every five frames as a test set, and the rest as a training set;

[0100] After dividing the test set and the training set, the above inputs each sample image in the training sample set into the YOLOv5 network model, including:

[0101] Each sample image in the training set is input into the YOLOv5 network model, that is, the YOLOv5 network model is trained through the training set. After the target detection model is obtained through training, the method further includes: testing the target detection model according to the test set, that is, testing the target detection model through the test set.

[0102] In the solution of the present application, after determining to obtain the first training sample and the second training sample, the YOLOv5 network model can be trained based on the first training sample and the second training sample, respectively, wherein the network structure of the YOLOv5 network model includes four parts connected in sequence, namely, the input network, Backbone (feedforward network), Neck (neck network) and Prediction (prediction network). In the optional solution of the present application, a channel space attention module can be connected after the feedforward network, and the method further includes:

[0103] For each sample image, the first feature map of the sample image is input into the channel space attention module to obtain the sixth feature map;

[0104] For each sample image, the first feature map and the third feature map are fused to obtain a fourth feature map, including:

[0105] The sixth feature map is fused with the third feature map to obtain a fourth feature map.

[0106] Among them, before the attention module predicts the network, the channel space attention module can make the YOLOv5 network model pay more attention to the characteristics of specific areas and specific channels, that is, the characteristics of the rotor target. The Attention mechanism can make the model pay more attention to important areas; at the same time, it also plays a role in combining global information. The channel dimension attention module makes the deep network more inclined to the characteristics of a certain dimension by adding weights to the channel features; the channel space combined attention module increases the attention of the model in two dimensions by adding weights to the channel features and spatial features. In the scheme of the present application, after each downsampling of the feedforward network (backbone) in YOLOv5, the first feature map is strengthened with the help of the channel space attention module to further enhance the characteristics of the small visible light rotor target.

[0107] Optionally, the feedforward network includes a BottleneckCsp module, and the method further includes:

[0108] Replace the BottleneckCsp module with the GhostBottleneck module;

[0109] For each sample image, the first feature map of the sample image outputted by the feedforward network in the YOLOv5 network model includes:

[0110] The first feature map of the sample image is output through the GhostBottleneck module.

[0111] Since the usual convolution operation will generate many redundant feature maps, these feature maps can be obtained by simple operations such as sharpening and rotation. Therefore, it is possible to use lightweight operations to replace some traditional convolutional layers to generate redundant features to reduce the amount of calculation. The present invention uses the GhostBottleneck module to replace the original BottleneckCsp module in the YOLOv5 downsampling, which improves the inference speed of the model while ensuring accuracy.

[0112] During the training process of the model, a reasonable detection threshold can also be set to accurately determine the final detection effect. That is, from multiple predicted detection results, a preset detection threshold is used to determine the predicted detection result that is greater than the preset detection threshold, that is, the predicted detection result with a higher confidence level. The predicted detection result that is greater than the preset detection threshold is used as the predicted detection result for comparison with the labeled result.

[0113] Among them, the predicted detection result can be represented by the target box.

[0114] In order to better illustrate and understand the principle of the method provided by the present invention, the scheme of the present invention is described below in conjunction with an optional specific embodiment. It should be noted that the specific implementation of each step in the specific embodiment should not be understood as a limitation to the scheme of the present invention. On the basis of the principle of the scheme provided by the present invention, other implementations that can be thought of by those skilled in the art should also be considered within the scope of protection of the present invention.

[0115] See also Figure 2 The training process of the target detection model shown in FIG. 1 includes the following steps:

[0116] Step 1: Collect image data of visible light rotor targets (for details, please refer to the previous article to obtain multiple sample videos, extract frames for each sample video, and obtain multiple frames of initial sample images), and create a data set according to the YOLO format (txt annotation format).

[0117] Step 2: Calculate the image brightness value of each frame of the initial sample image in the data set, and divide the data set into two categories: night data (first training sample) and day data (second training sample) according to the image brightness value of each frame of the initial sample image.

[0118] Step 3, determining a training set and a test set based on the first training sample and the second training sample, wherein the training set may include sample images in the first training sample and sample images in the second training sample, and the test set may also include sample images in the first training sample and sample images in the second training sample.

[0119] Step 4, preprocessing each frame sample image in the training set, including at least one of random splicing, brightness, hue adjustment, contrast adjustment, random cropping, random mirroring and random rotation.

[0120] Step 5, modify the YOLOv5 network model, add the processing method of the small target detection head, introduce the attention mechanism, replace the BottleneckCsp module with the GhostBottleneck module, and obtain the improved YOLOv5 network model.

[0121] Step 6, set a reasonable detection threshold to refine the final detection effect, introduce a single target detection threshold, that is, through the detection threshold, determine the predicted detection result that can be used to compare with the annotation result from each predicted detection result, thereby reducing the amount of data processing, where one rotor target can correspond to one detection threshold.

[0122] Step 7: Based on the improved YOLOv5 network model, the visible light rotor target data (training set and test set) are trained, detected and recognized to obtain a target detection model.

[0123] The solution of the present invention has the following advantages:

[0124] 1. The present invention improves the Yolov5 network structure and adds a small target detection head to make it more consistent with the proportion of visible light rotor targets in the image, and can detect small targets such as rotor targets more accurately and ensure a certain detection speed;

[0125] 2. The present invention improves the Yolov5 network structure, adds an attention mechanism, improves the model's detection accuracy for multi-scale targets, and improves the small target detection performance.

[0126] 3. The present invention improves the Yolov5 network structure and introduces the GhostBottleneck structure, which reduces the amount of calculation and speeds up the model reasoning speed;

[0127] 4. The present invention divides the data set into two types of data sets: night scene and day scene through data preprocessing, thereby improving the detection accuracy of rotor targets in night scenes and reducing the probability of missed detection of targets;

[0128] 5. Through multi-faceted optimization and improvement, the overall network of the present invention has achieved faster reasoning time and more accurate ability to detect and locate targets. It has strong practicality and real-time performance and can be applied to different devices to cope with different detection tasks.

[0129] Based on Figure 1 Based on the same principle as the method shown in , the embodiment of the present invention also provides an aerial visible light rotor single target detection device 20, such as Figure 3As shown in , the aerial visible light rotor single target detection device 20 may include a video acquisition module 210 and a target detection module 220, wherein:

[0130] The video acquisition module 210 is used to acquire the video to be detected, where the video to be detected includes a visible light rotor target;

[0131] The target detection module 220 is used to input the video to be detected into a pre-trained target detection model to obtain the detection result of the visible light rotor target in the video to be detected;

[0132] Among them, the target detection model is trained through the following model training modules:

[0133] A model training module is used to obtain a training sample set, wherein the training sample set includes a first training sample and a second training sample, wherein the first training sample includes a first sample image collected at night, and the second training sample includes a second sample image collected during the day, wherein each sample image includes a visible light rotor target, and each sample image corresponds to a labeling result, and for each sample image, the labeling result characterizes the position of the visible light rotor target in the sample image; each sample image in the training sample set is input into a YOLOv5 network model, and for each sample image, a first feature map of the sample image is output through a feedforward network in the YOLOv5 network model, and a second feature map corresponding to the first sample map is output through a neck network in the YOLOv5 network model; for each sample image, upsample the second feature map to obtain a third feature map; fuse the first feature map and the third feature map to obtain a fourth feature map; perform convolution processing on the fourth feature map to obtain a fifth feature map, and determine the predicted detection result of the sample image according to the fifth feature map; obtain the total loss value of the YOLOv5 network model according to the predicted detection results and annotation results of each sample image; if the total loss value meets the preset training end condition, the YOLOv5 network model that meets the training end condition is used as the target detection model; if the total loss value does not meet the preset training end condition, adjust the model parameters of the YOLOv5 network model, and train the YOLOv5 network model according to the adjusted model parameters until the total loss value meets the preset training end condition.

[0134] Optionally, the model training module may determine the training sample set in the following manner, wherein the following manner includes:

[0135] Acquire multiple sample videos, the multiple sample videos include night sample videos and day sample videos, and the multiple sample videos correspond to multiple scenes;

[0136] Perform frame extraction on each sample video to obtain multiple frames of initial sample images;

[0137] According to the set format, multiple frames of initial sample images are stored as a data set in a set format. The set format is a txt annotation format. The data set includes multiple lines of data. Each line of data includes the category and annotation results of a frame of initial sample image. The data set is used as a training sample set.

[0138] Optionally, when the data set is used as a training sample set, the model training module is specifically used to:

[0139] According to the brightness of each frame of the initial sample image in the data set, a first training sample and a second training sample are determined from the multiple frames of the initial sample image.

[0140] Optionally, when the model training module determines the first training sample and the second training sample from multiple frames of initial sample images according to the brightness of each frame of the initial sample image in the data set, it is specifically used to:

[0141] For each frame of the initial sample image, preprocessing the initial sample image to obtain a preprocessed image, wherein the preprocessing includes at least one of scaling, cropping, arranging, splicing, brightness adjustment, hue adjustment, contrast adjustment, mirroring and rotation;

[0142] According to the brightness of each preprocessed image in each frame of the preprocessed image, a first training sample and a second training sample are determined from the plurality of frames of the preprocessed image.

[0143] Optionally, a channel space attention module is connected after the feedforward network, and the device further includes:

[0144] An attention module, for each sample image, inputting the first feature map of the sample image into the channel space attention module to obtain a sixth feature map;

[0145] For each sample image, when the model training module fuses the first feature map and the third feature map to obtain the fourth feature map, it is specifically used to:

[0146] The sixth feature map is fused with the third feature map to obtain a fourth feature map.

[0147] Optionally, the feedforward network includes a BottleneckCsp module, and the device further includes:

[0148] Replacement module, used to replace the BottleneckCsp module with the GhostBottleneck module;

[0149] For each sample image, when the model training module outputs the first feature map of the sample image through the feedforward network in the YOLOv5 network model, it is specifically used to:

[0150] The first feature map of the sample image is output through the GhostBottleneck module.

[0151] Optionally, the device further comprises:

[0152] A partitioning module is used to divide the training sample set into a training set and a test set;

[0153] The above model training module is specifically used to input each sample image in the training sample set into the YOLOv5 network model:

[0154] Input each sample image in the training set into the YOLOv5 network model;

[0155] The device also includes:

[0156] The testing module is used to test the target detection model based on the test set.

[0157] The aerial visible light rotor single target detection device of the embodiment of the present invention can execute the aerial visible light rotor single target detection method provided by the embodiment of the present invention, and the implementation principle is similar. The actions performed by each module and unit in the aerial visible light rotor single target detection device in each embodiment of the present invention correspond to the steps in the aerial visible light rotor single target detection method in each embodiment of the present invention. For the detailed functional description of each module of the aerial visible light rotor single target detection device, please refer to the description in the corresponding aerial visible light rotor single target detection method shown in the previous text, which will not be repeated here.

[0158] Among them, the above-mentioned aerial visible light rotor single target detection device can be a computer program (including program code) running in a computer device, for example, the aerial visible light rotor single target detection device is an application software; the device can be used to execute the corresponding steps in the method provided in an embodiment of the present invention.

[0159] In some embodiments, the aerial visible light rotor single target detection device provided by an embodiment of the present invention can be implemented by a combination of software and hardware. As an example, the aerial visible light rotor single target detection device provided by an embodiment of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the aerial visible light rotor single target detection method provided by an embodiment of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0160] In other embodiments, the aerial visible light rotor single target detection device provided by the embodiments of the present invention can be implemented in a software manner. Figure 3 An aerial visible light rotor single target detection device stored in a memory is shown, which can be software in the form of a program and a plug-in, and includes a series of modules, including a video acquisition module 210 and a target detection module 220, for implementing the aerial visible light rotor single target detection method provided in an embodiment of the present invention.

[0161] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a module does not limit the module itself in some cases.

[0162] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0163] In an alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0164] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0165] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0166] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0167] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0168] The electronic device may also be a terminal device. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0169] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0170] According to another aspect of the present invention, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the implementation modes of the above-mentioned various embodiments.

[0171] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0172] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0173] The computer-readable storage medium provided by the embodiment of the present invention may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0174] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0175] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

Claims

1. A method for detecting single-target rotorcraft in air with visible light, characterized in that: The following steps are involved: Acquire a video to be detected, wherein the video to be detected includes a visible light rotor target; Inputting the video to be detected into a pre-trained target detection model to obtain a detection result of the visible light rotor target in the video to be detected; The target detection model is trained in the following way: Acquire a training sample set, wherein the training sample set includes a first training sample and a second training sample, wherein the first training sample includes a first sample image acquired at night, and the second training sample includes a second sample image acquired during the day, each sample image includes a visible light rotor target, and each sample image corresponds to a labeling result, and for each sample image, the labeling result represents a position of the visible light rotor target in the sample image; Input each of the sample images in the training sample set into the YOLOv5 network model, and for each of the sample images, output a first feature map of the sample image through a feedforward network in the YOLOv5 network model, and output a second feature map corresponding to the first sample image through a neck network in the YOLOv5 network model; For each of the sample images, upsampling the second feature map to obtain a third feature map; fusing the first feature map and the third feature map to obtain a fourth feature map; performing convolution processing on the fourth feature map to obtain a fifth feature map, and determining a predicted detection result of the sample image according to the fifth feature map; Obtaining a total loss value of the YOLOv5 network model according to the predicted detection results and the labeling results of each of the sample images; If the total loss value meets the preset training end condition, the YOLOv5 network model that meets the training end condition is used as the target detection model; if the total loss value does not meet the preset training end condition, the model parameters of the YOLOv5 network model are adjusted, and the YOLOv5 network model is trained according to the adjusted model parameters until the total loss value meets the preset training end condition.

2. The method according to claim 1, characterized in that The training sample set is determined in the following way: Acquire a plurality of sample videos, wherein the plurality of sample videos include night sample videos and day sample videos, and the plurality of sample videos correspond to a plurality of scenes; Performing frame extraction processing on each of the sample videos to obtain multiple frames of initial sample images; According to the set format, multiple frames of the initial sample images are stored as a data set in the set format, and the set format is a txt annotation format. The data set includes multiple lines of data, and each line of the data includes the category and annotation results of a frame of the initial sample image. The data set is used as the training sample set.

3. The method according to claim 2, characterized in that The step of using the data set as the training sample set includes: According to the brightness of each frame of the initial sample image in the data set, the first training sample and the second training sample are determined from multiple frames of the initial sample image.

4. The method according to claim 3, characterized in that The determining the first training sample and the second training sample from a plurality of frames of the initial sample images according to the brightness of each frame of the initial sample image in the data set comprises: For each frame of the initial sample image, preprocess the initial sample image to obtain a preprocessed image, wherein the preprocessing includes at least one of scaling, cropping, arranging, splicing, brightness adjustment, hue adjustment, contrast adjustment, mirroring and rotation; According to the brightness of each frame of the preprocessed image in each frame of the preprocessed image, the first training sample and the second training sample are determined from multiple frames of the preprocessed image.

5. The method according to any one of claims 1 to 4, characterized in that The feedforward network is connected to a channel space attention module, and the method further includes: For each sample image, inputting the first feature map of the sample image into the channel space attention module to obtain a sixth feature map; For each of the sample images, fusing the first feature map and the third feature map to obtain a fourth feature map includes: The sixth feature map and the third feature map are fused to obtain a fourth feature map.

6. The method according to any one of claims 1 to 4, characterized in that The feedforward network includes a BottleneckCsp module, and the method further includes: Replace the BottleneckCsp module with the GhostBottleneck module; For each of the sample images, outputting a first feature map of the sample image through a feedforward network in the YOLOv5 network model includes: The first feature map of the sample image is outputted through the GhostBottleneck module.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Dividing the training sample set into a training set and a test set; The step of inputting each sample image in the training sample set into the YOLOv5 network model comprises: Input each of the sample images in the training set into the YOLOv5 network model; The method further comprises: The target detection model is tested according to the test set.

8. An aerial visible light rotor single target detection device, characterized in that: include: A video acquisition module, used for acquiring a video to be detected, wherein the video to be detected includes a visible light rotor target; A target detection module, used for inputting the video to be detected into a pre-trained target detection model to obtain a detection result of a visible light rotor target in the video to be detected; The target detection model is trained in the following way: Acquire a training sample set, wherein the training sample set includes a first training sample and a second training sample, wherein the first training sample includes a first sample image acquired at night, and the second training sample includes a second sample image acquired during the day, each sample image includes a visible light rotor target, and each sample image corresponds to a labeling result, and for each sample image, the labeling result represents a position of the visible light rotor target in the sample image; Input each of the sample images in the training sample set into the YOLOv5 network model, and for each of the sample images, output a first feature map of the sample image through a feedforward network in the YOLOv5 network model, and output a second feature map corresponding to the first sample image through a neck network in the YOLOv5 network model; For each of the sample images, upsampling the second feature map to obtain a third feature map; fusing the first feature map and the third feature map to obtain a fourth feature map; performing convolution processing on the fourth feature map to obtain a fifth feature map, and determining a predicted detection result of the sample image according to the fifth feature map; Obtaining a total loss value of the YOLOv5 network model according to the predicted detection results and the labeling results of each of the sample images; If the total loss value meets the preset training end condition, the YOLOv5 network model that meets the training end condition is used as the target detection model; if the total loss value does not meet the preset training end condition, the model parameters of the YOLOv5 network model are adjusted, and the YOLOv5 network model is trained according to the adjusted model parameters until the total loss value meets the preset training end condition.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images