Target detection neural network training and target detection method, device and system

Abstract

The invention discloses a target detection neural network training method, a trained target detection neural network is improved based on a YOLO v7 algorithm, and the target detection neural network comprises a neck network and four detection heads which are respectively corresponding to a first level to a fourth level of a backbone network.

Description

Technical Field

[0001] The present application belongs to the field of image processing and relates to an image processing method, device and system for detecting small targets from a high altitude perspective. Background Art

[0002] In recent years, drone technology has been widely used in many fields, including public safety, smart traffic control and environmental protection. In actual scene applications, target detection is an important part of drone functions.

[0003] With the rapid development of computer vision technology, the way of target detection has changed from human eye observation to computer detection. Existing target detection methods focus on the ground perspective, such as Fast RCNN, Yolo (You Only Look Once), etc. However, for high-altitude perspective images taken by drones, the target size is small and the features are not obvious, resulting in the inability to directly migrate the target detection algorithm from the ground perspective to the high-altitude perspective.

[0004] For small target detection from the perspective of an aircraft, domestic and foreign scholars have explored how to change the neural network to obtain more target features. For example, the feature pyramid (FPN) network structure can detect targets on feature maps of different scales; data enhancement techniques such as Mixup are used to add noise to training samples to enhance data diversity. Although these methods improve the accuracy of target detection, the network is more complex and requires more computation, which is not conducive to the implementation of the algorithm.

[0005] Therefore, it is of great value to study a small target detection method from a high-altitude bird's-eye view. Summary of the invention

[0006] This application is to solve the deficiencies in the prior art and propose a method, device and system for processing overhead images for detecting small targets from a high altitude perspective. The purpose is to address the problem that targets are small and their features are not obvious from an aircraft's high altitude perspective, and propose a method for detecting small targets from a high altitude perspective. In addition, this application is improved on the basis of the YOLO v7 algorithm, aiming to improve the accuracy of target detection while avoiding a significant increase in the amount of model calculation.

[0007] In order to achieve the above purpose, this application adopts the following technical solutions:

[0008] According to one embodiment of the present application, a target detection neural network training method is provided, characterized in that it includes: receiving multiple overhead videos; dividing the frames according to the multiple overhead videos to obtain multiple overhead images; marking the targets of the multiple overhead images to obtain a training sample set; building a target detection neural network improved based on the YOLO v7 algorithm; training the target detection neural network using the training sample set; and testing the trained target detection neural network, wherein the target detection neural network includes a neck network and four detection heads corresponding to the first to fourth levels of the backbone network respectively.

[0009] Preferably, in order to improve the accuracy of multi-scale target detection, realize the interaction of local information and global information, effectively reduce the amount of calculation and save memory space, the first to fourth levels of the backbone network include one or more Swin-Transformer modules, each of which includes a window multi-head self-attention mechanism (W-MSA) module and a sliding window multi-head self-attention mechanism (SW-MSA) module.

[0010] Preferably, in order to better learn the spatial relationship and contextual information of the image, thereby extracting rich global information and improving the detection accuracy of the model, the neck network includes a C3TR module after the output of one or more aggregation modules to replace the ELAN module of the YOLO v7 algorithm, wherein the C3TR module includes a Transformer module.

[0011] Preferably, the above-mentioned Transformer module further includes multiple multi-head self-attention mechanism (MSA) modules in sequence.

[0012] Preferably, in order to improve the loss function of the detection head, the loss function of the detection head further includes The intersection-union ratio I IoU is the intersection ratio of the predicted box and the real box, and c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the real box.

[0013] Preferably, in order to enhance the algorithm to effectively learn the difficulty of the sample, the loss function of the detection head also includes a Focal-Loss function: L Flocal-EloU =(I IoU ) γ L EloU , where γ is a hyperparameter used to control the curvature of the curve.

[0014] Preferably, in order to better train the target detection neural network, the target detection neural network training method further comprises: screening the multiple overhead images obtained after framing, and removing blurred, repeated or low-quality overhead images.

[0015] Preferably, in order to better train the target detection neural network and thereby enhance the generalization ability and robustness of the model; the learning rate is adaptively adjusted to achieve more efficient training; the model training speed is increased and the computational pressure of a single node is reduced, wherein the target detection neural network training method further comprises one of the following steps or any combination thereof: a data enhancement step, changing the expression form of the multiple overhead images in the training sample set by geometric transformation, color transformation, or noise addition; a dynamic learning rate setting step, using a smaller learning rate for preheating at the beginning of training, and adaptively adjusting the learning rate as the training progresses; and a distributed training step, dividing the multiple overhead images of the training sample set into multiple batches, training them on different computing nodes respectively, and then aggregating the gradients of each node and updating the model parameters.

[0016] Preferably, in order to better test the target detection neural network, the target detection neural network training method further includes one of the following steps or any combination thereof: an algorithm accuracy testing step, using the aforementioned prepared test sample set as input, and using accuracy (Precision), recall rate (Recall), F1 score and mAP (mean Average Precision) as evaluation indicators to test the performance of the target detection neural network in the target detection task; an algorithm real-time testing step, using the aerial video stream as input, and recording the time required for the target detection neural network to process each frame of video, that is, using the frame rate (FPS) as a real-time evaluation indicator; and an algorithm complexity testing step, testing the computational complexity, evaluating the computational complexity during training and inference, including the number of floating-point operations (FLOPs) and the number of parameters.

[0017] According to one embodiment of the present application, a method for detecting a target in a bird's-eye view image is provided, which is characterized in that it includes: receiving a bird's-eye view video stream; capturing a bird's-eye view image from the bird's-eye view video stream; and inferring the bird's-eye view image using the target neural detection network trained by the aforementioned target neural detection network training method to obtain a target frame and its related information.

[0018] According to an embodiment of the present application, a bird's-eye view image processing device is provided, which is characterized by comprising: one or more processors for executing multiple instructions stored in a non-volatile memory to implement the aforementioned target detection neural network training method.

[0019] According to an embodiment of the present application, a bird's-eye view image processing device is provided, which is characterized by comprising: one or more processors for executing multiple instructions stored in a non-volatile memory to implement the aforementioned bird's-eye view image object detection method.

[0020] According to one embodiment of the present application, a bird's-eye view image processing system is provided, characterized in that it includes: an airborne device, including a first network device for connecting to a network and a camera device for shooting a bird's-eye view video stream; and the aforementioned bird's-eye view image processing device, including: a second network device for connecting to the network and receiving the bird's-eye view video stream from the airborne device via the network; and one or more processors for executing multiple instructions stored in a non-volatile memory to implement the aforementioned bird's-eye view image target detection method.

[0021] Due to the adoption of the above scheme, the beneficial effects of the present application are as follows: the improvements proposed in the present application include one of the above four improvements, or any combination thereof: (1) adding a detection head network and a neck network corresponding to the first level; (2) replacing one or more ELAN modules in the backbone network with Swin-Transformer modules; (3) replacing one or more ELAN-W modules in the neck network with C3TR modules; and (4) improving the loss function. The first three improvements further improve the detection effect of the algorithm for small targets with unclear features by adding a detection head and improving the network structure on the basis of the traditional YOLO v7 algorithm. The last improvement enhances the stability of the algorithm training gradient descent by improving the loss function of the YOLO v7 algorithm, and enhances the algorithm's targeted training and learning of difficult and easy samples. The present application proposes a method for detecting small targets from a high altitude perspective of a drone based on an improved YOLOv7 algorithm. Aiming at the problem that small objects are difficult to detect from the perspective of a drone, the traditional algorithm is improved, the accuracy of target detection is improved, and a significant increase in the amount of model calculation is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 1 is a flow chart of a method 100 for training a target detection neural network for target recognition in a bird's-eye view video according to an embodiment of the present application.

[0023] Figure 2 A block diagram of the YOLO v7 architecture.

[0024] Figure 3 A schematic diagram of an improved target detection neural network based on the YOLO v7 algorithm according to an embodiment of the present application.

[0025] Figure 4 Schematic diagram of a block diagram of an improved target detection neural network based on the YOLO v7 algorithm according to an embodiment of the present application.

[0026] Figure 5 4 is a block diagram of a Swin-Transformer module according to an embodiment of the present application.

[0027] Figure 6 Schematic diagram of a block diagram of an improved target detection neural network based on the YOLO v7 algorithm according to an embodiment of the present application.

[0028] Figure 7 4 is a block diagram of a C3TR module according to an embodiment of the present application.

[0029] Figure 8 4 is a block diagram of a Transformer module according to an embodiment of the present application.

[0030] Fig. 9 4 is a block diagram of a bird's-eye view image processing device 900 for detecting small targets from a high altitude perspective according to an embodiment of the present application.

[0031] Fig.10 FIG. 1 is a block diagram of a overhead image processing system 1000 according to an embodiment of the present application.

[0032] Fig.11 1 is a flow chart of a method 1100 for detecting an object in a top-view image according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other implementation methods obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The terms "first", "second", "third", etc. (if any) in the specification and claims of this application and the drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the objects described in these descriptions can be interchanged where appropriate. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware circuits or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present application.

[0036] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or mutual communication; it can be a direct connection, or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0037] In order to make the purpose, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with drawings and specific implementation methods.

[0038] This application is an image processing method, device and system for high-altitude small target detection proposed to solve the deficiencies in the prior art. The purpose is to propose a high-altitude small target detection method to address the problem that the target is small and the features are not obvious under the high-altitude perspective of the aircraft. In addition, this application is improved on the basis of the YOLO v7 algorithm, aiming to improve the accuracy of target detection while avoiding a significant increase in the amount of model calculation.

[0039] A person skilled in the art should be familiar with the related technologies in the fields of computer organization, architecture, machine learning, deep neural network models, image recognition, and YOLO.

[0040] Please refer to Figure 1 FIG. 1 is a flow chart of a target detection neural network training method 100 for target recognition in a bird's-eye view video according to an embodiment of the present application. The neural network training method 100 can be used to Fig. 9 The target detection neural network training method 100 can be implemented by the calculator 900 shown in the figure, and is used to train the target detection neural network improved based on the YOLO v7 algorithm. If there is no causal relationship between any two steps, the present application does not limit the execution order of the two steps. The target detection neural network training method 100 can start from step 110.

[0041] Step 110: Obtain multiple bird's-eye view videos. These bird's-eye view video files are video files shot by a camera module carried by a manned or unmanned aircraft at a high altitude of a ground or sea target. The video may contain a large number of small target objects, such as vehicles, pedestrians, etc. In one embodiment, these video files may be uncompressed source files without losing any information. In another embodiment, these video files may be compressed files to retain high-frequency information as much as possible.

[0042] Step 120: Create a training sample set and / or a test sample set based on the plurality of overhead videos. This step may further include the following optional steps.

[0043] Video framing: Frame the drone aerial video according to a certain time interval or frame number, extract continuous image frames, and convert the video data into image data suitable for processing.

[0044] Image filtering: Filter the image frames obtained after framing to remove blurry, duplicated or low-quality images.

[0045] Target labeling. Use a labeling tool such as LabelImg to label the filtered image frames and generate training samples in Yolo format. The labeling process includes drawing a bounding box in the image to indicate the location and size of the target and giving the target's category label. The labeled image frames can be assigned to a training sample set or a test sample set.

[0046] Step 130: Build an improved target detection neural network based on the YOLO v7 algorithm. In this application, this improved target detection neural network will be described in detail.

[0047] Step 140: Train the target detection neural network using the training sample set. This step may further include the following optional steps.

[0048] Data augmentation: By changing the representation of training samples through geometric transformation, color transformation, noise addition, etc., the generalization ability and robustness of the model are enhanced.

[0049] Set a dynamic learning rate. In the early stages of training, use a smaller learning rate for warm-up. As training progresses, adjust the learning rate adaptively to achieve more efficient training.

[0050] Distributed training. The data of the training sample set is divided into multiple batches, and training is performed on different computing nodes. The gradients of each node are then aggregated and the model parameters are updated. This improves the model training speed and reduces the computing pressure of a single node.

[0051] Step 150: Testing the target detection neural network. This step may further include the following optional steps.

[0052] Algorithm accuracy test: Use the prepared test sample set as input, and use accuracy, recall, F1 score, and mAP (mean Average Precision) as evaluation indicators to test the performance of the model in the target detection task.

[0053] Algorithm real-time test: Use drone aerial video stream as input and record the time required for model processing or each frame of video, that is, use frame rate (FPS) as the real-time evaluation indicator.

[0054] Algorithm complexity test. On the one hand, it tests computational complexity, that is, evaluates the computational complexity of the model during training and inference, including the number of floating-point operations (FLOPs), the number of parameters, etc. These indicators reflect the model's demand for computing resources. On the other hand, it tests storage complexity, that is, measures the memory and disk space required by the model to store model parameters and intermediate results.

[0055] Please refer to Figure 2 As shown in FIG. 1 , it is a block diagram of the YOLO v7 architecture. In order to illustrate the improvement of the YOLO v7 algorithm in this application, Figure 2 Describe the original architecture of YOLO v7. Figure 2 The YOLO algorithm is divided into three parts: the left, middle and right. The left is the backbone network, the middle is the neck network, and the right is the detection head network. The backbone network extracts the required features from different hidden layers, the neck network is used to organize the features, and the head network is used to make decisions.

[0056] The input of the backbone network is the Feature Pyramid Network (FPN), which includes the stem layer, stage 1, stage 2, stage 3, and stage 4 in sequence. The first stage does not have a corresponding neck network and detection head network. However, the second to fourth stages each have a corresponding neck network and detection head network.

[0057] The backbone network of the second to fourth layers contains MP-1 and ELAN modules respectively. The ELAN module is an Efficient Layer Aggregation Network module that uses 1x1 and 3x3 convolution and aggregation algorithms. The MP-1 module is a Max Pool and stride convolution module. The backbone layer contains four CBS convolution modules. The first layer contains CBS convolution modules and ELAN modules.

[0058] The neck network includes the ELAN-W module, the Up Sample upsampling module, the CAT aggregation module, another maximum pooling and stride convolution (Max Pool and stride convolution) MP-2 module and the SPPCPC module. The neck network obtains input data from the second to fourth level backbone network and outputs the processed data to the three detection heads.

[0059] The ELAN-W module in the neck network contains two convolution modules, the first one has only one convolution module, and the second one has five consecutive convolution modules. There is also an aggregation module for aggregating the outputs of the above six convolution modules and outputting them to one convolution module. The SPPCSPC module in the neck network is a Spatial Pynamid Pooling with Convolutional Sparse Coding module. By introducing sparsity constraints, the model can better learn the essential characteristics of the input data. This structure improves the performance of the model while enhancing the generalization ability of the model.

[0060] Each detection head includes a reparameterization (REP) module, a CBS convolution module and a Loss loss function module. A person skilled in the art should be able to understand the implementation of each of the above modules in order to implement the traditional Yolo v7 algorithm.

[0061] In addition, the traditional Yolo v7 algorithm can also have a variety of different variations. For example, the SPPCSPC module of the neck network is a spatial pyramid pooling module, which can be replaced with an SPP spatial pyramid pooling module that does not require sparse coding convolution, or other types of SPP spatial pyramid pooling modules, such as SimSPPF, ASPP (Atrous Spatial Pyramid Pooling), RFB (Receptive Field Block), SPPCSPC (Spatial Pyramid Pooling with Convolutional Sparse Coding) and SPPFCSPC (Spatial Pyramid Pooling with Fusion of Convolutional Sparse Coding and Pooling). If it does not conflict with the improvements provided by the present application, those of ordinary skill in the art should understand that the present application can be applied to the above-mentioned variations of the Yolo v7 algorithm.

[0062] Please refer to Figure 3 , which is a schematic diagram of an improved target detection neural network based on the YOLO v7 algorithm according to an embodiment of the present application. Figure 2 Compared with the YOLO v7 algorithm, Figure 3 The target detection neural network shown in the figure adds a detection head and a corresponding neck network. The target detection neural network constructed in the above step 130 can be Figure 3 The object detection neural network shown.

[0063] Figure 3 The embodiment shown adds a detection head and a corresponding neck network specifically for detecting small targets on the basis of traditional YOLO v7. The newly added detection head and the corresponding neck network obtain data from the first level. They are similar in structure to the original detection head and neck network of YOLO v7, but are optimized for the characteristics of small targets, with a smaller receptive field and higher resolution, so as to better capture the features of small targets. On the one hand, more convolutional layers are added to better capture the detailed features of small targets. On the other hand, the purpose of multi-scale feature fusion is achieved by upsampling or downsampling the feature maps of different levels and then splicing them.

[0064] In addition, there is a neck network corresponding to the second level, and an additional ELAN-W module is added to receive the output from the newly added MP-2 module.

[0065] Please refer to Figure 4, which is a block diagram of an improved target detection neural network based on the YOLO v7 algorithm according to an embodiment of the present application. Figure 3 Compared with the YOLO v7 algorithm, Figure 4 The target detection neural network shown replaces one or more ELAN modules in the YOLO v7 backbone network with Swin-Transformer modules. Considering that the targets in the image have multiple scales from the high-altitude perspective of the drone, these targets of different scales need to be processed when using a convolutional neural network for target detection. However, since the ELAN module passes through the CNN network when processing semantic information, it is affected by its structure or training data, resulting in inaccurate problems in multi-scale target detection. Therefore, this application introduces the Swin-Transformer module.

[0066] Despite Figure 4 In the embodiment of the present invention, the four ELAN modules in the backbone network are replaced with Swin-Transformer modules. However, the present application does not limit the backbone network to only use Swin-Transformer modules and not ELAN modules. In some embodiments of the present application, ELAN modules and Swin-Transformer modules can be mixed in the backbone network.

[0067] Please refer to Figure 5 As shown, it is a block diagram of a Swin-Transformer module according to an embodiment of the present application. This Swin-Transformer module can be applied to Figure 4In the backbone network of the target detection neural network shown in Figure 1. In addition to the LayerNorm layer normalization module and the MLP multi-layer perceptron module, this Swin-Transformer module also contains two attention modules, namely the Windows Multi-Head Self-Attention (W-MSA) module and the Shifted Windows Multi-Head Self-Attention (SW-MSA) module. The W-MSA module divides the feature map into multiple non-overlapping windows, and the multi-head self-attention mechanism SW-MSA calculates the attention score for the features in each independent window. However, there is a lack of global correlation between the windows in the W-MSA module, so the SW-MSA module changes the way W-MSA divides the windows, that is, it fuses the features of multiple windows by cyclically shifting the Shift Window sliding window, and uses the Mask mechanism to fuse context information of different scales to maintain the relative position relationship of the features. Compared with the ELAN module, the Swin-Transformer module not only has the advantage of self-attention and can realize the interaction between local information and global information, but also effectively reduces the amount of calculation and saves memory space.

[0068] Those skilled in the art will appreciate that the Swin-Transformer module has a variety of variations. Regardless of which Swin-Transformer module, all have the W-MSA module and the SW-MSA module, while the original ELAN module does not have the W-MSA module and the SW-MSA module.

[0069] Please refer to Figure 6 , which is a block diagram of an improved target detection neural network based on the YOLO v7 algorithm according to an embodiment of the present application. Figure 3 and Figure 4 Compared with the YOLO v7 algorithm, Figure 6 The object detection neural network shown replaces one or more ELAN-W modules in the neck network with C3TR modules.

[0070] Please refer to Figure 7 As shown, it is a block diagram of a C3TR module according to an embodiment of the present application. This C3TR module can be applied to Figure 6 The target detection neural network shown in Figure 2 is used in the target detection neural network shown in Figure 2. Figure 8As shown, it is a block diagram of a Transformer module according to an embodiment of the present application. This Transformer module can be applied to Figure 7 The C3TR module shown.

[0071] This application combines the Multi-Head Self-Attention (MSA) mechanism in the Transformer module with the CBS convolution module to form a new C3TR module. Compared with the original module ELAN-W, the C3TR module can better learn the spatial relationship and contextual information of the image, thereby extracting rich global information and improving the detection accuracy of the model.

[0072] To go further, the advantages of C3TR modules over ELAN-W modules are:

[0073] Enhanced feature extraction capability: The C3TR module may retain the ability to capture local features in detail through its convolutional part, while introducing global dependency modeling through the Transformer module part, thereby enhancing the model's ability to understand complex scenes and targets.

[0074] Improved performance: Since the C3TR module combines the advantages of two different architectures, it may show higher accuracy and robustness when dealing with tasks such as object detection. In particular, its performance advantage may be more obvious when dealing with scenes with complex backgrounds and variable object shapes.

[0075] Adapting to more diverse datasets: The introduction of the Transformer part enables the C3TR module to better handle long-distance dependencies, which may help the model show better generalization ability on datasets with greater diversity and complexity.

[0076] In one embodiment, it can be applied to Figure 3 , Figure 4 and Figure 6 One of the improvements to the target detection neural network is to improve the loss function of YOLO v7. The loss function L in the YOLO v7 algorithm includes the coordinate loss L loc , confidence loss L conf and classification loss L class It consists of three parts, as shown in formula (1).

[0077] L=L loc +L conf +L class (1)

[0078] Among them, the confidence loss and classification loss are calculated using the BCEWithLogits Loss function, and the coordinate loss is calculated using CloU, as shown in equations (2)-(5).

[0079]

[0080]

[0081] Among them, b represents the prediction box; b gt represents the real box; w is the width of the predicted box, h is the height of the predicted box; w gt is the actual frame width, h gt is the height of the real box; c is the diagonal distance of the minimum closure area that can contain both the predicted box and the real box; a is a balance parameter; v is used to measure whether the aspect ratio is consistent; p is the distance between the center of the predicted box and the center of the real box. From formula (4), it can be seen that when the aspect ratio of the predicted box is the same as that of the real box, v takes the value of 0. At this time, the penalty term of the aspect ratio does not play a role, and the CIoU loss function cannot be stably expressed.

[0082] Therefore, this patent improves the CIoU loss function by adding distance loss and edge length loss, as shown in formula (6). The denominator of the last term in formula (6) is the square of the distance between the predicted frame height and the true frame height, the denominator of the second to last term is the square of the distance between the predicted frame width and the true frame width, and the denominator of the third to last term is the square of the distance between the predicted frame center and the true frame center.

[0083]

[0084] On the other hand, this patent introduces the idea of ​​Focal-Loss to deal with imbalanced samples, and strengthens the algorithm to effectively learn according to the difficulty of samples, as shown in formula (7).

[0085] L Flocal-EloU =(I IoU ) γ L EloU (7)

[0086] Where γ is a hyperparameter used to control the curvature of the curve.

[0087] Advantages of adding Focal-Loss to handle imbalanced samples:

[0088] 1. Solve the problem of class imbalance: Focal-Loss introduces the modulation term hyperparameter γ, and the class balance factor enables the model to dynamically adjust the loss weight of each sample during training. Specifically, for samples that are easy to classify (i.e., samples with a predicted probability p close to 1), their loss contribution will be significantly reduced; while for samples that are difficult to classify, their loss contribution is relatively large. This mechanism helps the model focus more on samples that are difficult to distinguish, thereby alleviating the problem of class imbalance.

[0089] 2. Improve model performance: Optimize the training process: Since Focal-Loss can reduce the impact of easy-to-classify samples on training and make the model more focused on learning difficult-to-classify samples, it can optimize the training process and improve the training efficiency and convergence speed of the model. Improve generalization ability: By focusing on the learning of difficult-to-classify samples, Focal-Loss helps the model learn more generalized feature representations, thereby improving the model's performance on unseen test data.

[0090] In summary, based on the Yolo v7 algorithm, the improvements proposed in this application include one of the above four improvements, or any combination thereof: (1) adding a detection head network and a neck network corresponding to the first level; (2) replacing one or more ELAN modules in the backbone network with Swin-Transformer modules; (3) replacing one or more ELAN-W modules in the neck network with C3TR modules; and (4) improving the loss function. The first three improvements further improve the detection effect of the algorithm for small targets with unclear features by adding a detection head and improving the network structure on the basis of the traditional YOLO v7 algorithm. The last improvement enhances the stability of the algorithm training gradient descent by improving the loss function of the YOLO v7 algorithm, and enhances the algorithm's targeted training and learning of difficult and easy samples. This application proposes a method for detecting small targets from a high altitude perspective of a drone based on an improved YOLO v7 algorithm. Aiming at the problem that small objects are difficult to detect from the perspective of a drone, the traditional algorithm is improved, the accuracy of target detection is improved, and a significant increase in the amount of model calculation is avoided.

[0091] Please refer to Fig. 9 As shown, it is a block diagram of a bird's-eye view image processing device 900 for detecting small targets from a high altitude perspective according to an embodiment of the present application. The bird's-eye view image processing device 900 can be an airborne device, mounted on an aviation vehicle such as a manned or unmanned aircraft, or even on a space vehicle such as a satellite, and uses a battery or power supplied by an aerospace vehicle to perform real-time operations. The bird's-eye view image processing device 900 can also be a device used on the ground to detect non-real-time videos.

[0092] The overhead image processing device 900 may include a central processing unit 910, a memory 920, an input / output interface 930, a camera module 940, a network module 950, a storage module 960, an optional coprocessor 970, an optional graphics processor 980, and an optional display 990. The input / output interface 930 may include a proprietary interface or an interface that complies with industry standards, such as an interface such as USB, SCSI, PCI, and PCI-Express, for connecting various output and input devices, such as a keyboard, a mouse, a light pen, a touch panel, a camera module 940, a network module 950, and a storage module 960.

[0093] The camera module 940 can be used to shoot the aforementioned overhead image or a video containing a small target from a high altitude perspective. The camera module 940 can transmit the video stream to the central processor 910 for processing in real time through the input / output interface 930. The storage module 960 can include a hard disk, a CD or other forms of non-volatile memory, and the content includes an operating system for controlling the overhead image processing device 900, an application for implementing the target detection neural network provided in this application, a detection method using a trained target detection neural network, and / or an application for training the above-mentioned target detection neural network. The network module 950 can be used to communicate with the ground, to transmit video streams and / or detection results, and can also be connected to interfaces of other airborne equipment. For example, the attitude and position of an aircraft vehicle are collected. Accordingly, according to the data of the above-mentioned aircraft and the lens parameters of the camera module 940, the position of the small target found can be calibrated.

[0094] The optional coprocessor 970 may include an auxiliary processing unit, a neural processing unit specifically for accelerating neural network operations, or a field programmable gate array (FPGA) specifically for implementing the target detection neural network provided by the present application. The above-mentioned application may include instructions and data applicable to the coprocessor 970.

[0095] The optional graphics processor 980 is connected to the optional display 990, and can provide the on-board operator with the ability to operate the overhead image processing device 900. In addition, the logic operation and numerical calculation (ALU) circuit unit in the optional graphics processor 980 can also assist in implementing the target detection neural network provided in the present application. The above-mentioned application may include instructions and data applicable to the optional graphics processor 980.

[0096] The overhead image processing device 900 has a central processing unit 910, which is responsible for executing an operating system. In another embodiment, the overhead image processing device 900 may be an embedded system, such as a single-chip microcomputer, embedded in other airborne devices. In some embodiments, the overhead image processing device 900 may be a system on chip (SystemOnChip), which includes multiple chips in a package to implement the high-altitude small target detection provided by the present application. This system on chip can be embedded in other airborne devices, receive videos or video streams, and output results such as detected small targets. In the aforementioned embedded system, the system on chip may be omitted. Fig. 9 Some of the modules shown, such as the camera module 940 , the graphic processor 980 , and the display 990 , may be omitted.

[0097] Fig. 9 The embodiment shown is to perform target detection on the overhead image processing device 900 mounted on the aircraft. Fig.10 , which is a block diagram of a bird's-eye view image processing system 1000 according to an embodiment of the present application. The bird's-eye view image processing system 1000 includes an airborne device 1010, a network 1020, and a processing device 1030. The airborne device 1010 can be installed in various aviation and aerospace vehicles, such as lighter-than-air spacecraft, balloons, manned or unmanned aircraft, satellites, etc. The captured bird's-eye view image, bird's-eye view video stream, and / or other navigation and positioning information can be transmitted to the ground device 1030 via various wired or wireless networks 1020.

[0098] In one embodiment, the processing device 1030 may be Fig. 9 The overhead image processing device 900 or its variant is used to implement the high-altitude small target detection provided by the present application. For example, the processing device 1030 may not have the camera module 940, but receive the video stream captured by the airborne device 1010 from the network 1020. The processing device 1030 may be set on the ground, a sea vehicle, a land vehicle or an air vehicle. The present application does not limit the location of the processing device 1030. Fig.10 In the illustrated embodiment, any distributed device that is divided into two devices 1010 and 1030 and interconnected via a network 1030 is applicable to the present invention.

[0099] Please refer to Fig.11 As shown, it is a schematic diagram of the process of the target detection method 1100 of the overhead image according to an embodiment of the present application. The target detection method 1100 may be composed of Fig. 9 The overhead image processing device 900 or Fig.10The target detection method 1100 can be implemented by the processing device 1030 shown in the figure. For example, the target detection method 1100 can be a plurality of instructions and data stored in a non-volatile memory. After the CPU 910, the coprocessor 970, and / or the GPU 980 of the overhead image processing device 900 or the processing device 1030 load and execute the plurality of instructions and data, the target detection method 1100 can be implemented. The target detection method 1100 can be used for real-time target detection or for post-target detection. The target detection method 1100 can start from step 1110.

[0100] Step 1110: Receive the overhead video stream. The overhead video stream may be received from the camera module 940 or the onboard device 1010. This step may be performed in real time or after the video is shot.

[0101] Step 1120: Capture a bird's-eye view image from the bird's-eye view video stream. Then, the process may proceed to optional step 1130, or directly proceed to step 1140.

[0102] Optional step 1130: pre-process the overhead image. This step may perform one or more image processing steps on the overhead image, such as adjusting contrast, filtering, adjusting color, etc. Screening may also be performed to remove blurred, repeated or low-quality images.

[0103] Step 1140: Use the trained target neural detection network to infer the top view image to obtain the target frame and its related information. Then, the process can end or proceed to the optional step 1150.

[0104] Optional step 1150: Calculate the target-related position according to the shooting position and azimuth information corresponding to the overhead image, and the target frame and its related information.

[0105] According to one embodiment of the present application, a target detection neural network training method is provided, characterized in that it includes: receiving multiple overhead videos; dividing the frames according to the multiple overhead videos to obtain multiple overhead images; marking the targets of the multiple overhead images to obtain a training sample set; building a target detection neural network improved based on the YOLOv7 algorithm; training the target detection neural network using the training sample set; and testing the trained target detection neural network, wherein the target detection neural network includes a neck network and four detection heads corresponding to the first to fourth levels of the backbone network respectively.

[0106] Preferably, in order to improve the accuracy of multi-scale target detection, realize the interaction of local information and global information, effectively reduce the amount of calculation and save memory space, the first to fourth levels of the backbone network include one or more Swin-Transformer modules, each of which includes a window multi-head self-attention mechanism (W-MSA) module and a sliding window multi-head self-attention mechanism (SW-MSA) module.

[0107] Preferably, in order to better learn the spatial relationship and contextual information of the image, thereby extracting rich global information and improving the detection accuracy of the model, the neck network includes a C3TR module after the output of one or more aggregation modules to replace the ELAN module of the YOLO algorithm, wherein the C3TR module includes a Transformer module.

[0108] Preferably, the above-mentioned Transformer module further includes multiple multi-head self-attention mechanism (MSA) modules in sequence.

[0109] Preferably, in order to improve the loss function of the detection head, the loss function of the detection head further includes The intersection-union ratio I IoU is the intersection ratio of the predicted box and the real box, and c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the real box.

[0110] Preferably, in order to enhance the algorithm to effectively learn the difficulty of the sample, the loss function of the detection head also includes a Focal-Loss function: L Flocal-EloU =(I IoU ) γ L EloU , where γ is a hyperparameter used to control the curvature of the curve.

[0111] Preferably, in order to better train the target detection neural network, the target detection neural network training method further comprises: screening the multiple overhead images obtained after framing, and removing blurred, repeated or low-quality overhead images.

[0112] Preferably, in order to better train the target detection neural network and thereby enhance the generalization ability and robustness of the model; the learning rate is adaptively adjusted to achieve more efficient training; the model training speed is increased and the computational pressure of a single node is reduced, wherein the target detection neural network training method further comprises one of the following steps or any combination thereof: a data enhancement step, changing the expression form of the multiple overhead images in the training sample set by geometric transformation, color transformation, or noise addition; a dynamic learning rate setting step, using a smaller learning rate for preheating at the beginning of training, and adaptively adjusting the learning rate as the training progresses; and a distributed training step, dividing the multiple overhead images of the training sample set into multiple batches, training them on different computing nodes respectively, and then aggregating the gradients of each node and updating the model parameters.

[0113] Preferably, in order to better test the target detection neural network, the target detection neural network training method further includes one of the following steps or any combination thereof: an algorithm accuracy testing step, using the aforementioned prepared test sample set as input, and using accuracy (Precision), recall rate (Recall), F1 score and mAP (mean Average Precision) as evaluation indicators to test the performance of the target detection neural network in the target detection task; an algorithm real-time testing step, using the aerial video stream as input, and recording the time required for the target detection neural network to process each frame of video, that is, using the frame rate (FPS) as a real-time evaluation indicator; and an algorithm complexity testing step, testing the computational complexity, evaluating the computational complexity during training and inference, including the number of floating-point operations (FLOPs) and the number of parameters.

[0114] According to one embodiment of the present application, a method for detecting a target in a bird's-eye view image is provided, which is characterized in that it includes: receiving a bird's-eye view video stream; capturing a bird's-eye view image from the bird's-eye view video stream; and inferring the bird's-eye view image using the target neural detection network trained by the aforementioned target neural detection network training method to obtain a target frame and its related information.

[0115] According to an embodiment of the present application, a bird's-eye view image processing device is provided, which is characterized by comprising: one or more processors for executing multiple instructions stored in a non-volatile memory to implement the aforementioned target detection neural network training method.

[0116] According to an embodiment of the present application, a bird's-eye view image processing device is provided, which is characterized by comprising: one or more processors for executing multiple instructions stored in a non-volatile memory to implement the aforementioned bird's-eye view image object detection method.

[0117] According to one embodiment of the present application, a bird's-eye view image processing system is provided, characterized in that it includes: an airborne device, including a first network device for connecting to a network and a camera device for shooting a bird's-eye view video stream; and the aforementioned bird's-eye view image processing device, including: a second network device for connecting to the network and receiving the bird's-eye view video stream from the airborne device via the network; and one or more processors for executing multiple instructions stored in a non-volatile memory to implement the aforementioned bird's-eye view image target detection method.

[0118] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes within the technical scope disclosed in the present application according to the technical solution and application concept of the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for training a target detection neural network, characterized in that: Include: Receiving multiple overhead videos; According to the multiple bird's-eye view videos, frame division is performed to obtain multiple bird's-eye view images; Annotate the multiple overhead images to obtain a training sample set; Build an improved target detection neural network based on the YOLO v7 algorithm; Using the training sample set to train the target detection neural network; as well as Test the trained target detection neural network. The target detection neural network includes a neck network and four detection heads corresponding to the first to fourth levels of the backbone network respectively.

2. The target detection neural network training method according to claim 1, characterized in that: The first to fourth levels of the backbone network include one or more Swin-Transformer modules, each of which includes a window multi-head self-attention mechanism (W-MSA) module and a sliding window multi-head self-attention mechanism (SW-MSA) module.

3. The target detection neural network training method according to claim 1, characterized in that: The neck network includes a C3TR module after the output of one or more aggregation modules to replace the ELAN module of the YOLO v7 algorithm, wherein the C3TR module includes a Transformer module.

4. The target detection neural network training method according to claim 3, characterized in that: The above-mentioned Transformer module further includes multiple multi-head self-attention mechanism (MSA) modules in sequence.

5. The target detection neural network training method according to claim 1, characterized in that: The loss function of the detection head also includes The intersection and union ratio I IoU is the intersection ratio of the predicted box and the real box, and c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the real box.

6. The target detection neural network training method according to claim 5, characterized in that: The loss function of the detection head also includes the Focal-Loss function: L Flocal-EloU =(I IoU ) γ L EloU , where γ is a hyperparameter used to control the curvature of the curve.

7. The target detection neural network training method according to claim 1, characterized in that: The method also includes screening the plurality of overhead images obtained after the frame division to remove blurred, repeated or low-quality overhead images.

8. The target detection neural network training method according to claim 1, characterized in that: It also includes one or any combination of the following steps: A data enhancement step, changing the representation of the plurality of overhead images in the training sample set by geometric transformation, color transformation, or noise addition; Set a dynamic learning rate step. In the early stage of training, use a smaller learning rate for warm-up. As the training progresses, adjust the learning rate adaptively. as well as In the distributed training step, the multiple overhead images of the training sample set are divided into multiple batches, and training is performed on different computing nodes respectively. Then, the gradients of each node are summarized and the model parameters are updated.

9. The target detection neural network training method according to claim 1, characterized in that: It also includes one or any combination of the following steps: The algorithm accuracy test step uses the prepared test sample set as input, and uses accuracy, recall, F1 score, and mAP (mean Average Precision) as evaluation indicators to test the performance of the target detection neural network in the target detection task; The algorithm real-time test step uses the aerial video stream as input and records the time required for the target detection neural network to process each frame of video, that is, the frame rate (FPS) is used as the real-time evaluation indicator; as well as Algorithm complexity test steps, test computational complexity, and evaluate the computational complexity during training and inference, including the number of floating-point operations (FLOPs) and the number of parameters.

10. A method for detecting an object in a bird's-eye view image, characterized in that: It includes: receiving a bird's-eye view video stream; capturing a bird's-eye view image from the bird's-eye view video stream; and inferring the bird's-eye view image using the target neural detection network trained by the target neural detection network training method described in any one of claims 1 to 9 to obtain a target frame and related information.

11. A bird's-eye view image processing device, characterized in that: It comprises: one or more processors for executing a plurality of instructions stored in a non-volatile memory so as to implement the target detection neural network training method according to any one of claims 1 to 9.

12. A bird's-eye view image processing device, characterized in that: The system comprises: one or more processors for executing a plurality of instructions stored in a non-volatile memory so as to implement the object detection method of the overhead image as claimed in claim 10.

13. A bird's-eye view image processing system, characterized in that: Include: The onboard device includes a first network device for connecting to a network and a camera device for capturing a bird's-eye view video stream; and The aforementioned overhead image processing device comprises: A second network device for connecting the network and receiving the overhead video stream from the onboard device via the network; and One or more processors are used to execute a plurality of instructions stored in the non-volatile memory so as to implement the object detection method of the overhead image as claimed in claim 10.

Images