A real-time and efficient target detection method based on dynamic scale selection

Abstract

The present application discloses a real-time and efficient target detection method based on dynamic scale selection, which relates to the technical field of target detection. The method comprises the following steps: obtaining a first image taken by a drone at different heights and pitch angles, marking a target bounding box and a category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form a training sample; defining a target scale factor, a drone flight height range, and a pitch angle range, and calculating a dynamic scale factor of each training sample based on the drone height and pitch angle; adjusting the anchor frame size according to the dynamic scale factor to select a feature layer, calculating a loss function according to a dynamic weight, and using the training sample to train a dynamic scale selection model based on a Faster R-CNN model; obtaining the drone height and pitch angle in real time, inputting the obtained real-time image, the drone height and pitch angle into the dynamic scale selection model, outputting a target bounding box and category, and realizing target detection.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a real-time and efficient target detection method based on dynamic scale selection. Background Art

[0002] Object detection is an important research direction in the field of computer vision, which aims to identify and locate the target object of interest from images or videos. Object detection technology has a wide range of applications in the field of drone detection and tracking. Drones are widely used in many fields due to their flexibility, maneuverability and wide field of view. Object detection technology is the key to drones to achieve environmental perception, target recognition and tracking.

[0003] Existing drone target detection methods use fixed scale selection strategies, such as fixed image pyramids or predefined sizes, which may lead to suboptimal detection performance in different scenarios, especially for targets with large scale variations. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one purpose of the present application is to propose a real-time and efficient target detection method based on dynamic scale selection, so as to achieve real-time and efficient target detection.

[0005] One aspect of the present application provides a real-time and efficient target detection method based on dynamic scale selection, comprising:

[0006] Step S100: obtaining first images taken by a drone at different heights and pitch angles, marking a target bounding box and a category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form a training sample;

[0007] Step S200: define the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculate the dynamic scale factor of each training sample based on the UAV altitude and pitch angle;

[0008] Step S300: adjusting the anchor box size selection feature layer according to the dynamic scale factor, calculating the loss function according to the dynamic weight, and training the dynamic scale selection model based on the Faster R-CNN model using the training samples;

[0009] Step S400: acquiring the altitude and pitch angle of the drone in real time, inputting the acquired real-time image, the altitude and pitch angle of the drone into the dynamic scale selection model, and outputting the target bounding box and category;

[0010] The specific method of obtaining the first images taken by the drone at different heights and pitch angles, marking the target bounding box and category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form training samples is as follows:

[0011] Step S110: In an experimental environment, using a controllable UAV platform, photographing the target object at different heights and pitch angles to obtain a first image containing the target object;

[0012] Step S120: using a rectangular box to mark the position of the target object in the first image, obtaining a target bounding box, and assigning a predefined category label to the target object, wherein the annotation information (x, y, w, h) of the target bounding box includes the position (x, y) and size information (w, h) of the target object, wherein x and y are respectively the horizontal coordinate and the vertical coordinate of the upper left corner coordinate of the target bounding box, and w and h are respectively the width and height of the target bounding box;

[0013] Step S130: Recording the altitude G and pitch angle p of the drone corresponding to the first image;

[0014] Step S140: rotating the first image at a random angle to simulate different shooting angles, scaling the first image at a random scale to simulate different target sizes, and obtaining a data-enhanced first image;

[0015] Step S150: constructing training samples based on the data-enhanced first image, each training sample including the data-enhanced first image, a corresponding target bounding box and annotation information of the target bounding box, a category label, and a height and pitch angle of the drone;

[0016] The specific method of defining the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculating the dynamic scale factor of each training sample based on the UAV altitude and pitch angle is as follows:

[0017] Step S210: Setting target detection parameters according to the actual flight range and target scale of the drone, the target detection parameters including: minimum target scale factor s min , maximum target scale factor s max 、Maximum flight altitude of the droneG max , minimum flight altitude G min , maximum pitch angle p max and the minimum pitch angle p min ;

[0018] Step S220: Calculate the difference between the maximum target scale factor and the minimum target scale factor to obtain a scale factor difference, based on the height of the drone and the maximum flight height G of the drone. max , minimum flight altitude Gmin , calculate the impact of the drone height on the scale, based on the pitch angle and the maximum pitch angle p max , minimum pitch angle p min , calculate the influence of pitch angle on scale;

[0019] Step S230: designing a mathematical model for calculating the dynamic scale factor s based on the scale factor difference, the influence of the drone height on the scale, and the influence of the pitch angle on the scale;

[0020] The calculation formula of the dynamic scale factor s is: Among them, (s max -s min ) is the scale factor difference, Indicates the impact of drone height on scale, Indicates the effect of pitch angle on scale;

[0021] Step S240: in each training sample, according to the altitude and pitch angle of the drone in the corresponding first image, the dynamic scale factor of each training sample is calculated based on the mathematical model;

[0022] The specific method of adjusting the anchor frame size selection feature layer according to the dynamic scale factor, calculating the loss function according to the dynamic weight, and training the dynamic scale selection model based on the Faster R-CNN model using the training samples is as follows:

[0023] Step S310: preset an anchor frame reference size BS, and multiply the anchor frame reference size by the dynamic scale factor to obtain an adjusted anchor frame size, wherein the calculation formula of the adjusted anchor frame size is: DS=BS×s;

[0024] Step S320: selecting a feature layer according to the size of the dynamic scale factor, the feature layer including a conv3 layer, a conv4 layer, and a conv5 layer;

[0025] The method for selecting a feature layer according to the size of the dynamic scale factor is:

[0026] When the dynamic scale factor is less than or equal to 0.5, the conv3 layer is selected;

[0027] When the dynamic scale factor is greater than 0.5 and less than or equal to 1.5, the conv4 layer is selected;

[0028] When the dynamic scale factor is greater than 1.5, the conv5 layer is selected;

[0029] Step S330: Based on the RPN loss and the detection head loss, a loss function of the Faster R-CNN model is designed. The RPN loss includes the RPN target background binary classification loss L1 rpn and RPN bounding box regression loss L2 rpn , the detection head loss includes the detection head multi-class classification loss L3 and the detection head bounding box regression loss L4, and the dynamic weight of the loss function is calculated based on the dynamic scale factor;

[0030] The dynamic weight includes the RPN target background binary classification loss weight w1 rpn , RPN bounding box regression loss weight w2 rpn , the detection head multi-category classification loss weight w3 and the detection head bounding box regression loss weight w4, where the calculation formula of the RPN target background binary classification loss weight is: The calculation formula of the RPN bounding box regression loss weight is: The calculation formula of the multi-class classification loss weight of the detection head is w3=s, and the calculation formula of the bounding box regression loss weight of the detection head is w4=s;

[0031] Step S340: using the training samples to train the dynamic scale selection model based on the Faster R-CNN model. For each training sample, the anchor box size is adjusted according to its dynamic scale factor, the feature layer is selected, and the RPN loss is calculated. According to the candidate target area predicted by the RPN, the detection head performs a RoL pooling operation on the candidate target area. The detection head classifies and regresses the feature map after RoL pooling, calculates the detection head loss, calculates the loss function based on the RPN loss and the detection head loss, and uses the back propagation algorithm to update the model parameters to minimize the loss function until the loss function converges, thereby obtaining a trained dynamic scale selection model.

[0032] The specific method of acquiring the altitude and pitch angle of the drone in real time, inputting the acquired real-time image, the altitude and pitch angle of the drone into the dynamic scale selection model, and outputting the target bounding box and category is as follows:

[0033] Step S410: deploying the trained dynamic scale selection model on the UAV platform, obtaining the altitude and pitch angle of the UAV in real time during the UAV mission, and inputting the acquired real-time image, the altitude and pitch angle of the UAV as input data into the dynamic scale selection model;

[0034] Step S420: Calculate the dynamic scale factor of the real-time image based on the altitude and pitch angle of the drone, adjust the size of the anchor frame based on the dynamic scale factor, and select a feature layer for target detection;

[0035] Step S430: The dynamic scale selection model outputs the target bounding box and category of the real-time image.

[0036] One aspect of the present application provides a real-time and efficient target detection system based on dynamic scale selection, comprising:

[0037] A training sample acquisition module is used to acquire first images taken by the drone at different heights and pitch angles, mark the target bounding box and category label on the first image, and record the drone height and pitch angle corresponding to each first image to form training samples;

[0038] The scale factor calculation module is used to define the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculate the dynamic scale factor of each training sample based on the UAV altitude and pitch angle;

[0039] The model selection training module is used to adjust the anchor box size selection feature layer according to the dynamic scale factor, calculate the loss function according to the dynamic weight, and use the training samples to train the dynamic scale selection model based on the Faster R-CNN model;

[0040] The real-time target detection module is used to obtain the drone height and pitch angle in real time, input the acquired real-time image, drone height and pitch angle into the dynamic scale selection model, and output the target bounding box and category.

[0041] One aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, steps in a real-time and efficient target detection method based on dynamic scale selection are implemented.

[0042] One aspect of the present application provides a readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor to execute steps in a real-time and efficient target detection method based on dynamic scale selection.

[0043] Compared with the prior art, the real-time and efficient target detection method based on dynamic scale selection proposed in this application has the following advantages:

[0044] Traditional methods use fixed scale selection strategies, which cannot adapt well to changes in target scale in different scenarios. This application proposes a dynamic scale selection strategy, which calculates the dynamic scale factor in real time according to the height and pitch angle of the drone, and adjusts the anchor frame size and selects the feature layer based on the dynamic scale factor. This enables the detection model to adaptively adjust the detection parameters according to changes in shooting conditions, improving the detection performance of targets of different scales.

[0045] Different feature layers have different receptive fields and semantic information. This application adaptively selects the appropriate feature layer for target detection based on the size of the dynamic scale factor. When the target scale is small, shallow features with a smaller receptive field are selected; when the target scale is large, deep features with a larger receptive field are selected. This adaptive feature selection strategy can better utilize multi-scale features and improve detection performance.

[0046] This application introduces dynamic weights into the loss function of Faster R-CNN, including RPN target background binary classification loss weight, RPN bounding box regression loss weight, detection head multi-category classification loss weight, and detection head bounding box regression loss weight. These weights are calculated based on dynamic scale factors, and can dynamically adjust the importance of different loss items according to changes in target scale, making the loss function more reasonable. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A method flow chart of a real-time and efficient target detection method based on dynamic scale selection provided in this application;

[0048] Figure 2 A flow chart of the dynamic scale factor calculation method provided for this application;

[0049] Figure 3 A flow chart of a method for selecting a feature layer according to the size of a dynamic scale factor provided in this application;

[0050] Figure 4 A functional module diagram of a real-time and efficient target detection system based on dynamic scale selection provided by this application;

[0051] Figure 5 A schematic diagram of the structure of an electronic device provided in this application;

[0052] Figure 6 It is a schematic diagram of the structure of a readable storage medium provided by this application. DETAILED DESCRIPTION

[0053] In order to better understand the present application, various aspects of the present application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application and do not limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0054] In the accompanying drawings, the size, dimensions and shapes of the elements have been slightly adjusted for ease of illustration. The drawings are for illustration only and are not strictly drawn to scale. As used herein, the terms "substantially", "approximately" and similar terms are used as terms of approximation, not as terms of degree, and are intended to illustrate the inherent deviations in measurements or calculations that will be recognized by those of ordinary skill in the art. In addition, in this application, the order in which the steps are described does not necessarily represent the order in which these processes occur in actual operation, unless otherwise specified or can be derived from the context.

[0055] It should also be understood that expressions such as "including", "comprising", "having", "containing" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features rather than just the individual elements in the list. In addition, when describing embodiments of the present application, "may" is used to mean "one or more embodiments of the present application". And, the term "exemplary" is intended to refer to an example or illustration.

[0056] Unless otherwise specified, all words (including engineering terms and scientific and technological terms) used in this article have the same meaning as those commonly understood by ordinary technicians in the field to which this application belongs. It should also be understood that, unless clearly stated in this application, words defined in common dictionaries should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense.

[0057] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0058] Example 1

[0059] like Figure 1 As shown, a real-time and efficient target detection method based on dynamic scale selection provided by the present application includes:

[0060] Step S100: obtaining first images taken by a drone at different heights and pitch angles, marking a target bounding box and a category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form a training sample;

[0061] The specific method of obtaining the first images taken by the drone at different heights and pitch angles, marking the target bounding box and category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form training samples is as follows:

[0062] Step S110: In an experimental environment, using a controllable UAV platform, photographing the target object at different heights and pitch angles to obtain a first image containing the target object;

[0063] Step S120: using a rectangular box to mark the position of the target object in the first image, obtaining a target bounding box, and assigning a predefined category label to the target object, wherein the annotation information (x, y, w, h) of the target bounding box includes the position (x, y) and size information (w, h) of the target object, wherein x and y are respectively the horizontal coordinate and the vertical coordinate of the upper left corner coordinate of the target bounding box, and w and h are respectively the width and height of the target bounding box;

[0064] Step S130: Recording the altitude G and pitch angle p of the drone corresponding to the first image;

[0065] Step S140: rotating the first image at a random angle to simulate different shooting angles, scaling the first image at a random scale to simulate different target sizes, and obtaining a data-enhanced first image;

[0066] Step S150: constructing training samples based on the data-enhanced first image, each training sample including the data-enhanced first image, a corresponding target bounding box and annotation information of the target bounding box, a category label, and a height and pitch angle of the drone;

[0067] Step S200: define the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculate the dynamic scale factor of each training sample based on the UAV altitude and pitch angle;

[0068] The specific method of defining the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculating the dynamic scale factor of each training sample based on the UAV altitude and pitch angle is as follows:

[0069] Step S210: Setting target detection parameters according to the actual flight range and target scale of the drone, the target detection parameters including: minimum target scale factor s min , maximum target scale factor s max 、Maximum flight altitude of the droneG max , minimum flight altitude G min , maximum pitch angle p max and the minimum pitch angle p min ;

[0070] The minimum target scale factor represents the scale factor corresponding to the minimum size of the target object in the image when the drone is at the maximum flight altitude. The value of the minimum target scale factor is estimated and set according to the minimum size of the target object in the training sample and the maximum flight altitude of the drone.

[0071] The maximum target scale factor represents the scale factor corresponding to the maximum size of the target object in the image when the drone is at the minimum flight altitude. The value of the maximum target scale factor is estimated and set according to the maximum size of the target object in the training sample and the minimum flight altitude of the drone.

[0072] The maximum flight altitude, minimum flight altitude, maximum pitch angle and minimum pitch angle are set by those skilled in the art according to unmanned mission requirements;

[0073] Step S220: Calculate the difference between the maximum target scale factor and the minimum target scale factor to obtain a scale factor difference, based on the height of the drone and the maximum flight height G of the drone. max , minimum flight altitude G min , calculate the impact of the drone height on the scale, based on the pitch angle and the maximum pitch angle p max , minimum pitch angle p min , calculate the influence of pitch angle on scale;

[0074] Step S230: designing a mathematical model for calculating the dynamic scale factor s based on the scale factor difference, the influence of the drone height on the scale, and the influence of the pitch angle on the scale;

[0075] The calculation formula of the dynamic scale factor s is: Among them, (s max -s min ) is the scale factor difference, Indicates the impact of drone height on scale, Indicates the effect of pitch angle on scale;

[0076] Step S240: in each training sample, according to the altitude and pitch angle of the drone in the corresponding first image, the dynamic scale factor of each training sample is calculated based on the mathematical model;

[0077] Figure 2 This is a flow chart of the dynamic scale factor calculation method provided in this application.

[0078] Step S300: adjusting the anchor box size selection feature layer according to the dynamic scale factor, calculating the loss function according to the dynamic weight, and training the dynamic scale selection model based on the Faster R-CNN model using the training samples;

[0079] The input data of the dynamic scale selection model based on the Faster R-CNN model are the image taken by the drone, the altitude and pitch angle of the drone when the drone is taking the photo, and the output data are the target bounding box and the category of the target object in the image taken by the drone;

[0080] The specific method of adjusting the anchor frame size selection feature layer according to the dynamic scale factor, calculating the loss function according to the dynamic weight, and training the dynamic scale selection model based on the Faster R-CNN model using the training samples is as follows:

[0081] Step S310: preset an anchor frame reference size BS, and multiply the anchor frame reference size by the dynamic scale factor to obtain an adjusted anchor frame size, wherein the calculation formula of the adjusted anchor frame size is: DS=BS×s;

[0082] When the drone is at a high altitude and the pitch angle is small, the scale of the target object in the image is small, so the anchor frame size needs to be reduced accordingly; conversely, when the drone is at a low altitude and the pitch angle is large, the scale of the target object in the image is large, and the anchor frame size needs to be increased accordingly.

[0083] Step S320: selecting a feature layer according to the size of the dynamic scale factor, the feature layer including a conv3 layer, a conv4 layer, and a conv5 layer;

[0084] Figure 3 A flow chart of a method for selecting a feature layer according to the size of a dynamic scale factor provided in this application;

[0085] The method for selecting a feature layer according to the size of the dynamic scale factor is:

[0086] When the dynamic scale factor is less than or equal to 0.5, the conv3 layer is selected;

[0087] When the dynamic scale factor is greater than 0.5 and less than or equal to 1.5, the conv4 layer is selected;

[0088] When the dynamic scale factor is greater than 1.5, the conv5 layer is selected;

[0089] The principle of the method for selecting a feature layer according to the size of the dynamic scale factor is: when the scale of the target object is small, that is, the dynamic scale factor s is small, the conv3 layer with a smaller receptive field is selected to better capture the features of the small target; when the scale of the target object is moderate, that is, s is close to 1, the conv4 layer with a moderate receptive field is selected to balance the detection accuracy and efficiency; when the scale of the target object is large, that is, s is large, the conv5 layer with a larger receptive field is selected to better cover the area of ​​the large target.

[0090] Step S330: Based on the RPN loss and the detection head loss, a loss function of the Faster R-CNN model is designed. The RPN loss includes the RPN target background binary classification loss L1 rpn and RPN bounding box regression loss L2 rpn , the detection head loss includes the detection head multi-class classification loss L3 and the detection head bounding box regression loss L4, and the dynamic weight of the loss function is calculated based on the dynamic scale factor;

[0091] The dynamic weight includes the RPN target background binary classification loss weight w1 rpn , RPN bounding box regression loss weight w2 rpn , the detection head multi-category classification loss weight w3 and the detection head bounding box regression loss weight w4;

[0092] The calculation formula of the RPN target background binary classification loss weight is:

[0093] The calculation formula of the RPN bounding box regression loss weight is:

[0094] The calculation formula of the multi-category classification loss weight of the detection head is: w3=s;

[0095] The calculation formula of the detection head bounding box regression loss weight is: w4=s;

[0096] The calculation formula of the loss function of the Faster R-CNN model is: Loss = w1 rpn ×L1 rpn +w2 rpn ×L2 rpn +w3×L3+w4×L4;

[0097] The calculation formula of the RPN target background binary classification loss is: Among them, N1 represents the number of anchor boxes generated by RPN at each position of the feature map, and y i is the true label of the i-th anchor frame, which refers to the true label of the target background binary classification of the i-th anchor frame, P i is the probability that the i-th anchor box is predicted as a target;

[0098] The calculation formula of the RPN bounding box regression loss is: Among them, t i is the predicted bounding box regression value of the i-th anchor box, is the true target bounding box regression value of the i-th anchor box, smoothL1(·) refers to the smooth L1 loss function;

[0099] The predicted bounding box regression value of the i-th anchor box includes the center coordinate offset and width and height scaling of the predicted anchor box;

[0100] The true target bounding box regression value of the i-th anchor box includes the true center coordinate offset and width and height scaling;

[0101] The calculation formula of the multi-category classification loss of the detection head is: Among them, N2 is the number of candidate target areas, y jc is the true category label of the jth candidate target area. When the category label is c, y jc is equal to 1, otherwise it is equal to 0, P jc is the probability that the category label of the jth candidate target area is predicted to be c;

[0102] The calculation formula of the detection head bounding box regression loss is: Among them, t jc is the predicted bounding box regression value of the jth candidate target region corresponding to the category label c, is the true target bounding box regression value corresponding to the category label c of the jth candidate target area;

[0103] The predicted bounding box regression value and the true target bounding box regression value are converted according to the annotation information (x, y, w, h) of the target bounding box and the width and height of the anchor box and the candidate area;

[0104] Step S340: using the training samples to train the dynamic scale selection model based on the Faster R-CNN model. For each training sample, the anchor box size is adjusted according to its dynamic scale factor, the feature layer is selected, and the RPN loss is calculated. According to the candidate target area predicted by the RPN, the detection head performs a RoL pooling operation on the candidate target area. The detection head classifies and regresses the feature map after RoL pooling, calculates the detection head loss, calculates the loss function based on the RPN loss and the detection head loss, and uses the back propagation algorithm to update the model parameters to minimize the loss function until the loss function converges, thereby obtaining a trained dynamic scale selection model.

[0105] The Faster R-CNN model is mainly composed of a feature extraction network, a region proposal network RPN and a detection head;

[0106] RPN slides a small convolutional network on the feature map, generates multiple anchor boxes for each position, and performs target background classification and bounding box regression for each anchor box. The output of RPN is a set of candidate target regions, indicating the area that may contain the target;

[0107] The detection head receives the candidate target regions generated by RPN, classifies and regresses the bounding box for each candidate target region, and obtains the final detection result.

[0108] Among them, the feature extraction network adopts a convolutional neural network, which includes multiple convolutional layers and pooling layers. The feature layers conv3, conv4, and conv5 are the three convolutional layers in the convolutional neural network.

[0109] Step S400: acquiring the altitude and pitch angle of the drone in real time, inputting the acquired real-time image, the altitude and pitch angle of the drone into the dynamic scale selection model, and outputting the target bounding box and category;

[0110] The specific method of acquiring the altitude and pitch angle of the drone in real time, inputting the acquired real-time image, the altitude and pitch angle of the drone into the dynamic scale selection model, and outputting the target bounding box and category is as follows:

[0111] Step S410: deploying the trained dynamic scale selection model on the UAV platform, obtaining the altitude and pitch angle of the UAV in real time during the UAV mission, and inputting the acquired real-time image, the altitude and pitch angle of the UAV as input data into the dynamic scale selection model;

[0112] Step S420: Calculate the dynamic scale factor of the real-time image based on the altitude and pitch angle of the drone, adjust the size of the anchor frame based on the dynamic scale factor, and select a feature layer for target detection;

[0113] Step S430: The dynamic scale selection model outputs the target bounding box and category of the real-time image.

[0114] Example 2

[0115] like Figure 4 As shown, a real-time and efficient target detection system based on dynamic scale selection provided by the present application includes:

[0116] A training sample acquisition module is used to acquire first images taken by the drone at different heights and pitch angles, mark the target bounding box and category label on the first image, and record the drone height and pitch angle corresponding to each first image to form training samples;

[0117] The scale factor calculation module is used to define the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculate the dynamic scale factor of each training sample based on the UAV altitude and pitch angle;

[0118] The model selection training module is used to adjust the anchor box size selection feature layer according to the dynamic scale factor, calculate the loss function according to the dynamic weight, and use the training samples to train the dynamic scale selection model based on the Faster R-CNN model;

[0119] The real-time target detection module is used to obtain the drone height and pitch angle in real time, input the acquired real-time image, drone height and pitch angle into the dynamic scale selection model, and output the target bounding box and category.

[0120] Example 3

[0121] Figure 5 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 5 As shown, according to another aspect of the present application, an electronic device is also provided. The electronic device may include one or more processors and one or more memories. The memory stores a computer readable code, and when the computer readable code is executed by one or more processors, it can execute a real-time and efficient target detection method based on dynamic scale selection as described above.

[0122] The method or system according to the embodiment of the present application can also be used by Figure 5 The electronic device architecture shown in FIG. Figure 5 As shown, the electronic device may include a bus, one or more CPUs, a read-only memory (ROM), a random access memory (RAM), a communication port connected to a network, an input / output component, a hard disk, and the like. A storage device in the electronic device, such as a ROM or a hard disk, may store a real-time and efficient target detection method based on dynamic scale selection provided in the present application. A real-time and efficient target detection method based on dynamic scale selection may, for example, include: acquiring a first image taken by a drone at different heights and pitch angles, marking a target bounding box and a category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form a training sample; defining a target scale factor, a drone flight height range, and a pitch angle range, and calculating a dynamic scale factor for each training sample based on the drone height and pitch angle; adjusting the anchor frame size according to the dynamic scale factor to select a feature layer, calculating a loss function according to a dynamic weight, and using the training sample to train a dynamic scale selection model based on the Faster R-CNN model; acquiring the drone height and pitch angle in real time, inputting the acquired real-time image, drone height, and pitch angle into the dynamic scale selection model, and outputting a target bounding box and category. Furthermore, the electronic device may also include a user interface. Of course, Figure 5 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 5 One or more components of an electronic device are shown.

[0123] Example 4

[0124] Figure 6 Schematic diagram of a readable storage medium structure provided by an embodiment of the present application. Figure 6 As shown, it is a readable storage medium according to an embodiment of the present application. Computer-readable instructions are stored on the computer-readable storage medium. When the computer-readable instructions are executed by the processor, a real-time and efficient target detection method based on dynamic scale selection according to an embodiment of the present application described with reference to the above figures can be executed. The storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0125] In addition, according to the implementation of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided in the present application, for example: obtaining a first image taken by a drone at different heights and pitch angles, marking a target bounding box and a category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form a training sample; defining a target scale factor, a drone flight height range, and a pitch angle range, and calculating a dynamic scale factor for each training sample based on the drone height and pitch angle; adjusting the anchor box size selection feature layer according to the dynamic scale factor, calculating a loss function according to a dynamic weight, and using the training sample to train a dynamic scale selection model based on a Faster R-CNN model; obtaining the drone height and pitch angle in real time, inputting the acquired real-time image, drone height and pitch angle into the dynamic scale selection model, and outputting the target bounding box and category. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0126] The methods, apparatuses, and devices of the present application may be implemented in many ways. For example, the methods, apparatuses, and devices of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present application may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers recording media storing programs for executing the method according to the present application.

[0127] In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0128] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time and efficient target detection method based on dynamic scale selection, characterized in that: include: Step S100: obtaining first images taken by a drone at different heights and pitch angles, marking a target bounding box and a category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form a training sample; Step S200: define the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculate the dynamic scale factor of each training sample based on the UAV altitude and pitch angle; Step S300: adjusting the anchor box size selection feature layer according to the dynamic scale factor, calculating the loss function according to the dynamic weight, and training the dynamic scale selection model based on the Faster R-CNN model using the training samples; Step S400: Acquire the altitude and pitch angle of the drone in real time, input the acquired real-time image, the altitude and pitch angle of the drone into the dynamic scale selection model, and output the target bounding box and category.

2. A real-time and efficient target detection method based on dynamic scale selection as claimed in claim 1, characterized in that: The specific method of obtaining the first images taken by the drone at different heights and pitch angles, marking the target bounding box and category label on the first image, and recording the drone height and pitch angle corresponding to each first image to form training samples is as follows: Step S110: In an experimental environment, using a controllable UAV platform, photographing the target object at different heights and pitch angles to obtain a first image containing the target object; Step S120: using a rectangular box to mark the position of the target object in the first image, obtaining a target bounding box, and assigning a predefined category label to the target object, wherein the annotation information (x, y, w, h) of the target bounding box includes the position (x, y) and size information (w, h) of the target object, wherein x and y are respectively the horizontal coordinate and the vertical coordinate of the upper left corner coordinate of the target bounding box, and w and h are respectively the width and height of the target bounding box; Step S130: Recording the altitude G and pitch angle p of the drone corresponding to the first image; Step S140: rotating the first image at a random angle to simulate different shooting angles, scaling the first image at a random scale to simulate different target sizes, and obtaining a data-enhanced first image; Step S150: construct training samples based on the data-enhanced first image, each training sample includes the data-enhanced first image, the corresponding target bounding box and the annotation information of the target bounding box, the category label, the drone height and the pitch angle.

3. A real-time and efficient target detection method based on dynamic scale selection as claimed in claim 2, characterized in that: The specific method of defining the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculating the dynamic scale factor of each training sample based on the UAV altitude and pitch angle is as follows: Step S210: Setting target detection parameters according to the actual flight range and target scale of the drone, the target detection parameters including: minimum target scale factor s min , maximum target scale factor s max 、Maximum flight altitude of the droneG max , minimum flight altitude G min , maximum pitch angle p max and the minimum pitch angle p min ; Step S220: Calculate the difference between the maximum target scale factor and the minimum target scale factor to obtain a scale factor difference, based on the height of the drone and the maximum flight height G of the drone. max , minimum flight altitude G min , calculate the impact of the drone height on the scale, based on the pitch angle and the maximum pitch angle p max , minimum pitch angle p min , calculate the influence of pitch angle on scale; Step S230: designing a mathematical model for calculating the dynamic scale factor s based on the scale factor difference, the influence of the drone height on the scale, and the influence of the pitch angle on the scale; The calculation formula of the dynamic scale factor s is: Among them, (s max -s min ) is the scale factor difference, Indicates the impact of drone height on scale, Indicates the effect of pitch angle on scale; Step S240: In each training sample, the dynamic scale factor of each training sample is calculated based on the mathematical model according to the altitude and pitch angle of the drone in the corresponding first image.

4. A real-time and efficient target detection method based on dynamic scale selection as claimed in claim 3, characterized in that: The specific method of adjusting the anchor frame size selection feature layer according to the dynamic scale factor, calculating the loss function according to the dynamic weight, and training the dynamic scale selection model based on the Faster R-CNN model using the training samples is as follows: Step S310: preset an anchor frame reference size BS, and multiply the anchor frame reference size by the dynamic scale factor to obtain an adjusted anchor frame size, wherein the calculation formula of the adjusted anchor frame size is: DS=BS×s; Step S320: selecting a feature layer according to the size of the dynamic scale factor, the feature layer including a conv3 layer, a conv4 layer, and a conv5 layer; Step S330: Based on the RPN loss and the detection head loss, a loss function of the Faster R-CNN model is designed. The RPN loss includes the RPN target background binary classification loss L1 rpn and RPN bounding box regression loss L2 rpn , the detection head loss includes the detection head multi-class classification loss L3 and the detection head bounding box regression loss L4, and the dynamic weight of the loss function is calculated based on the dynamic scale factor; Step S340: Use the training samples to train the dynamic scale selection model based on the Faster R-CNN model. For each training sample, adjust the anchor box size according to its dynamic scale factor, select the feature layer, and calculate the RPN loss. According to the candidate target area predicted by the RPN, the detection head performs a RoL pooling operation on the candidate target area. The detection head classifies and regresses the feature map after RoL pooling, calculates the detection head loss, calculates the loss function based on the RPN loss and the detection head loss, and uses the back propagation algorithm to update the model parameters to minimize the loss function until the loss function converges to obtain a trained dynamic scale selection model.

5. A real-time and efficient target detection method based on dynamic scale selection as claimed in claim 4, characterized in that: The method for selecting a feature layer according to the size of the dynamic scale factor is: When the dynamic scale factor is less than or equal to 0.5, the conv3 layer is selected; When the dynamic scale factor is greater than 0.5 and less than or equal to 1.5, the conv4 layer is selected; When the dynamic scaling factor is greater than 1.5, the conv5 layer is selected.

6. A real-time and efficient target detection method based on dynamic scale selection as claimed in claim 5, characterized in that: The dynamic weight includes the RPN target background binary classification loss weight w1 rpn , RPN bounding box regression loss weight w2 rpn , the detection head multi-category classification loss weight w3 and the detection head bounding box regression loss weight w4, where the calculation formula of the RPN target background binary classification loss weight is: The calculation formula of the RPN bounding box regression loss weight is: The calculation formula for the multi-category classification loss weight of the detection head is w3=s, and the calculation formula for the bounding box regression loss weight of the detection head is w4=s.

7. A real-time and efficient target detection method based on dynamic scale selection as claimed in claim 6, characterized in that: The specific method of acquiring the altitude and pitch angle of the drone in real time, inputting the acquired real-time image, the altitude and pitch angle of the drone into the dynamic scale selection model, and outputting the target bounding box and category is as follows: Step S410: deploying the trained dynamic scale selection model on the UAV platform, obtaining the altitude and pitch angle of the UAV in real time during the UAV mission, and inputting the acquired real-time image, the altitude and pitch angle of the UAV as input data into the dynamic scale selection model; Step S420: Calculate the dynamic scale factor of the real-time image based on the altitude and pitch angle of the drone, adjust the size of the anchor frame based on the dynamic scale factor, and select a feature layer for target detection; Step S430: The dynamic scale selection model outputs the target bounding box and category of the real-time image.

8. A real-time and efficient target detection system based on dynamic scale selection, which is implemented based on a real-time and efficient target detection method based on dynamic scale selection according to any one of claims 1 to 7, characterized in that: include: A training sample acquisition module is used to acquire first images taken by the drone at different heights and pitch angles, mark the target bounding box and category label on the first image, and record the drone height and pitch angle corresponding to each first image to form training samples; The scale factor calculation module is used to define the target scale factor, the UAV flight altitude range, and the pitch angle range, and calculate the dynamic scale factor of each training sample based on the UAV altitude and pitch angle; The model selection training module is used to adjust the anchor box size selection feature layer according to the dynamic scale factor, calculate the loss function according to the dynamic weight, and use the training samples to train the dynamic scale selection model based on the Faster R-CNN model; The real-time target detection module is used to obtain the drone height and pitch angle in real time, input the acquired real-time image, drone height and pitch angle into the dynamic scale selection model, and output the target bounding box and category.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the real-time and efficient target detection method based on dynamic scale selection as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in a real-time and efficient target detection method based on dynamic scale selection as described in any one of claims 1 to 7.

Images