Target detection method, device and storage medium based on adaptive anchor alignment

Abstract

The present invention belongs to the technical field of image target detection and relates to a target detection method, device, and storage medium based on adaptive anchor alignment. The method comprises inputting an image to be detected into a feature extraction network, which outputs a positioning feature map and a classification feature map. The positioning feature map and the classification feature map are then input into an adaptive anchor alignment network, which outputs a target positioning feature map, a target classification feature map, and a target bias feature map. The adaptive anchor alignment network comprises N serially connected adaptive alignment modules, which generate adaptive anchors based on the target bias feature map and preset uniformly distributed anchors. The adaptive anchors, the target positioning feature map, and the target classification feature map are then input into a decoder to obtain predicted positioning boxes and predicted categories for each target in the image to be detected. This solution utilizes an adaptive anchor alignment network to generate adaptive anchors that converge to the target region, thereby improving target detection accuracy and being more applicable to various target classification and target location tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of image target detection, and in particular to a target detection method, device and computer-readable storage medium based on adaptive anchor alignment. Background Art

[0002] Object detection in natural images is an important means of acquiring image information. Object detection allows for the location and classification of objects within an image. Currently, object detection technology plays a vital role in a variety of fields. For example, in intelligent transportation and autonomous driving, object detection on road images allows for the identification and location of intelligent vehicles, pedestrians, and obstacles within the images. This allows for the calculation of distances between objects, enabling intelligent vehicles to plan driving routes and control them to avoid obstacles and pedestrians, thereby reducing accident rates. In biodiversity research, fine-grained recognition of images containing multispecies information allows for accurate statistical analysis of the distribution characteristics and abundance of different species within an image, providing critical data for biodiversity conservation and ecosystem research.

[0003] Existing object detection methods use a preset uniform anchor box mechanism to locate objects. First, image features are extracted to obtain a classification feature map containing the semantic information of each object and a localization feature map containing the positional information of each object. Multiple sets of fixed-size, uniformly distributed anchor boxes are then generated on the classification and localization feature maps to cover any objects that may appear in the image. Predictive classification is then performed on each anchor box in the classification and localization feature maps containing the anchor boxes to determine whether each anchor box contains the object and the object category it contains. Finally, the anchor box containing the object and the object category are output, resulting in the predicted target category and predicted localization box. However, since the anchor boxes are all of a preset fixed size, this can cause the anchor boxes to mismatch the shape and size of some real objects in the image. For example, when detecting a slender bridge in a drone image, the preset 2:1 ratio anchor box cannot effectively cover the slender bridge, resulting in missed detections or positioning errors. Furthermore, the uniformly distributed anchor boxes are strictly aligned with the feature map grid points, while the center of the target's true localization box may be located in the grid gaps. This results in a natural offset between the initial position of the anchor box and the true localization box, resulting in a large error in the final positioning of the object, reducing the accuracy of object detection.

[0004] In summary, existing target detection methods have the problem of low target detection accuracy, which affects the accuracy of various target classification and target positioning tasks. Summary of the Invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem that the target detection method in the prior art has low target detection accuracy, thereby affecting the accuracy of various target classification and target positioning tasks.

[0006] To solve the above technical problems, the present invention provides a target detection method based on adaptive anchor alignment, comprising:

[0007] Input the image to be detected into the feature extraction network and output the positioning feature map and classification feature map;

[0008] The positioning feature map and the classification feature map are input into the adaptive anchor alignment network, where the adaptive anchor alignment network includes N adaptive alignment modules connected in series. Each adaptive alignment module adaptively aligns the input feature map including:

[0009] Perform standard convolution on the input classification feature map and output the offset; obtain the bias feature map based on the offset and the preset hole prior;

[0010] Perform deformable convolution on the bias feature map and the input classification feature map at the same time, and output the classification feature map to the next adaptive alignment module;

[0011] Perform deformable convolution on the input positioning feature map and offset at the same time, and output the positioning feature map to the next adaptive alignment module;

[0012] The classification feature map, positioning feature map, and bias feature map output by the last adaptive alignment module are used as the target classification feature map, target positioning feature map, and target bias feature map, respectively; adaptive anchors are generated based on the target bias feature map and the preset uniform distribution anchors;

[0013] The adaptive anchor, target positioning feature map and target classification feature map are input into the decoder to obtain the predicted positioning box and predicted category of each target in the image to be detected.

[0014] Preferably, each adaptive alignment module performs standard convolution on the input classification feature map, and the output offset includes: performing standard convolution on the classification feature map using a first convolution unit and a second convolution unit with different convolution kernel sizes connected in series, and obtaining the offset based on the output of the second convolution unit.

[0015] Preferably, the offset output by each adaptive alignment module is expressed as:

[0016] ,

[0017] in, Indicates the The offset output by the adaptive alignment module, represents the number of adaptive alignment modules; represents a standard convolution unit for performing standard convolution; Indicates input The classification feature map of the adaptive alignment module, i.e. The classification feature map output by the adaptive alignment module is hour, The classification feature map output by the feature extraction network;

[0018] The bias feature map obtained by each adaptive alignment module is expressed as:

[0019] ,

[0020] in, Indicates the The bias feature map obtained by the adaptive alignment module; Indicates the preset empty prior;

[0021] The classification feature map output by each adaptive alignment module is expressed as:

[0022] ,

[0023] in, Indicates the The classification feature map output by the adaptive alignment module; represents a deformable convolution unit that performs deformable convolution;

[0024] The positioning feature map output by each adaptive alignment module is expressed as:

[0025] ,

[0026] in, Indicates the The positioning feature map output by the adaptive alignment module; Indicates input The positioning feature map of the adaptive alignment module, i.e. The positioning feature map output by the adaptive alignment module is hour, It is the positioning feature map output by the feature extraction network.

[0027] Preferably, inputting the image to be detected into a feature extraction network and outputting a positioning feature map and a classification feature map comprises:

[0028] The image to be detected is input into the encoder for multi-level feature extraction, and a multi-scale feature map is output;

[0029] The multi-scale feature maps are input into the parallel classification feature extraction network and positioning feature extraction network respectively, and the classification feature maps and positioning feature maps are output.

[0030] Preferably, generating an adaptive anchor based on a target bias feature map and a preset uniform distribution anchor includes: inputting the target bias feature map and the preset uniform distribution anchor into an alignment anchor generator, and outputting an adaptive anchor; wherein the adaptive anchor is represented as:

[0031] ,

[0032] in, represents the adaptive anchor; Indicates pixel-by-pixel addition; A list of anchor points representing preset evenly distributed anchors; represents the downsampling rate of each scale feature map in the multi-scale feature map, , Indicates the number of scale feature maps; Represents the average operator.

[0033] Preferably, when training the feature extraction network, the adaptive anchor alignment network, and the decoder using images in the training set, the process of constructing the object detection loss function includes:

[0034] The sample allocation strategy paradigm is used to divide each target in the image based on its predicted positioning box and predicted category, and obtain positive target sample set and negative target sample set;

[0035] Construct a classification loss function based on the distance measurement between the predicted category of each target in the positive target sample set and the negative target sample set and the target classification feature map of the image;

[0036] Construct a positioning loss function based on the distance measurement between the predicted positioning box of each target in the positive target sample set and its true positioning box;

[0037] Based on the weighted sum of the classification loss function and the positioning loss function, the target detection loss function is constructed.

[0038] Preferably, the classification loss function is expressed as:

[0039] ,

[0040] in, represents the classification loss function; represents a function for calculating the distance metric between the predicted category of each target in the positive target sample set and the negative target sample set and the target classification feature map of the image; represents the positive target sample set; represents the negative target sample set; Represents the target classification feature map;

[0041] The positioning loss function is expressed as:

[0042] ,

[0043] in, represents the positioning loss function; Represents a function for calculating the distance metric between the predicted positioning box of each target in the positive target sample set and its true positioning box; Represents the predicted positioning box; Represents the real positioning frame; Indicates filtering;

[0044] The target detection loss function is expressed as:

[0045] ,

[0046] in, represents the target detection loss function; Represents the weight of the positioning loss function.

[0047] Preferably, the adaptive anchor alignment network includes three adaptive alignment modules connected in series.

[0048] The present invention also provides a target detection device based on adaptive anchor alignment, comprising:

[0049] The feature extraction module is used to input the image to be detected into the feature extraction network and output the positioning feature map and the classification feature map;

[0050] The adaptive alignment module is used to input the positioning feature map and the classification feature map into the adaptive anchor alignment network, wherein the adaptive anchor alignment network includes N adaptive alignment modules connected in series, and each adaptive alignment module adaptively aligns the input feature map including:

[0051] Perform standard convolution on the input classification feature map and output the offset; obtain the bias feature map based on the offset and the preset hole prior;

[0052] Perform deformable convolution on the bias feature map and the input classification feature map at the same time, and output the classification feature map to the next adaptive alignment module;

[0053] Perform deformable convolution on the input positioning feature map and offset at the same time, and output the positioning feature map to the next adaptive alignment module;

[0054] The adaptive anchor generation module is used to use the classification feature map, positioning feature map and bias feature map output by the last adaptive alignment module as the target classification feature map, target positioning feature map and target bias feature map respectively; and generate adaptive anchors based on the target bias feature map and the preset uniform distribution anchor;

[0055] The target detection module is used to input the adaptive anchor, target positioning feature map and target classification feature map into the decoder to obtain the predicted positioning box and predicted category of each target in the image to be detected.

[0056] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned target detection method based on adaptive anchor alignment are implemented.

[0057] The object detection method based on adaptive anchor alignment provided by this application has the following beneficial effects:

[0058] After performing preliminary feature extraction on the image to obtain the classification feature map and the positioning feature map, the present application no longer directly generates anchors of fixed size and uniform distribution. Instead, the classification feature map and the positioning feature map are first input into the adaptive anchor alignment network. The adaptive anchor alignment network includes multiple cascaded adaptive alignment modules. When each adaptive alignment module uses standard convolution to perform a sliding window operation on the input classification feature map, it can output offsets such as the offset size of the sampling area relative to the center of the target area and the scaling ratio of the sampling width and height to the actual width and height of the target. These offsets can guide the subsequent deformable convolution to offset the sampling position on the feature map. At the same time, a basic sampling grid is provided through a preset hole prior, so that the deformable convolution can fine-tune the sampling area in the classification feature map based on the sampling grid and the offset to obtain feature information that is more focused on the target area. The offset and the input positioning feature map are then deformed. Deformed convolution, thereby fine-tuning the sampling area of the positioning feature map, also makes the output positioning feature map more focused on the target area; through the multi-stage series of adaptive alignment modules, the target distribution and scale in the classification feature map and the positioning feature map are extracted step by step, and finally the bias feature map containing the target core area position, the target classification feature map containing depth feature information and the target positioning feature map are output. Since the target bias feature map contains the target core area position, and the preset uniformly distributed anchor provides an initial anchor frame set that uniformly covers the image as the basic anchor frame grid, therefore, the target bias feature map is used to adjust the uniformly distributed adaptive anchor to generate an anchor frame (i.e., an adaptive anchor) that is highly matched with the scale, shape and position of the target in the image. The adaptive anchor can guide the model to focus on the target area when classifying and locating based on the target classification feature map and the target positioning feature map, thereby improving the classification accuracy and positioning accuracy of the model. The present application uses an adaptive anchor alignment network to generate adaptive anchors that converge to the target area, thereby improving target detection accuracy and being better applicable to various target classification and target positioning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0060] Figure 1 Flowchart of the target detection method based on adaptive anchor alignment provided by this application;

[0061] Figure 2 Flowchart of feature extraction of feature maps by the adaptive alignment module provided in this application;

[0062] Figure 3 Schematic diagram of the adaptive anchor alignment network structure provided by this application;

[0063] Figure 4 Schematic diagram of the adaptive alignment module structure provided by this application;

[0064] Figure 5 This is a schematic diagram comparing the convergence of the adaptive anchor points and the traditional uniformly distributed anchor points obtained in this application; Figure 5 (a) is a schematic diagram of traditional uniformly distributed anchor points generated on the image to be detected. Figure 5 (b) is a schematic diagram of the target detection result obtained by the target detection model in the prior art for detecting Figure (a). Figure 5 (c) in the figure is a schematic diagram of the adaptive anchor points generated on the image to be detected when using one adaptive alignment module. Figure 5 (d) is the target detection model provided by this application that includes an adaptive alignment module. Figure 5 Schematic diagram of target detection results obtained by performing detection on (c) in the figure. Figure 5 (e) in the figure is a schematic diagram of the adaptive anchor points generated on the image to be detected when two adaptive alignment modules are connected in series. Figure 5 (f) is the target detection model provided by this application, which includes two serially connected adaptive alignment modules. Figure 5 Schematic diagram of target detection results obtained by performing detection on (e) in FIG. Figure 5 (g) in the figure is a schematic diagram of adaptive anchor points generated on the image to be detected when using three adaptive alignment modules. Figure 5 (h) is the target detection model provided by this application, which includes three adaptive alignment modules. Figure 5 Schematic diagram of target detection results obtained by performing detection in (g);

[0065] Figure 6 This is a loss curve diagram of the target detection model provided by this application and the target detection model in the prior art when detecting the same image; wherein, Figure 6 (a) in the figure represents the positioning loss curve. Figure 6(b) shows the total detection loss curve based on positioning loss and classification loss;

[0066] Figure 7 This is a comparison chart of the detection results of the target detection model provided by this application and the target detection model in the prior art when detecting different images; wherein, Figure 7 (a) in the figure is a schematic diagram showing the result of detecting the first image using the target detection model in the prior art. Figure 7 (b) is a schematic diagram of the detection result of the first image using the target detection model of this application. Figure 7 (c) in the figure is a schematic diagram showing the result of detecting the second image using the target detection model in the prior art. Figure 7 (d) is a schematic diagram of the detection result of the second image using the target detection model of this application. Figure 7 (e) in the figure is a schematic diagram showing the result of detecting the third image using the target detection model in the prior art. Figure 7 (f) is a schematic diagram of the detection result of the third image using the target detection model of this application. Figure 7 (g) in the figure is a schematic diagram showing the detection result of the fourth image using the target detection model in the prior art. Figure 7 (h) is a schematic diagram of the detection results of the fourth image using the target detection model of this application. DETAILED DESCRIPTION

[0067] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0068] See also Figure 1 , Figure 1 The flowchart of the target detection method based on adaptive anchor alignment provided by this application is shown as follows. The method specifically includes:

[0069] S10: Input the image to be detected into the feature extraction network and output the positioning feature map and the classification feature map.

[0070] S20: Input the positioning feature map and the classification feature map into an adaptive anchor alignment network, wherein the adaptive anchor alignment network includes N adaptive alignment modules connected in series.

[0071] like Figure 2 As shown, each adaptive alignment module adaptively aligns the input feature map including:

[0072] S200: Perform standard convolution on the input classification feature map and output the offset.

[0073] S201: Perform deformable convolution on the bias feature map and the input classification feature map simultaneously, and output the classification feature map to the next adaptive alignment module.

[0074] S202: Perform deformable convolution on the input positioning feature map and offset at the same time, and output the positioning feature map to the next adaptive alignment module.

[0075] S30: The classification feature map, positioning feature map and bias feature map output by the last adaptive alignment module are used as the target classification feature map, target positioning feature map and target bias feature map respectively; and an adaptive anchor is generated based on the target bias feature map and the preset uniform distribution anchor.

[0076] S40: Input the adaptive anchor, the target positioning feature map and the target classification feature map into the decoder to obtain the predicted positioning box and predicted category of each target in the image to be detected.

[0077] Furthermore, in some embodiments of the present application, the specific implementation of step S10 is as follows:

[0078] S100: Input the image to be detected into the encoder for multi-level feature extraction and output a multi-scale feature map.

[0079] Alternatively, an encoder consisting of a backbone network and a neck structure can be used as a multi-scale feature extractor, and its feature extraction formula is expressed as:

[0080] ,

[0081] in, represents a multi-scale feature map, , Indicates the scale feature maps, Indicates the The height of the scale feature map, Indicates the The width of the scale feature map, Indicates the number of scale feature maps; represents the image to be detected, , Indicates the height of the image to be detected, Indicates the width of the image to be detected.

[0082] S101: Input the multi-scale feature map into the parallel classification feature extraction network and positioning feature extraction network respectively, and output the classification feature map and the positioning feature map.

[0083] Specifically, the classification feature map output by the classification feature extraction network Expressed as:

[0084] ,

[0085] in, Represents the convolutional layer of the classification feature extraction network;

[0086] Positioning feature map output by the positioning feature extraction network Expressed as:

[0087] ,

[0088] in, Represents the convolutional layer of the localization feature extraction network.

[0089] During image target detection, the model will evenly distribute some fixed-size anchor frames on the image based on the positioning feature map and the classification feature map. Subsequently, the probability and specific category of the target in each anchor frame are predicted by feature extraction of these anchor frames, and the offset of the anchor frame to the real target positioning frame is obtained, so that the model can adjust the anchor frame containing the target to a more accurate predicted positioning frame. In the existing technology, since the anchor frames are all preset fixed sizes, this will cause the shape and size of the anchor frame to not match the real targets in the image. For example, when detecting a slender bridge in a drone image, the preset 2:1 ratio anchor frame cannot effectively cover the slender bridge, which will lead to missed detection or positioning deviation. At the same time, the evenly distributed anchor frames will strictly align with the feature map grid points, but the center of the real positioning frame of the target may be located in the grid gap, which will lead to This causes a natural offset between the initial position of the anchor frame and the true positioning frame, which will also lead to a large error in the final positioning of the target. In addition, the number of evenly distributed anchor frames in an image may be as high as tens of thousands, but only a few anchor frames contain the real target, which will not only increase the time of target detection and affect the efficiency of target detection, but also make the model more affected by the background (no target anchor frame), reducing the accuracy of target detection. Therefore, if the existing preset evenly distributed anchor frame is used directly for target detection, the efficiency and accuracy of target detection will be affected. In actual applications, if it is necessary to perform target detection on an aerial vehicle, and then adjust the flight path and flight speed of the aerial vehicle based on the target detection results, this low timeliness and low precision detection result will provide incorrect data guidance for flight control.

[0090] Based on the above reasons, this application designs an adaptive anchor alignment network to adjust the preset uniformly distributed anchor frames. Specifically, Figure 3 As shown in the figure, the adaptive anchor alignment network consists of N adaptive alignment modules connected in series. The convolution blocks are the convolution layers in the classification feature extraction network and the positioning feature extraction network respectively. The last adaptive alignment module outputs the target classification feature map, the target bias feature map and the target positioning feature map.

[0091] A multi-level series-connected adaptive alignment module is used to extract the target distribution and scale in the classification feature map and the positioning feature map step by step, thereby generating an adaptive anchor that converges to the center of the target. The preset uniform distribution anchor is adjusted through the adaptive anchor, and the natural spatial misalignment problem between the target positioning box and the adjusted anchor box is gradually eliminated. At the same time, the multi-level adaptive alignment module continuously performs feature transformation on the classification feature map and the positioning feature map, and can also gradually enhance the feature details of the target and extract more detailed target feature information.

[0092] Specifically, the adaptive anchor alignment network is expressed as:

[0093] ,

[0094] in, represents the first adaptive alignment module; represents the second adaptive alignment module; Indicates the An adaptive alignment module; Represents a preset dilated prior, which is used to expand the receptive field without increasing the number of parameters based on the prior assumption of dilated convolution.

[0095] like Figure 4 The figure shows the structure of the adaptive alignment module. As can be seen from the figure, each adaptive alignment module includes standard convolution and deformable convolution. The steps of deep feature extraction of the input classification feature map and positioning feature map are as follows:

[0096] Step 1: Standard convolution First, perform a standard convolution operation on the input classification feature map and output the offset, which is expressed as:

[0097] ,

[0098] in, Indicates the The offset output by the adaptive alignment module, represents the number of adaptive alignment modules; represents a standard convolution unit for performing standard convolution; Indicates input The classification feature map of the adaptive alignment module, i.e. The classification feature map output by the adaptive alignment module is hour, It is the classification feature map output by the feature extraction network.

[0099] Optionally, each adaptive alignment module performs standard convolution on the input classification feature map, and the output offset includes: performing standard convolution on the classification feature map using a first convolution unit and a second convolution unit with different convolution kernel sizes connected in series, and obtaining the offset based on the output of the second convolution unit.

[0100] For example, the convolution kernel size of the first convolution unit is 3*3, and the convolution kernel size of the second convolution unit is 1*1.

[0101] First, since the classification feature map is used to encode the semantic information of the target, such as the target is a person, a car, and other categories, the offset generated based on the classification feature map can guide the sampling area to adjust to the area where the target of a specific category is located, thereby enhancing the purposefulness of the anchor frame adjustment, rather than guiding the anchor frame to move randomly to the area where each target is located. Therefore, this application obtains the offset directly based on the classification feature map.

[0102] Furthermore, when using standard convolution to perform a sliding window operation on the input classification feature map, the offset size of the sampling area relative to the center of the target area and the scaling ratio of the sampling width and height to the actual width and height of the target can be output. These offsets can guide the subsequent deformable convolution to offset the sampling position on the feature map.

[0103] Step 2: Add the offset to the preset hole prior to obtain the bias feature map, which is expressed as:

[0104] ,

[0105] in, Indicates the The bias feature map obtained by the adaptive alignment module; Indicates the preset hole prior.

[0106] Step 3: The first deformable convolution performs deformable convolution operations on the bias feature map and the input classification feature map at the same time, outputs the classification feature map of the current adaptive alignment module, and uses it as the input of the next adaptive alignment module. The classification feature map output by the current adaptive alignment module is expressed as:

[0107] ,

[0108] in, Indicates the The classification feature map output by the adaptive alignment module; Represents a deformable convolution unit that performs deformable convolution.

[0109] Since the role of the hole prior is to expand the receptive field without increasing the number of parameters based on the prior assumption of the hole convolution, the preset hole prior provides a basic sampling grid, so that the deformable convolution can fine-tune the sampling area in the classification feature map based on the sampling grid and the offset. For example, when detecting vehicles with different aspect ratios in the image, the hole prior selects the initial grid according to the target scale, and the offset shifts the grid points to key parts such as the front of the vehicle and the parking space, so that the output classification feature map is more focused on the core area of the target.

[0110] Step 4: The second deformable convolution performs deformable convolution operations on the offset and the input positioning feature map at the same time, outputs the positioning feature map of the current adaptive alignment module, and uses it as the input of the next adaptive alignment module. The positioning feature map output by the current adaptive alignment module is expressed as:

[0111] ,

[0112] in, Indicates the The positioning feature map output by the adaptive alignment module; Indicates input The positioning feature map of the adaptive alignment module, i.e. The positioning feature map output by the adaptive alignment module is hour, It is the positioning feature map output by the feature extraction network.

[0113] Specifically, the adjustment of the positioning feature map only requires accurately regressing the area where the target is located in the positioning feature map to the real positioning frame, and only fine-tuning is required based on the offset. There is no need to introduce a void prior to expand the receptive field. For example, when adjusting the sampling area containing the target in the positioning feature map, if a void prior is introduced, the sampling area in the adjusted positioning feature map may exceed the range of the real positioning frame, thereby affecting the positioning accuracy.

[0114] Furthermore, the target bias feature map and the preset uniformly distributed anchor are input into the alignment anchor generator to output the adaptive anchor. The adaptive anchor is expressed as:

[0115] ,

[0116] in, represents the adaptive anchor; Indicates pixel-by-pixel addition; A list of anchor points representing preset evenly distributed anchors; represents the downsampling rate of each scale feature map in the multi-scale feature map, , Indicates the number of scale feature maps; Represents the average operator.

[0117] Since the target bias feature map is generated by combining the hole prior and the offset, it contains the core area position of the target, and the preset uniformly distributed anchors provide an initial set of anchor frames that evenly cover the image as the basic anchor frame grid. Therefore, the target bias feature map is used to adjust the uniformly distributed adaptive anchors to generate anchor frames (i.e., adaptive anchors) that are highly matched with the scale, shape, and position of the target in the image. The adaptive anchor can guide the model to focus on the target area when performing classification and positioning based on the classification feature map and the positioning feature map, thereby improving the classification accuracy and positioning accuracy of the model.

[0118] It is worth noting that as the number of adaptive alignment modules increases, the anchor points of the generated adaptive anchors will converge more and more to the target center, thereby improving the target detection accuracy. However, this will also increase the complexity of the model and the computational cost, thereby affecting the target detection efficiency. Therefore, as a preferred option, the number of adaptive modules can be set to 3, which can ensure the target detection accuracy while taking into account the detection efficiency as much as possible.

[0119] Before using a target detection model including a feature extraction network, an adaptive anchor alignment network and a decoder to perform target detection on the image to be detected, it is necessary to train the target detection model using images in the training set. This application introduces a positive and negative sample allocation strategy when constructing the target detection loss function for training the model, so that the model can learn the common features of target samples of the same category from positive target samples, and learn the boundaries of targets of different categories from negative target samples, thereby capturing the subtle differences between targets of different categories.

[0120] Specifically, the construction process of the target detection loss function includes:

[0121] 1. Use the sample allocation strategy paradigm to divide each target in the image based on the predicted positioning box and predicted category of each target in the image to obtain a positive target sample set and a negative target sample set.

[0122] Optionally, in some embodiments, the step of dividing the targets based on the sample allocation strategy paradigm may include: obtaining the intersection-over-union (IoU) of the predicted positioning frame of each target and the true positioning frame of its corresponding target, as well as the predicted category confidence of each target; obtaining a positive target sample set based on targets whose IoU is greater than or equal to a preset IoU threshold and whose predicted category confidence is greater than or equal to a preset confidence threshold, and obtaining a negative target sample set based on targets outside the positive target sample set. Specifically, the true label of the image includes the true positioning frame and true category of each target, and the center coordinates, width, height, and rotation angle information can be obtained based on the coordinate information of its true positioning frame.

[0123] For example, the target can also be divided based only on the predicted positioning box and the true positioning box of the target, and the formula is expressed as: .

[0124] 2. Construct a classification loss function based on the distance measurement between the predicted category of each target in the positive target sample set and the negative target sample set and the target classification feature map of the image.

[0125] Specifically, the classification loss function is expressed as:

[0126] ,

[0127] in, represents the classification loss function; represents a function for calculating the distance metric between the predicted category of each target in the positive target sample set and the negative target sample set and the target classification feature map of the image; represents the positive target sample set; represents the negative target sample set; Represents the target classification feature map.

[0128] 3. Construct a positioning loss function based on the distance measurement between the predicted positioning box of each target in the positive target sample set and its true positioning box.

[0129] Specifically, the positioning loss function is expressed as:

[0130] ,

[0131] in, represents the positioning loss function; Represents a function for calculating the distance metric between the predicted positioning box of each target in the positive target sample set and its true positioning box; Represents the predicted positioning box; Represents the real positioning frame; Represents filtering, which means filtering from all predicted positioning frames in the image to obtain the predicted positioning frames corresponding to each target in the positive target sample set.

[0132] 4. Construct the target detection loss function based on the weighted sum of the classification loss function and the localization loss function.

[0133] Specifically, the target detection loss function is expressed as:

[0134] ,

[0135] in, represents the target detection loss function; Represents the weight of the positioning loss function.

[0136] In order to verify the effectiveness of the adaptive anchor alignment network provided by this application, this embodiment also provides a comparison diagram of the adaptive anchor points obtained by this application using the adaptive anchor alignment network and the traditional preset uniformly distributed anchors, such as Figure 5 The figure shows the convergence comparison of the adaptive anchor points obtained in this application and the traditional uniformly distributed anchor points; wherein, Figure 5 (a) is a schematic diagram of traditional uniformly distributed anchor points generated on the image to be detected. Figure 5 (b) is a schematic diagram of the target detection result obtained by the target detection model in the prior art for detecting Figure (a). Figure 5 (c) in the figure is a schematic diagram of the adaptive anchor points generated on the image to be detected when using one adaptive alignment module. Figure 5 (d) is the target detection model provided by this application that includes an adaptive alignment module. Figure 5 Schematic diagram of target detection results obtained by performing detection on (c) in the figure. Figure 5 (e) in the figure is a schematic diagram of the adaptive anchor points generated on the image to be detected when two adaptive alignment modules are connected in series. Figure 5 (f) is the target detection model provided by this application, which includes two serially connected adaptive alignment modules. Figure 4 Schematic diagram of target detection results obtained by performing detection on (e) in FIG. Figure 5 (g) in the figure is a schematic diagram of adaptive anchor points generated on the image to be detected when using three adaptive alignment modules. Figure 5 (h) is the target detection model provided by this application, which includes three adaptive alignment modules. Figure 5 Schematic diagram of target detection results obtained by performing detection in (g).

[0137] from Figure 5 As can be seen from (a) and (b) in the figure, the traditional uniform distribution anchor does not focus on the target area in the image. Figure 5 As can be seen from (c) and (d) in the figure, when using one adaptive alignment module, the adaptive anchor points generated on the image to be detected begin to gradually focus on the target in the image, starting from Figure 5 It can also be seen from (e), (f), (g) and (h) that as the number of adaptive alignment modules increases, the generated adaptive anchor points are increasingly focused on the center of the target. Therefore, the adaptive anchor alignment network introduced in this application can effectively make the anchor points converge to the target area, and once again proves that with the increase in the number of adaptive alignment modules, this convergence effect will become more obvious, so that the model can pay more attention to the anchor point dense area (i.e., the target area) during the detection process, thereby improving the accuracy of target detection.

[0138] Figure 6The figure shows the loss curves of the target detection model provided by the present application and the target detection model in the prior art when detecting the same image; wherein, Figure 6 (a) in the figure represents the positioning loss curve. Figure 6 (b) in the figure shows a total detection loss curve based on positioning loss and classification loss; the horizontal axis represents the number of training rounds, and the vertical axis represents the loss value. As can be seen from the figure, both in terms of positioning loss and total detection loss, the target detection model provided by this application has a significantly faster convergence speed and more stable and lower convergence results than target detection models in the prior art, demonstrating the effectiveness of the model provided by this application in the target detection process.

[0139] like Figure 7 The figure shows a comparison of the detection results of the target detection model provided by the present application and the target detection model in the prior art when detecting different images; wherein, Figure 7 (a) in the figure is a schematic diagram showing the result of detecting the first image using the target detection model in the prior art. Figure 7 (b) is a schematic diagram of the detection result of the first image using the target detection model of this application. Figure 7 (c) in the figure is a schematic diagram showing the result of detecting the second image using the target detection model in the prior art. Figure 7 (d) is a schematic diagram of the detection result of the second image using the target detection model of this application. Figure 7 (e) in the figure is a schematic diagram showing the result of detecting the third image using the target detection model in the prior art. Figure 7 (f) is a schematic diagram of the detection result of the third image using the target detection model of this application. Figure 7 (g) in the figure is a schematic diagram showing the detection result of the fourth image using the target detection model in the prior art. Figure 7 (h) is a schematic diagram of the detection results of the fourth image using the target detection model of this application.

[0140] pass Figure 7 It can be seen that the method provided by this application has a high detection accuracy when performing target detection on different images. For example, Figure 7 In (e) and (f), the number of targets that can be detected by this application when a large number of targets are gathered is much greater than that of existing methods.

[0141] Furthermore, this embodiment also compares the detection results of different targets of multiple existing baseline models before and after the introduction of the adaptive anchor alignment network. Specifically, the adaptive alignment module (A) is introduced into the three baseline models of RetinaNet network, adaptive training sample selection method and Faster-RCNN. 3M), and the test results are shown in Table 1:

[0142] Table 1

[0143]

[0144] As can be seen from Table 1, the RetinaNet network and the adaptive training sample selection method add A 3 The performance after M is improved by more than 3mAP. Specifically, small vehicles, ships and helicopters are relatively small in size and densely arranged, and there are obvious improvements. 3 M is deployed on the multi-stage detector Faster-RCNN, and an improvement of about 2mAP is also observed.

[0145] This embodiment also conducts ablation experiments on the number of standard convolutional units in the adaptive anchor alignment network, the number of adaptive alignment modules, and the expansion rate of a single adaptive alignment module. The experimental data are shown in Table 2:

[0146] Table 2

[0147]

[0148] From the data in Table 2, it can be seen that when the number of adaptive alignment modules is 2, the number of standard convolution units is 1, and the expansion rate of a single adaptive module is 2, the target detection model provided in this application has an mAP improvement of 2.6 compared to the baseline model in the prior art, and the number of parameters has only increased by 1.1MB. This also shows that the method provided in this application does not excessively increase the computational cost while improving the target detection accuracy, that is, it will not have a significant impact on the target detection efficiency.

[0149] Based on the target detection method based on adaptive anchor alignment provided in the above embodiment, an embodiment of the present application further provides a target detection device based on adaptive anchor alignment, which specifically includes:

[0150] The feature extraction module is used to input the image to be detected into the feature extraction network and output the positioning feature map and the classification feature map;

[0151] The adaptive alignment module is used to input the positioning feature map and the classification feature map into the adaptive anchor alignment network, wherein the adaptive anchor alignment network includes N adaptive alignment modules connected in series, and each adaptive alignment module adaptively aligns the input feature map including:

[0152] Perform standard convolution on the input classification feature map and output the offset; obtain the bias feature map based on the offset and the preset hole prior;

[0153] Perform deformable convolution on the bias feature map and the input classification feature map at the same time, and output the classification feature map to the next adaptive alignment module;

[0154] Perform deformable convolution on the input positioning feature map and offset at the same time, and output the positioning feature map to the next adaptive alignment module;

[0155] The adaptive anchor generation module is used to use the classification feature map, positioning feature map and bias feature map output by the last adaptive alignment module as the target classification feature map, target positioning feature map and target bias feature map respectively; and generate adaptive anchors based on the target bias feature map and the preset uniform distribution anchor;

[0156] The target detection module is used to obtain the predicted positioning box and predicted category of each target in the image to be detected based on the adaptive anchor, target positioning feature map and target classification feature map.

[0157] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned target detection method based on adaptive anchor alignment are implemented.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0160] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0162] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A target detection method based on adaptive anchor alignment, characterized in that: include: Input the image to be detected into the feature extraction network and output the positioning feature map and classification feature map; The positioning feature map and the classification feature map are input into the adaptive anchor alignment network, where the adaptive anchor alignment network includes N adaptive alignment modules connected in series. Each adaptive alignment module adaptively aligns the input feature map including: Perform standard convolution on the input classification feature map and output the offset; obtain the bias feature map based on the offset and the preset hole prior; Perform deformable convolution on the bias feature map and the input classification feature map at the same time, and output the classification feature map to the next adaptive alignment module; Perform deformable convolution on the input positioning feature map and offset at the same time, and output the positioning feature map to the next adaptive alignment module; the offset output by each adaptive alignment module is expressed as: ， in, Indicates the The offset output by the adaptive alignment module, represents the number of adaptive alignment modules; represents a standard convolution unit for performing standard convolution; Indicates input The classification feature map of the adaptive alignment module, i.e. The classification feature map output by the adaptive alignment module is hour, The classification feature map output by the feature extraction network; The bias feature map obtained by each adaptive alignment module is expressed as: ， in, Indicates the The bias feature map obtained by the adaptive alignment module; Indicates the presupposition of a void prior; The classification feature map output by each adaptive alignment module is expressed as: ， in, Indicates the The classification feature map output by the adaptive alignment module; represents a deformable convolution unit that performs deformable convolution; The positioning feature map output by each adaptive alignment module is expressed as: ， in, Indicates the The positioning feature map output by the adaptive alignment module; Indicates input The positioning feature map of the adaptive alignment module, i.e. The positioning feature map output by the adaptive alignment module is hour, It is the positioning feature map output by the feature extraction network; The classification feature map, positioning feature map, and bias feature map output by the last adaptive alignment module are used as the target classification feature map, target positioning feature map, and target bias feature map, respectively. The target bias feature map and the preset uniform distribution anchor are input into the alignment anchor generator to output the adaptive anchor. The adaptive anchor is expressed as: ， in, represents the adaptive anchor; Indicates pixel-by-pixel addition; A list of anchor points representing preset evenly distributed anchors; represents the downsampling rate of each scale feature map in the multi-scale feature map, , Indicates the number of scale feature maps; represents the average operator; The adaptive anchor, target positioning feature map and target classification feature map are input into the decoder to obtain the predicted positioning box and predicted category of each target in the image to be detected.

2. The object detection method based on adaptive anchor alignment according to claim 1, characterized in that Each adaptive alignment module performs standard convolution on the input classification feature map, and the output offset includes: performing standard convolution on the classification feature map using a first convolution unit and a second convolution unit with different convolution kernel sizes connected in series, and obtaining the offset based on the output of the second convolution unit.

3. The object detection method based on adaptive anchor alignment according to claim 1, characterized in that The image to be detected is input into the feature extraction network, and the output positioning feature map and classification feature map include: The image to be detected is input into the encoder for multi-level feature extraction, and a multi-scale feature map is output; The multi-scale feature maps are input into the parallel classification feature extraction network and positioning feature extraction network respectively, and the classification feature maps and positioning feature maps are output.

4. The object detection method based on adaptive anchor alignment according to claim 1, characterized in that When training the feature extraction network, adaptive anchor alignment network, and decoder using images from the training set, the object detection loss function is constructed as follows: The sample allocation strategy paradigm is used to divide each target in the image based on its predicted positioning box and predicted category, and obtain positive target sample set and negative target sample set; Construct a classification loss function based on the distance measurement between the predicted category of each target in the positive target sample set and the negative target sample set and the target classification feature map of the image; Construct a positioning loss function based on the distance measurement between the predicted positioning box of each target in the positive target sample set and its true positioning box; Based on the weighted sum of the classification loss function and the positioning loss function, the target detection loss function is constructed.

5. The object detection method based on adaptive anchor alignment according to claim 4, characterized in that: The classification loss function is expressed as: ， in, represents the classification loss function; represents a function for calculating the distance metric between the predicted category of each target in the positive target sample set and the negative target sample set and the target classification feature map of the image; represents the positive target sample set; represents the negative target sample set; Represents the target classification feature map; The positioning loss function is expressed as: ， in, represents the positioning loss function; Represents a function for calculating the distance metric between the predicted positioning box of each target in the positive target sample set and its true positioning box; Represents the predicted positioning box; Represents the real positioning frame; Indicates filtering; The target detection loss function is expressed as: ， in, represents the target detection loss function; Represents the weight of the positioning loss function.

6. The object detection method based on adaptive anchor alignment according to claim 1, characterized in that The adaptive anchor alignment network consists of three serially connected adaptive alignment modules.

7. An object detection device based on adaptive anchor alignment, characterized in that: include: The feature extraction module is used to input the image to be detected into the feature extraction network and output the positioning feature map and the classification feature map; The adaptive alignment module is used to input the positioning feature map and the classification feature map into the adaptive anchor alignment network, wherein the adaptive anchor alignment network includes N adaptive alignment modules connected in series, and each adaptive alignment module adaptively aligns the input feature map including: Perform standard convolution on the input classification feature map and output the offset; obtain the bias feature map based on the offset and the preset hole prior; Perform deformable convolution on the bias feature map and the input classification feature map at the same time, and output the classification feature map to the next adaptive alignment module; Perform deformable convolution on the input positioning feature map and offset at the same time, and output the positioning feature map to the next adaptive alignment module; the offset output by each adaptive alignment module is expressed as: ， in, Indicates the The offset output by the adaptive alignment module, represents the number of adaptive alignment modules; represents a standard convolution unit for performing standard convolution; Indicates input The classification feature map of the adaptive alignment module, i.e. The classification feature map output by the adaptive alignment module is hour, The classification feature map output by the feature extraction network; The bias feature map obtained by each adaptive alignment module is expressed as: ， in, Indicates the The bias feature map obtained by the adaptive alignment module; Indicates the presupposition of a void prior; The classification feature map output by each adaptive alignment module is expressed as: ， in, Indicates the The classification feature map output by the adaptive alignment module; represents a deformable convolution unit that performs deformable convolution; The positioning feature map output by each adaptive alignment module is expressed as: ， in, Indicates the The positioning feature map output by the adaptive alignment module; Indicates input The positioning feature map of the adaptive alignment module, i.e. The positioning feature map output by the adaptive alignment module is hour, It is the positioning feature map output by the feature extraction network; The adaptive anchor generation module is used to use the classification feature map, positioning feature map, and bias feature map output by the last adaptive alignment module as the target classification feature map, target positioning feature map, and target bias feature map, respectively; the target bias feature map and the preset uniform distribution anchor are input into the alignment anchor generator to output an adaptive anchor; wherein the adaptive anchor is expressed as: ， in, represents the adaptive anchor; Indicates pixel-by-pixel addition; A list of anchor points representing preset evenly distributed anchors; represents the downsampling rate of each scale feature map in the multi-scale feature map, , Indicates the number of scale feature maps; represents the average operator; The target detection module is used to input the adaptive anchor, target positioning feature map and target classification feature map into the decoder to obtain the predicted positioning box and predicted category of each target in the image to be detected.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the target detection method based on adaptive anchor alignment according to any one of claims 1 to 6 are implemented.

Images