A point positioning-based remote sensing image multi-class target counting method

Abstract

The application aims at the field of target counting and target positioning in remote sensing images, and discloses a remote sensing image multi-class target counting method based on point positioning. First, a feature extractor is used to extract features in the remote sensing image, then a scale adaptive feature enhancement module is used to complete rough prediction of the target scale in the remote sensing image, and the image features are fused and enhanced, next, the enhanced image features are used to predict the position, category, confidence and fine scale of the target in the image by using a prediction head, and the corresponding candidate target of the remote sensing image is generated. The candidate target contains repeated prediction of the same target, so it cannot be directly used as the prediction result of multi-class target counting and positioning. By using a non-maximum suppression post-processing means for the candidate target, the false prediction target and the repeated prediction target in the candidate target can be suppressed. The application improves the accuracy of network multi-class target counting and positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing image target counting and positioning, and particularly relates to a remote sensing image multi-class target counting method based on point positioning. BACKGROUND

[0002] Target counting and target positioning in remote sensing images is an effective means of large-scale target analysis. In recent years, with the development of deep learning, most of the most advanced target counting methods use to predict the density map of the target in the image to achieve the purpose of target counting. These density map-based methods do not need to label the bounding box of the target, and only through learning point-level labeling, the accurate number of single-class targets in the image can be obtained.

[0003] The target counting method based on predicted density map is composed of the following basic modules and steps: feature extraction, feature integration and target density map generation. First, the features of the image are extracted using a deep convolutional neural network, then a network module for feature integration is designed to address the difficulties of large target scale variation, complex background information and dense target distribution in the target counting task, and finally the network output result is up-sampled to obtain the final target density map. However, the target counting method based on density map can only obtain the number of targets, and cannot directly obtain the accurate position of the target through the density map. In addition, the existing target counting method can only count single-class targets, and cannot directly obtain the number of multiple-class targets in the image. However, in actual application scenarios, especially in remote sensing images, a picture often contains multiple classes of targets, and the existing counting method cannot meet the needs of actual application to some extent.

[0004] How to simultaneously complete the counting and positioning of multiple classes of targets in remote sensing images is a challenge. Most of the current deep learning-based target counting methods use the method of predicting target density map to realize the target counting task. However, these methods can only predict the target distribution density, and cannot directly obtain the specific position of the target in the image. In addition, most of the existing methods can only count single-class targets, and cannot meet the needs of actual application to some extent. SUMMARY

[0005] The present application provides a remote sensing image multi-class target counting method based on point positioning to simultaneously complete the counting and positioning of multiple classes of targets in remote sensing images.

[0006] The technical scheme adopted by the present application is a remote sensing image multi-class target counting method based on point positioning, and the specific steps are as follows:

[0007] Step 1, constructing a multi-class counting model based on point positioning,

[0008] The multi-class counting model comprises a feature extractor, a scale adaptive feature enhancement module and a prediction head, wherein the feature extractor is used to extract features of an input image, the extracted features are integrated and enhanced by the scale adaptive feature enhancement module, and the integrated and enhanced features are output by the prediction head to obtain position, class, scale information and confidence of a candidate target;

[0009] Step 2, based on the optimal transmission label allocation strategy SimOTA, positive and negative samples are allocated to the candidate targets predicted by the multi-class counting model, wherein the positive sample refers to a correctly predicted candidate target, and the negative sample refers to a wrongly predicted candidate target;

[0010] Step 3, a pseudo box is generated for a target to supervise the prediction of scale information, a rotating rectangular box is used to describe the scale of a labeled target, and the center point position of the rotating rectangular box is a point-level label of the target in the target counting task, a surrounding rectangular box generated for each candidate target corresponding to the labeled target is stretched in scale and rotated in angle to obtain the scale information of the labeled target;

[0011] Step 4, a multi-class target counting and positioning loss function is constructed, for a positive sample, the distance, class, scale and confidence loss of the positive sample and a corresponding labeled sample are calculated, and for a negative sample, only the confidence of the negative sample is constrained;

[0012] Step 5, non-maximum suppression is used to obtain the final prediction result, after the model training is completed, in the actual use of the inference stage, non-maximum suppression is used to delete repeated predicted candidate targets according to the position, confidence and scale information of the candidate targets, and finally the prediction result is output.

[0013] Further, the feature extractor adopts a VGG-16 model.

[0014] Further, the scale adaptive feature enhancement module is constructed as follows:

[0015] Step 1.2.1, first, the receptive field of the network is improved by using a hole convolution, and then a 3*3 convolution is used to obtain a feature map with a dimension of 18, which represents the rough scale information of the predicted candidate target, and the rough scale information of the candidate target Scale coarse which can be expressed as:

[0016]

[0017] wherein Feature is the image feature extracted by the feature extractor, is a convolution with a kernel size of 3*3 and an output feature map dimension of 18, Conv Dilated is a hole convolution, which is used to expand the receptive field of the network;

[0018] Step 1.2.2, according to the predicted target scale, extract the target key semantic information, further enhance the features extracted by the feature extractor, use the RepPoints structure, and use deformable convolution to extract the target corresponding features. Specifically, the extraction of target key semantic information can be expressed as:

[0019] Feature enhanced =Conv deformable (Scale coarse ,Feature)

[0020] where Feature enhanced is the enhanced feature, and Conv deformable (Scale coarse ,Feature) is a deformable convolution that inputs the predicted coarse scale information as an offset to process the features extracted by the feature extractor.

[0021] Further, the prediction of the specific information of the candidate target is completed using a prediction head composed of four branches, including a scale branch, a category branch, a coordinate branch, and a confidence branch, and the candidate target is represented as:

[0022]

[0023] where, is the jth candidate target in the image, is the position of the candidate target , is the category of the candidate target , is the confidence of the candidate target , is the scale of the candidate target , and the flow of the scale branch Scale fine , the category branch Cls, the coordinate branch Coord, and the confidence branch Conf can be expressed as:

[0024]

[0025]

[0026]

[0027]

[0028] where Softmax is a normalization exponential function used to normalize the module output so that the weight sum is 1, and Sigmoid is a normalization function used to normalize the input to the interval (0, 1);

[0029] In the scale-adaptive feature enhancement module, a coarse prediction of the scale of the candidate target is first performed. This coarse prediction of the candidate target is denoted as Scale. coarse In the prediction head, the scale will be... coarse Make corrections, the prediction head generates a scale coarse The correction is represented as Scale fine Final scale information of candidate targets final Represented as

[0030] Scale final =Scale coarse +Scale fine .

[0031] Furthermore, the specific implementation method of step 2 is as follows;

[0032] Step 2.1, preliminary screening of candidate targets; the purpose of preliminary screening of candidate targets is to delete negative samples with obvious prediction errors, so as to reduce the computational burden for subsequent fine matching; specifically, it is to set a distance threshold ε, and set the predicted values ​​of candidate targets that are more than the distance to the labeled target as negative samples.

[0033] Step 2.2, Fine-grained positive and negative sample allocation; Based on SimOTA, fine-grained positive and negative sample allocation is completed in two steps; First, the one-to-one cost distance between the candidate target and the labeled target is calculated. The cost distance is calculated by combining the distance cost between the candidate target and the labeled target, the category cost, and the confidence of the candidate target. The cost distance matrix is ​​then generated. Represented as:

[0034]

[0035] in, M1 is the initial set of candidate targets after preliminary screening. The number of candidate targets included, where N is the number of labeled targets in the image. Candidate target and labeled target p i The classification distance between them is calculated using cross-entropy. Candidate target and labeled target p i The positional distance between them is calculated based on Euclidean distance; for labeled sample p i The k candidate targets with the lowest cost are labeled as p. i The positive samples are identified, and the remaining candidate targets are labeled as negative samples. k is calculated using the Dynamic-k algorithm.

[0036] Furthermore, the specific implementation method of step 4 is as follows;

[0037] The positive sample obtained after step 2 can be represented as:

[0038]

[0039] Where K is the number of positive samples, for positive samples There exists a unique and definite labeled target that corresponds to it, which will be associated with The corresponding labeled target is represented as p l(j) Distance loss between positive samples and labeled targets Represented as:

[0040]

[0041] Where ε is the distance threshold, It is a positive sample The corresponding coordinates, (x l(j) ,y l(j) ) is a positive sample The corresponding labeled sample p l(j) The coordinates; the class loss between positive and labeled samples. Represented as:

[0042]

[0043] Where, N class It is the number of categories contained in the image, cls l(j),n It is p l(j) The corresponding nth category label, yes The corresponding nth category label; for positive samples, the confidence level should be as close to 1 as possible; for negative samples, the confidence level should be as close to 0 as possible. Based on this, the confidence loss is expressed as:

[0044]

[0045] in, It is the confidence loss corresponding to positive samples. This is the loss corresponding to negative samples. The confidence loss is specifically defined as follows:

[0046]

[0047]

[0048] Where K is the number of positive samples, and M is the number of candidate targets output by the model. It is the target of prediction The corresponding confidence level; for scale information, a coarse scale prediction needs to be calculated. coarse and final scale predictionfinal The scale loss between the target and the labeled target; the scale loss is expressed as:

[0049]

[0050] in It is a coarse-scale loss. This is the final scale loss; the coarse scale loss and fine scale loss can be further expressed as:

[0051]

[0052]

[0053] in, It is p l(j) Based on the scale annotations generated in step 3, the minimum bounding rectangle is represented as MERR. yes The corresponding coarse-scale prediction, yes The corresponding final scale prediction.

[0054] Furthermore, in step 3, for the labeled target p i In terms of p i The corresponding positive sample is represented as With p i Non-corresponding positive samples are represented as Include The smallest bounding rectangle is represented by box. i First, put the box i Scale the width or height using scaling factors of 0.5, 1, 2, and 3, and select the target area within the generated rectangle. The rectangle with the fewest elements is selected as the candidate bounding box with the correct scale; for each candidate bounding box with the correct scale, it is rotated 30 degrees clockwise or counterclockwise. ° Rotation includes The minimum number of bounding boxes is used as the pseudo-boundary of the final labeled target. The pseudo-boundary of the final labeled target is represented as follows:

[0055] Furthermore, in step 5, firstly, the output candidate targets are sorted in descending order of confidence level, and candidate targets with inaccurate predictions are deleted according to the set confidence threshold; then, the candidate target with the highest confidence level in the candidate target sequence is taken out, and the intersection-union ratio (IU) of this candidate target with other candidate targets in the same scale range is calculated, and candidate targets with IU exceeding the set scale threshold are deleted. At the same time, candidate targets whose predicted positions are within the scale range of candidate targets with high confidence are also deleted; finally, the candidate target with the highest confidence level is output to the sequence composed of the final results, and the above steps are repeated to complete non-maximum suppression.

[0056] Furthermore, in step 2.1, different distance thresholds ε are selected for candidate targets generated by different branches. For branches that are downsampled by 4 times, 8 times, and 16 times, the distance thresholds selected are 4, 8, and 16, respectively.

[0057] Furthermore, the confidence threshold was set to 0.7, and the scale threshold was set to 0.2.

[0058] Compared with existing technologies, this method uses a point-based localization framework to complete target counting and localization. Furthermore, by adding category information to the network prediction results, it achieves counting and localization of multiple target categories. To adapt to the characteristics of remote sensing images, such as large target scale variations, complex background information, and arbitrary target distribution angles, the proposed method employs a multi-branch structure to simultaneously localize targets of different scales in the remote sensing image. Simultaneously, the proposed method uses a rotated bounding box to represent the target scale and employs a point-set-based feature extraction structure to extract target information, significantly improving the accuracy of network target localization and counting. Attached Figure Description

[0059] Figure 1 This is a general framework diagram of the technology described in this invention.

[0060] Figure 2 This is a detailed model diagram of the technology described in this invention.

[0061] Figure 3 A diagram illustrating the candidate targets.

[0062] Figure 4 This is the overall flowchart. Detailed Implementation

[0063] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0064] The overall flowchart of the present invention is as follows: Figure 4 As shown, the specific implementation steps are as follows:

[0065] Step 1: Construction of a point-localization-based multi-class counting model. Point-localization-based counting methods predict the location of targets in an image to achieve the final counting objective. The general process involves extracting image features, enhancing and integrating the extracted features, using a prediction head to predict candidate targets in the image, and finally post-processing the predicted candidate targets to obtain the final result. In the field of deep learning research, convolutional neural networks are typically used to extract image features and then use these features to predict targets. This approximates a nonlinear function and can transform the image into a high-dimensional feature space for processing. For the candidate targets output by the convolutional network, post-processing techniques such as non-maximum suppression are used to remove duplicate predicted candidate targets, obtaining the final target prediction result.

[0066] Step 1.1: This invention uses a convolutional neural network to build the basic framework. The model input is remote sensing imagery. The commonly used image feature extractor VGG-16 is used to extract the features of the image. The extracted features need to be integrated and enhanced by a scale-adaptive feature enhancement module. The integrated and enhanced features output the location, category, scale information and confidence of the candidate targets through a specific prediction head. Finally, based on the candidate target information, a non-maximum suppression method is used to delete the candidate targets that are predicted repeatedly and retain the most accurate target prediction results.

[0067] Step 1.2: This invention provides a method for constructing a scale-adaptive feature enhancement module. Even with only point-level annotations, it can predict the scale of targets in an image. Furthermore, through a specific network structure, it can extract target features based on the predicted target scale information, improving the accuracy of the network's target prediction. The steps for constructing the scale-adaptive feature enhancement module are as follows:

[0068] Step 1.2.1: Predict the coarse scale of the target in the image. For each candidate target in the image, this invention uses a point set consisting of 9 skeleton points to describe the scale information of the candidate target. Unlike the method of directly using the bounding rectangle surrounding the target to represent the target scale, we use a point set covering the semantic key points of the target to represent the target scale information. The target scale is finally calculated by the minimum bounding rectangle surrounding the skeleton point set. For each skeleton point in the point set, two values, x and y, are predicted to represent the position of the skeleton point. Therefore, describing the scale information of a candidate target requires 18 variables. In the scale-adaptive feature enhancement module, the receptive field of the network is first increased through dilated convolution, and then a feature map with a dimension of 18 is obtained through 3×3 convolution. This feature map represents the predicted coarse scale information of the candidate target. (Coarse scale information of the candidate target) coarse It can be represented as

[0069]

[0070] Here, Feature refers to the image features extracted by the feature extractor. It is a convolution with a kernel size of 3×3 and an output feature map dimension of 18, Cobv Dilated It is a dilated convolution used to expand the receptive field of a network.

[0071] Step 1.2.2: Based on the predicted target scale, extract key semantic information of the target and further enhance the features extracted by the feature extractor. This invention uses the RepPoints structure and utilizes deformable convolution to extract the corresponding features of the target. Specifically, the extraction of key semantic information of the target can be represented as follows:

[0072] Feature enhanced =Conv deformable (Scale coarse (Feature)

[0073] Among them, Feature enhanced It is the enhanced feature, Conv deformable (Scale coarse The Feature() function takes the predicted coarse scale information as an offset input to a deformable convolution and processes the features extracted by the feature extractor.

[0074] Step 1.3: Use the prediction head to predict the location, category, scale, and confidence level of the candidate target. This invention uses a prediction head consisting of four branches to predict the specific information of the candidate target. The candidate target is represented as...

[0075]

[0076] in, For the j-th candidate target in the image, Candidate targets Location, Candidate targets Category Candidate targets confidence level Candidate targets The scale. The process of scale branching, category branching, coordinate branching, and confidence branching can be represented as follows:

[0077]

[0078]

[0079]

[0080]

[0081] Here, Softmax is a normalization exponential function used to standardize the module output so that the sum of the weights equals 1. Sigmoid is a normalization function used to standardize the input to the interval (0,1). It is important to note that in the scale-adaptive feature enhancement module, this invention performs a coarse prediction of the scale of the candidate target; this coarse prediction is denoted as Scale. coarse In the prediction head, the scale will be... coarse Make corrections, the prediction head generates a scale coarse The correction is represented as Scale fine Final scale information of candidate targets final Represented as

[0082] Scale final =Scale coarse +Scale fine .

[0083] Step 2: After the network outputs candidate targets, it needs to perform positive and negative sample matching with the labeled targets in the image. This invention uses the optimal transmission label allocation strategy SimOTA to complete the positive and negative sample allocation. The label allocation steps are as follows:

[0084] Step 2.1: Preliminary Screening of Candidate Targets. The purpose of preliminary screening of candidate targets is to remove negative samples with obvious prediction errors, thereby reducing the computational burden for subsequent fine-grained matching. Specifically, a distance threshold ε is set, and the predicted values ​​of candidate targets whose distances exceed the distance threshold are set as negative samples. In this implementation scheme, different distance thresholds ε are selected for candidate targets generated by different branches. For branches with downsampling by 4, 8, and 16 times, the distance thresholds selected are 4, 8, and 16, respectively.

[0085] Step 2.2: Fine-grained positive and negative sample allocation. Based on SimOTA, fine-grained positive and negative sample allocation is mainly achieved in two steps. First, the one-to-one cost distance between candidate targets and labeled targets needs to be calculated. In this invention, the cost distance is calculated by combining the distance cost between candidate targets and labeled targets, the category cost, and the confidence level of the candidate targets. Cost distance matrix Represented as

[0086]

[0087] in, M1 is the initial set of candidate targets after preliminary screening. The number of candidate targets included, where N is the number of labeled targets in the image. Candidate target and labeled target p iThe classification distance between them is calculated using cross-entropy in this invention. Candidate target and labeled target p i The positional distance between them is calculated based on Euclidean distance. For labeled sample p i The k candidate targets with the lowest cost are labeled as p. i The positive samples are identified, and the remaining candidate targets are labeled as negative samples. k is calculated using the Dynamic-k algorithm.

[0088] Since the target counting task only has target point-level annotations and lacks scale annotations for targets in the image, it is impossible to supervise the scale information predicted in steps 1.2 and 1.3. This invention designs a target pseudo-boundary generation algorithm to generate pseudo-boundaries for targets, which is used to supervise the prediction of scale information. The specific steps of the target pseudo-boundary generation algorithm are as follows:

[0089] Step 3: Generating Pseudo-Boundaries for Annotated Targets. Since most targets in remote sensing images have arbitrary orientations, this invention uses rotated rectangles to describe the scale of the annotated targets. There are three parameters to be determined for the rotated rectangle: the center point position, the length and width of the rotated rectangle, and the rotation angle. The center point position of the rotated rectangle is the point-level annotation of the target in the target counting task; therefore, only the length, width, and rotation angle need to be determined. After separating positive and negative samples, the candidate targets assigned to each annotated target can, to some extent, reflect the bounding rectangle of that annotated target. Based on this, scaling and rotating the bounding rectangles generated for each candidate target corresponding to annotated target yields the scale information of the annotated target. This scale information can be used to supervise the scale of the candidate targets predicted in steps 1.2 and 1.3. In this embodiment, for annotated target p... i In terms of p i The corresponding positive sample is represented as With p i Non-corresponding positive samples are represented as Include The smallest bounding rectangle is represented by box. i First, put the box i Scale the width or height using scaling factors of 0.5, 1, 2, and 3, and select the target area within the generated rectangle. The smallest rectangle is selected as the scaled candidate box. For each scaled candidate box, it is rotated 30° clockwise / counterclockwise, with the rotation including... The minimum number of bounding boxes is used as the pseudo-boundary of the final labeled target. The pseudo-boundary of the final labeled target is represented as follows:

[0090] Step 4: Construct a multi-class target counting and localization loss function. For positive samples, calculate the loss based on their distance to the corresponding labeled sample, class, scale, and confidence level; for negative samples, only their confidence level needs to be constrained. The positive sample obtained in Step 2 can be represented as:

[0091]

[0092] Where K is the number of positive samples. For positive samples There exists a unique and definite labeled target that corresponds to it, which will be associated with The corresponding labeled target is represented as p l(j) Distance loss between positive samples and labeled targets. Represented as:

[0093]

[0094] Where ε is the distance threshold defined in step 2.1, It is a positive sample The corresponding coordinates, (x l(j) ,y l(j) ) is a positive sample The corresponding labeled sample p l(j) The coordinates. The class loss between positive and labeled samples. Represented as:

[0095]

[0096] Where, N class It is the number of categories contained in the image, cls l(j),n It is p l(j) The corresponding nth category label, yes The corresponding nth category label. For positive samples, the confidence level should be as close to 1 as possible; for negative samples, the confidence level should be as close to 0 as possible. Based on this, the confidence loss... Represented as:

[0097]

[0098] in, It is the confidence loss corresponding to positive samples. This is the loss corresponding to negative samples. The confidence loss is specifically defined as follows:

[0099]

[0100]

[0101] Where K is the number of positive samples, and M is the number of candidate targets output by the network. It is the target of prediction The corresponding confidence level. For scale information, the coarse scale prediction output from step 1.2.1 needs to be calculated. coarse And the final scale prediction output of step 1.3 final The scale loss between the target and the labeled target. The scale loss is expressed as...

[0102]

[0103] in It is a coarse-scale loss. This is the final scale loss. The coarse scale loss and fine scale loss can be further expressed as...

[0104]

[0105]

[0106] in, It is p l(j) Based on the scale annotations generated in step 3, the minimum bounding rectangle is represented as MERR. yes The corresponding coarse-scale prediction, yes The corresponding final scale prediction.

[0107] Step 5: Non-maximum suppression to obtain the final prediction result. After model training, in the actual inference stage, non-maximum suppression is performed using the position, confidence, and scale information of candidate targets to remove duplicate predictions and output the final prediction result. First, the output candidate targets are sorted in descending order of confidence. Based on the set confidence threshold, inaccurate predictions are removed. Then, the candidate target with the highest confidence in the candidate target sequence is extracted, and the intersection-union ratio (IU) of this candidate target with the scale range of other candidate targets is calculated. Candidate targets with an IU exceeding the set scale threshold are removed. Simultaneously, candidate targets whose predicted positions fall within the scale range of high-confidence candidate targets are also removed. Finally, the candidate target with the highest confidence is output to the sequence of the final results, and the above steps are repeated to complete non-maximum suppression. In this implementation, the confidence threshold is set to 0.7, and the scale threshold is set to 0.2.

[0108] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, but without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for counting multi-class objects in remote sensing images based on point positioning, characterized in that, Comprising the following steps: Step 1, building a multi-class counting model based on point positioning, The multi-class counting model comprises a feature extractor, a scale adaptive feature enhancement module and a prediction head, wherein the feature extractor is used to extract the features of the input image, the extracted features are subjected to feature integration and enhancement by the scale adaptive feature enhancement module, and the integrated and enhanced features are output by the prediction head to obtain the position, class, scale information and confidence of the candidate target; Step 2, based on the optimal transmission label assignment strategy SimOTA, the positive and negative samples of the candidate target predicted by the multi-class counting model are assigned, the positive sample refers to the candidate target predicted correctly, and the negative sample refers to the candidate target predicted incorrectly; The specific implementation of step 2 is as follows: Step 2.1, preliminary screening of candidate targets; the purpose of the preliminary screening of candidate targets is to delete the negative samples with obvious prediction errors, so as to reduce the computational burden for the subsequent fine matching; specifically, a distance threshold is set The prediction value of the candidate target beyond the distance threshold from the labeled target is set as a negative sample. Step 2.2, fine positive and negative sample assignment; based on SimOTA, fine positive and negative sample assignment is completed in two steps; first, the one-to-one cost distance between the candidate target and the labeled target is calculated, and the cost distance is calculated by integrating the distance cost between the candidate target and the labeled target, the category cost and the confidence of the candidate target is represented as: in, This is the initial set of candidate targets. yes The number of candidate targets included. It is the number of marked targets in the image. Candidate target and labeled targets The classification distance between them is calculated using cross-entropy. Candidate target and labeled targets The positional distance between them is calculated based on Euclidean distance; for labeled samples With the lowest cost The candidate targets are marked as The positive samples are identified, and the remaining candidate targets are labeled as negative samples. Calculations are performed using the Dynamic-k algorithm; Step 3, generating a pseudo box for the target to supervise the prediction of the scale information, using a rotated rectangular box to describe the scale of the labeled target, the center point position of the rotated rectangular box is the point-level label of the target in the target counting task, the scale of the surrounding rectangular box generated for each labeled target corresponding candidate target is stretched and rotated, and the scale information of the labeled target is obtained; Step 4, building a multi-class target counting and positioning loss function, for the positive sample, the distance, class, scale and confidence loss of the positive sample and the corresponding labeled sample are calculated; for the negative sample, only the confidence of the negative sample is constrained; Step 5, non-maximum suppression to obtain the final prediction result, after the model training is completed, in the actual use of the inference stage, the position, confidence and scale information of the candidate target are used for non-maximum suppression, the candidate target with repeated prediction is deleted, and the final prediction result is output.

2. The method of claim 1, wherein: The feature extractor adopts a VGG-16 model.

3. The method of claim 1, wherein: The construction method of the scale adaptive feature enhancement module is as follows: Step 1.2.1, first improve the receptive field of the network by holey convolution, and then improve the resolution of the feature map by up-sampling convolution to obtain a feature map with a dimension of 18, which represents the coarse scale information of the predicted candidate target. The coarse scale information of the candidate target can be represented as: ​ wherein, is an image feature extracted by a feature extractor, is a convolution kernel with a size of , and the output feature map dimension is 18, is a dilated convolution used to expand the receptive field of the network; Step 1.2.2, according to the predicted target scale, the target key semantic information is extracted, the features extracted by the feature extractor are further enhanced, the RepPoints structure is used, and the target corresponding features are extracted by using the deformable convolution, specifically, the extraction of the target key semantic information can be expressed as: wherein, is the feature after enhancement, is to input the predicted coarse scale information as an offset to the deformable convolution, and process the feature extracted by the feature extractor.

4. The method of claim 1, wherein: The prediction head composed of four branches is used to complete the prediction of the specific information of the candidate target, including the scale branch, the class branch, the coordinate branch and the confidence branch, and the candidate target is expressed as: wherein, is a position of the th candidate object in the image, is a class of the candidate object , is a confidence of the candidate object , is a scale of the candidate object , is a dimension of the candidate object , the flow of the scale branch , the class branch , the coordinate branch , the confidence branch may be represented as: wherein, is a normalization exponential function used to normalize the module output such that the weight sum results in 1, is a normalization function used to normalize the input to the interval ; In the scale adaptive feature enhancement module, the scale of the candidate target is roughly predicted, and the rough prediction of the candidate target is represented as In the prediction head, the prediction head produces the correction of the candidate target scale information The correction of the candidate target scale information produced by the prediction head is represented as The final scale information of the candidate target is represented as 。 5. The method of claim 1, wherein: The specific implementation of step 4 is as follows: The positive sample obtained through step 2 can be expressed as: wherein is the number of positive samples, for a positive sample , there is a uniquely determined annotation target corresponding to it, and will be annotated The annotation target corresponding to is expressed as The distance loss between the positive sample and the annotation target is expressed as: wherein, is a distance threshold, is a positive sample corresponding coordinates, is a positive sample corresponding labeled sample coordinates; class loss between positive sample and labeled sample is expressed as: wherein, is the number of categories contained in the image, is the corresponding first category label, is the corresponding first category label; for the positive sample, the confidence should be as close to 1 as possible; for the negative sample, the confidence should be as close to 0 as possible, based on which, the confidence loss is expressed as: wherein, is the confidence loss corresponding to the positive sample, is the loss corresponding to the negative sample, and the confidence loss is specifically defined as follows: wherein, is the number of positive samples, is the number of candidate targets output by the model, is the predicted target the corresponding confidence; for scale information, a coarse scale prediction and a final scale prediction is the scale loss between the annotated target; the scale loss is represented as: wherein is a coarse scale loss, is a final scale loss; the coarse scale loss and the fine scale loss can further be expressed as: wherein, is According to the scale annotation generated in step 3, the minimum circumscribed rectangle is solved as , is The corresponding rough scale prediction, is The corresponding final scale prediction.

6. The method of claim 1, wherein: In step 3, the positive sample corresponding to the labeled target is represented as , the positive sample not corresponding to the labeled target is represented as , the minimum bounding rectangle containing the positive sample is represented as , and the minimum bounding rectangle containing the positive sample is represented as First, the width or height of the positive sample is scaled by the proportional coefficient 0.5, 1, 2, and 3, and the rectangular frame containing the labeled target with the least number of rectangular frames is selected as the scale-correct candidate frame. For the scale correct candidate box, rotate it clockwise or counterclockwise , the rotation includes The minimum rectangular frame as the final labeling target pseudo frame, the final labeling target pseudo frame is expressed as .

7. The method of claim 1, wherein: In step 5, first, the output candidate target is arranged in descending order according to the confidence, according to the set confidence threshold, the candidate target with inaccurate prediction is deleted; then, the candidate target with the maximum confidence in the candidate target sequence is taken out, the intersection over union of the scale range of the candidate target and other candidate targets is calculated, the candidate target with the intersection over union exceeding the set scale threshold is deleted, and the candidate target with the prediction position in the scale range of the candidate target with high confidence is deleted; finally, the candidate target with the maximum confidence is output to the sequence composed of the final result, and the above steps are repeated to complete the non-maximum suppression.

8. The method of claim 1, wherein: Step 2.1, different distance thresholds are selected for candidate target generation of different branches The distance thresholds selected for the branches down-sampled by 4, 8, and 16 times are 4, 8, and 16.

9. The method of claim 7, wherein: The confidence threshold is set to 0.7, and the scale threshold is set to 0.2.

Images