A point labeling remote sensing target directional detection method and device

Abstract

The application relates to a point labeling remote sensing target directional detection method and device, which comprises the following steps: acquiring a point labeling image, inputting the point labeling image into an improved ResNet50 model, and acquiring a class probability graph; the improved model comprises the following steps: connecting a hollow convolution layer, a feature extraction network based on canonical correlation analysis and a mixed channel attention mechanism in sequence at an output layer of the ResNet50 model to acquire the class probability graph; in the process of training the model, pseudo labels of the height and width of a target are acquired according to an original class probability graph, a dynamic radius adjustment mechanism of adjusting a positive label distribution radius is adjusted according to the target pseudo labels, and positive and negative label distribution is combined with point labeling information; the class probability graph is dimensionally reduced, each data point after the dimensional reduction is assigned a weight, a covariance matrix is constructed, eigenvalues of the covariance matrix are decomposed, the width and height directions of the target are determined according to the direction of the decomposition result, the target boundary is acquired by moving outward along the two directions, and a rotating frame is further generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a point labeling remote sensing target directional detection method and device. BACKGROUND

[0002] With the vigorous development of remote sensing technology, remote sensing data such as aerial images and satellite images are growing explosively, which contains a large amount of target information. Directional remote sensing target detection technology emerges as the times require in such a background, which plays a crucial role in accurately labeling small and densely arranged targets, and has a wide application prospect in many fields such as remote sensing image analysis, retail scene analysis and scene text detection. However, the traditional directional target detection method faces a serious challenge, that is, the work of labeling the oriented bounding box (OBB) needs to consume a large amount of manpower and time cost. This process is not only tedious, but also prone to errors, which greatly limits the large-scale application and development of target detection technology. In order to solve this problem, many weakly supervised methods have appeared in recent years, including horizontal box supervision and point supervision. Among them, the point supervision method greatly reduces the labeling cost because it only needs to label one point and the category of each target, so it has attracted widespread attention. This method greatly improves the labeling efficiency under the premise of ensuring a certain detection accuracy, and provides a new idea and direction for the development of directional remote sensing target detection technology.

[0003] The existing directional target detection method based on point supervision mainly includes three categories. The first category is based on a powerful SAM model. The P2RBox model proposed by Cao et al. generates mask proposals with the help of a mask generator such as SAM, filters high-quality masks according to the centroid offset penalty through the constraint module, uses fine point sampling and complex loss calculation in the inspector module to deeply evaluate the mask quality, and finally converts the mask into a rotated box label by relying on the symmetry axis estimation module. However, the SAM model is very effective on natural images, but when dealing with large-scale remote sensing images, this inefficiency is particularly evident, which seriously affects the efficiency of target detection, making it difficult for SAM-based methods to meet the requirements of speed and memory in practical applications. Secondly, there are still challenges in the accuracy of direction estimation for asymmetric objects. The PointSAM model proposed by Liu et al. is based on a self-training framework, which uses the zero-shot ability of SAM to train by iteratively generating pseudo-labels. The method includes prototype regularization (PBR), which reduces the error accumulation in self-training by generating target prototypes offline and dynamically updating predicted prototypes, and aligning the two using the Hungarian algorithm; it also proposes negative sample calibration (NPC), which uses the hint of overlapping masks as negative signals to adjust negative samples, so as to optimize mask prediction. However, the double-branch structure of the self-training framework leads to slow training speed, and the negative sample calibration does not work well when dealing with sparsely distributed objects, lacking a strategy that can accurately complete the allocation of positive and negative samples.

[0004] The second type of method is the method based on human prior knowledge. The Point2RBox proposed by Yu et al. is a representative method based on prior knowledge, which needs to integrate human prior knowledge. However, different data sets often have different characteristics and distributions, which means that specific prior knowledge is needed for each data set, greatly reducing the generality of the method. They are difficult to combine with more powerful detectors, so they cannot fully utilize the performance improvement potential of advanced detectors, and they are not capable enough when facing complex and variable target detection tasks.

[0005] The third type of method is the method based on modular structure. PointOBB proposed by Luo et al. does not rely on artificial design of prior knowledge, and provides greater flexibility by decoupling the pseudo-label generation from the detector, making it more suitable for efficient and scalable detection tasks. However, the teacher-student structure leads to a very slow pseudo-label generation process, which takes about 7-8 times the time of subsequent detector training. In addition, due to multiple view conversions, its training requires a large amount of GPU memory. In addition, the change in the number of regions of interest (Rol) can cause memory shortage problems, especially in dense object scenes. Although limiting the number of Rol can alleviate this problem, it will cause performance degradation. PointOBB-v2 proposed by Ren et al. as the most advanced point supervision oriented detection method at present generates class probability maps (CPM) from point annotations, and designs a new sample assignment strategy to capture the outline and direction of the object from CPM. Next, non-uniform sampling based on probability distribution is adopted, and principal component analysis (PCA) is used to determine the boundary and direction of the target. However, this method uses a fixed radius when assigning positive samples, which may need to be adjusted when applied to other targets or other data sets with different characteristics, otherwise it may not achieve optimal performance.

[0006] In summary, point supervision oriented detection methods are diverse, but each has its advantages and disadvantages. Although the SAM-based method is powerful, it faces challenges when dealing with cross-domain tasks such as aerial images. The prior knowledge-based method needs specific prior information, and its generality is limited and its flexibility is insufficient. The teacher-student structure-based method, such as PointOBB, has the defects of slow pseudo-label generation, large memory requirements, and easy memory problems. Although the PointOBB-v2 method improves PointOBB to some extent, it still has the limitation that the hyperparameters depend on the target and the data set. SUMMARY

[0007] In order to solve the problems existing in the prior art, the purpose of the present application is to provide a point labeling remote sensing target directional detection method and device, which has significant improvement in detection accuracy, memory efficiency, adaptability to different targets, and ability to handle complex scenes, and provides a more effective solution for directional remote sensing target detection.

[0008] In order to achieve the above purpose, the present application provides the following scheme:

[0009] A point labeling remote sensing target directional detection method comprises:

[0010] Obtaining a point labeling image, the point labeling image being an image with target center point labeling and target category labeling;

[0011] Inputting the point labeling image into an improved ResNet50 model to obtain a class probability map CPM; wherein the improved ResNet50 model comprises: connecting a dilated convolution layer to the output layer of the ResNet50 model to expand the receptive field, adding a feature extraction network based on canonical correlation analysis after the dilated convolution layer for feature fusion, introducing a secondary feature fusion based on a hybrid channel attention mechanism after the feature extraction network, and obtaining the class probability map CPM;

[0012] During training of the improved ResNet50 model, according to an original class probability map CPM0, obtaining a target height and width pseudo label, dynamically adjusting a dynamic radius adjustment mechanism of positive label distribution radius for different target pseudo labels, and combining point labeling information to perform positive and negative label distribution to determine the target morphology; the dynamic radius adjustment mechanism is to dynamically adjust the radius according to different width and height pseudo labels of different targets, and the point labeling information is the center point of the labeled target; during the process of positive and negative label distribution, a circle is drawn according to the labeled center point and the dynamic radius, and the inside of the circle is regarded as a positive label; a circle is drawn according to the distance between the center points of the two adjacent nearest same targets and the center point of one of the targets, and the outside of the circle is regarded as a negative label;

[0013] Reducing the dimension of the class probability map CPM, assigning a weight to each data point after dimension reduction, constructing a covariance matrix, decomposing the eigenvalues of the covariance matrix, determining the width and height direction of the target according to the direction of the decomposed eigenvalues, and moving outward along the width and height direction of the target, if the value of the current position is lower than a preset threshold, indicating the target boundary, thereby generating a rotated frame.

[0014] Optionally, the secondary feature fusion based on the hybrid channel attention mechanism to obtain the class probability map CPM comprises:

[0015] The input feature map is subjected to local average pooling and global average pooling processing, the features after local pooling and global pooling are input into a 1D convolution for feature conversion, after 1D convolution and rearrangement, the features after local pooling and the features after global pooling are added to obtain a first fusion result;

[0016] The first fusion result is subjected to an inverse pooling operation, and the first fusion result is multiplied by the input feature map to obtain a second fusion result, i.e., the class probability map CPM.

[0017] Optionally, obtaining the pseudo-label of the target height and width includes:

[0018] The horizontal and vertical directions of the original class probability map CPM0 are subjected to convolution calculation to obtain the gradient value of each pixel point in the horizontal direction and the gradient value in the vertical direction of the original class probability map CPM0, the gradient amplitude is calculated based on the gradient value, a first target threshold is set, and the gradient amplitude and the first target threshold are subjected to target boundary region judgment. If the gradient amplitude is greater than the first target threshold, the pixel point is regarded as a target boundary point, and if the gradient amplitude is less than the first target threshold, the pixel point is regarded as a background or noise point.

[0019] Based on the target boundary point, the outermost boundary point of the target in the horizontal and vertical directions is obtained, the difference between the horizontal coordinates of the leftmost and rightmost boundary points in the horizontal direction is taken as the width pseudo-label of the target, and the difference between the vertical coordinates of the uppermost and lowermost boundary points in the vertical direction is taken as the height pseudo-label of the target.

[0020] Optionally, obtaining the gradient value of each pixel point in the horizontal direction and the gradient value in the vertical direction of the original class probability map CPM0 includes:

[0021] The horizontal direction of the original class probability map CPM0 is subjected to convolution calculation, and the surrounding neighborhood pixel values of each pixel point in the original class probability map CPM0 are multiplied by the corresponding elements of the horizontal direction convolution kernel and then summed to obtain the gradient value of the pixel point in the horizontal direction.

[0022] The vertical direction of the original class probability map CPM0 is subjected to convolution calculation, and the surrounding neighborhood pixel values of each pixel point in the original class probability map CPM0 are multiplied by the corresponding elements of the vertical direction convolution kernel and then summed to obtain the gradient value of the pixel point in the vertical direction.

[0023] Optionally, determining the target morphology includes:

[0024] Based on the pseudo-label of the target height and width, the radius of the positive sample region is estimated:

[0025]

[0026] wherein r is a radius, k is a scaling factor, w is a width pseudo-label of the target, and h is a height pseudo-label of the target;

[0027] adjusting the radius of the positive sample region again according to a density of the target in the point-labeled image;

[0028] r adjusted =r×f(density)

[0029]

[0030] wherein r adjusted is the radius of the positive sample region adjusted again, r is the radius of the positive sample region, f(density) is a density factor, d min is a distance between two nearest center points of the same target, and T is a density threshold.

[0031] Optionally, obtaining the target boundary comprises:

[0032] obtaining a plurality of grid coordinates around the labeled point in the point-labeled image, performing dimension reduction on the class probability map CPM according to the plurality of grid coordinates, assigning a weight to each data point after the dimension reduction, constructing a covariance matrix, decomposing eigenvalues of the covariance matrix, taking a feature vector corresponding to the largest eigenvalue as a main direction and a position perpendicular to the main direction as a secondary direction, setting a second target threshold, and moving along the two directions, wherein if a value at a certain position is lower than the second target threshold, the position is regarded as the target boundary.

[0033] Optionally, constructing the covariance matrix comprises:

[0034]

[0035] wherein C z is the covariance matrix, p i is the weight, z i is the data point, μ z is a mean of the data point, and T is a transpose.

[0036] Optionally, decomposing the eigenvalues of the covariance matrix comprises:

[0037] C z v i =λ i v i

[0038] wherein v i is the feature vector, and λ i is the eigenvalue.

[0039] To achieve the above object, the application further provides a point labeling remote sensing target directional detection device, comprising:

[0040] A feature extraction module is configured to acquire a point labeling image, input the point labeling image into an improved ResNet50 model, and acquire a class probability map CPM, wherein the point labeling image is an image with target center point labeling and target category labeling.

[0041] A dynamic radius adjustment module is configured to acquire target height and width pseudo labels according to an original class probability map CPM0 during training of the improved ResNet50 model, dynamically adjust a dynamic radius adjustment mechanism of positive label distribution radius for different target pseudo labels, and combine point labeling information to perform positive and negative label distribution and determine target morphology.

[0042] A rotated box generation module is configured to perform dimension reduction on the class probability map CPM, assign weights to each data point after dimension reduction, construct a covariance matrix, decompose eigenvalues of the covariance matrix, determine a width and height direction of a target according to a direction of the decomposed eigenvalues, and move outward along the width and height direction of the target.

[0043] The application has the following beneficial effects:

[0044] The application proposes a new structure, which combines a feature optimization fusion module of ResNet50, a hollow convolution and a canonical correlation analysis, effectively enhances the feature extraction capability, and lays a foundation for generating high-quality class probability map CPM. Secondly, a hybrid local channel attention mechanism is used to replace the projection layer, which overcomes the problems of information loss and lack of dynamic adjustment capability of the projection layer, makes the class probability map CPM more accurate and accurate. Thirdly, the dynamic radius adjustment mechanism dynamically adjusts the size of the positive sample area according to the target density, reduces the overlap of high-density areas and the false detection of low-density areas, significantly improves the detection accuracy, especially when dealing with dense objects, and the mirror filling method for image edge targets also improves the accuracy and robustness of detection. Finally, the t-SNE method is used to determine the target direction and boundary, which can more effectively capture the nonlinear structure information of the target with complex shape. In summary, the method of the application has significantly improved in detection accuracy, memory efficiency, adaptability to different targets and the ability to handle complex scenes, and provides a more effective solution for directional remote sensing target detection. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1 A flow chart of a point labeling remote sensing target directional detection method according to an embodiment of the application;

[0047] Figure 2 A network model structure diagram of a point labeling remote sensing target directional detection method according to an embodiment of the application;

[0048] Figure 3 A feature optimization fusion module structure diagram of a point labeling remote sensing target directional detection method according to an embodiment of the application;

[0049] Figure 4 A hybrid channel attention mechanism structure diagram of a point labeling remote sensing target directional detection method according to an embodiment of the application;

[0050] Figure 5 A visualization result schematic diagram of each class of a point labeling remote sensing target directional detection method according to an embodiment of the application on the DOTA-V1.0 dataset. DETAILED DESCRIPTION

[0051] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0052] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] The embodiment discloses a point labeling remote sensing target directional detection method, comprising: acquiring a point labeling image, the point labeling image being an image with target center point labeling and target category labeling; inputting the point labeling image into an improved ResNet50 model to acquire a category probability map CPM; wherein the improved ResNet50 model comprises: connecting a dilated convolution layer to an output layer of the ResNet50 model to expand a receptive field, adding a feature extraction network based on canonical correlation analysis after the dilated convolution layer to perform feature fusion, and introducing a secondary feature fusion based on a hybrid channel attention mechanism after the feature extraction network to acquire the category probability map CPM; in the process of training the improved ResNet50 model, according to an original category probability map CPM0, a pseudo label of target height and width is acquired, a dynamic radius adjustment mechanism of dynamically adjusting a positive label distribution radius is adjusted according to different target pseudo labels, and positive and negative label distribution is performed combined with point labeling information to determine a target morphology; the dynamic radius adjustment mechanism is to dynamically adjust the radius according to different width and height pseudo labels of different targets, and the point labeling information is a center point of a labeled target; in the process of performing positive and negative label distribution, a circle is drawn according to the labeled center point and the dynamic radius, the inside of the circle is regarded as a positive label, and a circle is drawn according to the distance between the center points of two adjacent same targets and the center point of one of the targets, the outside of the circle is regarded as a negative label; the category probability map CPM is dimensionally reduced, each data point after dimension reduction is assigned a weight, a covariance matrix is constructed, eigenvalues of the covariance matrix are decomposed, the direction of the decomposed eigenvalues is determined to determine the width and height direction of the target, and the target is moved outward along the width and height direction; if the value of the current position is lower than a preset threshold, it indicates that the target boundary, so that a rotated frame is generated.

[0054] Specifically:

[0055] As Figure 1 shown, the embodiment discloses a point labeling remote sensing target directional detection method, comprising the following steps:

[0056] Acquire a point labeling dataset picture;

[0057] The picture is input into a feature extraction network based on canonical correlation analysis, and a class probability map CPM of the picture is obtained based on a hybrid channel attention mechanism;

[0058] Based on the class probability map CPM, pseudo labels of the height and width of the target are obtained through a Sobel operator;

[0059] A dynamic radius adjustment mechanism for dynamically adjusting the positive label assignment radius according to different target pseudo labels, and the center point information of the label, complete the positive and negative label assignment;

[0060] The class probability map CPM is reduced in dimension through a t-distributed random neighborhood method, a weight is assigned to each data point after dimension reduction, a covariance matrix is calculated, and the covariance matrix is eigenvalue decomposed;

[0061] A rotated frame is generated through the point labeling information, the decomposition result and the setting of a threshold.

[0062] Further, the class probability map CPM is obtained based on the hybrid channel attention mechanism for secondary feature fusion, including: performing local average pooling and global average pooling processing on the input feature map, inputting the features after local pooling and global pooling into 1D convolution for feature conversion, after 1D convolution and rearrangement, adding the features after local pooling and the features after global pooling to obtain a first fusion result; the first fusion result is subjected to an inverse pooling operation, and the first fusion result is multiplied by the input feature map to obtain a second fusion result, i.e. the class probability map CPM.

[0063] Specifically:

[0064] Step 1: Before inputting the point labeling data set into the trained PointOOD model, the trained PointOOD model weight needs to be loaded;

[0065] The point labeling remote sensing target directional detection method provided by the application does not depend on a rotated frame labeled data set, but a data set labeled with a target center point and a category. Features are learned from single point labeling and a rotated frame is generated to realize directional detection. Before using the model, the model needs to be trained according to a certain strategy, the trained model weight is loaded when used, and then the directional detection result is inferred according to the labeling information.

[0066] Step 2: The process of inputting the picture into the feature extraction network based on canonical correlation analysis and obtaining the class probability map CPM of the picture based on the hybrid channel attention mechanism includes: based on the features extracted based on RESNET, further fusion is performed through the feature optimization and fusion network based on canonical correlation analysis; through the hybrid channel attention mechanism based on global attention and local attention, the semantic information and expression ability are further fused, and are mapped to the category probability space to generate the class probability map CPM of the picture;

[0067] As a specific example: as shown in Figure 3 The three output layers of ResNet50 are each connected to three consecutive dilated convolutions. Dilated convolution is a special kind of convolution layer that allows the network to expand the receptive field without increasing the number of parameters. This means that the network can capture image information in a larger area without increasing the additional computational burden. Finally, the three-layer output is fused by using canonical correlation analysis. Canonical correlation analysis can help to fuse features from different levels by analyzing the correlation between features to select the most informative feature combination, thereby improving the quality of feature representation, reducing the dimension of features, and removing redundant features. This helps to reduce computational complexity and improve the efficiency of the model. By selecting the most relevant features, overfitting can also be reduced, which helps to improve the generalization ability of the model and make the model perform better on new data. Canonical correlation analysis provides a statistical measure of the relationship between features, which helps to understand the structure of the data and the relationship between features, thereby increasing the interpretability of the model. Let the feature vector be:

[0068] X = (x1, x2, … xp)

[0069] Y = (y1, y2, … yq)

[0070] After standardization, X * and Y * are denoted as X

[0071]

[0072] where X * and Y * are the vectors after standardization of X and Y, denoted as:

[0073]

[0074] ρ(U, V) represents the canonical correlation coefficient between two groups of variables U and V, with a value range of [-1, 1]. When ρ(U, V) = 1, it means that there is a complete positive correlation between U and V, i.e. the change of U can be completely explained by the change of V, and vice versa. When ρ(U, V) = -1, it represents a complete negative correlation. conv represents covariance, and var represents variance.

[0075] Then the optimization feature problem can be represented as That is, in image recognition tasks, one set of features is color information of the image, and the other set is texture information. Through canonical correlation analysis, a linear combination of color features and texture features can be found to discover the correlation between color changes and texture details, providing a basis for optimizing features. In terms of feature selection, canonical correlation analysis can judge the correlation between feature pairs, helping to determine which feature pairs are more important for model performance improvement. Highly correlated feature pairs can be represented by one feature, eliminating redundant features. When two features need to meet the constraint condition at the same time:

[0076] var(U)=a T ∑ XX a=1

[0077] var(V)=b T ∑ XX b=1

[0078] The linear correlation coefficients obtained by solving the optimization problem are set as a * and b * , and the fused features are represented as U * =Xa * and V * =Yb * , so that high-quality fused features can be obtained, and the output feature maps are fused two by two, at this time the output feature maps contain low-level to high-level features, rich semantic information and strong expression ability, and through the dimension reduction of canonical correlation analysis, the feature maps are more compact and efficient. At this time, the class probability map CPM obtained through the attention mechanism will be more rich and accurate, which is convenient for subsequent estimation of boundaries and angles.

[0079] The projection layer usually uses a fixed linear transformation to map high-dimensional features to a low-dimensional space. This simple mapping strategy is prone to information loss. When dealing with complex image data, the rich details and semantic information in high-dimensional features may not be fully preserved, which can affect the subsequent accurate detection and positioning of target objects. Moreover, the projection layer lacks dynamic adjustment capability for different features. It uses a unified mapping method for all input features, and cannot adaptively focus on important features according to the characteristics of target objects and scene changes, which is particularly limited when facing target objects with different feature distributions and complex backgrounds. In contrast, the mixed channel attention mechanism has obvious advantages. The mixed channel attention mechanism divides the feature map into local regions for channel-level attention calculation, which can effectively enhance local feature expression and fuse global information, enabling the model to focus on details while grasping the whole. Its lightweight design can improve performance without increasing much computational burden. It can highlight key features and improve the adaptability of the model to complex scenes, thereby overcoming the shortcomings of the projection layer and improving the accuracy and efficiency of target detection. The structure diagram of the mixed channel attention mechanism is as follows:Figure 4 As shown.

[0080] The mixed channel attention mechanism first processes the input feature map through local average pooling and global average pooling. The local pooling focuses on the features of the local region, while the global pooling captures the statistical information of the entire feature map. The features after local pooling and global pooling are both subjected to a 1D convolution (Conv1d) for feature conversion. The 1D convolution is used to compress the feature channels while keeping the spatial dimensions unchanged. After 1D convolution and rearrangement, the features after global pooling are combined with the local pooling features through an "addition" operation. This step fuses global context information in the feature map. Finally, the feature map after local and global attention processing is again subjected to an inverse pooling (UNAP) operation to restore the original spatial dimensions. Then it is combined with the original input feature through a "multiplication" operation. This process is equivalent to a kind of feature selection, which strengthens the attention to useful features.

[0081] Further, the pseudo label of the target height and width comprises: performing convolution calculation on the horizontal and vertical directions of the original class probability map CPM0 to obtain gradient values of each pixel point in the original class probability map CPM0 in the horizontal direction and in the vertical direction, calculating gradient amplitudes based on the gradient values, setting a first target threshold, and performing target boundary region judgment on the gradient amplitudes and the first target threshold. If the gradient amplitude is greater than the first target threshold, the pixel point is regarded as a boundary point of the target, and if the gradient amplitude is less than the first target threshold, the pixel point is regarded as a background or noise point; based on the boundary point of the target, the outermost boundary points of the target in the horizontal and vertical directions are obtained, the difference between the horizontal coordinates of the leftmost and rightmost boundary points in the horizontal direction is taken as the width pseudo label of the target, and the difference between the vertical coordinates of the topmost and bottommost boundary points in the vertical direction is taken as the height pseudo label of the target.

[0082] Further, the pseudo label of the target height and width comprises: performing convolution calculation on the horizontal and vertical directions of the original class probability map CPM0 to obtain gradient values of each pixel point in the original class probability map CPM0 in the horizontal direction and in the vertical direction, calculating gradient amplitudes based on the gradient values, setting a first target threshold, and performing target boundary region judgment on the gradient amplitudes and the first target threshold. If the gradient amplitude is greater than the first target threshold, the pixel point is regarded as a boundary point of the target, and if the gradient amplitude is less than the first target threshold, the pixel point is regarded as a background or noise point; based on the boundary point of the target, the outermost boundary points of the target in the horizontal and vertical directions are obtained, the difference between the horizontal coordinates of the leftmost and rightmost boundary points in the horizontal direction is taken as the width pseudo label of the target, and the difference between the vertical coordinates of the topmost and bottommost boundary points in the vertical direction is taken as the height pseudo label of the target.

[0083] Specifically:

[0084] Step 3: Based on the class probability map CPM, the process of obtaining the pseudo label of the target height and width by the Sobel operator includes: the Sobel operator processes the class probability map CPM of the picture, and calculates to obtain the pseudo label of the target height and width;

[0085] The Sobel operator is applied to the horizontal and vertical directions of the class probability map CPM respectively for convolution calculation;

[0086] The convolution kernel of the Sobel operator in the horizontal direction is usually

[0087] For each pixel point in the class probability map CPM, the gradient value of the pixel point in the horizontal direction is obtained by multiplying the surrounding neighborhood pixel values and the corresponding elements of the horizontal direction convolution kernel and then summing them up;

[0088] The convolution kernel of the Sobel operator in the vertical direction is usually

[0089] Similarly, for each pixel point in the class probability map CPM, the gradient value of the pixel point in the vertical direction is obtained by multiplying the surrounding neighborhood pixel values and the corresponding elements of the vertical direction convolution kernel and then summing them up;

[0090] For each pixel point in the class probability map CPM, the gradient amplitude G is calculated according to the gradient values in the horizontal and vertical directions. The calculation formula of the gradient amplitude is where G x is the gradient value in the horizontal direction, and G y is the gradient value in the vertical direction. This gradient amplitude reflects the degree of change of the image at the pixel point, and there is usually a larger gradient amplitude at the target boundary;

[0091] A suitable threshold is set to perform threshold processing on the calculated gradient amplitude. The pixel points with a gradient amplitude greater than the threshold are regarded as the boundary points of the target, and the pixel points with a gradient amplitude less than the threshold are considered as background or noise points and are ignored. Through threshold processing, the boundary region of the target can be preliminarily screened out, and the interference of the non-target region on the subsequent calculation is reduced;

[0092] In the boundary points after threshold processing, the outermost boundary points of the target in the horizontal and vertical directions are found. The difference between the horizontal coordinates of the leftmost and rightmost boundary points in the horizontal direction is the pseudo label of the width of the target w, and the difference between the vertical coordinates of the topmost and bottommost boundary points in the vertical direction is the pseudo label of the height of the target h.

[0093] Further, determining the target morphology includes: estimating the radius of the positive sample region based on the pseudo label of the target height and width:

[0094]

[0095] wherein r is the radius, k is the scaling factor, w is the width pseudo-label of the target, and h is the height pseudo-label of the target;

[0096] According to the density of the target in the point-labeled image, the radius of the positive sample region is adjusted again:

[0097] r adjusted = r x f(density)

[0098]

[0099] wherein r adjusted is the radius of the second adjustment of the positive sample region, and r is the radius of the positive sample region.

[0100] Specifically:

[0101] Step 4: The process of positive and negative label assignment is completed through a dynamic radius adjustment mechanism that can dynamically adjust the positive label assignment radius for different target pseudo-labels, and the center point information of the label, including: based on the consideration of the height and width difference of different targets and the target density, the radius of the positive label assignment is dynamically adjusted based on the pseudo-label, and the positive and negative label assignment is completed based on the point labeling information;

[0102] First, the pseudo-labels w and h are obtained from the class probability map CPM through the Sobel operator, which are used to estimate the radius:

[0103]

[0104] Then, considering the density of the target object in the image, the radius of the positive sample region is adjusted again based on the estimated density:

[0105] r adjusted = r x f(density)

[0106]

[0107] The radius is reduced when the target density is high to avoid overlap, and the radius is increased when the target density is low to cover more areas. The dynamic radius calculated is used to define the positive sample region, ensuring that the positive sample region can accurately reflect the actual size and shape of the target object.

[0108] Further, obtaining the target boundary includes: obtaining a plurality of grid coordinates around the labeled point in the point-labeled image, dimensionally reducing the class probability map CPM according to the plurality of grid coordinates, assigning a weight to each data point after dimensionally reducing, constructing a covariance matrix, decomposing the eigenvalues of the covariance matrix, taking the eigenvector corresponding to the maximum eigenvalue as the main direction and the perpendicular position as the auxiliary direction, setting a second target threshold, and moving along the two directions. When the value of a certain position is lower than the second target threshold, it is considered as the target boundary.

[0109] Furthermore, constructing the covariance matrix includes:

[0110]

[0111] Among them, C z Let p be the covariance matrix. i For the weight, z i For data points, μ z This represents the mean of the data points.

[0112] Furthermore, the eigenvalues ​​of the covariance matrix decomposed include:

[0113] C z v i =λ i v i

[0114] Among them, v i Let λ be the eigenvector. i These are the eigenvalues.

[0115] Specifically:

[0116] Step 5: Reduce the dimensionality of the class probabilistic graph CPM using the t-distribution random neighborhood method, assign a weight to each data point after dimensionality reduction, calculate the covariance matrix and decompose the eigenvalues ​​of the covariance matrix; generate a rotating box using the point label information, the decomposition results and the set threshold.

[0117] As a specific implementation, after obtaining the positive and negative label allocation areas in step 4, it is necessary to deduce the target's direction and boundary based on the center point of the label and the class probability map CPM. We first select a 7×7 grid around the label point, with the coordinates of 49 points as follows:

[0118] (x, y), x ∈ [-3, 3], y ∈ [-3, 3]

[0119] Next, the t-SNE method is used to reduce the dimensionality of the class probability graph CPM. In the high-dimensional space, for the data points of the class probability graph CPM, t-SNE calculates the similarity between data points using a specific formula to construct a joint probability distribution. The specific formula is:

[0120]

[0121] The formula is based on data point x. i With x j Euclidean distance between ||x i -x j || 2 And in combination with the bandwidth parameter σ, the point x is calculated. j At point x i Conditional probability p from the perspective j|i, which reflects the similarity between data points. By computing for all pairs of points, a joint probability distribution over the high-dimensional space is constructed, which embodies the local and global structure of the data.

[0122] In the low-dimensional space, t-SNE uses t-distribution to compute the similarity between data points, which is given by:

[0123]

[0124] Here y i and y j are the mapping of high-dimensional data points x i and x j in the low-dimensional space. The properties of t-distribution make it easier for similar data points to be close to each other and different clusters of points to be relatively separated in the low-dimensional space, which helps to reveal the intrinsic structure of the data.

[0125] Finally, t-SNE optimizes the layout of low-dimensional data points by minimizing a specific cost function, which is given by:

[0126]

[0127] where P and Q are the joint probability distributions over the high-dimensional and low-dimensional spaces, respectively. By minimizing this cost function, the similarity distribution of data points in the high-dimensional and low-dimensional spaces can be made as close as possible, so as to preserve the local and global structure of high-dimensional data in the low-dimensional space and achieve effective dimensionality reduction of class probability map CPM.

[0128] Then the covariance matrix of the low-dimensional data points after dimensionality reduction is calculated, and we assign a weight of p i to each point z i , and the mean of the data points is μ z , and the covariance matrix is defined as:

[0129]

[0130] Eigenvalue decomposition is performed on C z :

[0131] C z v i = λ i v i

[0132] The eigenvector v1 corresponding to the largest eigenvalue λ1 is selected as the principal direction. Since C z is a real symmetric matrix, it is guaranteed that the secondary direction is orthogonal to the principal direction. This orthogonality corresponds to the perpendicular relationship between two adjacent edges of the directional box. After determining the principal and secondary directions, starting from the labeled point, move along the two directions, and when the value at a certain position is lower than the set threshold, it indicates the target boundary.

[0133] As Figure 2 shown, the application also provides a point labeling remote sensing target directional detection device, comprising: a feature extraction module for obtaining a point labeling image, the point labeling image being an image with target center point labeling and target category labeling, inputting the point labeling image into an improved ResNet50 model to obtain a class probability map CPM; wherein the improved ResNet50 model comprises: connecting a dilated convolution layer at the output layer of the ResNet50 model for expanding the receptive field, adding a feature extraction network based on canonical correlation analysis after the dilated convolution layer for feature fusion, introducing a secondary feature fusion based on a mixed channel attention mechanism after the feature extraction network to obtain the class probability map CPM; a dynamic radius adjustment module for obtaining target height and width pseudo-labels according to an original class probability map CPM0 during training of the improved ResNet50 model, dynamically adjusting a dynamic radius adjustment mechanism of positive label distribution radius for different target pseudo-labels, and combining point labeling information for positive and negative label distribution to determine the target morphology; the dynamic radius adjustment mechanism is to dynamically adjust the radius according to different width and height pseudo-labels of different targets, and the point labeling information is the center point of the labeled target; during the process of positive and negative label distribution, a circle is drawn according to the labeled center point and the dynamic radius, the inside of the circle is the positive label, and a circle is drawn according to the distance between the center points of the two adjacent nearest same targets and the center point of one of the targets, the outside of the circle is the negative label; a rotated box generation module for dimension reduction of the class probability map CPM, assigning weights to each data point after dimension reduction, constructing a covariance matrix, decomposing the eigenvalues of the covariance matrix, determining the width and height direction of the target according to the direction of the decomposed eigenvalues, and moving outward along the width and height direction of the target, if the value at the current position is lower than a preset threshold, it indicates the target boundary, thereby generating a rotated box.

[0134] Specifically:

[0135] The embodiment also provides a point labeling remote sensing target directional detection device, and the use method of the point labeling remote sensing target directional detection device can be mutually referred to in effect with the point labeling remote sensing target directional detection method provided in the above embodiment; the point labeling remote sensing target directional detection device comprises: a feature extraction module, a feature extraction optimization and fusion module, a mixed channel attention mechanism, a dynamic radius adjustment mechanism, a prediction direction and boundary module;

[0136] The feature extraction module is used to extract basic features of the target;

[0137] The feature extraction optimization and fusion module is used to extract deep features of the target and optimize and fuse the features based on the method of canonical correlation analysis;

[0138] The mixed channel attention mechanism is used for further fusion of semantic information and spatial information, mapping the feature map to a category probability space, and generating a class probability map CPM of the picture.

[0139] The dynamic radius adjustment mechanism is used for dynamic adjustment of the radius of positive label assignment based on different target high and wide pseudo labels, so that the positive and negative label assignment work is better completed.

[0140] The prediction direction and boundary module is used for predicting the direction and boundary of each target based on point annotation information for the high-quality class probability map CPM, and generating a rotated frame.

[0141] The effects of the present application will be further described below in combination with simulation experiments and drawings.

[0142] As shown in Figure 5 , the simulation conditions of the present application are as follows:

[0143] All models in the experiment are performed on a server with 4 NVIDIA Tesla A100 40GB GPUs and 2 Intel Xeon 6330 CPUs. The experimental environment is Ubuntu 22.04, the programming language is Python 3.10, and the deep learning platform is PyTorch 2.0.1. In order to quickly converge, the AdamW optimizer is used to train all models in the experiment. The basic learning rate is set to 0.005, and the cosine strategy is used to adjust the learning rate. A total of 72 epochs are trained, and the learning rate is decayed to 0.1 times of the original every 6 epochs. The batch size of training is set to 16.

[0144] The evaluation metric (Evaluation Metric) uses the mean average precision (mAP) as the main metric to compare our method with existing alternative methods. In order to better evaluate the quality of the pseudo labels generated by points, we also compare the average intersection over union (mIoU) between the true frame and the predicted pseudo oobb.

[0145] Table 1 shows the influence of different fixed radii on different targets in the positive label assignment of the point annotation remote sensing target directional detection method of the embodiment of the present application.

[0146] Table 1

[0147]

[0148] Table 2 shows the results of each class on the DOTA-V1.0 dataset for the point annotation remote sensing target directional detection method of the embodiment of the present application and other point annotation directional detection methods.

[0149] Table 2

[0150]

[0151] Table 3 is the result of the point labeling remote sensing target directional detection method of the embodiment of the present application and other point labeling directional detection methods on DOTA-V1.0 / v1.5 / v2.0 three data sets.

[0152] Table 3

[0153]

[0154]

[0155] Table 4 is the ablation experiment of each module of the point labeling remote sensing target directional detection method of the embodiment of the present application.

[0156] Table 4

[0157]

[0158] Table 5 is the loss function weight coefficient ablation experiment of the point labeling remote sensing target directional detection method of the embodiment of the present application.

[0159] Table 5

[0160]

[0161] Table 6 is the scaling factor and density threshold ablation experiment of the point labeling remote sensing target directional detection method of the embodiment of the present application.

[0162] Table 6

[0163]

[0164] Table 7 is the t-SNE bandwidth parameter ablation experiment of the point labeling remote sensing target directional detection method of the embodiment of the present application.

[0165] Table 7

[0166]

[0167] Table 8 is the mIoU and detection accuracy comparison of the point labeling remote sensing target directional detection method of the embodiment of the present application and other point labeling directional target detection methods on DOTA-V1.0 / v1.5 / v2.0 three data sets.

[0168] Table 8

[0169]

[0170]

[0171] The above described embodiments are only to illustrate the preferred modes of the present application, and are not intended to limit the scope of the present application. Any modification and improvement made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A point annotation remote sensing target directional detection method, characterized in that, The method comprises the following steps: acquiring a point-labeled image, the point-labeled image being an image with a target center point label and a target category label; inputting the point-labeled image into an improved ResNet50 model to obtain a category probability map CPM; wherein the improved ResNet50 model comprises: connecting a dilated convolution layer to an output layer of the ResNet50 model to expand the receptive field, adding a feature extraction network based on canonical correlation analysis after the dilated convolution layer for feature fusion, and introducing a hybrid channel attention mechanism based on the feature extraction network for secondary feature fusion to obtain the category probability map CPM; during training of the improved ResNet50 model, obtaining a target height and width pseudo label according to an original category probability map CPM0, dynamically adjusting a positive label distribution radius dynamic radius adjustment mechanism for different target pseudo labels, and combining point-labeled information to perform positive and negative label distribution to determine the target morphology; the dynamic radius adjustment mechanism is to dynamically adjust the radius according to different width and height pseudo labels of different targets, and the point-labeled information is the center point of the labeled target; during the positive and negative label distribution process, a circle is drawn according to the labeled center point and the dynamic radius, and the inside of the circle is regarded as a positive label; a circle is drawn according to the distance between the center points of the nearest two same targets and the center point of one of the targets, and the outside of the circle is regarded as a negative label; dimensionality reduction is performed on the category probability map CPM, a weight is assigned to each data point after dimensionality reduction, a covariance matrix is constructed, eigenvalues of the covariance matrix are decomposed, the direction of the eigenvalues obtained by decomposition is used to determine the width and height direction of the target, and the target is moved outward along the width and height direction; if the value at the current position is lower than a preset threshold, it indicates that the target boundary, thereby generating a rotated frame.

2. The point annotated remote sensing target orientation detection method of claim 1, wherein, The hybrid channel attention mechanism is used for secondary feature fusion to obtain the category probability map CPM, which comprises: performing local average pooling and global average pooling on the input feature map, inputting the features after local and global pooling into a 1D convolution for feature conversion, adding the features after local and global pooling after 1D convolution and rearrangement, and obtaining a first fusion result; performing an inverse pooling operation on the first fusion result, multiplying the first fusion result by the input feature map to obtain a second fusion result, i.e., the category probability map CPM.

3. The point annotated remote sensing target orientation detection method of claim 1, wherein, The target height and width pseudo label is obtained by: performing convolution calculation on the horizontal and vertical directions of the original category probability map CPM0 to obtain the gradient value of each pixel point in the horizontal direction and the gradient value in the vertical direction of the original category probability map CPM0, calculating the gradient amplitude based on the gradient values, setting a first target threshold, performing target boundary region judgment on the gradient amplitude and the first target threshold, if the gradient amplitude of a pixel point is greater than the first target threshold, the pixel point is regarded as a target boundary point, and if the gradient amplitude of a pixel point is less than the first target threshold, the pixel point is regarded as a background or noise point. Based on the boundary points of the target, outermost boundary points of the target in horizontal and vertical directions are obtained, a difference between horizontal coordinates of leftmost and rightmost boundary points in the horizontal direction is taken as a width pseudo-label of the target, and a difference between vertical coordinates of uppermost and lowermost boundary points in the vertical direction is taken as a height pseudo-label of the target.

4. The point annotated remote sensing target orientation detection method of claim 3, wherein, The gradient values of each pixel point in the original class probability map CPM0 in the horizontal direction and the gradient values in the vertical direction include: The horizontal direction of the original class probability map CPM0 is convoluted to calculate, the surrounding neighborhood pixel values of each pixel point in the original class probability map CPM0 are multiplied by the corresponding elements of the horizontal direction convolution kernel respectively, and then summed to obtain the gradient value of the pixel point in the horizontal direction; The vertical direction of the original class probability map CPM0 is convoluted to calculate, the surrounding neighborhood pixel values of each pixel point in the original class probability map CPM0 are multiplied by the corresponding elements of the vertical direction convolution kernel respectively, and then summed to obtain the gradient value of the pixel point in the vertical direction.

5. The point annotated remote sensing target orientation detection method of claim 1, wherein, The target morphology is determined, including: Based on the pseudo-labels of the target height and width, the radius of the positive sample region is estimated: Wherein, r is the radius, k is the scaling factor, w is the width pseudo-label of the target, and h is the height pseudo-label of the target; According to the density of the target in the point-labeled image, the radius of the positive sample region is adjusted again: r adjusted = r x f(density) wherein r adjusted is the radius of the secondary adjustment positive sample region, r is the radius of the positive sample region, f(density) is a density factor, d min is the distance of two adjacent nearest same target center points, i.e., the distance of point labels, and T is a density threshold.

6. The point annotated remote sensing target orientation detection method of claim 1, wherein, The target boundary is obtained, including: A plurality of grid coordinates around the labeled point in the point-labeled image are obtained, the class probability map CPM is dimensionally reduced according to the plurality of grid coordinates, each data point after dimension reduction is assigned a weight, a covariance matrix is constructed, eigenvalues of the covariance matrix are decomposed, a feature vector corresponding to the largest eigenvalue is taken as a main direction and a vertical position thereof is taken as a secondary direction, a second target threshold is set, and the two directions are moved. When the value of a certain position is lower than the second target threshold, it is considered as the target boundary.

7. The point annotated remote sensing target orientation detection method of claim 6, wherein, The covariance matrix is constructed, including: where C z is the covariance matrix, p i is the weight, z i is the data point, μ z is the mean of the data points, and T is the transpose.

8. The point annotated remote sensing target orientation detection method of claim 6, wherein, The eigenvalues of the covariance matrix are decomposed, including: Czvi = λ i vi where v i is the eigenvector, and λ i is the eigenvalue.

9. A point labeling remote sensing target directional detection device, characterized in that, Including: The feature extraction module is configured to obtain a point-labeled image, the point-labeled image being an image with a target center point label and a target class label, input the point-labeled image into an improved ResNet50 model, and obtain a class probability map CPM; wherein the improved ResNet50 model includes: connecting a dilated convolution layer to an output layer of the ResNet50 model to expand the receptive field, adding a feature extraction network based on canonical correlation analysis after the dilated convolution layer for feature fusion, introducing a hybrid channel attention mechanism based on the feature extraction network for secondary feature fusion, and obtaining the class probability map CPM. The dynamic radius adjustment module is configured to, during training of the improved ResNet50 model, obtain target high and wide pseudo labels according to the original class probability map CPM0, dynamically adjust a dynamic radius adjustment mechanism of a positive label distribution radius according to different target pseudo labels, and combine point labeling information to determine a target morphology; the dynamic radius adjustment mechanism is to dynamically adjust the radius according to different high and wide pseudo labels of different targets, and the point labeling information is a center point of a labeled target; during the positive and negative label distribution process, a circle is drawn according to the labeled center point and the dynamic radius, the inside of the circle is regarded as a positive label, and a circle is drawn according to the distance between the center points of two adjacent nearest same targets and the center point of one of the targets, and the outside of the circle is regarded as a negative label. The rotated box generation module is configured to reduce the dimension of the class probability map CPM, assign a weight to each data point after the dimension reduction, construct a covariance matrix, decompose eigenvalues of the covariance matrix, determine a width and height direction of a target according to a direction of the decomposed eigenvalues, and move outward along the width and height direction of the target; if a value at a current position is lower than a preset threshold, it indicates a target boundary, so as to generate a rotated box.

Images