Space object ISAR image key point positioning and attitude estimation method

Abstract

The invention relates to a space object ISAR image key point positioning and attitude estimation method. The method comprises the following steps: acquiring an ISAR image of a to-be-processed space object; and inputting the ISAR image into a pre-trained attitude estimation model to obtain an attitude evaluation result of the to-be-processed space object, the attitude evaluation result including an object category, a detection frame, a key point position and a main body attitude angle, and performing parallel processing through the attitude estimation model to obtain four evaluation results. By means of the method, the technical problems that the universality is poor and the real-time requirement cannot be met due to the fact that an existing method only depends on a single feature are effectively solved.

Description

A method for key point localization and attitude estimation of space objects in ISAR images Technical Field

[0001] This invention relates to the field of spacecraft attitude estimation technology, and more specifically, to a method for locating key points and estimating the attitude of a space object in an ISAR image. Background Technology

[0002] Attitude estimation of space objects based on ISAR images can obtain features such as the flight status of space objects, which can effectively assist in many aerospace mission applications such as dynamic monitoring of key areas and situation analysis, and is of great significance to ensuring the safety of space assets.

[0003] Existing attitude estimation methods are usually based on the extraction of key points or linear components in images, which fails to make full use of the prior knowledge of the radar imaging characteristics of space objects. Current methods for extracting satellite target attitude based on key point extraction networks [1] are not universal for certain special attitudes; methods based on linear component extraction [2] require Hough transform or inverse Radon transform of the image sequence and perspective transformation with the ground truth to obtain the target attitude, which is difficult to meet the needs of real-time estimation of instantaneous attitude.

[0004] Among them, [1]XIE Pengfei, ZHANG Lei, DU Chuan, et al. Space targetattitude estimation from ISAR image sequences with key point extraction network[J]. IEEE Signal Processing Letters, 2021(28):1041-1045.

[0005] Huo Kai, Kou Peng, Liu Yongxiang, et al. On-orbit attitude estimation of complex structure spacecraft in sequence ISAR images [J]. Systems Engineering & Electronics, 2023, 45(8):2438-2445.

[0006] Therefore, there is an urgent need in the existing technology for a solution that is universal and meets the requirements for real-time estimation of instantaneous attitude. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method for key point localization and attitude estimation of space objects in ISAR images, aiming to solve at least one of the above-mentioned technical problems.

[0008] In a first aspect, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a method for key point localization and pose estimation of space objects in ISAR images, the method comprising: acquiring an ISAR image of a space object to be processed; inputting the ISAR image into a pre-trained pose estimation model to obtain a pose evaluation result of the space object to be processed, the pose evaluation result including object category, detection box, key point position and subject pose angle, the pose estimation model including a feature extraction layer, a feature fusion layer and a multi-task detection layer; the pose estimation model is trained based on the following method: acquiring training data, the training data including ISAR images corresponding to different types of space objects, each space object corresponding to a category label, a pose angle label, and a key point position. The system detects bounding box labels and keypoint labels; it extracts multi-scale basic features for each spatial object in the training data through a feature extraction layer, which includes features at different scales; it then fuses these multi-scale basic features across scales through a feature fusion layer to obtain fused features for each spatial object; for each spatial object, a multi-task detection layer processes the fused features in parallel to obtain a predicted pose assessment result, which includes the predicted object category, predicted bounding box, predicted keypoint locations, and predicted subject pose angle; and finally, it trains a pose estimation model based on each spatial object, the predicted pose assessment result, and the multiple labels corresponding to each spatial object.

[0009] The beneficial effects of this invention are as follows: by inputting the ISAR image of the spatial object to be processed into a pre-trained pose estimation model, end-to-end processing is achieved. The feature extraction layer and feature fusion layer in the model structure work together to fully explore the cross-scale correlation of multi-scale basic features. The multi-task detection layer outputs object category, detection box, key point position and subject pose in parallel, transforming the traditional pose calculation into a regression prediction problem with key point supervision. By using multiple labels in the training data as priors, the convergence speed and instantaneous pose estimation accuracy are improved. At the same time, the joint optimization of classification and detection tasks enhances the feature generalization ability. Finally, a unified framework that does not require complex post-processing and meets real-time requirements is formed, effectively solving the technical problem that existing methods rely on only a single feature, resulting in poor universality and inability to meet real-time requirements.

[0010] Based on the above technical solution, the present invention can be further improved as follows.

[0011] Furthermore, a feature enhancement layer is set between the aforementioned feature extraction layer and feature fusion layer. The feature enhancement layer includes two external memory units connected in sequence. The two external memory units are learnable weight matrices that describe the common features of multiple typical spatial objects. During the training process, the pose estimation model also includes: for each spatial object, matrix multiplication is performed on the multi-scale basic features of the spatial object through the first external memory unit of the two external memory units to calculate the similarity between the multi-scale basic features and the weight matrix corresponding to the first external memory unit, resulting in a similarity matrix. The similarity matrix is ​​then normalized to obtain a first attention weight matrix, which describes the similarity distribution between the features of each spatial location in the multi-scale basic features and the common features of multiple typical spatial objects; for each spatial object, the first attention weight matrix is ​​weighted and fused through the second external memory unit of the two external memory units to obtain the enhanced features; the multi-scale basic features of each spatial object are fused across scales through the feature fusion layer to obtain the fused features of each spatial object, including: the enhanced features of each spatial object are fused across scales through the feature fusion layer to obtain the fused features of each spatial object.

[0012] Furthermore, the above-mentioned pose estimation model is trained based on the evaluation results of each spatial object and the predicted pose, and the multiple labels corresponding to each spatial object. This includes: for each spatial object, determining the category classification loss based on the predicted object category and category label, determining the detection box regression loss based on the predicted detection box and detection box label, determining the keypoint localization loss based on the predicted keypoint location and keypoint label, and determining the pose regression loss based on the predicted subject pose angle and pose angle label; determining the total loss based on the category classification loss, detection box regression loss, keypoint localization loss, and pose regression loss of all spatial objects, and training the pose estimation model based on the total loss.

[0013] Furthermore, for each spatial object, the attitude angle labels include yaw angle labels, pitch angle labels, and roll angle labels. The predicted main attitude angles include predicted yaw angle, predicted pitch angle, and predicted roll angle. Based on the predicted main attitude angles and attitude angle labels, the attitude regression loss is determined, including: determining the yaw angle loss based on the yaw angle label and the predicted yaw angle; determining the pitch angle loss based on the pitch angle label and the predicted pitch angle; and determining the roll angle loss based on the roll angle label and the predicted roll angle. The attitude regression loss includes yaw angle loss, pitch angle loss, and roll angle loss.

[0014] Furthermore, the aforementioned space objects to be processed are three-axis attitude-stabilized satellites, including at least one of the Dragon spacecraft, space stations, and Earth observation satellites.

[0015] Secondly, to address the aforementioned technical problems, this invention also provides a device for keypoint localization and attitude estimation of ISAR images of space objects. This device includes: an acquisition module for acquiring ISAR images of the space object to be processed; and an attitude estimation module for inputting the ISAR images into a pre-trained attitude estimation model to obtain attitude evaluation results for the space object to be processed. The attitude evaluation results include object category, detection box, keypoint locations, and main body attitude angles. The attitude estimation model includes a feature extraction layer, a feature fusion layer, and a multi-task detection layer. The attitude estimation model is trained based on the following training module: the training module is used to: acquire training data, which includes ISAR images corresponding to different types of space objects, with each space object corresponding to one keypoint localization and attitude estimation model. The training data includes category labels, pose angle labels, bounding box labels, and keypoint labels. A feature extraction layer extracts multi-scale basic features for each spatial object from the training data, including features at different scales. A feature fusion layer fuses these multi-scale basic features across scales to obtain fused features for each spatial object. For each spatial object, a multi-task detection layer processes the fused features in parallel to obtain a predicted pose assessment result, which includes the predicted object category, predicted bounding box, predicted keypoint locations, and predicted subject pose angle. Based on each spatial object, the predicted pose assessment result, and the multiple labels corresponding to each spatial object, a pose estimation model is trained.

[0016] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for locating key points and estimating attitude of space objects in ISAR images according to the present application.

[0017] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for locating key points and estimating attitude of space objects in ISAR images according to the present application.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.

[0020] Figure 1 is a flowchart illustrating a method for locating key points and estimating attitude in ISAR images of space objects according to an embodiment of the present invention; Figure 2 is a schematic diagram of a three-dimensional geometric model of a Dragon spacecraft according to an embodiment of the present invention; Figure 3 is a schematic diagram of an ISAR image of a Dragon spacecraft according to an embodiment of the present invention; Figure 4 is a schematic diagram of a network structure according to an embodiment of the present invention; Figure 5 is a schematic diagram of an External-attention module according to an embodiment of the present invention; Figure 6 is a schematic diagram of a device for locating key points and estimating attitude in ISAR images of space objects according to an embodiment of the present invention; Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0021] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0022] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0023] The solution provided in this invention can be applied to any application scenario involving the estimation of the attitude of spatial objects. The solution provided in this invention can be executed by any electronic device, such as a user's terminal device, including at least one of the following: smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart TV, or smart in-vehicle device.

[0024] This invention provides a possible implementation, as shown in Figure 1, which is a flowchart of a method for keypoint localization and pose estimation of ISAR images of space objects. This method can be executed by any electronic device, such as a terminal device, or jointly executed by a terminal device and a server. For ease of description, the method provided by this invention will be described below using a terminal device as the execution subject. As shown in the flowchart in Figure 1, the method may include the following steps: S10, acquiring an ISAR image of the space object to be processed; S20, inputting the ISAR image into a pre-trained pose estimation model to obtain the pose evaluation result of the space object to be processed. The pose evaluation result includes the object category, detection box, keypoint position, and subject pose angle. The pose estimation model includes a feature extraction layer, a feature fusion layer, and a multi-task detection layer. The pose estimation model is trained based on the following method: acquiring training data, which includes ISAR images corresponding to different types of space objects, with each space object corresponding to a category label. The training data includes labels, pose angle labels, bounding box labels, and keypoint labels. A feature extraction layer extracts multi-scale basic features for each spatial object from the training data, including features at different scales. A feature fusion layer fuses these multi-scale basic features across scales to obtain fused features for each spatial object. For each spatial object, a multi-task detection layer processes the fused features in parallel to obtain a predicted pose assessment result, which includes the predicted object category, predicted bounding box, predicted keypoint locations, and predicted subject pose angle. Based on each spatial object, the predicted pose assessment result, and the multiple labels corresponding to each spatial object, a pose estimation model is trained.

[0025] The method of this invention achieves end-to-end processing by inputting the ISAR image of the spatial object to be processed into a pre-trained pose estimation model. The feature extraction layer and feature fusion layer in the model structure work together to fully explore the cross-scale correlation of multi-scale basic features. The multi-task detection layer outputs object category, detection box, key point position and subject pose in parallel, transforming the traditional pose calculation into a regression prediction problem with key point supervision. By using multiple labels in the training data as priors, the convergence speed and instantaneous pose estimation accuracy are improved. At the same time, the joint optimization of classification and detection tasks enhances the feature generalization ability. Finally, a unified framework that does not require complex post-processing and meets real-time requirements is formed, effectively solving the technical problem that existing methods rely on only a single feature, resulting in poor universality and inability to meet real-time requirements.

[0026] The following specific embodiments further illustrate the solution of the present invention. Addressing the problem that current methods for spatial object attitude estimation, which only utilize keypoints and linear features, do not fully leverage the prior characteristics of spatial object radar imaging, this invention proposes a spatial object attitude estimation method based on keypoint localization and regression networks. This method fully utilizes the prior characteristics of target imaging and additionally adds semantic keypoint location information of the spatial object. Based on an end-to-end learning method, it calculates the object category, detection box, keypoint positions, and subject attitude angle, achieving instantaneous attitude estimation of the spatial object. In this embodiment, the provided method for keypoint localization and attitude estimation of spatial objects in ISAR images may include the following steps: S10, acquiring the ISAR image of the spatial object to be processed; wherein, the ISAR image can be obtained through real-time observation of the spatial target using a deployed Ka-band inverse synthetic aperture radar.

[0027] Optionally, the aforementioned space object to be processed is a three-axis attitude-stabilized satellite, including at least one of the Dragon spacecraft, space station, and Earth observation satellite.

[0028] S20, the ISAR image is input into the pre-trained pose estimation model to obtain the pose evaluation result of the spatial object to be processed. The pose evaluation result includes the object category, detection box, key point position, and main body pose angle. The pose estimation model includes a feature extraction layer (see Backbone in Figure 4), a feature fusion layer (see Neck in Figure 4), and a multi-task detection layer (see Head in Figure 4). The above pose estimation model is trained in the following way: acquiring training data, which includes ISAR images corresponding to different types of spatial objects, each spatial object corresponding to a category label, pose angle label, detection box label, and key point label; and extracting the pose from the feature extraction layer. Multi-scale basic features of each spatial object in the training data are extracted. These multi-scale basic features include features at different scales, such as shallow, medium, and deep features. A feature fusion layer is used to fuse the multi-scale basic features of each spatial object across scales to obtain the fused features of each spatial object. For each spatial object, a multi-task detection layer is used to process the fused features of the spatial object in parallel to obtain the predicted pose evaluation result of the spatial object. The predicted pose evaluation result includes the predicted object category, predicted detection box, predicted keypoint position, and predicted subject pose angle. Based on each spatial object, the predicted pose evaluation result, and the multiple labels corresponding to each spatial object, a pose estimation model is trained.

[0029] Optionally, each ISAR image in the training data can be determined by generating a simulated ISAR image through echo simulation and imaging processing based on a three-dimensional model of the space object and radar parameters using electromagnetic simulation software (such as CST or FEKO).

[0030] Specifically, firstly, a three-dimensional geometric model of the space object is constructed using publicly available technical documents or CAD data, including details such as the main structure, solar panels, and their connecting components. Then, based on the relationship between orbital dynamics and imaging geometry, the yaw angle, pitch angle, and roll angle of the space object are set as attitude priors in the orbital coordinate system, and the attitude angles are discretized and sampled at 1° intervals within the ranges of 0~90°, 0~360°, and 0~90°. Finally, using electromagnetic simulation software (such as CST and FEKO), the incident angle and polarization mode of the Ka-band radar parameters are configured, and the time-varying scattering field of the target is calculated based on the physical optics method to obtain echo data. After deskewing, pulse compression, motion compensation, and azimuth matched filtering, the ISAR image is obtained.

[0031] As an example, the space object is the Dragon spacecraft. See Figure 2, which is a schematic diagram of the three-dimensional geometric model of the Dragon spacecraft, and Figure 3, which is a schematic diagram of the ISAR image of the Dragon spacecraft.

[0032] Optionally, the above-mentioned feature fusion layer performs cross-scale fusion of the multi-scale basic features of each spatial object to obtain the fused features of each spatial object. This includes: adopting a feature pyramid network structure, firstly performing a top-down upsampling operation, passing deep features layer by layer to shallow layers and adding them element-wise with the shallow features of the corresponding scale, thus achieving the fusion of high-level features to low-level spatial resolution; then performing a bottom-up downsampling operation, passing the fused shallow features layer by layer back to deep layers and adding them element-wise with the deep features of the corresponding scale, thus achieving the re-fusion of low-level spatial details to high-level semantics; and then adding the original features of each scale to the fused features element-wise through lateral connections, so that the output feature map of each scale simultaneously contains the original detailed information of that layer and the cross-layer context information, finally constructing a feature pyramid composed of multi-scale feature maps, and using the feature pyramid as the fused feature. Here, the fused feature refers to the intermediate feature map generated after bidirectional cross-scale propagation from top to bottom and bottom to top.

[0033] Optionally, the feature extraction layer adopts the CSPDarknet structure, and the multi-task detection layer includes parallel connected object classification branches, bounding box regression branches, key point localization branches, and pose angle regression branches.

[0034] For each spatial object, the fused features of the spatial object are processed in parallel through a multi-task detection layer to obtain the predicted pose evaluation result of the spatial object. This includes: for each spatial object, the fused features of the spatial object are processed through an object classification branch to obtain the predicted object category; the fused features of the spatial object are processed through a detection box regression branch to obtain the predicted detection box; the fused features of the spatial object are processed through a keypoint localization branch to obtain the predicted keypoint position; and the fused features of the spatial object are processed through a pose angle regression branch to obtain the predicted subject pose angle. These four branches are processed in parallel to accelerate the processing speed.

[0035] Optionally, this solution can be improved based on the YOLOv8 object detection algorithm, as shown in Figure 4. It can achieve tasks such as spatial object key point detection, classification, localization and pose estimation, and obtain the model category of different spatial objects, different semantic key point categories, semantic key point coordinate information and the pose orientation information of the main body of the spatial object.

[0036] When detecting keypoints of spatial objects, such as solar panels, an attention mechanism is needed to enhance the network's feature extraction capabilities in order to obtain local information about the solar panel structure and improve the keypoint detection effect. However, a significant drawback of the self-attention mechanism is its extremely high computational cost and redundancy. Furthermore, the self-attention mechanism only utilizes information within its own samples, ignoring the potential connections between different samples, which are meaningful in keypoint detection tasks. Therefore, optionally, a feature enhancement layer is set between the feature extraction layer and the feature fusion layer. The feature enhancement layer includes two external memory units connected in sequence. These two external memory units are learnable weight matrices that describe the common features of multiple typical spatial objects. Specifically, the first external memory unit describes the overall features of multiple typical spatial objects, and the second external memory unit describes the detailed features. The pose estimation model then further includes the following during training: for each spatial object, matrix multiplication is performed on the multi-scale basic features of the spatial object through the first external memory unit, and the similarity between the multi-scale basic features and the weight matrix corresponding to the first external memory unit is calculated to obtain the similarity score. The similarity matrix is ​​normalized to obtain the first attention weight matrix, which describes the similarity distribution between the features of each spatial location in the multi-scale basic features and the common features of multiple typical spatial objects. The similarity matrix describes the degree of similarity between a spatial object and each typical spatial object corresponding to the first external memory unit. For each spatial object, the first attention weight matrix is ​​weighted and fused through the second external memory unit of the two external memory units to obtain the enhanced features. The multi-scale basic features of each spatial object are fused across scales through a feature fusion layer to obtain the fused features of each spatial object. This includes fusing the enhanced features of each spatial object across scales through the feature fusion layer to obtain the fused features of each spatial object.

[0037] It needs to be explained that the first external memory unit is used to determine the similarity between a spatial object and multiple typical spatial objects, while the second external memory unit is used to learn the detailed features of different typical spatial objects. The purpose of weighted fusion of the first attention weight matrix is ​​to use the first attention weight matrix as weight coefficients and perform matrix multiplication with the weight matrix corresponding to the second external memory unit. This results in weighted aggregation of the common features (detail features) of multiple typical spatial objects stored in the second external memory unit, generating enhanced features. This ensures that the feature vector of each spatial location in the enhanced features is represented as an attention-weighted combination of the common features of multiple typical spatial objects, thereby integrating the global statistical information of the dataset into the representation of the current input features.

[0038] Specifically, the Norm method is used in each external memory unit to avoid attention failure caused by an excessively large feature vector.

[0039] ; Where F represents the input feature matrix, obtained by flattening the multi-scale basic features extracted by Backbone, with dimensions [N, d] (N=H×W is the number of spatial locations, d is the feature dimension); M k T Let represent the transpose matrix of the first external memory unit, with dimensions [d, S] (where S is the size of the memory unit, and S < d / s). <N);FM k T The matrix multiplication result is represented by the dimension [N, S], which indicates the similarity between the input feature and S global memory patterns; Norm represents a normalization function (such as Softmax) that converts the similarity into a probability distribution. Let F represent the first attention weight matrix, with dimensions [N, S], describing the attention distribution of each spatial location to the common features of S typical spatial objects; out =AM v In the diagram: A represents the first attention weight matrix, which serves as the weighting coefficient; M... v Let AM represent the weight matrix of the second external memory unit, with dimensions [S, d]. v The result of matrix multiplication is represented by [N, d], indicating attention-weighted aggregation of S global feature patterns; F out This represents the enhanced feature, which has the same dimension as the input feature F, so that the feature vector at each position incorporates global statistical information from the dataset.

[0040] This new attention mechanism can also be called External-attention. External attention implicitly learns the features of the entire dataset by introducing two external memory units. For the specific structure, please refer to the schematic diagram shown in Figure 5.

[0041] Optionally, the above-mentioned training of the pose estimation model based on the evaluation results of each spatial object and the predicted pose, and the multiple labels corresponding to each spatial object, includes: for each spatial object, determining the category classification loss based on the predicted object category and category label, determining the detection box regression loss based on the predicted detection box and detection box label, determining the keypoint localization loss based on the predicted keypoint location and keypoint label, and determining the pose regression loss based on the predicted subject pose angle and pose angle label; determining the total loss based on the category classification loss, detection box regression loss, keypoint localization loss, and pose regression loss of all spatial objects, and training the pose estimation model based on the total loss.

[0042] Furthermore, for each spatial object, the attitude angle labels include yaw angle labels, pitch angle labels, and roll angle labels. The predicted main attitude angles include predicted yaw angle, predicted pitch angle, and predicted roll angle. Based on the predicted main attitude angles and attitude angle labels, the attitude regression loss is determined, including: determining the yaw angle loss based on the yaw angle label and predicted yaw angle; determining the pitch angle loss based on the pitch angle label and predicted pitch angle; and determining the roll angle loss based on the roll angle label and predicted roll angle. The attitude regression loss includes the yaw angle loss, pitch angle loss, and roll angle loss, which are the sum of the three.

[0043] Specifically, the total loss mentioned above can be expressed as: Where L represents the total loss, , , These represent the yaw angle loss, pitch angle loss, and roll angle loss, respectively. Represents the category classification loss. This indicates the loss from key point location. This represents the regression loss of the detection box. , , , , , The weights for each loss.

[0044] , , They are represented as follows: ; ; ;in, , and These are the network pairs for the first... The predicted values ​​of yaw angle, pitch angle and roll angle of a spatial object, namely the predicted yaw angle, predicted pitch angle and predicted roll angle; The total number of samples, i.e., the number of spatial objects in the training data, is calculated using the mean squared variance. , , .

[0045] Category classification loss Represented as: ;in, Category label (can be represented by 0 or 1, indicating the first category) (Whether each category is a positive sample) To predict the object category.

[0046] The loss for keypoint localization is expressed as: ;in, It's the number of key points. , The first The first spatial object The predicted values ​​of the x-coordinate and y-coordinate of each key point, i.e., the predicted location of the key point. , The first The first spatial object The true values ​​of the x and y coordinates of each key point, that is, the true values ​​corresponding to the key point labels.

[0047] The bounding box regression loss is expressed as: ;in, This is the predicted value of the detection box, i.e., the predicted detection box. This represents the ground truth bounding box corresponding to the bounding box label. CIoU (Complete IoU) is a loss function used for bounding box regression in object detection.

[0048] Based on the ISAR imaging characteristics of space objects, this scheme can simulate labeled radar ISAR image datasets (images). Yaw angle Pitch angle Roll angle Meanwhile, unlike traditional classification and object detection problems, pose estimation, as a regression problem, requires more difficult feature extraction. In order to improve the accuracy of pose estimation, this invention introduces prior information on the location of key points of spatial objects to assist network training and convergence.

[0049] The advantages of this application compared to existing technologies are: 1) For the spatial object pose estimation problem, the traditional pose calculation function optimization problem is transformed into a pose angle regression prediction problem, which makes full use of the prior characteristics of spatial object imaging; 2) Compared with the regression prediction method, prior information of the key point position of the solar panel of the spatial object is introduced, which adds additional information to assist network training convergence and improves the instantaneous pose estimation accuracy; 3) The single spatial object pose estimation problem is transformed into a multi-task learning and prediction problem, which can simultaneously calculate the spatial object category, detection box, semantic key point position, and subject pose information, providing an end-to-end learning solution.

[0050] Based on the same principle as the method shown in Figure 1, this embodiment of the invention also provides a space object ISAR image keypoint localization and attitude estimation device 20, as shown in Figure 6. This space object ISAR image keypoint localization and attitude estimation device 20 may include an acquisition module 210 and an attitude estimation module 220, wherein: the acquisition module 210 is used to acquire the ISAR image of the space object to be processed; the attitude estimation module 220 is used to input the ISAR image into a pre-trained attitude estimation model to obtain the attitude evaluation result of the space object to be processed. The attitude evaluation result includes the object category, detection box, keypoint position, and main body attitude angle. The attitude estimation model includes a feature extraction layer, a feature fusion layer, and a multi-task detection layer. The attitude estimation model is trained based on the following training module: the training module is used to: acquire training data, training data... The training data includes ISAR images of different types of space objects, each with a category label, pose angle label, bounding box label, and keypoint label. A feature extraction layer extracts multi-scale basic features for each space object from the training data, including features at different scales. A feature fusion layer fuses these multi-scale basic features across scales to obtain fused features for each space object. For each space object, a multi-task detection layer processes the fused features in parallel to obtain a predicted pose assessment result, which includes the predicted object category, predicted bounding box, predicted keypoint location, and predicted subject pose angle. Based on each space object, the predicted pose assessment result, and the multiple labels corresponding to each space object, a pose estimation model is trained.

[0051] Optionally, a feature enhancement layer is further provided between the aforementioned feature extraction layer and feature fusion layer. The feature enhancement layer includes two external memory units connected in sequence. The two external memory units are learnable weight matrices that describe the common features of multiple typical spatial objects. During the training process, the pose estimation model also includes an enhancement module, which is used to: for each spatial object, perform matrix multiplication on the multi-scale basic features of the spatial object through the first external memory unit of the two external memory units, calculate the similarity between the multi-scale basic features and the weight matrix corresponding to the first external memory unit, obtain a similarity matrix, normalize the similarity matrix to obtain a first attention weight matrix, which is used to describe the similarity distribution between the features of each spatial location in the multi-scale basic features and the common features of multiple typical spatial objects; for each spatial object, perform weighted fusion on the first attention weight matrix through the second external memory unit of the two external memory units to obtain the enhanced features; the feature fusion layer specifically performs cross-scale fusion on the enhanced features of each spatial object to obtain the fused features of each spatial object.

[0052] Optionally, when the above training module trains the pose estimation model based on the evaluation results of each spatial object and the multiple labels corresponding to each spatial object, it is specifically used for: for each spatial object, determining the category classification loss based on the predicted object category and category label, determining the detection box regression loss based on the predicted detection box and detection box label, determining the keypoint localization loss based on the predicted keypoint location and keypoint label, and determining the pose regression loss based on the predicted subject pose angle and pose angle label; determining the total loss based on the category classification loss, detection box regression loss, keypoint localization loss and pose regression loss of all spatial objects, and training the pose estimation model based on the total loss.

[0053] Optionally, for each spatial object, the attitude angle labels include yaw angle labels, pitch angle labels, and roll angle labels, and the predicted main attitude angles include predicted yaw angle, predicted pitch angle, and predicted roll angle. When the training module determines the attitude regression loss based on the predicted main attitude angles and attitude angle labels, it is specifically used to: determine the yaw angle loss based on the yaw angle label and the predicted yaw angle; determine the pitch angle loss based on the pitch angle label and the predicted pitch angle; and determine the roll angle loss based on the roll angle label and the predicted roll angle. The attitude regression loss includes yaw angle loss, pitch angle loss, and roll angle loss.

[0054] Optionally, the aforementioned space object to be processed is a three-axis attitude-stabilized satellite, including at least one of the Dragon spacecraft, space station, and Earth observation satellite.

[0055] The space object ISAR image key point localization and attitude estimation device of the present invention can execute the space object ISAR image key point localization and attitude estimation method provided in the present invention. The implementation principle is similar. The actions performed by each module and unit in the space object ISAR image key point localization and attitude estimation device in each embodiment of the present invention correspond to the steps in the space object ISAR image key point localization and attitude estimation method in each embodiment of the present invention. For detailed functional descriptions of each module of the space object ISAR image key point localization and attitude estimation device, please refer to the descriptions in the corresponding space object ISAR image key point localization and attitude estimation methods shown above. They will not be repeated here.

[0056] The aforementioned space object ISAR image key point localization and attitude estimation device can be a computer program (including program code) running on a computer device. For example, the space object ISAR image key point localization and attitude estimation device is an application software; the device can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.

[0057] In some embodiments, the space object ISAR image key point localization and attitude estimation device provided in this invention can be implemented in a combination of hardware and software. As an example, the space object ISAR image key point localization and attitude estimation device provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the space object ISAR image key point localization and attitude estimation method provided in this invention. For example, the processor in the form of a hardware decoding processor can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0058] In other embodiments, the space object ISAR image key point localization and attitude estimation device provided in the embodiments of the present invention can be implemented in software. Figure 6 shows the space object ISAR image key point localization and attitude estimation device stored in the memory. It can be software in the form of programs and plug-ins, and includes a series of modules, including an acquisition module 210 and an attitude estimation module 220, for implementing the space object ISAR image key point localization and attitude estimation method provided in the embodiments of the present invention.

[0059] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0060] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.

[0061] In one optional embodiment, an electronic device is provided, as shown in FIG7. The electronic device 4000 shown in FIG7 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0062] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0063] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in Figure 7, but this does not indicate that there is only one bus or one type of bus.

[0064] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0065] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0066] The electronic device may also be a terminal device. The electronic device shown in Figure 7 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0067] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0068] According to another aspect of the present invention, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0069] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0070] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0071] The computer-readable storage medium provided in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0072] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0073] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for key point localization and attitude estimation of space objects in ISAR images, characterized in that, include: Acquire ISAR images of the space objects to be processed; The ISAR image is input into a pre-trained pose estimation model to obtain the pose evaluation result of the spatial object to be processed. The pose evaluation result includes object category, bounding box, key point location, and subject pose angle. The pose estimation model includes a feature extraction layer, a feature fusion layer, and a multi-task detection layer. The pose estimation model is trained in the following way: acquiring training data, which includes ISAR images corresponding to different types of spatial objects, each spatial object corresponding to a category label, pose angle label, bounding box label, and key point label; and extracting multi-scale basic features of each spatial object in the training data through the feature extraction layer, the multi-scale basic features including features at different scales. The feature fusion layer performs cross-scale fusion of the multi-scale basic features of each spatial object to obtain the fused features of each spatial object. For each spatial object, the multi-task detection layer performs parallel processing on the fused features of the spatial object to obtain the predicted pose evaluation result of the spatial object. The predicted pose evaluation result includes the predicted object category, predicted detection box, predicted key point position, and predicted subject pose angle of the spatial object. Based on each spatial object, the predicted pose evaluation result, and multiple labels corresponding to each spatial object, the pose estimation model is trained.

2. The method according to claim 1, characterized in that, A feature enhancement layer is further provided between the feature extraction layer and the feature fusion layer. This feature enhancement layer includes two external memory units connected in sequence. These two external memory units are learnable weight matrices that describe the common features of multiple typical spatial objects. During training, the pose estimation model further includes: for each spatial object, performing matrix multiplication on the multi-scale basic features of the spatial object through the first external memory unit of the two external memory units; calculating the similarity between the multi-scale basic features and the weight matrix corresponding to the first external memory unit to obtain a similarity matrix; and normalizing the similarity matrix to obtain the first attention. The first attention weight matrix is ​​used to describe the similarity distribution between the features of each spatial location in the multi-scale basic features and the common features of multiple typical spatial objects. For each spatial object, the first attention weight matrix is ​​weighted and fused through the second external memory unit of the two external memory units to obtain the enhanced features. The step of fusing the multi-scale basic features of each spatial object across scales through the feature fusion layer to obtain the fused features of each spatial object includes: fusing the enhanced features of each spatial object across scales through the feature fusion layer to obtain the fused features of each spatial object.

3. The method according to claim 1, characterized in that, The step of training the pose estimation model based on the predicted pose evaluation results and multiple labels corresponding to each spatial object includes: for each spatial object, determining a category classification loss based on the predicted object category and the category label; determining a detection box regression loss based on the predicted detection box and the detection box label; determining a keypoint localization loss based on the predicted keypoint location and the keypoint label; determining a pose regression loss based on the predicted subject pose angle and the pose angle label; determining a total loss based on the category classification loss, detection box regression loss, keypoint localization loss, and pose regression loss of all spatial objects; and training the pose estimation model based on the total loss.

4. The method according to claim 3, characterized in that, For each of the aforementioned spatial objects, the attitude angle labels include yaw angle labels, pitch angle labels, and roll angle labels; the predicted subject attitude angles include predicted yaw angle, predicted pitch angle, and predicted roll angle; and the determination of attitude regression loss based on the predicted subject attitude angles and the attitude angle labels includes: determining yaw angle loss based on the yaw angle labels and the predicted yaw angle; determining pitch angle loss based on the pitch angle labels and the predicted pitch angle; and determining roll angle loss based on the roll angle labels and the predicted roll angle. The attitude regression loss includes the yaw angle loss, the pitch angle loss, and the roll angle loss.

5. The method according to any one of claims 1 to 4, characterized in that, The space object to be processed is a three-axis attitude-stabilized satellite, including at least one of the Dragon spacecraft, space station, and Earth observation satellite.

6. A device for key point localization and attitude estimation of a space object in an ISAR image, characterized in that, include: The acquisition module is used to acquire ISAR images of the space objects to be processed. The pose estimation module is used to input the ISAR image into a pre-trained pose estimation model to obtain the pose evaluation result of the spatial object to be processed. The pose evaluation result includes the object category, detection box, key point position, and main body pose angle. The pose estimation model includes a feature extraction layer, a feature fusion layer, and a multi-task detection layer. The pose estimation model is trained based on the following training module, which is used to: acquire training data, which includes ISAR images corresponding to different types of spatial objects, each spatial object corresponding to a category label, pose angle label, detection box label, and key point label; and extract multi-scale basic features of each spatial object in the training data through the feature extraction layer, the multi-scale basic features including features at different scales. The feature fusion layer performs cross-scale fusion of the multi-scale basic features of each spatial object to obtain the fused features of each spatial object. For each spatial object, the multi-task detection layer performs parallel processing on the fused features of the spatial object to obtain the predicted pose evaluation result of the spatial object. The predicted pose evaluation result includes the predicted object category, predicted detection box, predicted key point position, and predicted subject pose angle of the spatial object. Based on each spatial object, the predicted pose evaluation result, and multiple labels corresponding to each spatial object, the pose estimation model is trained.

7. The apparatus according to claim 6, characterized in that, A feature enhancement layer is further provided between the feature extraction layer and the feature fusion layer. The feature enhancement layer includes two external memory units connected in sequence. The two external memory units are learnable weight matrices that describe the common features of multiple typical spatial objects. The pose estimation model also includes an enhancement module during training, which is used to: for each spatial object, perform matrix multiplication on the multi-scale basic features of the spatial object through the first external memory unit of the two external memory units, calculate the similarity between the multi-scale basic features and the weight matrix corresponding to the first external memory unit, obtain a similarity matrix, normalize the similarity matrix to obtain a first attention weight matrix, the first attention weight matrix is ​​used to describe the similarity distribution between the features of each spatial location in the multi-scale basic features and the common features of multiple typical spatial objects; for each spatial object, perform weighted fusion on the first attention weight matrix through the second external memory unit of the two external memory units to obtain the enhanced features; the feature fusion layer specifically performs cross-scale fusion on the enhanced features of each spatial object to obtain the fused features of each spatial object.

8. The apparatus according to claim 6, characterized in that, When the training module trains the pose estimation model based on the evaluation results of each spatial object and the multiple labels corresponding to each spatial object, it specifically performs the following steps: For each spatial object, it determines the category classification loss based on the predicted object category and the category label; it determines the detection box regression loss based on the predicted detection box and the detection box label; it determines the keypoint localization loss based on the predicted keypoint location and the keypoint label; it determines the pose regression loss based on the predicted subject pose angle and the pose angle label; it determines the total loss based on the category classification loss, detection box regression loss, keypoint localization loss, and pose regression loss of all spatial objects; and it trains the pose estimation model based on the total loss.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images