In-orbit space target semantic feature extraction method and device, equipment and storage medium

Abstract

The application relates to an on-orbit space target semantic feature extraction method, device, equipment and storage medium. Each optical image in an image sequence set of an on-orbit space target is subjected to multiple random homography transformation to generate target transformed images under different perspectives, a trained semantic feature extractor is used to extract a predicted heat map of at least two feature components on the on-orbit space target in each optical image and the corresponding multiple target transformed images, multiple predicted heat maps obtained according to the same optical image are subjected to inverse homography transformation, the multiple transformed predicted heat maps corresponding to the same feature component are superimposed, the preliminary semantic feature of the feature component is extracted on the comprehensive heat map obtained after superimposition according to the geometric constraint of the corresponding feature component, and finally the preliminary semantic features extracted from all the optical images in the image sequence set are optimized based on a motion constraint to obtain the semantic feature of the on-orbit space target. The method can be used for feature extraction of multiple different space targets with similar structures, and can also be used for feature extraction of unknown on-orbit space targets.

Description

Technical Field

[0001] This application relates to the field of space surveillance technology, and in particular to a method, apparatus, device, and storage medium for extracting semantic features of on-orbit space targets. Background Technology

[0002] Space targets, broadly speaking, encompass all objects within Earth's orbit, including spacecraft, launch vehicles, ballistic missiles, space debris, and celestial bodies. Based on their ability to provide interactive information, space targets can be categorized into cooperative space targets (own spacecraft) and non-cooperative space targets (failed satellites, foreign spacecraft, space debris, etc.). Space surveillance refers to the use of observation equipment to track and measure space targets, acquiring information such as their orbital position, dimensions, and motion status. Three-dimensional attitude is a crucial attribute of spacecraft, and space target attitude estimation is fundamental to many space missions and a vital aspect of space surveillance. Attitude estimation for cooperative space targets can be achieved using signals emitted by the target itself or artificial targets, a relatively mature technology. Attitude estimation for non-cooperative space targets relies solely on external observation, significantly increasing the difficulty compared to cooperative targets. However, non-cooperative space targets have broader applications and more pressing needs, such as fault diagnosis, reconnaissance and surveillance, and behavior prediction, making them a hot research topic in the aerospace field.

[0003] Traditional space surveillance relies primarily on ground-based methods, and the deployment of these stations significantly limits the observation range and timeliness. Space-based optical imaging, with its advantages of flexible platform mobility, highly interpretable optical sensors, and lightweight, low-cost operation, has gained increasing attention from major spacefaring nations and has now become a backbone of space surveillance.

[0004] The core of non-cooperative space target attitude estimation is establishing feature correlations between the observed image sequence and the ground truth representation space. Compared to ground-based imaging, attitude estimation based on orbital intersection has significant advantages in terms of imaging environment, target structure, and prior information. The capability and accuracy of attitude estimation methods depend on the applicability of the features adopted in the specific space-based observation scenarios.

[0005] Existing methods can be categorized based on the different features used in the feature association stage, broadly into traditional manual feature extraction and deep learning feature extraction methods. Traditional manual feature extraction methods mainly include template matching based on the overall features of the target, as well as extracting target point features, rectangular features, or elliptical features. However, in space-based imaging scenarios, the highly dynamic imaging environment leads to significant variations in target representation, making it difficult to ensure robustness and accuracy of overall features. Deep learning-based methods primarily extract feature points of specific spacecraft using CNNs, offering greater robustness compared to traditional methods. However, the unique nature of space-based observation scenarios results in extremely limited data acquisition, restricting the application of deep learning methods. Furthermore, existing algorithms are all based on single spacecraft targets and cannot generalize to multiple targets; therefore, the feature extraction problem for unknown spacecraft remains unsolved. Summary of the Invention

[0006] Therefore, it is necessary to provide an on-orbit space target semantic feature extraction method, apparatus, device, and storage medium capable of extracting semantic features from multiple space targets, in order to address the aforementioned technical problems.

[0007] A method for extracting semantic features of on-orbit space targets, the method comprising:

[0008] Acquire an image sequence set of an in-orbit space target, the image sequence set including multiple optical images of the in-orbit space target ordered by time;

[0009] Each optical image in the image sequence set is subjected to multiple random homography transformations to generate target transformation images from different viewpoints;

[0010] Each optical image and the corresponding multiple target transformation images are input into a trained semantic feature extractor to extract predicted heatmaps of at least two feature components on the on-orbit space target from each image.

[0011] After performing inverse homography transformation on multiple predicted heatmaps obtained from the same optical image, the multiple transformed predicted heatmaps corresponding to the same feature component are superimposed, and the preliminary semantic features of the feature component are extracted based on the geometric constraints of the corresponding feature component.

[0012] Based on motion constraints, the preliminary semantic features extracted from all optical images in the image sequence set are optimized to obtain the semantic features of the on-orbit space target.

[0013] In one embodiment, the characteristic component of the on-orbit space target is: the same component on a class of on-orbit space targets with similar geometry.

[0014] In one embodiment, the method further includes training the semantic feature extractor:

[0015] Obtain optical simulation training images of various on-orbit space targets with similar geometric structures in all attitudes, as well as the feature component labels of the targets in each optical training image;

[0016] Each of the optical training images is input into the semantic feature extractor for training. The semantic feature extractor includes a shared feature encoder and a multi-task decoder. The shared feature encoder extracts features from the optical training images, and the extracted features are input into different task decoders to obtain a predicted heatmap of at least two feature components.

[0017] The loss function is calculated based on the predicted heatmaps and the corresponding feature component labels. The learnable parameters in the semantic feature extractor are adjusted according to the calculation results until the loss function converges, thus obtaining the trained semantic feature extractor.

[0018] In one embodiment, in the semantic feature extractor:

[0019] The shared feature encoder includes an improved EfficientNet backbone network and a bidirectional feature pyramid structure BiFPN network.

[0020] The multi-task decoder includes at least two decoder units that perform heatmap prediction for different feature components, and a decoder unit that performs foreground segmentation prediction for the optical training image.

[0021] In one embodiment, the loss function is obtained by weighted summation of the prediction errors of each decoder unit in the multi-task decoder.

[0022] In one embodiment, the semantic features of the on-orbit space target are the semantic features of at least two feature components. According to the geometric structure of the feature components, the semantic features of the feature components include features representing the corner points of the plate-like structure and features representing the axis of the main structure.

[0023] In one embodiment, the step of overlaying multiple transformed predicted heatmaps corresponding to the same feature component and extracting preliminary semantic features of the feature component based on its geometric constraints includes:

[0024] When the feature component is a plate-like structure:

[0025] Each homography-transformed predicted heatmap is back-projected onto the original image coordinate system, and a preset number of points are selected as candidate points according to a preset threshold.

[0026] Superimpose all candidate points to obtain the candidate point set;

[0027] Taking each candidate point in the candidate point set as the center, if the number of candidate points within a preset size range is less than a preset threshold, the candidate point at the center is removed, thus obtaining a point that is robust to homography transformation.

[0028] Based on the geometric constraints after the projection of the plate-like structure, multiple final feature points of the feature component are finally determined among robust points;

[0029] The position coordinates of each of the final feature points are used as the preliminary semantic features of the feature component;

[0030] When the feature portion is a main structure:

[0031] Multiple homography-transformed predicted heatmaps are superimposed to obtain a superimposed predicted heatmap.

[0032] The Radon transform is applied to the superimposed predicted heatmap, and the extreme points in the transform matrix are found;

[0033] The coordinates corresponding to the extreme point with the highest probability are selected as the preliminary semantic features of the feature component.

[0034] This application also provides an on-orbit space target semantic feature extraction device, the device comprising:

[0035] The on-orbit space target image acquisition module is used to acquire an image sequence set of on-orbit space targets, the image sequence set including multiple optical images of on-orbit space targets ordered by time;

[0036] The homography transformation module is used to perform multiple random homography transformations on each optical image in the image sequence set to generate target transformation images from different viewpoints;

[0037] The predictive heatmap acquisition module is used to input each of the optical images and the corresponding multiple target transformation images into a trained semantic feature extractor, and extract the predicted heatmaps of at least two feature components on the on-orbit space target from each image.

[0038] The preliminary semantic feature acquisition module is used to perform inverse homography transformation on multiple predicted heat maps obtained from the same optical image, then superimpose multiple transformed predicted heat maps corresponding to the same feature component, and extract the preliminary semantic features of the feature component based on the geometric constraints of the corresponding feature component.

[0039] The spatial target semantic feature acquisition module is used to optimize the preliminary semantic features extracted from all optical images in the image sequence set based on motion constraints, so as to obtain the semantic features of the on-orbit space target.

[0040] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0041] Acquire an image sequence set of an in-orbit space target, the image sequence set including multiple optical images of the in-orbit space target ordered by time;

[0042] Each optical image in the image sequence set is subjected to multiple random homography transformations to generate target transformation images from different viewpoints;

[0043] Each optical image and the corresponding multiple target transformation images are input into a trained semantic feature extractor to extract predicted heatmaps of at least two feature components on the on-orbit space target from each image.

[0044] After performing inverse homography transformation on multiple predicted heatmaps obtained from the same optical image, the multiple transformed predicted heatmaps corresponding to the same feature component are superimposed, and the preliminary semantic features of the feature component are extracted based on the geometric constraints of the corresponding feature component.

[0045] Based on motion constraints, the preliminary semantic features extracted from all optical images in the image sequence set are optimized to obtain the semantic features of the on-orbit space target.

[0046] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0047] Acquire an image sequence set of an in-orbit space target, the image sequence set including multiple optical images of the in-orbit space target ordered by time;

[0048] Each optical image in the image sequence set is subjected to multiple random homography transformations to generate target transformation images from different viewpoints;

[0049] Each optical image and the corresponding multiple target transformation images are input into a trained semantic feature extractor to extract predicted heatmaps of at least two feature components on the on-orbit space target from each image.

[0050] After performing inverse homography transformation on multiple predicted heatmaps obtained from the same optical image, the multiple transformed predicted heatmaps corresponding to the same feature component are superimposed, and the preliminary semantic features of the feature component are extracted based on the geometric constraints of the corresponding feature component.

[0051] Based on motion constraints, the preliminary semantic features extracted from all optical images in the image sequence set are optimized to obtain the semantic features of the on-orbit space target.

[0052] The aforementioned method, apparatus, device, and storage medium for extracting semantic features of on-orbit space targets involve performing multiple random homography transformations on each optical image in an image sequence set of on-orbit space targets to generate target transformation images from different perspectives. A trained semantic feature extractor then extracts predicted heatmaps of at least two feature components on the on-orbit space target from the multiple target transformation images corresponding to each optical image. After performing inverse homography transformations on multiple predicted heatmaps obtained from the same optical image, the transformed predicted heatmaps corresponding to the same feature component are superimposed. Based on the geometric constraints of the corresponding feature component, preliminary semantic features of that feature component are extracted. Finally, based on motion constraints, the preliminary semantic features extracted from all optical images in the image sequence set are optimized to obtain the semantic features of the on-orbit space target. This method can extract features from multiple different space targets with similar structures, and it can also extract features from unknown on-orbit space targets. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating an on-orbit spatial target semantic feature extraction method in one embodiment;

[0054] Figure 2 This is a schematic diagram of the flow framework of a complete method, including training a semantic feature extractor, in one embodiment.

[0055] Figure 3 This is a schematic diagram of the semantic feature extractor in one embodiment;

[0056] Figure 4 This is a schematic diagram of the process for extracting the corner points of a solar panel using homography transformation in one embodiment.

[0057] Figure 5 This is a schematic diagram illustrating the generation of random homography transformation in one embodiment;

[0058] Figure 6 This is a schematic diagram of the training images in the training dataset of an experiment;

[0059] Figure 7 This is a schematic diagram of test images in a test dataset used in an experiment, where... Figure 7 (a) is the 15th partial intersection image of the KH11 model. Figure 7 (b) is the 17th intersection image of the TG model;

[0060] Figure 8 This is a diagram illustrating the experimental results in an experiment, where... Figure 8 (a) is a schematic diagram of feature extraction of the first intersection image using the KH11 model. Figure 8 (b) is a schematic diagram of feature extraction of the 16th intersection image of the TG model;

[0061] Figure 9 This is a histogram of estimated errors in an experiment.

[0062] Figure 10 This is a structural block diagram of an on-orbit spatial target semantic feature extraction device in one embodiment;

[0063] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] In existing technologies, deep learning-based methods, using CNNs to extract feature points of specific spatial targets, are more robust than traditional methods. However, the unique nature of space-based observation scenarios leads to extremely limited data acquisition, restricting the application of deep learning methods. Furthermore, existing algorithms are all based on single targets and cannot generalize to multiple targets. Therefore, the problem of feature extraction for unknown targets remains unresolved. In this application, as... Figure 1 As shown, an on-orbit space target semantic feature extraction method is provided, including the following steps:

[0066] Step S100: Obtain an image sequence set of the on-orbit space target, the image sequence set including multiple optical images of the on-orbit space target sorted by time;

[0067] Step S110: Perform multiple random homography transformations on each optical image in the image sequence set to generate target transformation images from different viewpoints.

[0068] Step S120: Input each optical image and the corresponding multiple target transformation images into the trained semantic feature extractor to extract the predicted heat map of at least two feature components on the on-orbit space target from each image.

[0069] Step S130: After performing anti-homography transformation on multiple predicted heat maps obtained from the same optical image, the multiple transformed predicted heat maps corresponding to the same feature component are superimposed, and the preliminary semantic features of the feature component are extracted according to the geometric constraints of the corresponding feature component.

[0070] Step S140: Based on motion constraints, optimize the preliminary semantic features extracted from all optical images in the image sequence set to obtain the semantic features of the on-orbit space target.

[0071] In this embodiment, the semantic feature extraction method for on-orbit space target optical images based on multi-task convolutional neural networks can extract semantic features of a class of target feature components with similar structures in space-based optical intersection scenarios. The feature extractor is trained using full-attitude optical images of multiple known space targets, tested on a sequence of images of unknown space targets, and the feature extraction effect is optimized through homography transformation and geometric constraints of features, as well as motion constraints between the sequence images.

[0072] It should be noted here that in this method, the semantic features extracted are not traditional SIFT features, but rather the semantic features that are selected so that when this method is used for subsequent pose estimation tasks, it can estimate the absolute pose, rather than the relative pose.

[0073] In this embodiment, the on-orbit space target for feature extraction can be a spacecraft, ballistic missile, or launch vehicle.

[0074] In this embodiment, the semantic features of an on-orbit space target are the semantic features of at least two feature components. It should be noted that the feature components refer to similar components on an on-orbit space target that have similar geometric structures.

[0075] Based on the geometric structure of the feature components, their semantic features include features representing the corner points of the plate-like structure and features representing the axis of the main structure. For example, when the on-orbit space target is a spacecraft, most spacecraft have two similar feature components: the main structure in the middle of the spacecraft and the solar panels on both sides of the limb structure. If the solar panels on both sides are considered as plate-like structures, then the position coordinates of the four corner points of the solar panels can be used as the semantic features of the solar panel component, while the axis of the main structure can be used as the semantic features of the main structure.

[0076] When the target in orbit is a ballistic missile or a launch vehicle, the main axis can be used as the semantic feature to be extracted.

[0077] In fact, based on the geometry of each on-orbit space target, most component features can be represented by points or lines. Therefore, the following text will use a spacecraft as an example to illustrate this method.

[0078] In fact, the method in this paper was originally designed for feature extraction of unknown targets. Because this method has good generalization ability, it can effectively extract features from both known and unknown targets of the same class that have similar structures.

[0079] According to such Figure 2 The complete method flowchart shown first explains the training process of the semantic feature extractor.

[0080] In this embodiment, the semantic feature extractor is first trained. This semantic feature extractor, SSF-Net, is a multi-scale, multi-task convolutional neural network, consisting of a shared feature encoder and multiple prediction heads performing different tasks, i.e., multi-task decoders. Its structure is as follows: Figure 3 As shown, features are extracted from the optical training image using a shared feature encoder, and the extracted features are input into different task decoders to obtain predicted heatmaps of at least two feature components.

[0081] In this embodiment, the shared feature encoder includes an improved EfficientNet backbone network and a bidirectional feature pyramid structure BiFPN network to fuse features at different scales. The multi-task decoder includes at least two decoder units for heatmap prediction corresponding to different feature components, and a decoder unit for foreground segmentation prediction of the optical training image, respectively realizing three tasks: heatmap prediction of the four corner points of the spacecraft's solar panel, heatmap prediction of the main axis, and mask prediction of the target. The multi-task network avoids overfitting on specific tasks and improves the network's generalization ability through joint optimization of the three tasks, thus achieving better performance even on unknown targets.

[0082] In this embodiment, optical training images of various spacecraft with similar geometric structures in all attitudes, along with the labels of the spacecraft's solar panel corner points and main axis in each optical training image, are used to train the semantic feature extractor. After inputting each optical training image into the semantic feature extractor, predicted heatmaps of the solar panel corner points and main axis, as well as predicted foreground segmentation maps of the optical training images, are obtained.

[0083] Next, the predicted semantic features of the corresponding feature components are extracted by calculating the confidence level of each pixel in the image from the predicted heatmap of the corner point of the sailboard and the main axis.

[0084] Specifically, the heatmap-based feature extraction method determines whether a point is a feature point or lies on a feature line by calculating the confidence score of each pixel in the image. Compared with traditional direct regression feature extraction methods, the heatmap-based method avoids direct gradient jumps during backpropagation, resulting in better feature extraction performance.

[0085] Assuming the image size is N×N, the coordinates of the i-th corner point of the sailboard are (h i ,g i If the value is zero, then its corresponding heatmap exhibits a two-dimensional Gaussian distribution within a range of 6σ+3 centered at that point, with zero values ​​in other regions. This can be represented as:

[0086]

[0087] in,

[0088] Since the windsurfing board has four corner points, the true value of its heatmap can be obtained directly by summing the four heatmaps, that is:

[0089]

[0090] The generation method of the main axis heatmap is similar to that of the windsurfing corner points, using each point (h) on the axis as the basis. j ,g j A heat map S is generated centered on this point. j Then, the heatmaps corresponding to different points are overlaid. Since the points on the straight line are adjacent, when generating the heatmap, the same location may be covered by multiple Gaussian windows. In this case, the largest number is selected as the value at that location.

[0091] Because the original EfficientNet backbone network performs downsampling twice before being connected to the BiFPN network, the maximum size of the feature maps learned by the network is smaller than that of the original image. In subsequent tasks, the results can only be upsampled to obtain a heatmap or mask with the same size as the original image. The simple interpolation operation in the upsampling process will cause the region where the feature points are located to be smoothly expanded, making it difficult to accurately locate the location of the features. In addition, some spacecraft have narrow main structures, and the prediction results after upsampling small feature maps will cause the axis range to cover or even exceed the main body area, instead of extracting a clear straight line, thus causing greater errors.

[0092] Based on the above considerations, the original EfficientNet network is improved in this embodiment by removing the two downsampling operations, so that the maximum feature map size of the BiFPN module is consistent with the original image size, thus ensuring the accuracy of the network prediction results.

[0093] In this embodiment, the loss function is calculated based on the predicted semantic features and the corresponding feature component labels. The loss function is obtained by weighted summation of the prediction errors of each decoder unit in the multi-task decoder.

[0094] Specifically, for the two heatmap decoding units, it can be seen from the process of generating their true values ​​that their purpose is to locate the feature by finding the maximum value in the predicted heatmap. Therefore, the prediction error can be described by the mean square error (MSE) between the network's regression results and the true value.

[0095] The loss function for the heatmap of the windsurfing corner points can be expressed as:

[0096]

[0097] Specifically, in addition to considering the direct difference between predicted and actual values, the main axis heatmap should also maintain a straight line as much as possible. Therefore, in this embodiment, the Pearson correlation coefficient ρ is used to measure whether the coordinates of pixels in the image greater than the threshold ε are linear. The closer |ρ| is to 1, the greater the linear correlation between the variables. Let represent the value at (x,y) in the normalized prediction heatmap. Then, the Pearson correlation coefficient can be expressed as:

[0098]

[0099] Using λ mb Let the weights be represented, then the loss function of the main axis heatmap is expressed as:

[0100]

[0101] Specifically, the target mask prediction error can be described by the binary cross-entropy (BCE) activated by sigmoid:

[0102]

[0103] In this embodiment, λ1, λ2, and λ3 are used to balance the final loss function, resulting in the overall loss function representation:

[0104]

[0105] During the iterative training of the semantic feature extractor, the total loss function is calculated until it converges, thus obtaining the trained semantic feature extractor.

[0106] In fact, the semantic feature extractor trained using the above method can accurately extract the semantic features of known spatial targets. Here, known spatial targets refer to targets that appeared in the training images used to train the semantic feature extractor. However, the semantic feature extractor alone cannot accurately extract semantic features for unknown spatial targets. Therefore, in this method, the trained semantic feature extractor is used to robustly extract features from unknown targets.

[0107] In step S100, if the spatial target for semantic feature extraction is unknown, then a sequence of images of the unknown spatial target is acquired, which is multiple images of the unknown target arranged in chronological order over a period of time. When using this method for experimental simulation, the spatial unknown target images used in the experiment are obtained through simulation. The simulation process considers the special characteristics of the space-based observation scenario, performs image degradation processing, and exhibits uneven brightness.

[0108] In step S110, homography transformation, also known as projection transformation, is used to describe the two-dimensional transformation of mapping points in one image to another. Multiple random homography transformations are performed on a single image to generate semantic features from different perspectives. The final location of the features can then be determined by filtering and merging them after inverse transformation.

[0109] In this embodiment, homography transformation can be used to map corresponding two-dimensional coordinate points (in homogeneous coordinates) in two images according to formula (8):

[0110]

[0111] In formula (8), x = (x1, x2) are the points before the transformation, and x′ = (x1′, x2′) are the points after the transformation. Let H be the 3×3 homography transformation matrix, then x′ = Hx. Let f θ (·) represents the previously trained semantic feature extractor, I is the input image, and kp is the semantic point. Therefore:

[0112] kp = f θ (I) (9)

[0113] Ideally, the homography transformation of a feature with respect to the image should be covariant, meaning that points in the image undergoing the homography transformation should coincide with points in the original image after the inverse transformation. Therefore, for all homography transformation matrices H, the detector should satisfy:

[0114] H·kp=f θ (H(I)) (10)

[0115] Moving all homography-related terms to the right side of the equation, we get:

[0116] kp = H -1 f θ (H(I)) (11)

[0117] After performing multiple random homography transformations on each optical image in the image sequence set to generate target transformation images from different perspectives, a trained semantic feature extractor is used to extract predicted heatmaps of the solar panel corner points and the main axis of the spacecraft from each optical image and the corresponding multiple target transformation images. Then, after performing inverse homography transformations on the multiple predicted heatmaps obtained from the same optical image, the multiple transformed predicted heatmaps corresponding to the solar panel corner points and the main axis are superimposed. Based on the geometric constraints of the corresponding feature components, preliminary semantic features of the solar panel corner points and the main axis are extracted.

[0118] In practice, it is difficult for the extractor to achieve complete covariance, meaning that different homography matrices in formula (11) will produce semantic points kp with slight positional deviations. To improve the robustness of semantic point extraction, in this embodiment, N... h The detection results generated by H from the random sampling are empirically summed to obtain the improved extractor. Let the merge and filter operations be... but

[0119]

[0120] In formula (12), when i = 0, H0 is a diagonal matrix. At this time, the input of the extractor is the original image, and there is no need to perform homography transformation.

[0121] like Figure 4 As shown, in the initial semantic feature extraction of the corner points of the solar panel, each transformed predicted heatmap is back-projected onto the original image coordinate system, and a preset number of points are selected as candidate points according to a preset threshold. All candidate points are then superimposed to obtain a candidate point set. Taking each candidate point in the candidate point set as the center, if the number of candidate points within a preset size range is less than a preset threshold, the candidate point at the center is removed, resulting in points robust to homography transformation. Based on the geometric constraints of the projected solar panel structure, multiple final feature points of the feature component are finally determined from these robust points, and the position coordinates of each final feature point are used as the initial semantic features of the feature component.

[0122] Specifically:

[0123] 1. Combine the original image I and the homography-transformed image H. i (I) Input the detector to obtain the heatmap of the predicted corner point position of the sail.

[0124] 2. Predict the heatmap through Backprojecting onto the original image coordinate system yields...

[0125] 3. Set a threshold γ to filter out... For points with intensity greater than γ, if there are many, only the first 6 are retained, resulting in c. i ={c i1 ,c i2 ,L,c im}, m≤6.

[0126] 4. Superposition of N h +1 candidate point c i This yields all candidate points.

[0127] 5. Set the range threshold α, if C i Centered on a grid of size α×α, the number of candidate points is less than the threshold N. s If the candidate point is eliminated, the final point P that is robust to homography transformation is obtained. robust .

[0128] 6. Since parallel projection does not change the parallelism of lines, the sail remains a parallelogram in the imaging plane. In P robust By selecting the points that form the parallelogram with the largest area, we can obtain the corner point P of the windsurfing board. sp .

[0129] In this embodiment, taking the solar panel of a spacecraft as an example, the positions of the four corner points can be found as semantic features of the feature component.

[0130] In this embodiment, when extracting the preliminary semantic features of the main axis, multiple homography-transformed prediction heatmaps are superimposed to obtain a superimposed prediction heatmap. Radon transform is applied to the superimposed prediction heatmap, and extreme points in the transformation matrix are found. The coordinates corresponding to the extreme point with the highest probability are selected as the preliminary semantic features of the feature component.

[0131] Specifically, first compare the original image I with the homography-transformed image H. i (I) Input the detector to obtain a heat map of the predicted main axis position. right through Backproject to the original image coordinate system and directly sum all N values. h +1 result, resulting in right Perform Radon transform and find the extreme points in the transform matrix R. Select the coordinates corresponding to the extreme points with the highest probability as the parameters of the main axis.

[0132] It should be noted that not all 3×3 matrices are suitable for the homography transformation in this method. To obtain a matrix that better represents the camera transformation, this embodiment decomposes the potential homography into simpler transformation forms. Using a truncated normal distribution, operations such as translation, scaling, in-plane rotation, and perspective projection are sampled within a certain range. These transformations are combined with image center cropping to achieve homography while avoiding edge artifacts. The generation of random homography transformation is as follows: Figure 5 As shown.

[0133] Finally, in step S140, the preliminary semantic features are further optimized to obtain more accurate features for the unknown target.

[0134] Some structures of unknown targets that are similar to the corner points or main axis features of the sailboard may interfere with the network’s feature extraction performance. However, the errors caused by these interferences are only a minority in a set of sequence images. Therefore, after obtaining the preliminary semantic features, the proposed features can be further optimized based on the motion constraints of the sequence images.

[0135] In this embodiment, assuming a set of image sequences of an unknown target in orbit has M frames, the optimization steps for the corner features of the solar panel include: determining whether the feature points in all consecutive frames of the sequence are within a certain range, i.e., whether they satisfy motion constraints. If there are consecutive preset frames that satisfy the motion constraints, the first frame of these preset frames is used as the base, and its index is denoted as base; if not, the first image in the segment that satisfies the motion constraints the most consecutively is used as the base, and its index is denoted as base. The M-base frames with indices from base+1 to M are optimized sequentially. The optimization method is to first determine whether the current image and the previous frame satisfy the motion constraints; if so, skip them and optimize the next frame; if not, use the previous frame as the base and optimize the next frame. robust Find other points in the array that can form a parallelogram and satisfy the motion constraints. If none can be found, then directly check point P. robust and P all Find the nearest neighbor point to the previous frame. Optimize the base-1 frames from index base-1 to 1 sequentially. The optimization method is to first determine if the current image and the next frame satisfy the motion constraints; if they do, skip them and optimize the previous frame; otherwise, use the next frame as a reference and optimize the next frame. robust Find other points in the array that can form a parallelogram and satisfy the motion constraints. If none can be found, then directly check point P. robust and P all Find the point that is closest to the previous frame.

[0136] In this embodiment, the maximum range of change Δs that feature points in two consecutive frames need to satisfy in the image is:

[0137]

[0138] In formula (13), ω=(v t -v o ) / d is the angular velocity obtained by dividing the relative velocity between the target and the observation platform by the distance, Δt is the time interval between adjacent target frames, L is the direction vector corresponding to the maximum length of the target, and D is the vector of the target's line of sight.

[0139] In this embodiment, assuming a set of image sequences of an unknown target in orbit contains M frames, the optimization steps for the main axis features include: determining whether the angles of straight lines in all consecutive frames of the sequence are within a certain range, i.e., whether they satisfy motion constraints. If there are consecutive preset frames that satisfy the motion constraints, the first frame of these preset frames is taken as the reference, and its index is denoted as base; if not, the first frame of the set of images that consecutively satisfies the motion constraints the most is taken as the reference, and its index is denoted as base. The M-base frames with indices from base+1 to M are optimized sequentially. The optimization method is to first determine whether the current image and the previous frame satisfy the motion constraints. If they do, skip them and optimize the next frame; if not, use the previous frame as the reference and search for straight lines that satisfy the angle constraints among the other maxima found after the Radon transform of the current image's heatmap. If no such line is found, the axis of the previous image is directly used as the axis of the current image to avoid the incorrect prediction results having too much impact on the next frame. The base-1 frame images with indices from base-1 to 1 are optimized sequentially. The optimization method is as follows: first, it is determined whether the current image and the next frame image can satisfy the motion constraints. If they can, the process is skipped and the previous frame is optimized. If not, the next frame image is used as a reference, and a straight line that satisfies the angle constraints is found among the other maxima found after the Radon transform of the current image heatmap. If no straight line is found, the axis of the next image is directly used as the axis of the current image to avoid the influence of incorrect prediction results on the next frame image.

[0140] In one embodiment, the preset number of frames can be set according to specific circumstances; in some embodiments, it can be set to 4 frames.

[0141] This paper also uses experimental data to illustrate the effectiveness of the proposed method. The specific experimental steps and results are as follows:

[0142] Fourteen spacecraft models with similar structures (bi-sail and clearly defined main structures) were selected, and images were rendered using Blender + Python to obtain 512×512 images. To ensure diversity of observation perspectives, 36×36 = 1296 relative attitudes were generated at 10° intervals across the entire attitude space. The simulation scene was designed with ideal lighting conditions, and the camera captured images of the target using parallel projection. The rendered dataset was used for training SSF-Net, such as... Figure 6 As shown.

[0143] Two spacecraft models, KH11 and TG, were selected as unknown targets. The rendezvous time and relative attitude were calculated based on the orbital elements of the observation and target satellites. A total of 388 simulated images of the rendezvous moments were acquired for each spacecraft across 17 arc segments. The scene was relatively dark, with lighting provided by point light sources to simulate the uneven brightness in the space environment. Motion blur and salt-and-pepper noise were added to the images to simulate blurring caused by relative movement and bad pixels due to CMOS damage. Example images are shown below. Figure 7 As shown, the time directly below each image is the intersection time corresponding to that image.

[0144] Using this method to extract semantic features of the sailboard corner points and the main axis of an unknown target sequence image, we can obtain, for example: Figure 8 The detection results are shown, where the time directly below each image is the intersection moment corresponding to that image, the yellow dots in the images represent the corner points of the sailboard, and the pink lines represent the direction of the main axis. The results show that after steps 1 and 2 of the method proposed in this patent, most of the features in the sequence images are correct, while a small portion will produce significant deviations. After step 3, which involves motion constraints on the sequence images, almost all features are optimized correctly.

[0145] The true values ​​of features in the image can be obtained by 3D to 2D projection. Calculating the distance between the optimized features and their true values ​​yields the results shown in Figure 1. The average estimation errors of the sail corner points for the KH11 model and the TG model are 13 pixels and 4 pixels, respectively, with an average error of 7.7993 pixels for both models. The average angle errors of the main axis are 1.2163° and 0.6597°, respectively, with an average error of 0.9380° for both models.

[0146] Table 1 shows the distances between the proposed features and the true values.

[0147] KH11 12.2576 1.2163 TG 3.3410 0.6597

[0148] The histogram of the estimation error is as follows Figure 9 As shown, from Figure 9The results show that for the corner features of the solar panel, the KH11 model has a larger estimation error than the TG model. This is because the solar panel in the KH11 model is shorter than that in the TG model, resulting in more instances where it is completely occluded by the main subject, thus leading to a larger error. For the main subject axis features, the TG model has a larger estimation error because the solar panel in the TG model is longer, which affects the network's judgment of the main subject. The experimental results demonstrate the effectiveness and accuracy of the proposed method.

[0149] The aforementioned on-orbit space target semantic feature extraction method utilizes SSF-Net, specifically designed for space target semantic feature extraction, and geometric constraints on the solar panel corner points and the main axis to achieve initial feature extraction. Further optimization of the extracted features is achieved through homography transformation, geometric constraints on the features, and motion constraints between adjacent frames of the image sequence, thereby improving the accuracy of semantic feature extraction. This method can extract the solar panel corner points and main axis features of a class of unknown spacecraft with similar structures in space-based optical intersection scenarios, laying the foundation for attitude estimation tasks in space surveillance missions.

[0150] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0151] In one embodiment, such as Figure 10 As shown, an on-orbit space target semantic feature extraction device is provided, comprising: an on-orbit space target image acquisition module 200, a homography transformation module 210, a predicted heatmap acquisition module 220, a preliminary semantic feature acquisition module 230, and a space target semantic feature acquisition module 240, wherein:

[0152] The on-orbit space target image acquisition module 200 is used to acquire an image sequence set of on-orbit space targets, the image sequence set including multiple optical images of on-orbit space targets ordered by time.

[0153] The homography transformation module 210 is used to perform multiple random homography transformations on each optical image in the image sequence set to generate target transformation images from different viewpoints.

[0154] The predictive heatmap acquisition module 220 is used to input each of the optical images and the corresponding multiple target transformation images into a trained semantic feature extractor, and extract the predicted heatmaps of at least two feature components on the on-orbit space target from each image.

[0155] The preliminary semantic feature acquisition module 230 is used to perform inverse homography transformation on multiple predicted heat maps obtained from the same optical image, then superimpose multiple transformed predicted heat maps corresponding to the same feature component, and extract the preliminary semantic features of the feature component based on the geometric constraints of the corresponding feature component.

[0156] The spatial target semantic feature acquisition module 240 is used to optimize the preliminary semantic features extracted from all optical images in the image sequence set based on motion constraints to obtain the semantic features of the on-orbit space target.

[0157] Specific limitations regarding the on-orbit space target semantic feature extraction device can be found in the limitations of the on-orbit space target semantic feature extraction method described above, and will not be repeated here. Each module in the aforementioned on-orbit space target semantic feature extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0158] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an on-orbit space target semantic feature extraction method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0159] Those skilled in the art will understand that Figure 11The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0161] Acquire an image sequence set of an in-orbit space target, the image sequence set including multiple optical images of the in-orbit space target ordered by time;

[0162] Each optical image in the image sequence set is subjected to multiple random homography transformations to generate target transformation images from different viewpoints;

[0163] Each optical image and the corresponding multiple target transformation images are input into a trained semantic feature extractor to extract predicted heatmaps of at least two feature components on the on-orbit space target from each image.

[0164] After performing inverse homography transformation on multiple predicted heatmaps obtained from the same optical image, the multiple transformed predicted heatmaps corresponding to the same feature component are superimposed, and the preliminary semantic features of the feature component are extracted based on the geometric constraints of the corresponding feature component.

[0165] Based on motion constraints, the preliminary semantic features extracted from all optical images in the image sequence set are optimized to obtain the semantic features of the on-orbit space target.

[0166] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0167] Acquire an image sequence set of an in-orbit space target, the image sequence set including multiple optical images of the in-orbit space target ordered by time;

[0168] Each optical image in the image sequence set is subjected to multiple random homography transformations to generate target transformation images from different viewpoints;

[0169] Each optical image and the corresponding multiple target transformation images are input into a trained semantic feature extractor to extract predicted heatmaps of at least two feature components on the on-orbit space target from each image.

[0170] After performing inverse homography transformation on multiple predicted heatmaps obtained from the same optical image, the multiple transformed predicted heatmaps corresponding to the same feature component are superimposed, and the preliminary semantic features of the feature component are extracted based on the geometric constraints of the corresponding feature component.

[0171] Based on motion constraints, the preliminary semantic features extracted from all optical images in the image sequence set are optimized to obtain the semantic features of the on-orbit space target.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for extracting semantic features of on-orbit space targets, characterized in that, The method includes: Acquire an image sequence set of an in-orbit space target, the image sequence set including multiple optical images of the in-orbit space target ordered by time; Each optical image in the image sequence set is subjected to multiple random homography transformations to generate target transformation images from different viewpoints; Each optical image and the corresponding multiple target transformation images are input into a trained semantic feature extractor to extract predicted heatmaps of at least two feature components on the on-orbit space target from each image. After performing inverse homography transformation on multiple predicted heatmaps obtained from the same optical image, the multiple transformed predicted heatmaps corresponding to the same feature component are superimposed. Preliminary semantic features of the feature component are then extracted based on its geometric constraints. Specifically, when the feature component is a plate-like structure: Each homography-transformed predicted heatmap is back-projected onto the original image coordinate system, and a preset number of points are selected as candidate points according to a preset threshold. All candidate points are superimposed to obtain a candidate point set. Taking each candidate point in the candidate point set as the center, if the number of candidate points within a preset size range is less than a preset threshold, the candidate point at the center is removed, resulting in points robust to homography transformation. Based on the geometric constraints after the plate-like structure is projected, multiple final feature points of the feature component are finally determined from the robust points. The position coordinates of each final feature point are used as the preliminary semantic features of the feature component. When the feature portion is a main structure: Multiple homography-transformed predicted heatmaps are superimposed to obtain a superimposed predicted heatmap; Radon transform is applied to the superimposed predicted heatmap, and extreme points in the transformation matrix are found; the coordinates corresponding to the extreme point with the highest probability are selected as the preliminary semantic features of the feature component. Based on motion constraints, the preliminary semantic features extracted from all optical images in the image sequence set are optimized to obtain the semantic features of the on-orbit space target.

2. The method for extracting semantic features of on-orbit space targets according to claim 1, characterized in that, The characteristic components of the on-orbit space target are: identical components on a class of on-orbit space targets with similar geometric structures.

3. The method for extracting semantic features of on-orbit space targets according to claim 2, characterized in that, The method further includes training the semantic feature extractor: Obtain optical simulation training images of various on-orbit space targets with similar geometric structures in all attitudes, as well as the feature component labels of the targets in each optical training image; Each of the optical training images is input into the semantic feature extractor for training. The semantic feature extractor includes a shared feature encoder and a multi-task decoder. The shared feature encoder extracts features from the optical training images, and the extracted features are input into different task decoders to obtain a predicted heatmap of at least two feature components. The loss function is calculated based on the predicted heatmaps and the corresponding feature component labels. The learnable parameters in the semantic feature extractor are adjusted according to the calculation results until the loss function converges, thus obtaining the trained semantic feature extractor.

4. The method for extracting semantic features of on-orbit space targets according to claim 3, characterized in that, In the semantic feature extractor: The shared feature encoder includes an improved EfficientNet backbone network and a bidirectional feature pyramid structure BiFPN network. The multi-task decoder includes at least two decoder units that perform heatmap prediction for different feature components, and a decoder unit that performs foreground segmentation prediction for the optical training image.

5. The method for extracting semantic features of on-orbit space targets according to claim 4, characterized in that, The loss function is obtained by weighted summation of the prediction errors of each decoder unit in the multi-task decoder.

6. The method for extracting semantic features of on-orbit space targets according to any one of claims 1-5, characterized in that, The semantic features of the on-orbit space target are the semantic features of at least two feature components. According to the geometric structure of the feature components, the semantic features of the feature components include features representing the corner points of the plate-like structure and features representing the axis of the main structure.

7. An on-orbit spatial target semantic feature extraction device, characterized in that, The device implements the on-orbit space target semantic feature extraction method according to any one of claims 1-6, including: The on-orbit space target image acquisition module is used to acquire an image sequence set of on-orbit space targets, the image sequence set including multiple optical images of on-orbit space targets ordered by time; The homography transformation module is used to perform multiple random homography transformations on each optical image in the image sequence set to generate target transformation images from different viewpoints; The predictive heatmap acquisition module is used to input each of the optical images and the corresponding multiple target transformation images into a trained semantic feature extractor, and extract the predicted heatmaps of at least two feature components on the on-orbit space target from each image. The preliminary semantic feature acquisition module is used to perform inverse homography transformation on multiple predicted heat maps obtained from the same optical image, then superimpose multiple transformed predicted heat maps corresponding to the same feature component, and extract the preliminary semantic features of the feature component based on the geometric constraints of the corresponding feature component. The spatial target semantic feature acquisition module is used to optimize the preliminary semantic features extracted from all optical images in the image sequence set based on motion constraints, so as to obtain the semantic features of the on-orbit space target.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images