Space target behavior recognition method based on multi-modal data fusion

Abstract

The application relates to a space target behavior recognition method based on multi-modal data fusion, and belongs to the fields of artificial intelligence, computer vision, spaceflight and simulation technology. The method comprises multi-modal data set construction based on a generative adversarial network, multi-modal data feature fusion, and target behavior recognition based on trajectory fitting correlation. The application realizes high-accuracy recognition of target on-orbit behavior in a complex space environment, and provides important support for space obstacle avoidance, autonomous detection and intelligent decision-making of a spacecraft.

Description

Technical Field

[0001] This invention relates to a spatial target behavior recognition method based on multimodal data fusion, belonging to the fields of artificial intelligence, computer vision, aerospace, and simulation technology. Background Technology

[0002] In recent years, the rapid increase in the number of space debris and spacecraft in orbit has brought more safety hazards. Target behavior recognition in the space environment helps to clarify the target's intentions in orbit in a timely manner, thereby improving the spacecraft's ability to avoid collision risks.

[0003] Data in space is often comprised of large-field-of-view high-resolution images or high-resolution optical images stitched together from multiple cameras. Single optical sensors face several specific challenges in space target detection. Since optical sensors rely on visible light for imaging, their imaging performance deteriorates significantly under low-light or no-light conditions in space, leading to blurred images or missing information. In the rapidly moving space environment, optical sensors struggle to capture dynamic targets, and motion blur affects image clarity. Optical sensors also have a limited field of view; targets located in blind spots or edge areas may not be effectively detected. Furthermore, optical sensors primarily provide two-dimensional image information, lacking depth information and understanding of spatial relationships. Therefore, a space target behavior recognition method based on multimodal data fusion was designed.

[0004] Therefore, there is an urgent need to propose a spatial target behavior recognition method based on multimodal data fusion to solve the above-mentioned technical problems. Summary of the Invention

[0005] To address the aforementioned problems, a spatial target behavior recognition method based on multimodal data fusion is provided. A brief overview of this invention is given below to provide a basic understanding of certain aspects of the invention. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention.

[0006] The technical solution of the present invention:

[0007] A spatial target behavior recognition method based on multimodal data fusion includes the following steps:

[0008] Step 1: Construction and processing of multimodal datasets:

[0009] A multimodal dataset was constructed through satellite collaborative observation. The acquired data was preprocessed by denoising and normalization. Features of different modalities were extracted using CNN. Additional conditional information was introduced through cGAN. The generator parameters were optimized by comparing the generated images with real images through a discriminator. The Adam optimizer was used during cGAN training. The quality of the generated images was evaluated and the model parameters were adjusted using SSIM and PSNR metrics.

[0010] Step 2, Multimodal Image Registration and Feature Processing:

[0011] Multimodal image registration algorithms are used to ensure spatial consistency of images from different sources. The registration process includes: feature detection using the SIFT algorithm, transformation model estimation using the RANSAC algorithm, and image resampling using bilinear interpolation.

[0012] Step 3: Target Behavior Identification and Association

[0013] A scenario centered on spacecraft S and a non-cooperative target G is constructed. The motion state of the spacecraft is described using a geocentric inertial frame and local vertical and local horizontal coordinate systems. On-orbit behavior is defined and simulated using corresponding formulas. A trajectory association algorithm based on nearest neighbor data association strategy is used to solve the target tracking loss problem. The fitted orbital state is associated with the defined events.

[0014] Preferably, the modules used in the method include:

[0015] A multimodal dataset construction module based on generative adversarial networks: Multiple satellites equipped with high-resolution optical telescopes, spaceborne high-resolution radars, and infrared telescopes observe the same target in space from different angles at multiple distance scales to obtain small-scale multimodal data. By introducing conditional generative adversarial networks and self-supervised learning, large-scale image datasets can be constructed.

[0016] Multimodal data feature fusion module: For multimodal data measured over a period of time, image registration and deep and shallow feature extraction are performed. Multi-head attention mechanism and graph neural network are introduced to fuse features. The fused features are input into the classifier and classified through a series of fully connected layers. The output is a multidimensional feature vector containing multimodal comprehensive information for subsequent target behavior recognition.

[0017] The target behavior recognition module based on trajectory fitting and association defines four typical behaviors for general scenarios of space targets and gives the corresponding orbital dynamics formulas. To address the target tracking loss problem, a trajectory association algorithm based on nearest neighbor data association is introduced. Through data import, interruption introduction, and fitting function definition, the module achieves accurate recognition of the target's on-orbit behavior.

[0018] Preferred: In step 1, the multimodal dataset includes optical, spaceborne high-resolution radar and infrared images, and target image data is acquired under different distances, different lighting conditions, different angles and different spatial backgrounds;

[0019] Additional conditional information includes target category and observation angle;

[0020] cGAN generates modal data that is missing due to the sensor's own performance limitations; this missing modal data includes infrared images.

[0021] Preferred: Step 2 includes:

[0022] Feature detection: Key points are extracted from the image using the SIFT algorithm, a Gaussian pyramid is constructed, and the difference Gaussian is calculated to detect extreme points;

[0023] Transformation model estimation: The RANSAC algorithm is used to estimate the transformation model, matching points are randomly selected and the transformation matrix is ​​calculated, and finally a 3x3 homography matrix is ​​obtained to ensure that the source image and the target image are aligned.

[0024] Image resampling: The source image is resampled using bilinear interpolation. After image registration, a ResNet structure is used for feature extraction. The extracted features are divided into shallow and deep features. A multi-head self-attention mechanism is introduced to enhance the feature extraction effect. By combining spatial attention and channel attention, the importance of each channel is calculated, enabling the model to adaptively focus on high-value features. The fused features are processed through GCN. A graph structure is constructed, where nodes represent features and edges are established based on the similarity between features. Feature propagation is performed through the graph Laplacian operator to enhance the correlation between modal features. The fused features are used as input and processed through a fully connected layer for final classification. A Dropout layer is introduced into the classifier.

[0025] Preferred configuration: In step 3, spacecraft S is equipped with multi-modal onboard detection equipment, capable of acquiring real-time orbit, position, and velocity information of the non-cooperative target G when it enters its observable region; a geocentric inertial frame is adopted, with the origin set at the Earth's center, and a relative motion coordinate system is established through local vertical and local horizontal coordinate systems; in terms of behavior modes, four on-orbit behaviors are defined, including launch into orbit, on-orbit operation, maneuvering and orbit change, and re-entry into orbit; the launch into orbit process is simulated using the gravitational formula of the two-body problem, covering the basic dynamic principles of rocket thrust and gravitational acceleration; regarding on-orbit operation, [the following is omitted as the text is incomplete and requires further context]. The motion of spacecraft in circular or elliptical orbits is described using the classic Kepler orbital equations. For maneuvering orbit changes, a Hohmann transfer orbit is introduced to represent the transfer from a low Earth orbit to a higher orbit, while the orbital decay formula is used to simulate the re-entry process of the spacecraft. To address the problem of target tracking loss, a trajectory association algorithm based on the nearest neighbor data association strategy is employed. This algorithm includes steps such as adding Gaussian noise to the existing multimodal fusion information of the target, introducing data interruptions, defining a fitting function, performing parameter fitting, and calculating errors. The algorithm associates the fitted orbital state with various defined events.

[0026] Preferred option: In step 1, in order to meet the mission requirements of satellites identifying the behavior of specific targets in space, multiple satellites equipped with advanced sensors are selected to obtain optical, spaceborne high-resolution radar and infrared image data.

[0027] After data collection is complete, the images are preprocessed; a Gaussian filter is used to denoise the images, and the denoising is achieved using the following formula:

[0028]

[0029] Where (x,y) are the pixel coordinates for denoising, k is the radius of the Gaussian kernel (representing the range of the filter), and G... σ (m,n) is the Gaussian kernel, where m and n are the offsets of the kernel relative to the center point. The formula for calculating the pixel value of the normalized image is:

[0030]

[0031] In the formula, I(x,y) and I normalized (x, y) represent the pixel values ​​at coordinates (x, y) of the original image and the normalized image, respectively. min It is the minimum value of the image pixels, I max It is the maximum value of the image pixels;

[0032] After data preprocessing, CNN is used for feature extraction. The outputs of the optical image, radar image, and infrared image after feature extraction by the convolutional neural network are represented as follows:

[0033] Foptical =CNN(I optical )

[0034] F SAR =CNN(I SAR )

[0035] F IR =CNN(I IR )

[0036] In the formula, I optical I represents the three-dimensional tensor of the input optical image. SAR I represents the three-dimensional tensor of the input radar image. IR Represents a three-dimensional tensor of the input infrared image;

[0037] After feature extraction is completed, a conditional generative adversarial network (cGAN) is introduced to achieve data fitting and generation for targets with variable behavior in dynamic environments. It mainly generates missing modal data based on specific conditional information. The network includes a generator and a discriminator.

[0038] In the generator of the model, random noise z and conditional information c are combined as input to generate high-quality images related to the target satellite;

[0039] The generator's network architecture includes multiple deconvolutional layers, progressively transforming low-dimensional noise into a high-dimensional image. After each deconvolutional layer, batch normalization and ReLU activation functions are applied to enhance the network's expressive power and stability. The high-resolution image output by the generator is represented as follows:

[0040] F generated =G(z,c)

[0041] The discriminator's network architecture employs multiple convolutional layers to progressively extract image features and ultimately outputs a probability value representing the likelihood that the input image is a real image. The discriminator's loss function is designed as follows:

[0042]

[0043] In the formula, D(x|c) represents the discriminator's probability of classifying a real sample under given condition c; D(G(z,c)|c) represents the discriminator's probability of classifying a generated image under given condition c; logD(x|c) and log(1-D(G(z,c)|c)) represent the log loss of the discriminator's probability of classifying real samples and generated samples, respectively. and These represent the expected values ​​of the logarithmic loss of the discrimination probability when sampling from real data samples and randomly generated samples, respectively.

[0044] The training process of cGAN involves alternating training of the generator and the discriminator, with a particular focus on how to handle constantly moving targets in dynamic environments. In each training cycle, the generator is fixed and the parameters of the discriminator are updated to maximize its accuracy in judging real images. The discriminator is fixed and the parameters of the generator are updated to minimize the probability that the generated image is judged as "fake".

[0045] SSIM is used to measure the similarity between two images, and its formula is:

[0046]

[0047] In the formula, v and w are two images, and μ v It is the average value of image v, μ w It is the average value of image w. It is the variance of the image v. σ is the variance of the image w. vw C1 and C2 are the covariances of images v and w, and small constants are introduced to stabilize the computation.

[0048] PSNR is used to evaluate the quality of reconstructed images, and its formula is:

[0049]

[0050] Where MAX is the maximum possible pixel value in the image, and MSE is the mean squared error, calculated using the following formula:

[0051]

[0052] Where N is the total number of pixels in the image, a i and b i These are the values ​​of the i-th pixel in each of the two images.

[0053] Preferably, in step 2, the multimodal data features are fused to obtain effective real-time information on the target state;

[0054] The SIFT feature detection algorithm is used to extract key points from an image. The specific steps are as follows: The input image is Gaussian blurred to create images at different scales to construct a Gaussian pyramid; at each scale, the DoG (Domain of Gaussian) is calculated to detect extreme points, using the following formula:

[0055] D(x,y,σ)=G(x,y,kσ)-G(x,y,σ)

[0056] In the formula, G is the Gaussian function, σ is the standard deviation, and k is the scaling factor;

[0057] After obtaining the keypoints and their descriptors, the transformation model is estimated using the random sampling consensus algorithm. The KNN nearest neighbor algorithm is used to match the feature descriptors, obtaining a preliminary set of matching points. A small subset of matching points is randomly selected, and the transformation model is calculated. The distances from all matching points to the estimated transformation model are calculated to determine the interior points, using the following formula:

[0058] inlier={(x i y i )||(Ax i +b, y i )||<∈}

[0059] In the formula, A is the transformation matrix, b is the translation vector, ∈ is the distance threshold, and inlier is the inlier; this process is repeated until the optimal model is found to maximize the number of inliers; the resulting transformation model homography matrix H represents the mapping relationship from the source image to the target image:

[0060]

[0061] The estimated transformation model is applied to resample the source image to ensure spatial alignment of features between different modalities; the transformation matrix H is used to resample the source image, and bilinear interpolation is used to calculate pixel values; for each coordinate (x', y') in the target image, the coordinates (x, y) in the source image can be calculated through reverse mapping.

[0062]

[0063] By solving H -1 This allows us to obtain the source coordinates, and then use interpolation methods to obtain the pixel value of that point.

[0064] The processing is performed using a previous convolutional operation and an activation function, as shown in the following formula:

[0065]

[0066] In the formula, Indicates from image I optical Extracted shallow features, Indicates from image I SAR Extracted shallow features, Indicates from image I IR The extracted shallow features, where σ is the activation function, and W shallow Let b be the weight matrix of the shallow convolution kernel. shallow This refers to the bias term for shallow convolutional layers;

[0067] The process is performed using subsequent convolutional operations and activation functions, as shown in the following formula:

[0068]

[0069]

[0070]

[0071] in, Indicates from image I optical Extracted deep features, Indicates from image I SAR Extracted deep features, Indicates from image I IR The extracted deep features, where σ is the activation function and W deep Let b be the weight matrix of the deep convolution kernel. deep For bias terms of deep convolutional layers;

[0072] Generate query (Q), key (K), and value (V) vectors through linear transformation:

[0073]

[0074] Where F represents the features extracted in the previous step, and Q... i This is the query vector in the attention mechanism, representing the input feature that needs to be focused on at the moment, K. i It is a key vector, which is compared with the query vector to determine relevance. V i It is a value vector, which is the attention score of local image features after attention mechanism processing. W is the weight matrix that maps the input features to the query vector space. k i W is the weight matrix that maps the input features to the key vector space. v i It is the weight matrix that maps the input features to the value vector space, and h is the dimension of the features, i.e. the number of heads in the multi-head attention mechanism;

[0075] Calculate the attention score for each attention head, normalize these scores using the Softmax function to obtain the output for each head. The formula for calculating the attention score is:

[0076]

[0077] In the formula, d k Given the dimension of the key vector, the outputs of each head are concatenated to form a fused feature:

[0078] F fused =Concat(A1V1,A2V2,...,A h V h )

[0079] h is the dimension of the feature, i.e., the number of heads in the multi-head attention mechanism, which generates the fused feature F. fused Next, features from different modalities need to be effectively integrated. A Graphical Network (GCN) is introduced to process the fused features and capture higher-order feature relationships. A graph structure is constructed, where nodes represent features, and edges are established based on the similarity between features. Feature propagation is performed using the graph Laplacian operator, with the following formula:

[0080] F combined =σ(D -1 / 2 AD -1 / 2 F fused W)

[0081] In the formula, A is the adjacency matrix, D is the degree matrix, and W is the learnable weight matrix;

[0082] After feature fusion is completed, the fused feature F is used. combined As input, the data is processed through a series of fully connected layers for final classification. To prevent overfitting, a Dropout layer is introduced into the classifier. The classifier formula is as follows:

[0083] Y = Softmax(W fc F combined +b)

[0084] In the formula, the weight W fc The bias parameter b is a learnable parameter obtained through training;

[0085] The final output after classification is labeled F. combined Eigenvector time series.

[0086] Preferred Step 3: First, establish a space target behavior recognition scenario. Using S and G as observation satellites and non-cooperative targets, spacecraft S is equipped with multi-modal onboard detection equipment, capable of detecting spacecraft G when it enters its observable region. The observable region refers to the range of spacecraft S's detection equipment; within this range, spacecraft G can be detected, and its orbit, position, and velocity information can be obtained through steps 1 and 2. To describe the spacecraft's motion in space, a geocentric inertial frame is used, with the origin at the Earth's center. The X-axis points to the vernal equinox, where the Earth's equatorial plane intersects the ecliptic plane; the Z-axis points to the North Pole, and the Y-axis is perpendicular to the X-axis of the equatorial plane, following the right-hand rule. A relative motion coordinate system is established in the local vertical and local horizontal coordinate systems of the central spacecraft. The starting point of the coordinate system is the spacecraft's center of mass, and the x-axis is aligned with the geocentric vector of spacecraft S, pointing from the Earth's center to S. On the orbital plane, the y-axis is perpendicular to the x-axis, pointing in the direction of motion. Finally, the z-axis is aligned with the spacecraft's angular momentum.

[0087] The configuration of a spacecraft relative to a reference orbit can change naturally due to orbital dynamics; without external disturbances or control, the relative configuration of the spacecraft remains unchanged; in the absence of orbital disturbances and control, the geometric configuration of relative motion includes two elements: ① the geometry of the relative motion trajectory of spacecraft G; ② the positional relationship between the relative motion trajectory and spacecraft S; combining the spacecraft dynamics model, the definitions of four on-orbit behavior modes are realized: launch into orbit, on-orbit operation, orbit change, and reentry into orbit.

[0088] i. Launch into orbit

[0089] The orbit is approximated using the gravitational formula of the two-body problem:

[0090]

[0091] In the formula, G is the gravitational constant, M is the mass of the Earth, m is the mass of the spacecraft, and r is the distance from the spacecraft to the central celestial body. The launch and orbit insertion process can be simulated using the simple numerical integration Euler method.

[0092] ii. In-orbit operation

[0093] The motion of a spacecraft in an elliptical orbit is described using the general Kepler orbital equations:

[0094]

[0095] In the formula, a is the semi-major axis of the orbit, e is the orbital eccentricity, and θ is the perigee angle.

[0096] iii. Motorized track changing

[0097] A general Hohmann transfer orbit is used;

[0098] Transfer from circular orbit to elliptical orbit:

[0099]

[0100]

[0101] In the formula, μ is the gravitational parameter of the central celestial body, r1 and r2 are the radii of the initial low orbit and the transfer orbit, respectively, v1 is the initial velocity in the low orbit, and v t It is the speed of the transfer orbit;

[0102] The elliptical track has been transformed into a circular track.

[0103]

[0104]

[0105] In the formula, μ is the gravitational parameter of the central celestial body, r1 and r2 are the radii of the initial low orbit and the transferred high orbit, respectively, v2 is the velocity after reaching the target orbit, and v t 'It is the velocity at the apogee of the orbit;

[0106] iv. Deorbiting

[0107] The orbital decay formula demonstrates the decreasing trend of the orbital radius under the influence of upper atmospheric drag:

[0108]

[0109] In the formula, D is the force exerted on the spacecraft by air resistance;

[0110] After characterizing and representing the on-orbit behavior of space targets, the trajectory behavior is correlated based on a trajectory fitting correlation method.

[0111] The target's behavior is determined by analyzing the solution of the Clohessy-Wiltshire equation and the characteristics of the feature points, based on the position and velocity sequences obtained in steps 1 and 2, and matching them with dynamic conditions.

[0112] By setting orbital parameters to correspond to different on-orbit events, trajectory diagrams in different states under the inertial coordinate system are obtained. Based on the nearest neighbor data association strategy, a trajectory association algorithm is used to simulate the missing state of target trajectory tracking in a spatial scenario. The original trajectory loses part of the data according to time intervals to simulate multiple interruptions in the trajectory tracking process. The trajectory event association status is determined by the nearest neighbor data association strategy. The specific implementation process is as follows:

[0113] (1) Import the merged data:

[0114] The multimodal fusion data obtained through steps 1 and 2 is imported into the model, and Gaussian noise is added to the generated data to simulate the errors that may exist in real measurements.

[0115] (2) Introducing data interruption:

[0116] Randomly select a time period and set that data segment as missing. Simulate potential data transmission interruptions or obstructions.

[0117] (3) Define the fitting function:

[0118] Define a model function to generate simulated orbital data. This function takes orbital parameters and time t as input and returns simulated orbital data; use this model function to generate simulated data to be fitted.

[0119] (4) Associate nearest neighbor data:

[0120] The nearest neighbor data is correlated and fitted by combining relative motion modes; the optimal parameters are found to make the generated simulated data as close as possible to the actual orbit; setting the initial parameters to known orbit parameters can improve the convergence speed and accuracy of the fitting.

[0121] (5) Evaluate the fitting results:

[0122] Save the fitted parameters for later comparison and verification;

[0123] For orbitals that cannot be successfully fitted, record the failure and continue processing other orbitals;

[0124] (6) Matching orbital data:

[0125] Calculate the root mean square error between the new trajectory data and the data generated by the fitted trajectory parameters; find the fitted parameters with the smallest error, which is considered to be the trajectory model that best fits the trajectory data, and give the accuracy of judging that the trajectory is the trajectory model within a certain error range;

[0126] The above algorithm is used to obtain the fitted trajectory of the target's absolute orbital state and relative orbital state and to identify the association between it and several defined behaviors.

[0127] The present invention has the following beneficial effects:

[0128] This invention achieves high-accuracy identification of target behavior in complex space environments, providing important support for spacecraft obstacle avoidance, autonomous detection, and intelligent decision-making.

[0129] This invention optimizes model performance, and the dataset design fully considers the complexity of dynamic environments, enhancing the model's generalization ability under different conditions, enabling it to better adapt to different types of original image inputs to expand practical multimodal datasets.

[0130] This invention enables fused features to effectively propagate information within a graph structure, thereby enhancing the correlation between modal features, improving the effect of feature integration, and enabling the model to effectively identify targets and provide a credibility score for each category.

[0131] This invention solves the problem of tracking loss that may occur during space target detection due to the dynamic characteristics of the target or other reasons. Attached Figure Description

[0132] Figure 1 This is a schematic diagram of the overall process of the spatial target behavior recognition method based on multimodal data fusion of the present invention;

[0133] Figure 2 This is a schematic diagram of the multimodal data feature fusion process of the present invention;

[0134] Figure 3 This is a schematic diagram of a spatial target behavior recognition scenario according to the present invention;

[0135] Figure 4 This is a schematic diagram of the trajectory association algorithm based on the nearest neighbor data association strategy of the present invention;

[0136] Figure 5 This is a schematic diagram of the absolute trajectory fitting of a space target centered on the Earth, according to the present invention.

[0137] Figure 6 This is a schematic diagram of the relative trajectory fitting of a space target centered on an observation satellite according to the present invention;

[0138] Figure 7 This is a schematic diagram of the spatial target behavior association recognition results of the present invention. Detailed Implementation

[0139] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention is described below with reference to specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the present invention.

[0140] Specific implementation method one: Combining Figure 1-7 This embodiment describes a spatial target behavior recognition method based on multimodal data fusion, which includes the following steps:

[0141] Step 1: Construction and processing of multimodal datasets:

[0142] A multimodal dataset was constructed through satellite collaborative observation. The acquired data underwent preprocessing operations such as denoising and normalization to ensure image clarity and reliability. Convolutional Neural Networks (CNNs) were used to extract features from different modalities. Additional conditional information was introduced through Conditional Generative Adversarial Networks (cGANs). The generator parameters were optimized by comparing the generated images with real images through a discriminator to improve image quality and realism. During cGAN training, the Adam optimizer was used to alternately update the generator and discriminator to optimize model performance. The quality of generated images was evaluated using metrics such as Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), and the model parameters were adjusted based on the evaluation results to enhance the robustness and generalization ability of the image data generation model, enabling it to better adapt to different types of original image inputs and expand the practical multimodal dataset.

[0143] In step 1, the multimodal dataset includes optical, spaceborne high-resolution radar, and infrared images, acquiring target image data under different distances (less than 20km, 20-50km, 50-100km), different lighting conditions, different angles, and different spatial background environments;

[0144] Additional conditional information, including target category and observation angle, makes the generated images more suitable for the specific scene, thereby improving the relevance and accuracy of the generated images;

[0145] cGAN is also used to generate modal data that is missing due to the sensor's own performance at certain specific angles. The missing modal data includes infrared images, thus completing and enhancing the dataset.

[0146] Step 2, Multimodal Image Registration and Feature Processing:

[0147] Multimodal image registration algorithms are used to ensure spatial consistency of images from different sources. The registration process includes: feature detection using the SIFT algorithm, transformation model estimation using the RANSAC algorithm, and image resampling using bilinear interpolation.

[0148] Step 2 includes:

[0149] Feature detection: Key points are extracted from the image using the SIFT algorithm, a Gaussian pyramid is constructed, and the Difference of Gaussians (DoG) is calculated to detect extreme points;

[0150] Transformation model estimation: The RANSAC algorithm is used to estimate the transformation model, matching points are randomly selected and the transformation matrix is ​​calculated, and finally a 3x3 homography matrix is ​​obtained to ensure that the source image and the target image are aligned.

[0151] Image resampling: The source image is resampled using bilinear interpolation. After image registration, a ResNet structure is used for feature extraction. The extracted features are divided into shallow and deep features. Shallow features focus on basic features (such as edges and textures), while deep features capture more complex patterns and semantic information of objects. A multi-head self-attention mechanism is introduced to enhance the feature extraction effect. By combining spatial attention and channel attention, the importance of each channel is calculated, enabling the model to adaptively focus on high-value features. The fused features are processed through a graph convolutional network (GCN) to capture higher-order feature relationships. A graph structure is constructed, where nodes represent features and edges are established based on the similarity between features. Feature propagation is performed through the graph Laplacian operator to enhance the correlation between modal features. The fused features are used as input and processed through a fully connected layer for final classification. A Dropout layer is introduced into the classifier to prevent overfitting. The final output is a labeled feature vector containing time-series information such as the target's relative satellite azimuth, distance, and velocity, providing a data foundation for subsequent target behavior recognition.

[0152] Step 3: Target Behavior Identification and Association

[0153] A scenario centered on spacecraft S and non-cooperative target G is constructed. The motion state of the spacecraft is accurately described using a geocentric inertial frame and local vertical and local horizontal coordinate systems. On-orbit behavior is defined and simulated using corresponding formulas. A trajectory association algorithm based on nearest neighbor data association strategy is used to solve the target tracking loss problem. The fitted orbital state is associated with a variety of defined events.

[0154] In step 3, spacecraft S is equipped with multi-modal onboard detection equipment, capable of acquiring real-time orbit, position, and velocity information when a non-cooperative target G enters its observable region. To accurately describe the spacecraft's motion in space, a geocentric inertial frame is adopted, with the origin set at the Earth's center, and a relative motion coordinate system is established through local vertical and local horizontal coordinate systems. In terms of behavior modes, four main on-orbit behaviors are defined: launch into orbit, on-orbit operation, maneuvering, and reentry into orbit. The launch into orbit process is simulated using the gravitational formula of the two-body problem, covering fundamental dynamic principles such as rocket thrust and gravitational acceleration. For on-orbit operation, the classical Kepler orbital equations are used to describe the spacecraft's motion in circular or elliptical orbits. For maneuvering... This invention introduces the Hohmann transfer orbit as a common method for transferring from a low Earth orbit to a higher orbit (such as a geostationary orbit), and simulates the reentry process of a spacecraft using an orbital decay formula. To address the problem of target tracking loss, a trajectory association algorithm based on a nearest-neighbor data association strategy is employed. This algorithm includes steps such as adding Gaussian noise to existing multimodal fusion information of the target, introducing data interruptions, defining a fitting function, performing parameter fitting, and calculating errors. By associating the fitted orbital state with various defined events, the algorithm achieves an orbital accuracy of over 90%. This series of processes ensures the effective identification and association of target behavior, providing crucial support for autonomous detection and decision-making by spacecraft in complex space environments.

[0155] The modules used in the method include:

[0156] A multimodal dataset construction module based on generative adversarial networks (GANs) is used. Multiple satellites equipped with high-resolution optical telescopes, spaceborne high-resolution radar, and infrared telescopes observe the same target in space from different angles at multiple distance scales (less than 20km, 20-100km, 100-500km) to obtain small-scale multimodal data. By introducing conditional generative adversarial networks and self-supervised learning, a large-scale image dataset is constructed. First, step 1, the multimodal dataset construction module based on GANs, can efficiently establish a large-scale multimodal dataset containing high-resolution optical, spaceborne high-resolution radar, and infrared images for comprehensive perception of space targets.

[0157] Multimodal data feature fusion module: For multimodal data measured over a period of time, image registration and deep / shallow feature extraction are performed. A multi-head attention mechanism and graph neural network are introduced to fuse the features. The fused features are input into a classifier and classified through a series of fully connected layers. The output is a multidimensional feature vector containing comprehensive multimodal information for subsequent target behavior recognition. Secondly, based on the multimodal data feature fusion module, deep / shallow feature extraction and fusion are used to generate a multidimensional feature vector for subsequent behavior recognition.

[0158] The target behavior recognition module based on trajectory fitting and association defines four typical behaviors for general space target scenarios and provides corresponding orbital dynamics formulas. To address the target tracking loss problem, a trajectory association algorithm based on nearest neighbor data association is introduced. Through data import, interruption introduction, and fitting function definition, accurate identification of the target's on-orbit behavior is achieved. Finally, step 3 of the target behavior recognition module based on trajectory fitting and association defines four typical orbital behaviors and achieves accurate identification through the nearest neighbor data association algorithm. This invention achieves high-accuracy identification of target on-orbit behavior in complex space environments, providing important support for spacecraft obstacle avoidance, autonomous detection, and intelligent decision-making.

[0159] Specific Implementation Method Two: Combining Figure 1-7 This embodiment describes a space target behavior recognition method based on multimodal data fusion, applicable to the fields of artificial intelligence, computer vision, and aerospace technology. The method comprises three modules: 1) a multimodal dataset construction module based on generative adversarial networks, which efficiently establishes a large-scale multimodal dataset including high-resolution optical, spaceborne high-resolution radar, and infrared images for comprehensive perception of space targets; 2) a multimodal data feature fusion module, which generates multidimensional feature time series for subsequent behavior recognition through deep and shallow feature extraction and fusion; and 3) a target behavior recognition module based on trajectory fitting and association, which defines four typical orbital behaviors and achieves accurate recognition through a nearest neighbor data association algorithm. This invention achieves high-accuracy recognition of target behavior in orbit within complex space environments, providing crucial support for spacecraft obstacle avoidance, autonomous detection, and intelligent decision-making. The method includes the following steps:

[0160] In step 1, to meet the mission requirements of satellites identifying specific target behavior in space, multiple satellites equipped with advanced sensors are selected to acquire optical, spaceborne high-resolution radar (SAR), and infrared image data. Observations cover different distances: less than 20km (high-resolution images, resolution less than 1 meter) for acquiring detailed features when approaching the target satellite; 20-100km (medium-to-high resolution images, resolution between 1 and 10 meters) for initial target identification and tracking; and 100-500km (low-resolution images, resolution greater than 10 meters) for large-scale monitoring and background identification. At least three different angles (such as top-down, oblique-down, and side-down views) are set to ensure comprehensive feature capture of the target satellite.

[0161] After data collection, the images are preprocessed to improve their quality and usability, ensuring they meet the requirements of image-based target behavior recognition tasks. First, a Gaussian filter is used to denoise the images, with the specific parameters set to a standard deviation σ between 1 and 2 pixels to control the denoising intensity and ensure clear identification of target satellites even against complex backgrounds. The denoising is achieved using the following formula:

[0162]

[0163] Where (x,y) are the pixel coordinates for denoising, k is the radius of the Gaussian kernel (representing the range of the filter), and G... σ (m,n) is the Gaussian kernel, where m and n are the offsets of the kernel relative to the center point. The formula for calculating the pixel value of the normalized image is:

[0164]

[0165] In the formula, I(x,y) and I normalized (x, y) represent the pixel values ​​at coordinates (x, y) of the original image and the normalized image, respectively. min It is the minimum value of the image pixels, I max It is the maximum value of the image pixels; through the above preprocessing steps, the clarity and reliability of the image are ensured, providing high-quality data for the subsequent feature extraction and generation process, and ultimately supporting the accurate identification and tracking of target satellites by multiple satellites;

[0166] After data preprocessing, a Convolutional Neural Network (CNN) is used for feature extraction to extract useful features from images of different modalities. The feature extraction process also optimizes for specific attributes of the target (such as shape, color, and thermal features). The designed CNN model includes multiple convolutional and pooling layers. Specifically, the input layer receives optical, radar, and infrared images, followed by 5 to 10 convolutional layers, each using a different kernel size (3×3 or 5×5) to capture different image features. After each convolutional layer, a max-pooling layer is used for downsampling to reduce feature dimensionality while preserving key features. The outputs of the optical, radar, and infrared images after feature extraction by the CNN are represented as follows:

[0167] F optical =CNN(I optical )

[0168] F SAR =CNN(I SAR )

[0169] F IR =CNN(I IR )

[0170] In the formula, I optical I represents the three-dimensional tensor of the input optical image. SAR I represents the three-dimensional tensor of the input radar image. IR The input infrared image is represented by a three-dimensional tensor, which includes the image's height, width, and number of channels. These extracted features are used for subsequent training of the Conditional Generative Adversarial Network (cGAN) to help the generator and discriminator better understand the features of different modal data, thereby improving the quality and relevance of the generated images.

[0171] After feature extraction is completed, a conditional generative adversarial network (cGAN) is introduced to achieve data fitting and generation for targets with variable behavior in dynamic environments. It mainly generates missing modal data based on specific conditional information (especially the need for capturing fast-moving targets and recognizing behavior in complex backgrounds). The network includes a generator and a discriminator.

[0172] In the generator of the model, random noise z (sampled from a standard normal distribution to increase the diversity of generated images) and conditional information c (such as the target satellite's category, orbital information, velocity, observation angle, and timestamp) are combined as input to generate high-quality images related to the target satellite. In dynamic environments, orbital information and velocity changes are more important and can help the generator predict the appearance of the target satellite at different points in time.

[0173] The generator's network architecture includes multiple deconvolutional layers, progressively transforming low-dimensional noise into a high-dimensional image. After each deconvolutional layer, batch normalization and ReLU activation functions are applied to enhance the network's expressive power and stability. The high-resolution image output by the generator is represented as follows:

[0174] F generated =G(z,c)

[0175] To ensure that the generated images are visually similar to real images, the generator continuously adjusts its parameters during training to minimize the differences from real images, especially the changes in the appearance of spatial targets from specific angles due to their dynamic motion.

[0176] The main function of the discriminator is to evaluate the authenticity of the input images (including real images and generated images). The input of the discriminator includes: real images (from real samples known in the dataset, covering satellite images under different orbits and velocities), generated images (images output by the generator), and conditional information c (such as the target satellite's category, orbital information, velocity, observation angle, and timestamp, etc.) to ensure that the discriminator can correctly judge the usability of the image under specific conditions.

[0177] The discriminator's network architecture employs multiple convolutional layers to progressively extract image features and ultimately outputs a probability value representing the likelihood that the input image is a real image. The discriminator's loss function is designed as follows:

[0178]

[0179] In the formula, D(x|c) represents the discriminator's probability of classifying a real sample under given condition c; D(G(z,c)|c) represents the discriminator's probability of classifying a generated image under given condition c; logD(x|c) and log(1-D(G(z,c)|c)) represent the log loss of the discriminator's probability of classifying real samples and generated samples, respectively. and E z～p(z) [log(1-D(G(z,c)|c))] represents the expected value of the logarithmic loss of the discrimination probability when sampling from real data samples and randomly generated samples, respectively; by continuously optimizing the discriminator, its sensitivity to the differences between real and generated images is improved, ensuring that targets on different orbits can be accurately identified in rapidly changing spatial environments.

[0180] The training process of cGAN involves alternating training of the generator and discriminator, with a particular focus on handling constantly moving targets in dynamic environments. In each training epoch, the generator is first fixed, and the discriminator's parameters are updated to maximize its accuracy in identifying real images. Next, the discriminator is fixed, and the generator's parameters are updated to minimize the probability of generated images being judged as "fake." The specific implementation is as follows:

[0181] Update the discriminator: from the real image distribution p data A batch of samples x are randomly selected from the middle i Ensure the samples cover satellite images with different speeds and orbits; extract a batch of generated samples G(z0,c0) from the generator, where z0 is random noise and c0 is the conditional information at the current time point; calculate the discriminator loss L. D And update its parameters to improve the accuracy of the judgment;

[0182] Update generator: Draw a batch of samples z from a random noise distribution z. i Using dynamic condition information c i Generate image G(z) i ,c i To maintain consistency between the generated image and the appearance of the target satellite at different times, the generator's loss L is calculated. G It updates its parameters to increase the probability that the generated image will be judged as "real".

[0183] This alternating optimization training strategy continuously improves the performance of the generator and discriminator, enabling the generator to produce more realistic images that better meet dynamic conditions.

[0184] To evaluate the performance of cGAN, structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) will be used periodically during training to ensure that the generated images are visually similar to real images. The adaptability of the generated images under different tracks and speeds will also be considered to ensure the model has good generalization ability and robustness. Based on the evaluation results, the model's hyperparameters (such as learning rate and batch size) will be adjusted in a timely manner, and the network structure will be optimized to improve the quality and diversity of generated images in dynamic environments.

[0185] SSIM is used to measure the similarity between two images, and its formula is:

[0186]

[0187] In the formula, v and w are two images, and μ v It is the average value of image v, μ w It is the average value of image w. It is the variance of the image v. σ is the variance of the image w. vw C1 and C2 are the covariances of images v and w, and small constants are introduced to stabilize the computation.

[0188] PSNR is used to evaluate the quality of reconstructed images, and its formula is:

[0189]

[0190] Where MAX is the maximum possible pixel value in the image, and MSE is the mean squared error, calculated using the following formula:

[0191]

[0192] Where N is the total number of pixels in the image, a i and b i These are the values ​​of the i-th pixel in each of the two images;

[0193] The design and training process of this cGAN model is suitable for mission requirements of multiple satellites collaboratively identifying dynamic space targets in space. cGAN can improve the data sharing and fusion capabilities between different satellites, enabling more accurate identification and tracking of fast-moving targets in complex environments. The introduction of dynamic conditions ensures that the generated images are relevant to specific observation tasks, improving the efficiency and success rate of mission execution.

[0194] The large-scale dataset expected to be built through this step includes high-resolution optical images, spaceborne high-resolution radar images, and infrared images. The optical images, sourced from high-resolution optical telescopes, are characterized by high clarity and rich detail, suitable for the identification and feature extraction of close-range targets, and cover target features under different lighting conditions. Radar images provide the reflection characteristics of targets, suitable for monitoring and tracking the real-time motion of dynamic targets. Infrared images can capture the thermal characteristics of targets, especially suitable for capturing the state of target propulsion components. The dataset will cover different types of space targets (such as satellites, space debris, man-made objects, etc.) and various scenarios (such as launches). To ensure broad applicability of the data, the dataset covers different distance scales (less than 20km, 20-100km, 100-500km) and observation angles (top view, oblique view, side view, etc.), providing comprehensive range of distance and perspective. Generative Adversarial Networks (cGANs) are used to ensure that the generated images are visually highly similar to real images, and they are accompanied by detailed annotation information, including target category, orbital parameters, relative position, velocity, and observation time, providing a training foundation for subsequent fusion models. Furthermore, the dataset design fully considers the complexity of dynamic environments, enhancing the model's generalization ability under different conditions.

[0195] In step 2, the multimodal data features are fused to obtain effective real-time target state information. The flowchart is shown in the attached figure. Figure 2 As shown;

[0196] Before multimodal data fusion, multimodal images are first registered to ensure spatial consistency between images from different sources. The registration process mainly includes three steps: feature detection, transform model estimation, and image resampling.

[0197] The SIFT feature detection algorithm is used to extract key points (saliency and stability points in an image that can be found in images of different modalities). The specific steps are as follows: The input image is Gaussian blurred to create images at different scales to construct a Gaussian pyramid; at each scale, the DoG (Difference of Gaussian) of the image is calculated to detect extreme points, using the following formula:

[0198] D(x,y,σ)=G(x,y,kσ)-G(x,y,σ)

[0199] In the formula, G is the Gaussian function, σ is the standard deviation, and k is the scale factor; D is a descriptor calculated for each key point, set as a 128-dimensional vector, describing the local image features around it, for pixels with coordinates (x,y);

[0200] After obtaining the keypoints and their descriptors, the transformation model is estimated using the Random Sample Consensus Algorithm (RANSAC). The KNN nearest neighbor algorithm is used to match the feature descriptors, obtaining a preliminary set of matching points. A small subset of matching points (usually four points) is randomly selected, and the transformation model (such as an affine transformation or homography matrix) is calculated. The distances from all matching points to the estimated transformation model are calculated to determine the interior points. The formula is as follows:

[0201] inlier={(x i y i )||(Ax i +b, y i )||<∈}

[0202] In the formula, A is the transformation matrix, b is the translation vector, ∈ is the distance threshold, and inlier is the inlier. This process is repeated until the optimal model is found to maximize the number of inliers. The final transformation model is usually a 3x3 homography matrix H, representing the mapping relationship from the source image to the target image.

[0203]

[0204] The estimated transformation model is applied to resample the source image to ensure spatial alignment of features between different modalities; the transformation matrix H is used to resample the source image, and bilinear interpolation is used to calculate pixel values; for each coordinate (x', y') in the target image, the coordinates (x, y) in the source image can be calculated through reverse mapping.

[0205]

[0206] By solving H -1 The source coordinates can be obtained, and then the pixel value of the point can be obtained using interpolation methods. Through these three steps (feature detection, transform model estimation, and image resampling), images of different modalities can be effectively aligned.

[0207] Based on the already registered images, feature extraction is performed on each modality image, and the ResNet structure is used to capture multi-level feature information; the feature information is divided into shallow features and deep features.

[0208] Shallow features mainly refer to basic features in an image, such as edges, textures, and colors. These features are extracted through the first few layers of a convolutional neural network (CNN), such as layers C1 and C2. Shallow features typically have low spatial resolution but contain rich local information. They are processed using early convolutional operations and activation functions, as shown in the following formula:

[0209]

[0210] In the formula, Indicates from image I optical Extracted shallow features, Indicates from image I SAR Extracted shallow features, Indicates from image I IR The extracted shallow features, where σ is the activation function, and W shallow b is the weight matrix of the shallow convolutional kernel (containing multiple filters for extracting different features). shallow This is the bias term for the shallow convolutional layer; (for other parameters, refer to the feature extraction section in step 1);

[0211] Deeper features focus on higher-level semantic information, located in later layers of the network (such as layers C4 and C5), and can capture more complex patterns and semantic relationships between objects. They are processed using later convolutional operations and activation functions, as shown in the following formula:

[0212]

[0213]

[0214]

[0215] in, Indicates from image I optical Extracted deep features, Indicates from image I SAR Extracted deep features, Indicates from image I IR The extracted deep features, where σ is the activation function and W deep b is the weight matrix of the deep convolutional kernel (containing multiple filters for extracting different features). deep This is the bias term for deep convolutional layers; (for other parameters, refer to the feature extraction section in step 1);

[0216] To further enhance the feature extraction effect, a multi-head self-attention mechanism is introduced to focus on important feature regions. By combining spatial attention and channel attention mechanisms and calculating the importance of each channel, the model can adaptively focus on high-value features. On this basis, a multi-head attention mechanism is introduced to enable the model to learn feature representations from multiple subspaces in parallel.

[0217] Generate query (Q), key (K), and value (V) vectors through linear transformation:

[0218]

[0219]

[0220]

[0221] (i = 1, 2, ..., h)

[0222] In the formula, F represents the features extracted in the previous step, and Q... i This is the query vector in the attention mechanism, representing the input feature that needs to be focused on at the moment, K. i It is a key vector, which is compared with the query vector to determine relevance. V i It is a value vector, which is the attention score of local image features after attention mechanism processing. It is the weight matrix that maps the input features to the query vector space. It is the weight matrix that maps the input features to the key vector space. It is the weight matrix that maps the input features to the value vector space, and h is the dimension of the features, i.e. the number of heads in the multi-head attention mechanism;

[0223] Calculate the attention score for each attention head, normalize these scores using the Softmax function to obtain the output for each head. The formula for calculating the attention score is:

[0224]

[0225] In the formula, d k Given the dimension of the key vector, the outputs of each head are concatenated to form a fused feature:

[0226] F fused =Concat(A1V1,A2V2,...,A h V h )

[0227] Generate fusion feature F fused Next, features from different modalities need to be effectively integrated. A Graph Convolutional Network (GCN) is introduced to process the fused features and capture higher-order feature relationships. A graph structure is constructed, where nodes represent features, and edges are established based on the similarity between features (cosine similarity can be used). Feature propagation is performed using the Graph Laplacian operator, with the following formula:

[0228] F combined =σ(D -1 / 2 AD -1 / 2 F fused W)

[0229] In the formula, A is the adjacency matrix, D is the degree matrix, and W is the learnable weight matrix; this process enables the fused features to effectively propagate information in the graph structure, thereby enhancing the correlation between modal features and improving the effect of feature integration.

[0230] After feature fusion is completed, the fused feature F is used.combined As input, the data is processed through a series of fully connected layers for final classification. To prevent overfitting, a Dropout layer is introduced into the classifier. The classifier formula is as follows:

[0231] Y = Softmax(W fc F combined +b)

[0232] In the formula, the weight W fc The bias parameter b is a learnable parameter obtained through training; in this way, the model can effectively identify targets and provide a confidence score for each category.

[0233] The final output after classification is labeled F. combined The feature vector time series contains the target's azimuth sequence relative to the observation satellite, the target's distance sequence relative to the observation satellite, and the target's velocity sequence relative to the observation satellite, extracted based on three modal data.

[0234] Step 3: First, establish a spatial target behavior recognition scenario; as shown in the attached image. Figure 3 As shown, S and G are used as observation satellites and non-cooperative targets. Spacecraft S is equipped with multi-modal onboard detection equipment, which can detect spacecraft G when it enters its observable region. The observable region refers to the range of spacecraft S's detection equipment, within which spacecraft G can be detected, and its orbit, position, and velocity information can be obtained through steps 1 and 2. To describe the spacecraft's motion in space, a geocentric inertial frame is used, with the origin located at the center of the Earth. The X-axis points to the vernal equinox, where the Earth's equatorial plane intersects the ecliptic plane. The Z-axis points to the North Pole, and the Y-axis is perpendicular to the X-axis of the equatorial plane, following the right-hand rule. A relative motion coordinate system is established in the local vertical and local horizontal coordinate systems of the central spacecraft. The starting point of the coordinate system is the spacecraft's center of mass, and the x-axis is aligned with the geocentric vector of spacecraft S, pointing from the Earth's center to S. On the orbital plane, the y-axis is perpendicular to the x-axis and points in the direction of motion. Finally, the z-axis is aligned with the spacecraft's angular momentum.

[0235] The configuration of a spacecraft relative to a reference orbit can change naturally due to orbital dynamics; without external disturbances or control, the relative configuration of the spacecraft remains unchanged; in the absence of orbital disturbances and control, the geometric configuration of relative motion includes two elements: ① the geometry of the relative motion trajectory of spacecraft G; ② the positional relationship between the relative motion trajectory and spacecraft S; combining the spacecraft dynamics model, the definitions of four on-orbit behavior modes are realized: launch into orbit, on-orbit operation, orbit change, and reentry into orbit.

[0236] i. Launch into orbit

[0237] Launch into orbit refers to the process by which a spacecraft is launched from Earth into outer space and enters a specific orbit; it mainly involves basic dynamic properties such as rocket thrust and gravitational acceleration; the orbit is approximated using the gravitational formula of the two-body problem.

[0238]

[0239] In the formula, G is the gravitational constant, M is the mass of the Earth, m is the mass of the spacecraft, and r is the distance from the spacecraft to the central celestial body. The launch and orbit insertion process can be simulated using the simple numerical integration Euler method.

[0240] ii. In-orbit operation

[0241] On-orbit operation can be understood as a spacecraft moving in a circular or elliptical orbit along a specific path; on-orbit operation is simulated based on classical trajectory equations; the general Kepler orbital equations are used to describe the motion of the spacecraft in an elliptical orbit:

[0242]

[0243] In the formula, a is the semi-major axis of the orbit, e is the orbital eccentricity, and θ is the perigee angle.

[0244] iii. Motorized track changing

[0245] Maneuvering refers to the process by which a spacecraft moves from one orbit to another. Maneuvering usually requires orbital adjustments, which can be achieved by impacting a ground station or by briefly propelling the spacecraft with its own engines. Here, we use the general Hohmann transfer orbit (a common method for moving from a low Earth orbit to a higher orbit, such as a geostationary orbit).

[0246] Circular orbit to elliptical orbit transition (first injection, injection at half focal length):

[0247]

[0248]

[0249] In the formula, μ is the gravitational parameter of the central celestial body, r1 and r2 are the radii of the initial low orbit and the transfer orbit, respectively, v1 is the initial velocity in the low orbit, and v t It is the speed of the transfer orbit;

[0250] The elliptical orbit transitions to a circular orbit (second jet, jet towards apogee):

[0251]

[0252]

[0253] In the formula, μ is the gravitational parameter of the central celestial body, r1 and r2 are the radii of the initial low orbit and the transferred high orbit, respectively, v2 is the velocity after reaching the target orbit, and v t 'It is the orbital apogee velocity;

[0254] iv. Deorbiting

[0255] To make a spacecraft re-enter Earth's orbit means to make it leave Earth's orbit, re-enter the atmosphere, and burn up. This can be achieved by increasing the orbital decay rate; the orbital decay formula shows the law of decreasing orbital radius under the drag of the upper atmosphere:

[0256]

[0257] In the formula, D is the force exerted on the spacecraft by air resistance;

[0258] After characterizing and representing the on-orbit behavior of space targets, the trajectory behavior is correlated based on a trajectory fitting correlation method.

[0259] The target's behavior is determined by analyzing the solution of the Clohessy-Wiltshire equation and the characteristics of the feature points, based on the position and velocity sequences obtained in steps 1 and 2, and matching them with dynamic conditions.

[0260] By setting orbital parameters to correspond to different on-orbit events, trajectory diagrams in different states in the inertial coordinate system are obtained. During space target detection, tracking loss may occur due to the target's dynamic characteristics or other reasons. This high dynamic characteristic often causes target tracking instability or tracking loss. To address this problem, a trajectory association algorithm based on the nearest neighbor data association strategy is studied, and the process is shown in the attached figure. Figure 4 As shown, a simulation of missing data in target trajectory tracking was performed in a spatial scenario. Partial data was lost from the original trajectory at time intervals to simulate multiple interruptions in the trajectory tracking process. The correlation of trajectory events was determined using a nearest neighbor data association strategy. The specific implementation process is as follows:

[0261] (1) Import the merged data:

[0262] The multimodal fusion data obtained through steps 1 and 2 is imported into the model, and Gaussian noise is added to the generated data to simulate the errors that may exist in real measurements.

[0263] (2) Introducing data interruption:

[0264] Randomly select a time period and set that data segment as missing. Simulate potential data transmission interruptions or obstructions.

[0265] (3) Define the fitting function:

[0266] Define a model function to generate simulated orbital data. This function takes orbital parameters (semi-major axis a, orbital inclination, and orbital eccentricity) and time t as input and returns simulated orbital data; use this model function to generate simulated data to be fitted.

[0267] (4) Associate nearest neighbor data:

[0268] Nearest neighbor data correlation fitting is performed by combining relative motion methods. The optimal parameters are sought to make the generated simulated data as close as possible to the actual orbit; setting the initial parameters to known orbital parameters can improve the convergence speed and accuracy of the fitting.

[0269] (5) Evaluate the fitting results:

[0270] Save the fitted parameters for later comparison and verification;

[0271] For orbitals that cannot be successfully fitted, record the failure and continue processing other orbitals;

[0272] (6) Matching orbital data:

[0273] Calculate the root mean square error between the new trajectory data and the data generated by the fitted trajectory parameters; find the fitted parameters with the smallest error, which is considered to be the trajectory model that best fits the trajectory data, and give the accuracy of judging that the trajectory is the trajectory model within a certain error range;

[0274] Using the above algorithm, the fitted trajectories of the target's absolute orbital state and relative orbital state are obtained as shown in the attached figures. Figure 5 Appendix Figure 6 As shown in the attached figure, the identification results associated with the defined behaviors are as follows. Figure 7 As shown, the orbital shape may change slightly over different time periods, and the accuracy of orbit determination is over 90% for each event.

[0275] It should be noted that in the above embodiments, as long as the technical solutions are not contradictory, they can be permuted and combined. Those skilled in the art can exhaust all possibilities based on the mathematical knowledge of permutation and combination. Therefore, the present invention will not describe the technical solutions after permutation and combination one by one, but it should be understood that the technical solutions after permutation and combination have been disclosed by the present invention.

[0276] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A space target behavior recognition method based on multi-modal data fusion, characterized in that: The method comprises the following steps: Step 1, multi-modal dataset construction based on generative adversarial network: A multi-modal dataset is constructed through satellite cooperative observation, and the obtained data is denoised and normalized. The features of different modalities are extracted using CNN, additional condition information is introduced through cGAN, the generator parameters are optimized by comparing the generated image with the real image through the discriminator, the Adam optimizer is used in the cGAN training process, and the SSIM and PSNR indicators are used to evaluate the quality of the generated image and adjust the model parameters; Step 2, multi-modal data feature fusion: The spatial consistency of images from different sources is ensured through a multi-modal image registration algorithm. The registration process includes feature detection using the SIFT algorithm, transformation model estimation using the RANSAC algorithm, and image resampling using the bilinear interpolation method; Step 3, target behavior recognition based on trajectory fitting correlation: A scene centered on spacecraft and non-cooperative targets is constructed. The spacecraft motion state is described using the geocentric inertial frame and the local vertical and local horizontal coordinate system. The on-orbit behavior is defined and simulated using the corresponding formula. The nearest neighbor data association strategy is used to solve the target tracking loss problem. The fitted orbit state is correlated with the defined events; In step 3, the spacecraft is equipped with multi-modal satellite-borne detection equipment, which can obtain real-time orbit, position and velocity information of the non-cooperative target when it enters the observable area. The geocentric inertial frame is used, with the origin at the center of the Earth, and a relative motion coordinate system is established through the local vertical and local horizontal coordinate system. Four on-orbit behaviors are defined, including launch into orbit, on-orbit operation, maneuvering orbit transfer and re-entry orbit departure. The launch into orbit process is simulated using the gravitational formula of the two-body problem, covering the basic dynamics principles of rocket thrust and gravitational acceleration. In terms of on-orbit operation, the classical Kepler orbit equation is used to describe the spacecraft's motion on a circular or elliptical orbit. For maneuvering orbit transfer, the Hohmann transfer orbit is introduced as a transfer from low Earth orbit to higher orbit, and the spacecraft's re-entry process is simulated through the orbit decay formula. To solve the target tracking loss problem, the nearest neighbor data association strategy-based trajectory association algorithm includes adding Gaussian noise to the existing multi-modal fusion information, introducing data interruption, defining the fitting function, performing parameter fitting and error calculation steps. The algorithm correlates the fitted orbit state with the defined multiple events; The method uses the following modules: Multi-modal dataset construction module based on generative adversarial network: multiple satellites equipped with high-resolution optical telescopes, satellite-borne high-resolution radars and infrared telescopes observe the same target from different angles at multiple distance scales to obtain small-scale multi-modal data. The conditional generative adversarial network and self-supervised learning are introduced to realize large-scale image dataset construction. The multi-modal data feature fusion module: image registration and deep and shallow level feature extraction are performed on the multi-modal data measured in a period of time, a multi-head attention mechanism and a graph neural network are introduced to fuse the features, the fused features are input into a classifier to realize classification through a series of fully connected layers, and a multi-dimensional feature vector containing multi-modal comprehensive information is output, which is used for subsequent target behavior recognition; The target behavior recognition module based on trajectory fitting correlation: four typical behaviors of space targets in general scenarios are defined, and the corresponding orbit dynamics formulas are given. To deal with the problem of target tracking loss, a trajectory correlation algorithm based on nearest neighbor data correlation is introduced. Through data import, interruption introduction, and fitting function definition, the orbit behavior of the target is accurately recognized.

2. The space target behavior recognition method based on multi-modal data fusion according to claim 1, characterized in that: In step 1, the multi-modal data set includes optical, spaceborne high-resolution radar, and infrared images. Target image data is obtained under different distances, different lighting, different angles, and different spatial background environments. Additional condition information includes target category and observation angle. The cGAN generates missing modal data of sensor performance, including infrared images.

3. The space target behavior recognition method based on multi-modal data fusion according to claim 2, characterized in that: Step 2 includes: Feature detection: SIFT algorithm is used to extract key points from the image, and a Gaussian pyramid is constructed to calculate the difference Gaussian to detect extreme points; Transformation model estimation: RANSAC algorithm is used to estimate the transformation model, and a 3x3 homography matrix is obtained by randomly selecting matching points and calculating the transformation matrix to ensure that the source image is aligned with the target image; Image resampling: bilinear interpolation method is used for resampling the source image; after image registration, ResNet structure is used for feature extraction, and the extracted features are divided into shallow and deep features. A multi-head self-attention mechanism is introduced to enhance feature extraction, combined with spatial attention and channel attention, to enable the model to adaptively focus on high-value features. The fused features are processed by GCN; a graph structure is constructed, with nodes representing features and edges connecting features based on their similarity. Feature propagation is performed through graph Laplacian to enhance the correlation between modal features; the fused features are used as input and classified through fully connected layers. A Dropout layer is introduced in the classifier.

4. The space target behavior recognition method based on multi-modal data fusion according to claim 3, characterized in that: In step 1, to meet the task requirements of satellite target behavior recognition in space, multiple satellites with advanced sensors are selected to obtain optical, spaceborne high-resolution radar, and infrared image data; After data collection is completed, the images are preprocessed; a Gaussian filter is used for denoising, and the denoising is realized using the following formula: wherein, is a pixel coordinate to which the de-noising process is performed, is a radius of a Gaussian kernel, indicating a range of the filter, is a Gaussian kernel, and is an offset of the kernel with respect to a center point, and a pixel value of the normalized image is calculated by the following formula: wherein, and are pixel values of the original image and the normalized image at coordinates respectively, is the minimum value of the image pixels, is the maximum value of the image pixels; After data preprocessing is completed, CNN is used for feature extraction, and the outputs of the optical image, radar image, and infrared image after convolutional neural network feature extraction are represented as: wherein, represents a three-dimensional tensor of the input optical image, represents a three-dimensional tensor of the input radar image, represents a three-dimensional tensor of the input infrared image; After feature extraction is completed, a conditional generative adversarial network (cGAN) is introduced to generate missing modal data for behaviorally variable targets in dynamic environments. The network includes a generator and a discriminator. In the generator of the model, random noise and conditional information is combined as input to generate a high-quality image related to the target satellite; The network architecture of the generator includes multiple deconvolutional layers, which gradually convert the low-dimensional noise into high-dimensional images; after each deconvolutional layer, batch normalization and ReLU activation functions are applied to enhance the expressive power and stability of the network; the high-resolution image output by the generator is represented as: The network architecture of the discriminator adopts multiple convolutional layers, which gradually extract image features and finally output a probability value representing the likelihood of the input image being a real image; the loss function of the discriminator is designed as: wherein denotes the discriminative probability of the discriminator for real samples under given conditions; denotes the discriminative probability of the discriminator for generated images under given conditions; denotes the discriminative probability of the discriminator for real samples under given conditions; denotes the discriminative probability of the discriminator for generated images under given conditions; and denote the log-loss of the discriminative probability of the discriminator for real samples and generated samples, respectively; and denote the expected value of the log-loss of the discriminative probability for samples drawn from real data samples and randomly generated samples, respectively. The training process of cGAN is an alternating training of the generator and the discriminator, focusing on how to handle moving targets in a dynamic environment; in each training cycle, the generator is fixed and the parameters of the discriminator are updated to maximize its accuracy in judging real images; the discriminator is fixed and the parameters of the generator are updated to minimize the probability of the generated image being judged as "fake"; SSIM is used to measure the similarity between two images, and its formula is: wherein v and w are two images, is the average value of the image v , is the average value of the image w , is the variance of the image v , is the variance of the image w , is the covariance of the images v and w , and are small constants introduced for stabilization of the calculation; PSNR is used to evaluate the quality of the reconstructed image, and its formula is: wherein is the maximum pixel value possible in the image, is the mean square error, calculated as: wherein is the total number of pixels of the image, a i and b i are the values of the two images at the pixel.

5. The space target behavior recognition method based on multi-modal data fusion according to claim 4, characterized in that: In step 2, the features of the multi-modal data are fused to obtain effective real-time information of the target state; SIFT feature detection algorithm is used to extract key points from the image, the specific implementation steps are as follows: Gaussian blur processing is performed on the input image, and images of different scales are created to construct a Gaussian pyramid; on each scale, the DoG of the image is calculated to detect extreme points, and the formula is: wherein is a Gaussian function, σ is the standard deviation, is a scale factor; After obtaining the key points and their descriptors, the random sample consensus algorithm is used to estimate the transformation model; KNN nearest neighbor algorithm is used to match the feature descriptors to obtain the preliminary matching point set, a small part of the matching points are randomly selected, the transformation model is calculated, the distance of all matching points to the estimated transformation model is calculated, and the inliers are determined, and the formula is: wherein, is a translation vector, is a translation vector, is a distance threshold; the process is repeated until the best model is found to maximize the number of inliers; the resulting transformation model homography matrix H represents the mapping from the source image to the target image: The estimated transformation model is applied to resample the source image to ensure that the features between different modalities are spatially aligned; the transformation matrix The source image is resampled and the pixel value is calculated using the bilinear interpolation method; for each coordinate in the target image The coordinates in the source image can be calculated by inverse mapping : By solving the source coordinates can be obtained, and then the pixel value of the point is obtained by using an interpolation method; The front layer convolution operation and the activation function are used for processing, and the formula is as follows: wherein, represents the shallow features extracted from the image represents the shallow features extracted from the image represents the shallow features extracted from the image represents the shallow features extracted from the image represents the shallow features extracted from the image represents the shallow features extracted from the image is a weight matrix of the shallow convolution kernel, is a bias term of the shallow convolution layer; The back layer convolution operation and the activation function are used for processing, and the formula is as follows: wherein, represents deep features extracted from the image, represents deep features extracted from the image, represents deep features extracted from the image, represents deep features extracted from the image, represents deep features extracted from the image, represents deep features extracted from the image, σ is an activation function, is a weight matrix of the deep convolution kernel, is a bias term of the deep convolution layer; The query (Q), key (K), and value (V) vectors are generated through linear transformation: wherein, are the respective features extracted from the previous step, is the query vector in the attention mechanism, representing the input features that need to be focused on currently, is the key vector, which is compared with the query vector to determine the relevance, is the value vector, which is the attention score of the local features of the image after being processed by the attention mechanism, is the weight matrix that maps the input features to the query vector space, is the weight matrix that maps the input features to the key vector space, is the weight matrix that maps the input features to the value vector space, is the dimension of the feature, i.e., the number of heads in the multi-head attention mechanism. The attention scores of each attention head are calculated, and the Softmax function is used to normalize these scores to obtain the output of each head, and the calculation formula of the attention score is: In the formula, is the dimension of the bond vector, and the outputs of each head are spliced to form a fusion feature: Generating fused features After that, the features of different modalities need to be effectively integrated, and GCN is introduced to process the fused features to capture higher-order feature relationships; a graph structure is constructed, in which nodes represent features and edges are established according to the similarity between features; feature propagation is performed through the graph Laplacian operator, and the formula is: wherein is an adjacency matrix, is a degree matrix, is a learnable weight matrix; After completing the feature fusion, the fused features are used As input, a series of fully connected layers are used for final classification; to prevent model overfitting, a Dropout layer is introduced in the classifier, and the classifier formula is: where the weights and the bias parameter are learned parameters obtained by training. The final output after classification is a labeled feature vector time series.

6. The space target behavior recognition method based on multi-modal data fusion according to claim 5, characterized in that: Step 3, first establish the space target behavior recognition scene; take S and G as the observation satellite and the non-cooperative target, and the S spacecraft is equipped with multi-modal satellite-borne detection equipment, which can detect the spacecraft G when it enters its observable area; the observable area refers to the range of the detection equipment of the spacecraft S, within which the spacecraft G can be detected, and the information of its orbit, position and velocity can be obtained through steps 1 and 2; in order to describe the motion state of the spacecraft in space, the geocentric inertial frame is adopted, with the origin at the center of the Earth; the X-axis points to the vernal equinox, i.e. the intersection of the Earth's equatorial plane and the ecliptic plane; the Z-axis points to the North Pole, and the Y-axis is perpendicular to the X-axis of the equatorial plane, following the right-hand rule; a relative motion coordinate system is established in the local vertical and local horizontal coordinate system of the central spacecraft, with the origin at the center of mass of the spacecraft, and the x-axis aligned with the geocentric vector of the spacecraft S, pointing from the geocenter to S; In the orbital plane, the y-axis is perpendicular to the x-axis, pointing in the direction of motion; finally, the z-axis is aligned with the spacecraft's angular momentum; The spacecraft's configuration relative to the reference orbit can change naturally due to orbital dynamics; without external disturbances or control, the relative configuration of the spacecraft remains unchanged; without orbital disturbances and control, the geometric configuration of relative motion includes two elements: ① the geometric shape of the spacecraft G's relative motion trajectory; ② the positional relationship between the relative motion trajectory and the spacecraft S; combined with the spacecraft dynamics model, the definitions of the four on-orbit behavior modes of launch into orbit, on-orbit operation, orbit transfer, and re-entry de-orbit are realized; ⅰ Launch into orbit The gravitational formula of the two-body problem is used to approximate the orbit: In the formula, G is the gravitational constant, M is the mass of the Earth, m is the mass of the spacecraft, and r is the distance from the spacecraft to the central celestial body. The launch into orbit process can be simulated using the simple numerical integral Euler method; ⅱ On-orbit operation The spacecraft's motion on an elliptical orbit is described using the general Keplerian orbit equation: In the formula, a is the semi-major axis of the orbit, e is the orbital eccentricity, and θ is the perigee angle; ⅲ Maneuvering orbit transfer A general Hohmann transfer orbit is used; Circular to elliptical orbit transfer: where μ is the gravitational parameter of the central body, and are the radii of the initial low orbit and the transfer orbit, respectively, is the initial velocity at the low orbit, is the velocity at the transfer orbit; Elliptical to circular orbit completion: where μ is the gravitational parameter of the central body, and are the radii of the initial low orbit and the transfer high orbit, respectively, is the velocity after reaching the target orbit, is the apogee velocity of the orbit. ⅳ Re-entry de-orbit The orbit decay formula shows the law of the reduction of the orbit radius under the action of the high-altitude atmospheric resistance: In the formula, D is the force exerted on the spacecraft by air resistance; After the characterization and representation of the on-orbit behavior of space targets, the trajectory behavior is associated based on the trajectory fitting correlation method; According to the analytical solution of the Clohessy-Wiltshire equation and the characteristics of the characteristic points, the position and velocity sequences obtained in steps 1 and 2 are matched with the dynamic conditions to determine the behavior of the target; By setting the orbit parameters corresponding to different on-orbit events, the trajectory graphs in different states in the inertial coordinate system are obtained; Based on the nearest neighbor data association strategy, the trajectory association algorithm is used to simulate the missing state of the target trajectory in the space scene. The original trajectory is missing part of the data according to the time interval to simulate the multiple interruptions in the trajectory tracking process. The nearest neighbor data association strategy is used to determine the trajectory event association. The specific implementation process is as follows: (1) Import fusion data: import the multi-modal fusion data obtained in steps 1 and 2 into the model, and add Gaussian noise to the generated data to simulate the errors that may exist in real measurement; (2) Introduce data interruption: randomly select a time period and set the data in that period as missing; simulate the possible data transmission interruption or occlusion phenomenon; (3) Define the fitting function: define a model function that takes the orbit parameters and time t as input and returns the simulated orbit data; use this model function to generate the simulated data to be fitted; (4) Associate the nearest neighbor data: Combine the relative motion mode to perform nearest neighbor data association fitting; find the best parameters to make the generated simulated data as close as possible to the actual orbit; the initial parameters are set to the known orbit parameters, which can improve the convergence speed and accuracy of the fitting; (5) Evaluate the fitting results: Save the fitted parameters for subsequent comparison and verification; For orbits that cannot be successfully fitted, record the failure and continue processing other orbits; (6) Match the orbit data: Calculate the root mean square error between the new trajectory data and the data generated by the fitted orbit parameters; find the least error fitting parameters, that is, the orbit model that the trajectory data best fits, and give the accuracy of judging the trajectory as the orbit model within a certain error range; Through the above algorithm, the fitted trajectories of the target absolute and relative orbit states are obtained, and the correlation recognition of the defined behaviors is achieved.

Images