A Cooperative Sensing Method and System for Space Non-Cooperative Targets Based on Multi-Satellite Networking

Abstract

The present invention discloses a method and system for collaborative perception of non-cooperative space targets based on multi-satellite networking. The method of the present invention includes generating a spatial confidence map, extracting key regions in the image, and regions with a higher probability of target appearance in the input image will obtain higher confidence scores. Packing high-confidence key features into compact image information and transmitting messages through a sparsely connected communication graph; using a multi-head attention mechanism to fuse target features sent by each spacecraft to generate an enhanced target feature map, initially estimating the relative pose of the target and the camera through a 2D-2D method, establishing a target coordinate system, constructing a joint state variable, and completing online estimation through a distributed non-linear filter. By adopting distributed information filtering, the state estimation values of each satellite node can quickly converge. The present invention can maximize the perception performance under limited bandwidth and can also quickly and consistently estimate the geometric features and relative pose state of the target.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-satellite networking, and particularly to a method and system for collaborative perception of space non-cooperative targets based on multi-satellite networking. Background Technique

[0002] With the increasingly frequent space activities of humans, the number of on-orbit objects is also increasing, including not only in-service and failed satellites, but also a large number of space debris. According to the statistics of the European Space Agency (ESA), as of March 2023, there are approximately 9,800 satellites in orbit, of which only about 7,200 are in normal service. The remaining nearly 30% of the failed satellites are moving at high speeds in orbit at 20 - 30,000 km / h, which will pose a potential threat to the normally operating satellites. In fact, there have been more than 640 sudden or man-made spacecraft space collision events, generating tens of thousands of space debris with a total weight of about 10,700 tons. Among them, 32,300 larger debris are routinely tracked by the Space Surveillance Network (SSN). In tasks such as on-orbit servicing and debris cleaning, the space perception and relative measurement of non-cooperative targets such as failed satellites and space debris are important prerequisites for spacecraft rendezvous and proximity operations. Visual cameras are suitable for small spacecraft due to their advantages such as light weight, small size, low cost, and low power consumption, and have become an important means for spacecraft relative measurement.

[0003] Due to the dynamic changes in the relative attitude and orbit states between the detecting spacecraft and the target, the traditional single-spacecraft target perception method is easily affected by target occlusion, cannot continuously observe a specified area, and the complex and variable space illumination conditions, such as low contrast, reflection, and earth background interference, will result in unclear target imaging. With the diverse, complex, and refined requirements of space manipulation tasks, the traditional single-spacecraft working mode will be difficult to meet the requirements. Multi-spacecraft collaborative observation, especially the collaborative perception technology of small satellite clusters, uses multi-satellite collaboration to complete multi-view and all-round observation of the target. How to efficiently transmit and share multi-view data between different spacecraft and complete the fusion and reconstruction of target visual features is a challenge faced by multi-spacecraft collaborative perception.

[0004] When multiple networked satellites cooperate to detect space targets, if each satellite node equally exchanges all observed space information, it will consume a large amount of communication bandwidth. In fact, each node is more concerned about the foreground area of the image that contains the target, as well as the areas that are missing due to occlusion or lighting effects. In a cyber-physical system of space information, fully exploiting the heterogeneity of the spatial distribution of sensing information, transmitting sparse and critical information, and achieving over-the-horizon sensing and ultra-high communication efficiency are the keys to cooperative sensing. Therefore, how to make the most of limited communication resources, distill concise and important sensing information, select individuals with interaction needs in a spacecraft cluster, and fully integrate the information shared by neighboring spacecraft to effectively balance the cooperative sensing performance and communication consumption of multiple spacecraft is a key problem that urgently needs to be solved. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems in the related art to some extent.

[0006] The present invention proposes a method for cooperative sensing of space non-cooperative targets based on multi-satellite networking, which can effectively balance the cooperative sensing performance and communication consumption of multiple spacecraft. At the same time, it realizes effective sensing of non-cooperative targets under limited communication resources, not only improving the efficiency of information transmission, but also enhancing the cooperative operation ability of multiple spacecraft.

[0007] Another object of the present invention is to propose a cooperative sensing system for space non-cooperative targets based on multi-satellite networking.

[0008] To achieve the above object, on the one hand, the present invention proposes a method for cooperative sensing of space non-cooperative targets based on multi-satellite networking, including:

[0009] Construct a spatial confidence map based on the target features of the input image, and extract the key area where the target spacecraft is located in the input image according to the spatial confidence map;

[0010] According to the processing result of the spatial confidence map, extract the area where the confidence score is higher than the preset confidence score threshold from the key area, and obtain the feature map of the corresponding area;

[0011] Through multiple rounds of inter-satellite communication, each node satellite aggregates the target information from other satellites to iteratively update the feature map, packs the iteratively updated feature map to obtain fused image information, and constructs a sparsely connected communication graph according to the fused image information;

[0012] Adopt a multi-head attention mechanism and fuse the target features transmitted from each communication node through the communication graph to update its own sensing information, and obtain an enhanced target feature map;

[0013] The preliminary estimation of feature points based on the enhanced target feature map requires the relative pose of the uncooperative target and the camera to be perceived and modeled to establish the target coordinate system;

[0014] Based on the target coordinate system, a joint state variable including relative motion information and target geometric features is constructed, and the joint state variable, prediction model, and measurement model are used for joint estimation of geometric features and motion states to generate an updated joint state variable and covariance matrix;

[0015] Using the updated joint state variable and covariance matrix, multi-spacecraft consensus state estimation is performed through distributed information filtering, and newly detected feature points are added to the joint state variable to update the state variable in real time.

[0016] The cooperative perception method for space non-cooperative targets based on multi-satellite networking in the embodiments of the present invention may have the following additional technical features:

[0017] In an embodiment of the present invention, constructing a spatial confidence map based on the target features of the input image includes:

[0018] Assume that the image input in the k-th communication cycle of spacecraft i is I i , and the feature map of the image is obtained through image encoding Feature map P i (k) The corresponding confidence map is generated by the target detection decoder Generate:

[0019]

[0020] where h, w, and d represent the height, width, and channels of the image matrix, respectively.

[0021] In an embodiment of the present invention, the information for packaging, including the data packet encapsulated by a certain spacecraft i, includes:

[0022] The feature information that the spacecraft needs to supplement after this round of observation, used to request valid information from other spacecraft, is expressed as:

[0023]

[0024] And the feature information that other spacecraft need and spacecraft i currently has.

[0025] In an embodiment of the present invention, the confidence map generated by spacecraft i and the request map sent by spacecraft j determine the image area to be transmitted, and the area is selected through a binary matrix:

[0026]

[0027] The sparse feature map transmitted from spacecraft i to spacecraft j is denoted as It provides spatially sparse but perceptually crucial information; the information packed by spacecraft i in the k - th round of communication is finally determined as

[0028] In an embodiment of the present invention, the necessity of communication between spacecraft i and spacecraft j is determined according to the maximum value of the binary matrix Suppose A (k) is the adjacency matrix of the communication topology in the k - th round of communication cycle, then the (i, j) - th element is:

[0029]

[0030] In an embodiment of the present invention, the multi - head attention mechanism is adopted, and the target features transmitted from each communication node are fused through the communication graph to update its own perceptual information, obtaining an enhanced target feature map, including:

[0031] Calculate the scaled dot - product attention weights according to the image feature information of spacecraft i itself and the image information sent by neighboring spacecraft j:

[0032]

[0033] where MHA(·) represents the multi - head attention mechanism, which fuses multi - source image features at each image spatial position, is the spatial confidence map sent by spacecraft j, representing the key image regions from the perspective of spacecraft j, P i (k) is the image feature map of spacecraft i, represents the sparse feature map transmitted from spacecraft j to spacecraft i;

[0034] After the k - th round of communication, an enhanced feature map of spacecraft i is obtained by fusing multi - source image information using a feed - forward neural network:

[0035]

[0036] where FFN(·) is the feed - forward neural network, and FFN(·) is the neighboring spacecraft of spacecraft i defined in the adjacency matrix A (k) The fused image feature map P i (k+1) is used as the feature map of spacecraft i for the (k + 1) - th round of communication and as the measurement input for non - cooperative target three - dimensional modeling.

[0037] In an embodiment of the present invention, based on the feature points of the enhanced target feature map, the relative pose of the non - cooperative target and the camera to be perceived and modeled is initially estimated to establish a target coordinate system, including:

[0038] Estimate the relative pose between the target and the camera initially by matching the feature points of the target image and the feature points of the camera image;

[0039] Receive the target image features and the target coordinate system defined by them transmitted by other spacecrafts through the inter-satellite link;

[0040] Select a certain spacecraft in the cluster as the reference star, and convert the solved relative pose to the target coordinate system defined by the reference star.

[0041] To achieve the above object, the present invention also proposes a space non-cooperative target collaborative perception system based on multi-satellite networking, including:

[0042] A key area extraction module, configured to construct a spatial confidence map based on the target features of the input image, and extract the key area where the target spacecraft is located in the input image according to the spatial confidence map;

[0043] A feature map extraction module, configured to extract the area where the confidence score is higher than the preset confidence score threshold from the key area according to the processing result of the spatial confidence map, and obtain the feature map of the corresponding area;

[0044] An information packaging module, configured to, through multi-round inter-satellite communication, each node satellite aggregates the target information from other satellites to iteratively update the feature map, package the iteratively updated feature map to obtain fused image information, and construct a sparsely connected communication graph according to the fused image information;

[0045] A feature map enhancement module, configured to adopt a multi-head attention mechanism and fuse the target features transmitted by each communication node through the communication graph and update its own perception information to obtain an enhanced target feature map;

[0046] A target coordinate system construction module, configured to initially estimate the relative pose between the non-cooperative target to be perceived and modeled and the camera based on the feature points of the enhanced target feature map to establish a target coordinate system;

[0047] A state joint estimation module, configured to construct a joint state variable including relative motion information and target geometric features based on the target coordinate system, and perform joint estimation of geometric features and motion states by using the joint state variable, a prediction model, and a measurement model to generate an updated joint state variable and covariance matrix;

[0048] A consistency state estimation module, configured to utilize the updated joint state variable and covariance matrix, and perform multi-spacecraft consistency state estimation through distributed information filtering, and add the newly detected feature points to the joint state variable to update the state variable in real time.

[0049] The collaborative sensing method and system for space non-cooperative targets based on multi-satellite networking according to the embodiments of the present invention are closely connected through a spatial confidence map, forming a dynamically adjustable process, which not only ensures the effective transmission of key information, but also promotes the efficient cooperation between multi-spacecraft systems to jointly complete complex space target detection tasks.

[0050] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the drawings, wherein:

[0052] Figure 1 is a flowchart of the collaborative sensing method for space non-cooperative targets based on multi-satellite networking according to the embodiments of the present invention;

[0053] Figure 2 is a schematic diagram of the collaborative sensing task for space non-cooperative targets according to the embodiments of the present invention;

[0054] Figure 3 is a schematic diagram of 3D modeling of non-cooperative targets under non-coplanar viewing conditions of multi-spacecraft according to the embodiments of the present invention;

[0055] Figure 4 is a structural diagram of the collaborative sensing system for space non-cooperative targets based on multi-satellite networking according to the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0057] In order to enable those skilled in the art of the present technology to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0058] The collaborative sensing method and system for space non-cooperative targets based on multi-satellite networking according to the embodiments of the present invention will be described below with reference to the drawings.

[0059] Figure 1 is the collaborative sensing method for space non-cooperative targets based on multi-satellite networking according to the embodiments of the present invention, as Figure 1As shown, the method includes:

[0060] S1. Construct a spatial confidence map based on the target features of the input image, and extract the key region where the target spacecraft is located in the input image according to the spatial confidence map;

[0061] S2. According to the processing result of the spatial confidence map, extract the regions with confidence scores higher than the preset confidence score threshold from the key region, and obtain the feature maps of the corresponding regions;

[0062] S3. Through multiple rounds of inter-satellite communication, each node satellite aggregates the target information from other satellites to iteratively update the feature map, packs the iteratively updated feature map to obtain fused image information, and constructs a sparsely connected communication graph according to the fused image information;

[0063] S4. Adopt a multi-head attention mechanism and fuse the target features transmitted from each communication node through the communication graph to update its own perception information, and obtain an enhanced target feature map;

[0064] S5. Based on the feature points of the enhanced target feature map, preliminarily estimate the relative pose of the non-cooperative target and the camera to be perceived and modeled to establish a target coordinate system;

[0065] S6. Based on the target coordinate system, construct a joint state variable including relative motion information and target geometric features, and use the joint state variable, prediction model, and measurement model to jointly estimate the geometric features and motion states to generate an updated joint state variable and covariance matrix;

[0066] S7. Utilize the updated joint state variable and covariance matrix, and perform multi-spacecraft consensus state estimation through distributed information filtering, and add the newly detected feature points to the joint state variable to update the state variable in real time.

[0067] The following are the noun terms in the embodiments of the present invention:

[0068] Spaceborne camera: A spaceborne camera is a device installed on a spacecraft for capturing images in space. The main function of these cameras is to image the target spacecraft or other space targets to obtain their appearance and motion information. Spaceborne cameras are the basic tools for realizing space target perception.

[0069] Target spacecraft: The target spacecraft refers to the object that the multi-spacecraft collaborative perception system attempts to detect, track, and analyze. It can be a non-cooperative target, such as a failed satellite, space debris, etc. The target spacecraft is the core object of the entire perception process, and all the collected data revolves around it.

[0070] Spacecraft: Spacecraft refers to all satellites participating in the sensing mission. They are equipped with on-board cameras and other necessary sensors and can perform tasks autonomously or semi-autonomously in space. Spacecraft are not only responsible for capturing images of the target spacecraft but also undertake tasks such as communicating with other spacecraft, data processing, and decision-making.

[0071] Node satellite: In a multi-spacecraft collaborative sensing system, each participating spacecraft can be called a node satellite. They form a network, communicate through inter-satellite links, and share the data they capture respectively to achieve a comprehensive perception of the target spacecraft. Through multiple rounds of communication among node satellites, the target information is continuously aggregated and updated to improve the understanding and positioning accuracy of the target spacecraft.

[0072] In summary, the on-board camera is a tool on the spacecraft (or node satellite) used to capture images of the target spacecraft. Multiple spacecraft (i.e., node satellites) form a network through inter-satellite communication and cooperate with each other to achieve efficient and accurate perception of the target spacecraft. In this process, each node satellite will use its on-board camera to obtain image data and share this data with other node satellites, and improve the overall sensing effect through iterative updates.

[0073] It can be understood that in the space non-cooperative target detection mission, using a cluster of small satellites to form a spacecraft cluster for collaborative observation of the target can significantly improve the global sensing ability. However, the communication resources of the space network system are severely limited, and it is necessary to effectively reduce the communication bandwidth while ensuring the sensing performance. For this reason, the present invention proposes a collaborative sensing framework suitable for distributed detection of multiple spacecraft.

[0074] In an embodiment of the present invention, by constructing a spatial confidence map and extracting key regions in the image, the communication bandwidth is saved, and only key information such as the target spacecraft is transmitted.

[0075] Specifically, for the space target detection task, the area where the target spacecraft is located in the space image captured by the spaceborne camera is obviously more important than the background area. Removing the perceptual information of the background area can save valuable communication bandwidth. Only the key information such as the target spacecraft in the image is retained and transmitted to other spacecraft through the inter-satellite link, which can help restore the full-scale information of the target spacecraft that is missing due to reasons such as limited viewing angle, target self-occlusion, and space illumination effects. The present invention constructs a spatial confidence map to characterize the importance of image information in different regions for the space perception task. Regions in the input image where the detected target has a higher probability of appearance can obtain higher confidence scores. Through multiple rounds of inter-satellite communication, each node satellite can aggregate target information from other satellites to iteratively update the feature map. Using the generated spatial confidence to achieve the separation and extraction of key regions in the image can provide a solid foundation for subsequent efficient communication and information fusion. The spatial non-cooperative collaborative perception task of the present invention is as Figure 2 shown.

[0076] It can be understood that, in order to optimize the communication efficiency and reduce unnecessary data transmission in the space target detection task, it is first necessary to construct a spatial confidence map. This map can distinguish different regions in the image, especially indicating which regions contain high-value target information (such as the target spacecraft), and which regions are mainly background information.

[0077] Specifically, by analyzing the space image captured by the spaceborne camera, the probability of the target spacecraft appearing in each region is calculated, thereby assigning a confidence score to each region. The higher the confidence score, the greater the importance of the region for the space perception task.

[0078] It can be understood that based on the constructed spatial confidence map, the key regions containing high-value information are separated and extracted from the original image. This step aims to further reduce the amount of data to be transmitted and ensure the effective utilization of communication resources.

[0079] Specifically, a specific algorithm is used to process the spatial confidence map to identify and extract the regions with high confidence scores as key information. Subsequently, the information of these key regions is efficiently transmitted to other spacecraft participating in the task through the inter-satellite link. In this process, multiple rounds of communication will be carried out among the node satellites to share the key information captured by each of them and jointly construct a more complete image of the target spacecraft.

[0080] It can be understood that the key region information obtained from multiple spacecraft is used for information fusion to make up for the information missing that may exist under a single viewing angle, such as problems like viewing angle limitation, target self-occlusion, and space illumination effects, and finally achieve the reconstruction of the full-scale information of the target spacecraft.

[0081] Specifically, each node satellite updates through multiple rounds of iteration, integrates the key area information from other satellites, processes this data using advanced algorithms, and gradually improves the three-dimensional model of the target spacecraft. This process can not only improve the accuracy of target recognition but also enhance the monitoring ability of the motion state of the target spacecraft. Eventually, all participating spacecraft can obtain a more comprehensive and accurate view of the target spacecraft.

[0082] Preferably, the process of generating the spatial perception confidence map of the image can be realized by a series of image encoding and decoding. Assume that the image input in the k-th communication cycle of spacecraft i is I i , then the feature map of this image is obtained through image encoding h, w, and d represent the height, width, and channels of the image matrix respectively, and the feature map P i (k) The corresponding confidence map can be generated by a target detection decoder :

[0083]

[0084] The spatial perception confidence map reflects the importance of different regions in the image, that is, the regions containing the target spacecraft, and the parts with clear features have higher confidence scores. The regions with high confidence scores are more worthy of being transmitted to other neighboring spacecraft during the inter-satellite communication process.

[0085] In an embodiment of the present invention, the key features with high confidence are packed into compact image information and transmitted through a sparsely connected communication graph, reducing unnecessary communication and improving communication efficiency.

[0086] It can be understood that according to the spatial confidence map, the key regions with high confidence are extracted. These regions contain information crucial for the spatial perception task. The extracted key features are packed into a set of compact image information. This information usually includes key feature points, feature descriptors, etc. in the image. A sparsely connected communication graph is constructed to determine when and with which spacecraft to communicate. By reducing unnecessary communication, the waste of precious bandwidth resources is avoided.

[0087] It is understandable that through a sparsely connected communication graph, efficient message transmission is achieved to ensure that each spacecraft can timely obtain necessary key feature information. In the first round of communication, each spacecraft broadcasts its own information to all other spacecrafts through a fully connected undirected communication graph to ensure the sharing of initial information. In subsequent communication rounds, each spacecraft determines whether further communication is needed based on its own spatial confidence map and the request map of neighboring spacecrafts. Only when there is a need for information interaction between both sides, a communication connection is established to achieve efficient data transmission. Each node satellite aggregates the key feature information from other satellites through multiple rounds of communication, iteratively updates its own feature map, and gradually improves the full-scale information of the target spacecraft.

[0088] Specifically, in order to reduce the communication bandwidth without affecting the perception performance, the present invention establishes an efficient communication mechanism based on the generated spatial confidence map, packs the key features with high confidence into a set of compact image information, and transmits messages through a sparsely connected communication graph.

[0089] Information packing determines which image information should be included in the message to be sent. The data packet encapsulated by a certain spacecraft i consists of two parts: one is the feature information that the spacecraft needs to supplement after this round of observation, which is used to efficiently and specifically request valid information from other spacecrafts, and is specifically expressed as The request information is negatively correlated with the spatial confidence map because the areas with lower confidence scores are likely to be due to information loss caused by self-occlusion of the target body (such as large attachments like solar panels), solar reflection, and low light. The other part of the content is the feature information that other spacecrafts need and spacecraft i currently has. In order to reduce the amount of data transmitted, the information provided by spacecraft i is spatially sparse but perceptually critical feature information.

[0090] Preferably, the confidence map generated by spacecraft i and the request map sent by spacecraft j determine which image regions need to be transmitted, and these regions can be selected through a binary matrix:

[0091]

[0092] The sparse feature map transmitted from spacecraft i to spacecraft j is expressed as It provides spatially sparse but perceptually critical information. The information packed by spacecraft i in the kth round of communication is finally determined as Since only the request map R negatively correlated with the confidence is transmitted i (k) and non-zero features greatly reduces the communication bandwidth required for collaborative perception.

[0093] After the information is packaged, a sparsely connected communication graph needs to be constructed to determine when and with which spacecraft to communicate, avoiding wasting precious bandwidth resources by reducing unnecessary communication. In the first round of communication, since each spacecraft does not have the observation information of other spacecraft, a fully connected undirected communication needs to be established first to broadcast its own information to all other spacecraft. In subsequent communication rounds, it will be judged whether further communication is needed according to the request graph of each node satellite and its own spatial confidence graph. Communication will only be constructed when there is a need for information interaction between both sides, aiming to minimize the amount of data transmitted between satellites, so as to achieve more efficient and accurate target collaborative perception.

[0094] Preferably, according to the maximum value of the binary matrix to determine the necessity of communication between spacecraft i and spacecraft j. Assume that A (k) is the adjacency matrix of the communication topology in the kth communication cycle, then the (i, j)th element is:

[0095]

[0096] If an element in the adjacency matrix is 1, it means that there is a need for communication between the two spacecraft corresponding to this element, and this communication link will be activated. The communication graph constructed in this way is sparse and efficient, which can save communication resources while ensuring the transmission of effective information.

[0097] In an embodiment of the present invention, aiming at problems such as the unknown geometric model of the detected target, the dynamic change of the reference coordinate system, the large dimension of the state variables to be estimated and the change with the addition of new target features, etc., based on the inter-satellite sharing of space target perception information achieved in the previous research content, a three-dimensional modeling method for non-cooperative targets under non-line-of-sight conditions is proposed, and the technical route is as Figure 3 shown.

[0098] Specifically, a multi-head attention mechanism is used to fuse the target features sent by each spacecraft to generate an enhanced target feature map.

[0099] Specifically, after receiving the perception information from other spacecraft, the present invention intends to use a multi-head attention mechanism to fuse the target features sent by each communication node and update its own perception information to obtain an enhanced target feature map. The spatial confidence map in the data packet reflects the perception level of each spacecraft from different perspectives of the target, providing prior information for attention learning. The multi-head attention mechanism can calculate the scaled dot product attention weights of the inputs from different channels separately and in parallel. The input information includes the image features perceived by spacecraft i itself, the sparse features sent by other spacecraft, and the corresponding confidence scores. The obtained attention weights and sparse features of each channel are input into a feed-forward neural network for information fusion, and an enhanced target image feature map is output.

[0100] Based on the image feature information of spacecraft i itself and the image information sent by neighboring spacecraft j, the scaled dot-product attention weights can be calculated:

[0101]

[0102] where MHA(·) represents the multi-head attention mechanism, which can fuse multi-source image features at each image spatial position. is the spatial confidence map sent by spacecraft j, representing the key image regions from the perspective of spacecraft j, and contributing to the calculation of the attention weights of spacecraft i. (P i (k) is the image feature map of spacecraft i. represents the sparse feature map transmitted from spacecraft j to spacecraft i)

[0103] After the k-th round of communication, through multi-source image information fusion, an enhanced feature map of spacecraft i can be obtained using a feed-forward neural network:

[0104]

[0105] where FFN(·) is the feed-forward neural network, and FFN(·) is the neighboring spacecraft of spacecraft i defined in the adjacency matrix A (k) The fused image feature map P i (k+1) can be used as the feature map of spacecraft i during the (k + 1)-th round of communication and will be used as the measurement input for non-cooperative target three-dimensional modeling.

[0106] In an embodiment of the present invention, the relative pose between the target and the camera is initially estimated by a 2D-2D method to establish a target coordinate system. A joint state variable is constructed, and online estimation is completed through a distributed nonlinear filter. Distributed information filtering is adopted to enable the state estimation values of each satellite node to quickly converge.

[0107] Specifically, due to the lack of model information for space non-cooperative targets, after target imaging and feature extraction are completed, it is impossible to directly establish the mapping relationship between the target image and its physical entity. Therefore, at the initial stage of visual perception, it is necessary to determine a target coordinate system as the spatial reference to define the geometric positions of feature points. For a multi-spacecraft collaborative perception system, the initialization process of the target coordinate system mainly consists of three parts: each spacecraft independently defines the target coordinate system, information sharing is achieved through inter-satellite communication, and the unification of the target coordinate system at each spacecraft node is completed. First, the relative pose between the target and the camera is roughly estimated by 2D-2D methods such as the 5-point method, the 8-point method, and the homography matrix. Assume that a certain feature point is the centroid of the target and establish a coordinate system with this as the origin; receive the target image features transmitted by other spacecraft and the information of the target coordinate system defined by them through the inter-satellite link; select a certain spacecraft in the cluster as the reference satellite, convert the solved relative pose to the target coordinate system defined by the reference satellite, and complete the unified representation of the feature points in this coordinate system.

[0108] Specifically, a 10 + 3n-dimensional joint state variable x1 including relative motion information and target geometric features is constructed in the target coordinate system: relative position, relative velocity, relative attitude (described by quaternion), and the three-dimensional coordinates of the feature points. To keep the unified state variables to be estimated at each satellite node, it is necessary to convert the relative motion state into the relative position, relative velocity, and relative attitude between the target coordinate system and the reference satellite coordinate system through the absolute position information obtained by GNSS and the pose information of the reference satellite transmitted between satellites. The prediction model of the joint estimation algorithm consists of relative orbital kinematics and dynamics, and attitude kinematics. The measurement model is represented by the pinhole camera model, which describes the mapping from the 3D coordinates of the feature points to the 2D pixel coordinates. An online estimation of the joint state variable x1 is completed using a distributed non-linear filter. Considering that the angular velocity estimation needs to be based on attitude dynamics and the moment of inertia information of non-cooperative targets cannot be obtained in advance, the uncertainty of the moment of inertia will greatly affect the robustness of the filter. Therefore, this scheme intends to separately estimate the 6-dimensional state variable x2 composed of the angular velocity and the centroid position. A new measurement model is constructed based on the optical flow and star sensor information, and a non-linear filter is used to estimate the target angular velocity and centroid position. Since the dimension of the state vector is relatively low, the centroid / angular velocity estimator has a higher solution rate than other state variable x1 estimators. When the dynamic changes between the spacecraft and the target are large, the higher-rate angular velocity output can improve the calculation efficiency and stability of the optical flow.

[0109] Specifically, the centralized filtering structure needs to define a main spacecraft to collect all the measurement information sent by the slave satellites for centralized fusion processing. Although this structure is simple in design and has guaranteed accuracy, it has a large amount of calculation and poor robustness, and is not suitable for small satellite clusters with limited resources. The present invention intends to adopt distributed information filtering to jointly solve the relative pose and geometric features. Apply graph theory to model the network topology structure. Starting from the established sparse-connected network communication graph, if there is inter-satellite communication between spacecraft i and spacecraft j, an edge is formed, and spacecraft i and spacecraft j are the network node sets. Each node first completes the time update of the information matrix and information vector according to its own state information and covariance information at the previous moment, and performs consistent state estimation after receiving the data from neighboring nodes. Through the consistency protocol, the state estimation values of each satellite node in the network can quickly tend to be consistent. During the target perception process, new feature points may be added at any time. Initialize the newly detected target feature points and add them to the joint state variable x1. At this time, x1 will be augmented to 10 + 3(n + 1) dimensions, and the information matrix also needs to incorporate the new covariance matrix accordingly.

[0110] According to the method for collaborative perception of space non-cooperative targets based on multi-satellite networking according to an embodiment of the present invention, aiming at problems such as limited communication resources and large data transmission requirements of small satellites, an integrated coordination mechanism for inter-satellite communication-visual perception is proposed. Based on the spatial perception confidence of the target image, a sparse-connected inter-satellite communication topology is constructed to maximize the perception performance under limited bandwidth; aiming at problems such as weak correlation of image features from different perspectives and dynamic time-varying communication topology, an attention mechanism is used to intelligently fuse multi-perspective images, and a distributed filtering method suitable for sparse dynamic space networks is designed to quickly and consistently estimate the target geometric features and relative pose states.

[0111] To achieve the above object, the present invention also proposes a system 10 for collaborative perception of space non-cooperative targets based on multi-satellite networking, as Figure 4 shown, including:

[0112] A key area extraction module 100, configured to construct a spatial confidence map based on the target features of the input image, and extract the key area where the target spacecraft is located in the input image according to the spatial confidence map;

[0113] A feature map extraction module 200, configured to extract the area where the confidence score is higher than the preset confidence score threshold from the key area according to the processing result of the spatial confidence map, and obtain the feature map of the corresponding area;

[0114] An information packaging module 300 is configured to aggregate target information from other satellites through multiple rounds of inter-satellite communication at each node satellite to iteratively update a feature map, package the iteratively updated feature map to obtain fused image information, and construct a sparsely connected communication graph according to the fused image information;

[0115] A feature map enhancement module 400 is configured to use a multi-head attention mechanism and fuse and update its own perception information for the target features transmitted from each communication node through the communication graph to obtain an enhanced target feature map;

[0116] A target coordinate system construction module 500 is configured to preliminarily estimate the relative pose of a non-cooperative target and a camera to be perceived and modeled based on the feature points of the enhanced target feature map to establish a target coordinate system;

[0117] A state joint estimation module 600 is configured to construct a joint state variable including relative motion information and target geometric features based on the target coordinate system, and perform joint estimation of geometric features and motion states by using the joint state variable, a prediction model, and a measurement model to generate an updated joint state variable and covariance matrix;

[0118] A consensus state estimation module 700 is configured to use the updated joint state variable and covariance matrix, and perform multi-spacecraft consensus state estimation through distributed information filtering, and add newly detected feature points to the joint state variable to update the state variable in real time.

[0119] The space non-cooperative target collaborative perception system based on multi-satellite networking according to an embodiment of the present invention proposes an inter-satellite communication-visual perception integrated coordination mechanism for problems such as limited communication resources and large data transmission requirements of small satellites, constructs a sparsely connected inter-satellite communication topology based on the spatial perception confidence of target images, and maximizes the perception performance under limited bandwidth; for problems such as weak correlation of image features from different perspectives and dynamic time-varying communication topologies, uses an attention mechanism to intelligently fuse multi-perspective images, designs distributed filtering applicable to sparse dynamic spatial networks, and performs fast consensus estimation of target geometric features and relative pose states.

[0120] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0121] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

Claims

1. A collaborative perception method for non-cooperative space targets based on multi-satellite networking, characterized in that Including: Construct a spatial confidence map based on the target features of the input image, and extract the key region where the target spacecraft is located in the input image according to the spatial confidence map; According to the processing result of the spatial confidence map, extract the region with a confidence score higher than the preset confidence score threshold from the key region, and obtain the feature map of the corresponding region; Through multiple rounds of inter-satellite communication, each node satellite aggregates the target information from other satellites to iteratively update the feature map, packs the iteratively updated feature map to obtain fused image information, and constructs a sparsely connected communication map according to the fused image information; Adopt a multi-head attention mechanism and fuse the target features transmitted by each communication node through the communication map to update its own perception information, and obtain an enhanced target feature map; Based on the feature points of the enhanced target feature map, preliminarily estimate the relative pose of the non-cooperative target and the camera that need to be perceived and modeled to establish a target coordinate system; Construct a joint state variable including relative motion information and target geometric features based on the target coordinate system, and use the joint state variable, prediction model, and measurement model for joint estimation of geometric features and motion states to generate an updated joint state variable and covariance matrix; Utilize the updated joint state variable and covariance matrix, and perform multi-spacecraft consistent state estimation through distributed information filtering, and add the newly detected feature points to the joint state variable to update the state variable in real time; The construction of the spatial confidence map based on the target features of the input image includes: Assume a spacecraft i The k image input in the nth communication cycle is , and the feature map of the image is obtained through image encoding . The corresponding confidence map is generated by the object detection decoder : (1) Where h, w, and d respectively represent the height, width, and channels of the image matrix.

2. The method according to claim 1, characterized in that, Information for packaging, including a certain spacecraft i Encapsulated data packets, including: The feature information that the spacecraft needs to supplement after this round of observation, used to request effective information from other spacecraft, is expressed as: Expressed as: and other spacecraft require and the spacecraft i currently possessed characteristic information.

3. The method according to claim 2, characterized in that Spacecraft i Generated confidence map and spacecraft j The sent request map determines the image area to be transmitted, and the area is selected through a binary matrix: (2) Spacecraft i To the spacecraft j The transmitted sparse feature map is represented as , providing spatially sparse but perceptually critical information; In k spacecraft in round-robin communication i the packaged information is finally determined as .

4. The method according to claim 3, characterized in that, Determine the necessity of communication between spacecraft based on the maximum value of the binary matrix Suppose i is the adjacency matrix of the communication topology in the j th round of communication cycle. Then the element at the ([[]] th row and k th column) is: i , j ) (3)。 5. The method according to claim 4, characterized in that Adopt a multi-head attention mechanism and fuse the target features transmitted by each communication node through the communication map to update its own perception information, and obtain an enhanced target feature map, including: According to the spacecraft i 's own image feature information and the image information sent by neighboring spacecraft j calculate the scaled dot product attention weights: (4) Among them represents the multi-head attention mechanism, which fuses multi-source image features at each image spatial position is the spacecraft j sent the spatial confidence map, representing the key image area from the perspective of the spacecraft j viewpoint is the image feature map of spacecraft i represents the sparse feature map transmitted from spacecraft j to spacecraft i After the k round of communication, an enhanced feature map of the spacecraft is obtained through multi-source image information fusion using a feedforward neural network: i ​ (5) in, is a feed-forward neural network, is the adjacency matrix The neighboring spacecraft of spacecraft i defined in , the image feature map obtained by fusion For the k +1 communication time for the spacecraft i feature map for measurement input of non-cooperative target 3D modeling.

6. The method according to claim 5, wherein Based on the feature points of the enhanced target feature map, preliminarily estimate the relative pose of the non-cooperative target and the camera that need to be perceived and modeled to establish a target coordinate system, including: Preliminarily estimate the relative pose between the target and the camera by matching the target image feature points and the camera image feature points; Receive the target image features and the defined target coordinate system transmitted by other spacecraft through the inter-satellite link; Select a certain spacecraft in the cluster as the reference satellite, and convert the solved relative pose to the target coordinate system defined by the reference satellite.

7. A multi-star networking-based space non-cooperative target collaborative sensing system using the method according to claim 1, characterized in that, Including: A key region extraction module, used to construct a spatial confidence map based on the target features of the input image, and extract the key region where the target spacecraft is located in the input image according to the spatial confidence map; A feature map extraction module, used to extract the region with a confidence score higher than the preset confidence score threshold from the key region according to the processing result of the spatial confidence map, and obtain the feature map of the corresponding region; An information packaging module, which is used to aggregate the target information from other satellites by each node satellite through multiple rounds of inter-satellite communication to iteratively update the feature map, package the iteratively updated feature map to obtain fused image information, and construct a sparsely connected communication graph according to the fused image information; A feature map enhancement module, which is used to adopt a multi-head attention mechanism and fuse the target features transmitted from each communication node through the communication graph to update its own perception information, and obtain an enhanced target feature map; A target coordinate system construction module, which is used to preliminarily estimate the relative pose of the non-cooperative target and the camera to be perceived and modeled based on the feature points of the enhanced target feature map to establish a target coordinate system; A state joint estimation module, which is used to construct a joint state variable including relative motion information and target geometric features based on the target coordinate system, and perform joint estimation of geometric features and motion states by using the joint state variable, a prediction model, and a measurement model to generate an updated joint state variable and covariance matrix; A consensus state estimation module, which is used to utilize the updated joint state variable and covariance matrix, and perform multi-spacecraft consensus state estimation through distributed information filtering, and add newly detected feature points to the joint state variable to update the state variable in real time.

Images