Spatial non-cooperative target cooperative sensing method and system based on multi-satellite networking

Abstract

The invention discloses a spatial non-cooperative target cooperative sensing method and system based on multi-satellite networking, and the method comprises the steps: extracting a key region in an image through generating a spatial confidence map, inputting a region with a higher probability of occurrence of a target in the image to obtain a higher confidence score, and carrying out the recognition of the target. High-confidence key features are packaged into compact image information, and messages are transmitted through a sparsely connected communication graph; a multi-head attention mechanism is utilized to fuse target features sent by each spacecraft, an enhanced target feature map is generated, relative poses of a target and a camera are preliminarily estimated through a 2D-2D method, a target coordinate system is established, a joint state variable is constructed, online estimation is completed through a distributed nonlinear filter, and distributed information filtering is adopted to obtain a target feature map. And the state estimation values of the satellite nodes quickly tend to be consistent. According to the method, the perception performance maximization under the limited bandwidth can be realized, and the consistency rapid estimation can be carried out on the geometrical characteristics and the relative pose state of the target.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-satellite networking, and in particular to a method and system for collaborative perception of space non-cooperative targets based on multi-satellite networking. Background Art

[0002] With the increasing frequency of human space activities, the number of objects in orbit is also increasing, including not only active and failed satellites, but also a large number of space debris. According to statistics from the European Space Agency (ESA), as of March 2023, there are about 9,800 satellites in orbit, of which only about 7,200 are in normal service, and the remaining nearly 30% of failed satellites are moving at high speeds in orbit at a speed of 20 to 30,000 km / h, which will pose a potential threat to satellites in normal operation. In fact, more than 640 sudden or man-made spacecraft collisions have occurred, and tens of thousands of space debris with a total weight of about 10,700 tons have been generated, of which 32,300 are larger debris tracked daily by the Space Surveillance Network (SSN). In tasks such as on-orbit servicing and debris cleaning, spatial perception and relative measurement of non-cooperative targets such as failed satellites and space debris are important prerequisites for achieving spacecraft rendezvous and approach operations. Visual cameras are suitable for small spacecraft due to their light weight, small size, low cost and power consumption, and have become an important means of relative measurement of spacecraft.

[0003] Due to the dynamic changes in the relative attitude and orbit between the detection spacecraft and the target, the traditional single-spacecraft target perception method is easily affected by the target occlusion and cannot continuously observe the designated area. In addition, the space lighting conditions are complex and changeable, and low contrast, reflection, and interference from the earth's background will cause unclear target imaging. With the diversity, complexity, and refinement of space control tasks, the traditional single-spacecraft working mode will be difficult to meet the requirements. Multi-spacecraft collaborative observation, especially micro-satellite cluster collaborative perception technology, uses multi-satellite collaboration to complete multi-perspective, all-round observation of the target. How to achieve efficient transmission and sharing of multi-perspective data between different spacecraft, and complete the fusion and reconstruction of target visual features is a challenge faced by multi-spacecraft collaborative perception.

[0004] When multiple networked satellites conduct collaborative detection of space targets, if all satellite nodes exchange all observed space information equally, a large amount of communication bandwidth will be occupied. In fact, each node pays more attention to the foreground area of ​​the image containing the target, as well as the area missing due to occlusion or illumination. In space information-physical systems, fully tapping the heterogeneity of the spatial distribution of perception information, transmitting sparse and critical information, and achieving perception beyond line of sight and ultra-efficient communication are the keys to collaborative perception. Therefore, how to maximize the use of limited communication resources, condense concise and important perception information, select individuals with interaction needs in the spacecraft cluster, and fully integrate the information shared by neighboring spacecraft to achieve an effective trade-off between multi-spacecraft collaborative perception performance and communication consumption is a key issue that needs to be solved urgently. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] The present invention proposes a collaborative perception method for non-cooperative targets in space based on multi-satellite networking, which can achieve an effective trade-off between the collaborative perception performance of multiple spacecraft and communication consumption, and at the same time realizes effective perception of non-cooperative targets under conditions of limited communication resources, which not only improves the efficiency of information transmission, but also enhances the ability of multiple spacecraft to work collaboratively.

[0007] Another object of the present invention is to propose a space non-cooperative target collaborative perception system based on multi-satellite networking.

[0008] To achieve the above-mentioned object, the present invention proposes a method for collaboratively sensing space non-cooperative targets based on multi-satellite networking, comprising:

[0009] Constructing a spatial confidence map based on target features of the input image, and extracting a key area where the target spacecraft is located in the input image according to the spatial confidence map;

[0010] According to the processing result of the spatial confidence map, extracting the area whose confidence score is higher than the preset confidence score threshold from the key area, and obtaining the feature map of the corresponding area;

[0011] Through multiple rounds of inter-satellite communications, each node satellite aggregates target information from other satellites to iteratively update feature maps, packages the iteratively updated feature maps to obtain fused image information, and constructs a sparsely connected communication graph based on the fused image information;

[0012] A multi-head attention mechanism is used to fuse the target features transmitted by each communication node through the communication graph and update the own perception information to obtain an enhanced target feature graph;

[0013] The feature points based on the enhanced target feature map preliminarily estimate the relative poses of the non-cooperative target and the camera that need to be perceived and modeled to establish the target coordinate system;

[0014] Constructing a joint state variable including relative motion information and target geometric features based on the target coordinate system, and jointly estimating the geometric features and motion state using the joint state variable, the prediction model, and the measurement model to generate an updated joint state variable and a covariance matrix;

[0015] The updated joint state variables and covariance matrix are used to estimate the consistency state of multiple spacecraft through distributed information filtering, and the newly detected feature points are added to the joint state variables to update the state variables in real time.

[0016] The space non-cooperative target collaborative perception method based on multi-satellite networking in an embodiment of the present invention may have the following additional technical features:

[0017] In one embodiment of the present invention, constructing a spatial confidence map based on target features of an input image includes:

[0018] Assume that the image input by spacecraft i in the kth communication cycle is I i , obtain the feature map of the image through image encoding Feature map P i (k) The corresponding confidence map passes through the object detection decoder generate:

[0019]

[0020] Among them, h, w, and d represent the height, width, and channels of the image matrix respectively.

[0021] In one embodiment of the present invention, the information for packaging includes a data packet packaged by a certain spacecraft i, including:

[0022] The feature information that the spacecraft needs to supplement after this round of observation is used to request valid information from other spacecraft, expressed as:

[0023]

[0024] And, the feature information required by other spacecraft and currently possessed by spacecraft i.

[0025] In one embodiment of the present invention, the confidence map generated by spacecraft i and the request map sent by spacecraft j determine the image area to be transmitted, and the area is selected by a binary matrix:

[0026]

[0027] The sparse feature map transmitted from spacecraft i to spacecraft j is expressed as It provides spatially sparse but perceptually critical information; the information packaged by spacecraft i in round k of communication is finally determined as

[0028] In one embodiment of the present invention, according to the binary matrix The maximum value of determines the necessity of communication between spacecraft i and spacecraft j. Assume that A (k) is the adjacency matrix of the communication topology in the kth communication cycle, then the (i, j)th element is:

[0029]

[0030] In one embodiment of the present invention, a multi-head attention mechanism is adopted to fuse the target features transmitted by each communication node through the communication graph and update its own perception information to obtain an enhanced target feature graph, including:

[0031] The scaled dot product attention weight is calculated based on the image feature information of spacecraft i and the image information sent by neighboring spacecraft j:

[0032]

[0033] Where MHA(·) represents the multi-head attention mechanism, which fuses multi-source image features at each image spatial position. is the spatial confidence map sent by spacecraft j, representing the key image area from the perspective of spacecraft j, P i (k) is the image feature map of spacecraft i, represents the sparse feature map transmitted from spacecraft j to spacecraft i;

[0034] After the kth round of communication, the enhanced feature map of spacecraft i is obtained by fusion of multi-source image information using a feedforward neural network:

[0035]

[0036] Among them, FFN(·) is a feedforward neural network, FFN(·) is the adjacency matrix A (k) The neighboring spacecraft of spacecraft i defined in , the image feature map P obtained by fusion i (k+1) The feature map of spacecraft i during the k+1th round of communication is used as the measurement input for 3D modeling of non-cooperative targets.

[0037] In one embodiment of the present invention, preliminarily estimating the relative pose of the non-cooperative target and the camera that needs to be sensed and modeled based on the feature points of the enhanced target feature map to establish a target coordinate system includes:

[0038] Preliminary estimate of the relative pose between the target and the camera by matching the target image feature points with the camera image feature points;

[0039] Receive target image features and their defined target coordinate systems transmitted by other spacecraft via intersatellite links;

[0040] A spacecraft in the cluster is selected as a reference star, and the solved relative pose is converted to the target coordinate system defined by the reference star.

[0041] To achieve the above-mentioned purpose, the present invention also proposes a space non-cooperative target collaborative perception system based on multi-satellite networking, comprising:

[0042] A key area extraction module, used to construct a spatial confidence map based on the target features of the input image, and extract the key area where the target spacecraft is located in the input image according to the spatial confidence map;

[0043] A feature map extraction module, used to extract an area whose confidence score is higher than a preset confidence score threshold from the key area according to the processing result of the spatial confidence map, and obtain a feature map of the corresponding area;

[0044] An information packaging module is used to aggregate target information from other satellites through multiple rounds of inter-satellite communications, so that each node satellite can iteratively update the feature map, package the iteratively updated feature map to obtain fused image information, and construct a sparsely connected communication graph based on the fused image information;

[0045] A feature map enhancement module is used to adopt a multi-head attention mechanism and fuse the target features transmitted by each communication node through the communication map and update its own perception information to obtain an enhanced target feature map;

[0046] A target coordinate system building module is used to preliminarily estimate the relative pose of the non-cooperative target and the camera that needs to be perceived and modeled based on the feature points of the enhanced target feature map to establish the target coordinate system;

[0047] A state joint estimation module, used to construct a joint state variable containing relative motion information and target geometric features based on the target coordinate system, and use the joint state variable, prediction model and measurement model to perform joint estimation of geometric features and motion state to generate an updated joint state variable and covariance matrix;

[0048] The consistency state estimation module is used to utilize the updated joint state variables and covariance matrix and perform multi-spacecraft consistency state estimation through distributed information filtering, and add the newly detected feature points to the joint state variables to update the state variables in real time.

[0049] The collaborative perception method and system for non-cooperative space targets based on multi-satellite networking in the embodiments of the present invention are closely connected through a space confidence map to form a dynamic adjustment process, which not only ensures the effective transmission of key information, but also promotes efficient collaboration between multiple spacecraft systems to jointly complete complex space target detection tasks.

[0050] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0052] Figure 1 is a flow chart of a method for collaboratively sensing a space non-cooperative target based on a multi-satellite network according to an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of a spatial non-cooperative target collaborative perception task according to an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of three-dimensional modeling of a non-cooperative target under a non-co-visual condition of multiple spacecraft according to an embodiment of the present invention;

[0055] Figure 4 It is a structural diagram of a space non-cooperative target collaborative perception system based on multi-satellite networking according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0057] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0058] The following describes a method and system for collaborative perception of non-cooperative space targets based on multi-satellite networking according to an embodiment of the present invention with reference to the accompanying drawings.

[0059] Figure 1 A method for collaboratively sensing non-cooperative space targets based on multi-satellite networking according to an embodiment of the present invention is provided. Figure 1As shown, the method includes:

[0060] S1, constructing a spatial confidence map based on target features of an input image, and extracting a key area where a target spacecraft is located in the input image according to the spatial confidence map;

[0061] S2, extracting an area whose confidence score is higher than a preset confidence score threshold from the key area according to the processing result of the spatial confidence map, and obtaining a feature map of the corresponding area;

[0062] S3, through multiple rounds of inter-satellite communication, each node satellite aggregates target information from other satellites to iteratively update the feature map, packages the iteratively updated feature map to obtain fused image information, and constructs a sparsely connected communication graph based on the fused image information;

[0063] S4, adopting a multi-head attention mechanism to fuse the target features transmitted by each communication node through the communication graph and update its own perception information to obtain an enhanced target feature graph;

[0064] S5, based on the feature points of the enhanced target feature map, the relative poses of the non-cooperative target and the camera that need to be perceived and modeled are preliminarily estimated to establish the target coordinate system;

[0065] S6, constructing a joint state variable including relative motion information and target geometric features based on the target coordinate system, and using the joint state variable, the prediction model and the measurement model to jointly estimate the geometric features and the motion state to generate an updated joint state variable and a covariance matrix;

[0066] S7, using the updated joint state variables and covariance matrix, and performing multi-spacecraft consistency state estimation through distributed information filtering, and adding the newly detected feature points to the joint state variables to update the state variables in real time.

[0067] The following are the terms used in the embodiments of the present invention:

[0068] Spaceborne cameras: Spaceborne cameras are devices installed on spacecraft to capture images in space. The main function of these cameras is to image target spacecraft or other space objects and obtain their appearance and motion information. Spaceborne cameras are the basic tools for achieving space object perception.

[0069] Target spacecraft: The target spacecraft refers to the object that the multi-spacecraft cooperative perception system attempts to detect, track and analyze. It can be a non-cooperative target, such as a failed satellite, space debris, etc. The target spacecraft is the core object of the entire perception process, and all collected data revolve around it.

[0070] Spacecraft: Spacecraft refers to all satellites involved in the perception mission. They are equipped with onboard cameras and other necessary sensors and can perform tasks autonomously or semi-autonomously in space. Spacecraft are not only responsible for capturing images of target spacecraft, but also for communicating with other spacecraft, data processing, and decision-making.

[0071] Node satellite: In a multi-spacecraft collaborative perception system, each participating spacecraft can be called a node satellite. They form a network, communicate through intersatellite links, and share their captured data to achieve comprehensive perception of the target spacecraft. Through multiple rounds of communication, node satellites continuously aggregate and update target information to improve the understanding of the target spacecraft and the positioning accuracy.

[0072] In summary, a spaceborne camera is a tool used on a spacecraft (or node satellite) to capture images of a target spacecraft. Multiple spacecraft (i.e., node satellites) form a network through intersatellite communications and work together to achieve efficient and accurate perception of the target spacecraft. In this process, each node satellite uses its own spaceborne camera to obtain image data and share this data with other node satellites to improve the overall perception effect through iterative updates.

[0073] It is understandable that in the space non-cooperative target detection mission, using multiple micro-satellites to form a spacecraft cluster to conduct collaborative observation of the target can significantly improve the global perception capability, while the communication resources of the space network system are severely limited, and it is necessary to effectively reduce the communication bandwidth while ensuring the perception performance. To this end, the present invention proposes a collaborative perception framework suitable for multi-spacecraft distributed detection.

[0074] In one embodiment of the present invention, a spatial confidence map is constructed to extract key areas in an image, thereby saving communication bandwidth and transmitting only key information such as the target spacecraft.

[0075] Specifically, for space target detection tasks, the area where the target spacecraft is located in the space image taken by the onboard camera is obviously more important than the background area. Removing the perception information of the background area can save precious communication bandwidth. Only key information such as the target spacecraft in the image is retained and transmitted to other spacecraft through intersatellite links. It can help restore the full-scale information of the target spacecraft that is missing due to limited viewing angle, target self-occlusion, space illumination influence, etc. The present invention constructs a spatial confidence map to characterize the importance of image information in different areas to the space perception task, and areas in the input image with a higher probability of the detected target appearing can obtain a higher confidence score. Through multiple rounds of intersatellite communication, each node satellite can aggregate target information from other satellites to iteratively update the feature map. Using the generated spatial confidence to achieve the separation and extraction of key areas in the image can provide a solid foundation for subsequent efficient communication and information fusion. The space non-cooperative collaborative perception task of the present invention is such as Figure 2 shown.

[0076] It is understandable that in order to optimize communication efficiency and reduce unnecessary data transmission in space target detection missions, it is necessary to first construct a spatial confidence map. This map can distinguish different areas in the image, especially indicating which areas contain high-value target information (such as target spacecraft) and which areas are mainly background information.

[0077] Specifically, by analyzing the space images taken by the onboard camera, the probability of the target spacecraft appearing in each area is calculated, and each area is assigned a confidence score. The higher the confidence score, the more important the area is for the space perception task.

[0078] It can be understood that based on the constructed spatial confidence map, the key area containing high-value information is separated and extracted from the original image. This step is intended to further reduce the amount of data that needs to be transmitted and ensure the efficient use of communication resources.

[0079] Specifically, a specific algorithm is used to process the spatial confidence map to identify and extract areas with high confidence scores as key information. Subsequently, the information of these key areas is efficiently transmitted to other spacecraft participating in the mission through intersatellite links. During this process, multiple rounds of communication will be carried out between the node satellites to share the key information they have captured and jointly build a more complete image of the target spacecraft.

[0080] It is understandable that information fusion is performed using key area information obtained from multiple spacecraft to make up for possible information loss under a single perspective, such as perspective limitation, target self-occlusion, space illumination influence, etc., and ultimately achieve full-scale information reconstruction of the target spacecraft.

[0081] Specifically, each node satellite integrates key area information from other satellites through multiple rounds of iterative updates, uses advanced algorithms to process this data, and gradually improves the three-dimensional model of the target spacecraft. This process can not only improve the accuracy of target recognition, but also enhance the ability to monitor the motion status of the target spacecraft. Ultimately, all participating spacecraft can obtain a more comprehensive and accurate view of the target spacecraft.

[0082] Preferably, the process of generating the spatial perception confidence map of an image can be implemented by a series of image encoding and decoding. Assuming that the image input in the kth communication cycle of spacecraft i is I i , then the feature map of the image is obtained by image encoding h, w and d represent the height, width and channel of the image matrix respectively, and the feature map P i (k) The corresponding confidence map can be passed through an object detection decoder generate:

[0083]

[0084] The spatial perception confidence map reflects the importance of different areas in the image, that is, the area containing the target spacecraft and the part with clear features have higher confidence scores, and the areas with high confidence scores are more worthy of being transmitted to other neighboring spacecraft during intersatellite communication.

[0085] In one embodiment of the present invention, key features with high confidence are packed into compact image information and transmitted through a sparsely connected communication graph, thereby reducing unnecessary communication and improving communication efficiency.

[0086] It can be understood that based on the spatial confidence map, high confidence key areas are extracted. These areas contain information that is critical to the space perception task. The extracted key features are packaged into a set of compact image information. This information usually includes key feature points in the image, feature descriptors, etc. A sparsely connected communication graph is constructed to determine when and with which spacecraft to communicate. By reducing unnecessary communications, precious bandwidth resources are avoided.

[0087] It can be understood that efficient message transmission is achieved through a sparsely connected communication graph, ensuring that each spacecraft can obtain the necessary key feature information in a timely manner. In the first round of communication, each spacecraft broadcasts its own information to all other spacecraft through a fully connected undirected communication graph to ensure the sharing of initial information. In subsequent communication rounds, each spacecraft determines whether further communication is needed based on its own spatial confidence graph and the request graph of neighboring spacecraft. Only when both parties have information exchange needs, a communication connection is established to achieve efficient data transmission. Through multiple rounds of communication, each node satellite aggregates key feature information from other satellites, iteratively updates its own feature graph, and gradually improves the full-scale information of the target spacecraft.

[0088] Specifically, in order to reduce the communication bandwidth without affecting the perception performance, the present invention establishes an efficient communication mechanism based on the generated spatial confidence map, packs the key features with high confidence into a set of compact image information, and transmits the messages through a sparsely connected communication graph.

[0089] Information packaging determines what image information should be included in the message to be sent. The data package of a spacecraft i consists of two parts: one is the feature information that the spacecraft needs to supplement after this round of observation, which is used to efficiently and specifically request valid information from other spacecraft, specifically expressed as The requested information is negatively correlated with the spatial confidence map, because areas with lower confidence scores are likely to be missing information due to self-occlusion of the target body (large accessories such as solar panels), solar reflection, and low light. The other part of the content is the feature information required by other spacecraft and currently possessed by spacecraft i. In order to reduce the amount of transmitted data, the information provided by spacecraft i is spatially sparse but perceptually critical feature information.

[0090] Preferably, the confidence map generated by spacecraft i and the request map sent by spacecraft j determine which image regions need to be transmitted, and these regions can be selected through a binary matrix:

[0091]

[0092] The sparse feature map transmitted from spacecraft i to spacecraft j is expressed as It provides spatially sparse but perceptually critical information. The information packaged by spacecraft i in round k of communication is finally determined as Since only the request graph R with negative correlation to confidence is transmitted i (k) and non-zero features The communication bandwidth required for collaborative sensing is greatly reduced.

[0093] After the information is packaged, a sparsely connected communication graph needs to be constructed to determine when and with which spacecraft to communicate, so as to avoid wasting precious bandwidth resources by reducing unnecessary communications. In the first round of communication, since each spacecraft does not have observation information of other spacecraft, it is necessary to first establish a fully connected undirected communication to broadcast its own information to all other spacecraft. Subsequent communication rounds will determine whether further communication is needed based on the request graph of each node satellite and its own spatial confidence graph. Communication will only be established when both parties have information exchange needs, aiming to minimize the amount of data transmitted between satellites, thereby achieving more efficient and accurate target collaborative perception.

[0094] Preferably, according to the binary matrix The maximum value of determines the necessity of communication between spacecraft i and spacecraft j. Assume that A (k) is the adjacency matrix of the communication topology in the kth communication cycle, then the (i, j)th element is:

[0095]

[0096] If the adjacency matrix If an element in is 1, it means that there is a need for communication between the two spacecraft corresponding to the element, and this communication link will be activated. The communication graph constructed in this way is sparse and efficient, which can save communication resources while ensuring effective information transmission.

[0097] In one embodiment of the present invention, in order to solve the problems that the geometric model of the detected target is unknown, the reference coordinate system changes dynamically, the dimension of the state variable to be estimated is large and will change with the addition of new target features, this section proposes a non-cooperative target 3D modeling method under non-common view conditions based on the inter-satellite sharing of space target perception information achieved in the previous research content. The technical route is as follows Figure 3 shown.

[0098] Specifically, a multi-head attention mechanism is used to fuse the target features sent by each spacecraft to generate an enhanced target feature map.

[0099] Specifically, after receiving the perception information from other spacecraft, the present invention intends to use a multi-head attention mechanism to fuse the target features sent by each communication node and update its own perception information to obtain an enhanced target feature map. The spatial confidence map in the data packet reflects the perception level of each spacecraft at different perspectives of the target, providing prior information for attention learning. The multi-head attention mechanism can calculate the scaled dot product attention weights from different channel inputs separately and in parallel. The input information includes the image features perceived by spacecraft i itself, the sparse features sent by other spacecraft, and the corresponding confidence scores. The obtained attention weights and sparse features of each channel are input into a feedforward neural network for information fusion, and the enhanced target image feature map is output.

[0100] The scaled dot product attention weight can be calculated based on the image feature information of spacecraft i and the image information sent by neighboring spacecraft j:

[0101]

[0102] Where MHA(·) represents the multi-head attention mechanism, which can fuse multi-source image features at each image spatial position. The spatial confidence map sent by spacecraft j represents the key image area from the perspective of spacecraft j and contributes to the calculation of the attention weight of spacecraft i. (P i (k) is the image feature map of spacecraft i, represents the sparse feature map transmitted from spacecraft j to spacecraft i)

[0103] After the kth round of communication, a feedforward neural network can be used to obtain the enhanced feature map of spacecraft i through multi-source image information fusion:

[0104]

[0105] Among them, FFN(·) is a feedforward neural network, FFN(·) is the adjacency matrix A (k) The neighboring spacecraft of spacecraft i defined in , the image feature map P obtained by fusion i (k+1) It can be used for the characteristic map of spacecraft i in the k+1th round of communication, and will be used as the measurement input for the 3D modeling of non-cooperative targets.

[0106] In one embodiment of the present invention, the relative position of the target and the camera is preliminarily estimated by a 2D-2D method, and a target coordinate system is established. A joint state variable is constructed, and online estimation is performed by a distributed nonlinear filter. Distributed information filtering is used to achieve rapid convergence of the state estimation values ​​of each satellite node.

[0107] Specifically, due to the lack of model information of non-cooperative targets in space, after completing target imaging and feature extraction, it is impossible to directly establish the mapping relationship between the target image and its physical body. Therefore, in the initial stage of visual perception, it is necessary to determine a target coordinate system as a spatial reference to define the geometric position of feature points. For multi-spacecraft collaborative perception systems, the target coordinate system initialization process mainly consists of three parts: each spacecraft independently defines the target coordinate system, realizes information sharing through inter-satellite communication, and completes the unification of the target coordinate system at each spacecraft node. First, the relative pose between the target and the camera is roughly estimated by 2D-2D methods such as the 5-point method, the 8-point method, and the homography matrix. A feature point is assumed to be the target center of mass and the coordinate system is established with this as the origin; the target image features transmitted by other spacecraft and the target coordinate system information defined by them are received through the inter-satellite link; a spacecraft in the cluster is selected as a reference star, the solved relative pose is converted to the target coordinate system defined by the reference star, and the unified representation of the feature points in this coordinate system is completed.

[0108] Specifically, a set of 10+3n-dimensional joint state variables x1 containing relative motion information and target geometric features is constructed in the target coordinate system: relative position, relative velocity, relative attitude (described by quaternion), and three-dimensional coordinates of feature points. In order to maintain the unity of the state variables to be estimated for each satellite node, it is necessary to convert the relative motion state into the relative position, relative velocity and relative attitude between the target coordinate system and the reference satellite coordinate system through the absolute position information obtained by GNSS and the reference satellite position information transmitted between satellites. The prediction model of the joint estimation algorithm consists of relative orbital kinematics and dynamics, and attitude kinematics. The measurement model is represented by a pinhole camera model, which describes the mapping of 3D coordinates of feature points to 2D pixel coordinates. A distributed nonlinear filter is used to complete the online estimation of the joint state variable x1. Considering that the angular velocity estimation needs to be based on attitude dynamics, and the moment of inertia information of non-cooperative targets cannot be obtained in advance, the uncertainty of the moment of inertia will greatly affect the robustness of the filter. Therefore, this scheme proposes to estimate the 6-dimensional state variable x2 composed of angular velocity and center of mass position separately. A new measurement model is constructed based on optical flow and star-sensing information, and a nonlinear filter is used to estimate the target angular velocity and center of mass position. Due to the low dimension of the state vector, the center of mass / angular velocity estimator has a higher solution rate than other state variable x1 estimators. When the dynamic changes between the spacecraft and the target are large, the higher rate of angular velocity output can improve the calculation efficiency and stability of the optical flow.

[0109] Specifically, the centralized filtering structure needs to define a main spacecraft to collect all the measurement information sent from the satellite for centralized fusion processing. Although the structure is simple in design and has guaranteed accuracy, it has a large amount of calculation and poor robustness, and is not suitable for resource-constrained micro-satellite clusters. The present invention intends to use distributed information filtering to jointly solve relative pose and geometric features. Graph theory is used to model the network topology. Starting from the established sparsely connected network communication graph, if there is intersatellite communication between spacecraft i and spacecraft j, an edge is formed, and spacecraft i and spacecraft j are network node sets. Each node first completes the time update of the information matrix and information vector according to its own state information and covariance information at the last moment, and performs consistency state estimation after receiving the data of the neighboring nodes. The consistency protocol can make the state estimation values ​​of each satellite node in the network quickly converge to consistency. In the process of target perception, new feature points may be added at any time. The newly detected target feature points are initialized and added to the joint state variable x1. At this time, x1 will be expanded to 10+3(n+1) dimensions, and the information matrix also needs to include the new covariance matrix accordingly.

[0110] According to the collaborative perception method for non-cooperative targets in space based on multi-satellite networking in an embodiment of the present invention, in order to solve the problems of limited communication resources of micro-satellites and large data transmission requirements, an integrated coordination mechanism of inter-satellite communication and visual perception is proposed. Based on the spatial perception confidence of the target image, a sparsely connected inter-satellite communication topology is constructed to maximize the perception performance under limited bandwidth. In order to solve the problems of weak correlation between image features from different perspectives and dynamic time-varying communication topology, the attention mechanism is used to intelligently fuse multi-perspective images, and a distributed filtering method suitable for sparse dynamic spatial networks is designed to quickly estimate the consistency of target geometric features and relative posture states.

[0111] To achieve the above objectives, the present invention also proposes a space non-cooperative target collaborative perception system 10 based on multi-satellite networking, such as Figure 4 As shown, including:

[0112] A key area extraction module 100 is used to construct a spatial confidence map based on the target features of the input image, and extract the key area where the target spacecraft is located in the input image according to the spatial confidence map;

[0113] A feature map extraction module 200 is used to extract an area having a confidence score higher than a preset confidence score threshold from the key area according to the processing result of the spatial confidence map, and obtain a feature map of the corresponding area;

[0114] The information packaging module 300 is used to aggregate target information from other satellites through multiple rounds of inter-satellite communications, so that each node satellite can iteratively update the feature map, package the iteratively updated feature map to obtain fused image information, and construct a sparsely connected communication graph based on the fused image information;

[0115] The feature map enhancement module 400 is used to adopt a multi-head attention mechanism and fuse the target features transmitted by each communication node through the communication map and update its own perception information to obtain an enhanced target feature map;

[0116] A target coordinate system building module 500 is used to preliminarily estimate the relative position of the non-cooperative target and the camera that needs to be sensed and modeled based on the feature points of the enhanced target feature map to establish a target coordinate system;

[0117] A state joint estimation module 600 is used to construct a joint state variable including relative motion information and target geometric features based on the target coordinate system, and to perform joint estimation of geometric features and motion states using the joint state variable, prediction model, and measurement model to generate an updated joint state variable and covariance matrix;

[0118] The consistency state estimation module 700 is used to use the updated joint state variables and covariance matrix to perform multi-spacecraft consistency state estimation through distributed information filtering, and add the newly detected feature points to the joint state variables to update the state variables in real time.

[0119] According to the space non-cooperative target collaborative perception system based on multi-satellite networking of the embodiment of the present invention, in order to solve the problems of limited communication resources of micro-satellite and large data transmission demand, an inter-satellite communication-visual perception integrated coordination mechanism is proposed, and a sparsely connected inter-satellite communication topology is constructed based on the spatial perception confidence of the target image to maximize the perception performance under limited bandwidth; in order to solve the problems of weak correlation between image features from different perspectives and dynamic time-varying communication topology, the attention mechanism is used to intelligently fuse multi-perspective images, and a distributed filter suitable for sparse dynamic spatial networks is designed to quickly estimate the consistency of target geometric features and relative posture states.

[0120] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0121] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A method for collaborative perception of non-cooperative space targets based on multi-satellite networking, characterized in that: include: Constructing a spatial confidence map based on target features of the input image, and extracting a key area where the target spacecraft is located in the input image according to the spatial confidence map; According to the processing result of the spatial confidence map, extracting the area whose confidence score is higher than the preset confidence score threshold from the key area, and obtaining the feature map of the corresponding area; Through multiple rounds of inter-satellite communications, each node satellite aggregates target information from other satellites to iteratively update feature maps, packages the iteratively updated feature maps to obtain fused image information, and constructs a sparsely connected communication graph based on the fused image information; A multi-head attention mechanism is used to fuse the target features transmitted by each communication node through the communication graph and update the own perception information to obtain an enhanced target feature graph; The feature points based on the enhanced target feature map preliminarily estimate the relative poses of the non-cooperative target and the camera that need to be perceived and modeled to establish the target coordinate system; Constructing a joint state variable including relative motion information and target geometric features based on the target coordinate system, and jointly estimating the geometric features and motion state using the joint state variable, the prediction model, and the measurement model to generate an updated joint state variable and a covariance matrix; The updated joint state variables and covariance matrix are used to estimate the consistency state of multiple spacecraft through distributed information filtering, and the newly detected feature points are added to the joint state variables to update the state variables in real time.

2. The method according to claim 1, characterized in that The step of constructing a spatial confidence map based on the target features of the input image includes: Assume that the image input by spacecraft i in the kth communication cycle is I i , obtain the feature map of the image through image encoding Feature map P i (k) The corresponding confidence map passes through the object detection decoder generate: Among them, h, w, and d represent the height, width, and channels of the image matrix respectively.

3. The method according to claim 2, characterized in that The information used for packaging includes a data packet packaged by a spacecraft i, including: The feature information that the spacecraft needs to supplement after this round of observation is used to request valid information from other spacecraft, expressed as: And, the feature information required by other spacecraft and currently possessed by spacecraft i.

4. The method according to claim 3, characterized in that The confidence map generated by spacecraft i and the request map sent by spacecraft j determine the image area that needs to be transmitted. The area is selected through a binary matrix: The sparse feature map transmitted from spacecraft i to spacecraft j is expressed as It provides spatially sparse but perceptually critical information; the information packaged by spacecraft i in round k of communication is finally determined as 5. The method according to claim 4, characterized in that According to the binary matrix The maximum value of determines the necessity of communication between spacecraft i and spacecraft j. Assume that A (k) is the adjacency matrix of the communication topology in the kth communication cycle, then the (i, j)th element is:

6. The method according to claim 5, characterized in that The multi-head attention mechanism is adopted to fuse the target features transmitted by each communication node through the communication graph and update its own perception information to obtain an enhanced target feature graph, including: The scaled dot product attention weight is calculated based on the image feature information of spacecraft i and the image information sent by neighboring spacecraft j: Where MHA(·) represents the multi-head attention mechanism, which fuses multi-source image features at each image spatial position. is the spatial confidence map sent by spacecraft j, representing the key image area from the perspective of spacecraft j, P i (k) is the image feature map of spacecraft i, represents the sparse feature map transmitted from spacecraft j to spacecraft i; After the kth round of communication, the enhanced feature map of spacecraft i is obtained by fusion of multi-source image information using a feedforward neural network: Among them, FFN(·) is a feedforward neural network, FFN(·) is the adjacency matrix A (k) The neighboring spacecraft of spacecraft i defined in , the image feature map P obtained by fusion i (k+1) The feature map of spacecraft i during the k+1th round of communication is used as the measurement input for 3D modeling of non-cooperative targets.

7. The method according to claim 6, characterized in that The feature points based on the enhanced target feature map preliminarily estimate the relative poses of the non-cooperative target and the camera that need to be perceived and modeled to establish the target coordinate system, including: Preliminary estimate of the relative pose between the target and the camera by matching the target image feature points with the camera image feature points; Receive target image features and their defined target coordinate systems transmitted by other spacecraft via intersatellite links; A spacecraft in the cluster is selected as a reference star, and the solved relative pose is converted to the target coordinate system defined by the reference star.

8. A space non-cooperative target collaborative perception system based on multi-satellite networking, characterized in that: include: A key area extraction module, used to construct a spatial confidence map based on the target features of the input image, and extract the key area where the target spacecraft is located in the input image according to the spatial confidence map; A feature map extraction module, used to extract an area whose confidence score is higher than a preset confidence score threshold from the key area according to the processing result of the spatial confidence map, and obtain a feature map of the corresponding area; An information packaging module is used to aggregate target information from other satellites through multiple rounds of inter-satellite communications, so that each node satellite can iteratively update the feature map, package the iteratively updated feature map to obtain fused image information, and construct a sparsely connected communication graph based on the fused image information; A feature map enhancement module is used to adopt a multi-head attention mechanism and fuse the target features transmitted by each communication node through the communication map and update its own perception information to obtain an enhanced target feature map; A target coordinate system building module is used to preliminarily estimate the relative pose of the non-cooperative target and the camera that needs to be perceived and modeled based on the feature points of the enhanced target feature map to establish the target coordinate system; A state joint estimation module, used to construct a joint state variable containing relative motion information and target geometric features based on the target coordinate system, and use the joint state variable, prediction model and measurement model to perform joint estimation of geometric features and motion state to generate an updated joint state variable and covariance matrix; The consistency state estimation module is used to utilize the updated joint state variables and covariance matrix and perform multi-spacecraft consistency state estimation through distributed information filtering, and add the newly detected feature points to the joint state variables to update the state variables in real time.

Images