Satellite behavior state diagram generation method, device, electronic device and storage medium

Abstract

The present application provides a method, device, electronic device, and storage medium for generating a satellite behavior state diagram. The method includes: acquiring an image sequence for a target satellite; performing component target detection on each frame of the image in the image sequence to obtain a target detection result, wherein the target detection result includes a detection frame; cropping each frame of the image according to the area indicated by each detection frame to obtain a component image, and extracting features from the component image using an image coding network to obtain a visual feature vector; constructing a spatiotemporal dependency relationship for each visual feature vector using a spatiotemporal modeling model to obtain a spatiotemporal dependency result; the spatiotemporal dependency result includes the spatial dependency relationship between each component in the same frame image and the temporal dependency relationship between the same component in different frame images; and generating a behavior state diagram for the target satellite based on the spatiotemporal dependency result. In this way, the analysis accuracy of the target satellite's behavior state can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of satellite status analysis, and in particular to a method, device, electronic device, and storage medium for generating a satellite behavior status diagram. Background Art

[0002] With the advancement of aerospace technology, not only has the number of satellites increased, but their applications have also become increasingly diverse. For example, they are used in communications, remote sensing, weather monitoring, scientific experiments, and navigation and positioning. Currently, satellites are required to operate within their designated orbits and remain operational. Satellite failures can affect their functionality. If a satellite deviates from its designated orbit, it poses a threat to other satellites. Therefore, analyzing satellite behavior is crucial. Summary of the Invention

[0003] In view of this, the present application provides a method, device, equipment and storage medium for generating a satellite behavior state diagram, which can not only monitor the satellite behavior state, but also improve the analysis accuracy of the satellite behavior state.

[0004] According to a first aspect of the present application, a method for generating a satellite behavior state diagram is provided, comprising:

[0005] Acquire an image sequence of a target satellite;

[0006] For each frame of the image in the image sequence, performing component target detection on the image to obtain a target detection result, wherein the target detection result includes a detection frame;

[0007] Cropping the image according to the area indicated by each detection frame to obtain component images, and extracting features from the component images using an image coding network to obtain visual feature vectors;

[0008] Using a spatiotemporal modeling model to construct a spatiotemporal dependency relationship for each of the visual feature vectors, thereby obtaining a spatiotemporal dependency result; the spatiotemporal dependency result includes a spatial dependency relationship between components within the same frame image and a temporal dependency relationship between the same components in different frame images;

[0009] Based on the spatiotemporal relationship dependency result, a behavior state diagram for the target satellite is generated.

[0010] In some possible embodiments, the spatiotemporal modeling model includes a spatial coding model and a temporal coding model; and using the spatiotemporal modeling model to construct a spatiotemporal dependency relationship for each of the visual feature vectors to obtain a spatiotemporal dependency result includes:

[0011] Using the spatial coding model to model the spatial relationship of each visual feature vector in the same frame image, to obtain a spatial dependency result;

[0012] The time coding model is used to perform time relationship modeling on the same component in the spatial dependency results corresponding to different frame images to obtain the time-space dependency results.

[0013] In some possible embodiments, the spatiotemporal modeling model is a model based on the Transformer architecture, and the spatial encoding model is a spatial Transformer encoding model;

[0014] The spatial coding model is used to model the spatial relationship of each visual feature vector in the same frame image to obtain a spatial dependency result, including:

[0015] Based on the spatial position coding information corresponding to each visual feature vector, the spatial position coding information is fused with the corresponding visual feature vector to generate a position-aware feature representation;

[0016] The position-aware feature representation is input into the spatial Transformer encoding model, and modeling is performed in the spatial dimension through a multi-head self-attention mechanism to obtain the spatial dependency result.

[0017] In some possible embodiments, the temporal encoding model is a temporal Transformer encoding model; and using the temporal encoding model to perform temporal relationship modeling on the same component in the spatial dependency results corresponding to different images to obtain the spatiotemporal dependency result includes:

[0018] Arrange the visual feature vectors of the same satellite component in each spatial dependency result in chronological order to form a relationship sequence, generate a time position code according to the time position information corresponding to each frame image, and fuse the time position code into the relationship sequence;

[0019] The multi-head self-attention mechanism of the temporal Transformer encoding model is used to model the time dimension of the relationship sequence fused with the time position encoding to obtain the spatiotemporal relationship dependency result.

[0020] In some possible embodiments, generating a behavior state diagram for the target satellite based on the spatiotemporal relationship dependency result includes:

[0021] Based on the spatiotemporal relationship dependency result, predicting a behavior state of the target satellite, the behavior state comprising at least one of an attitude adjustment state, a component deployment state, and an orbit change state;

[0022] A behavior state diagram for the target satellite is generated according to the behavior state prediction result.

[0023] In some possible embodiments, the target detection result also includes the category of the detected satellite component; the behavioral state diagram includes a multi-frame scene diagram, each frame of the scene diagram includes nodes and connecting edges between nodes, the nodes are used to indicate the category of the satellite component and the state attribute information of the satellite component, and the connecting edges are used to indicate the spatial connection relationship between different components.

[0024] In some possible embodiments, the behavior state diagram further includes a time edge, where the time edge is used to indicate a state change of the same satellite component between adjacent scene graphs.

[0025] According to a second aspect of the present application, a device for generating a satellite behavior state diagram is provided, comprising:

[0026] An image acquisition module, used to acquire an image sequence of a target satellite;

[0027] an object detection module, configured to perform component object detection on each frame of the image in the image sequence to obtain an object detection result, wherein the object detection result includes a detection frame;

[0028] a feature extraction module, configured to crop each frame of the image according to the area indicated by each detection frame to obtain a component image, and perform feature extraction on the component image using an image coding network to obtain a visual feature vector;

[0029] a spatiotemporal modeling module for constructing a spatiotemporal dependency relationship for each of the visual feature vectors using a spatiotemporal modeling model to obtain a spatiotemporal dependency result; the spatiotemporal dependency result includes a spatial dependency relationship between components within the same frame image and a temporal dependency relationship between the same components in different frame images;

[0030] A state generation module is used to generate a behavior state diagram for the target satellite based on the spatiotemporal relationship dependency result.

[0031] In some possible embodiments, the spatiotemporal modeling model includes a spatial coding model and a temporal coding model; and the spatiotemporal modeling module is specifically configured to:

[0032] Using the spatial coding model to model the spatial relationship of each visual feature vector in the same frame image, to obtain a spatial dependency result;

[0033] The time coding model is used to perform time relationship modeling on the same component in the spatial dependency results corresponding to different frame images to obtain the time-space dependency results.

[0034] In some possible embodiments, the spatiotemporal modeling model is a model based on the Transformer architecture, the spatial encoding model is a spatial Transformer encoding model; and the spatiotemporal modeling module is specifically configured to:

[0035] Based on the spatial position coding information corresponding to each visual feature vector, the spatial position coding information is fused with the corresponding visual feature vector to generate a position-aware feature representation;

[0036] The position-aware feature representation is input into the spatial Transformer encoding model, and modeling is performed in the spatial dimension through a multi-head self-attention mechanism to obtain the spatial dependency result.

[0037] In some possible embodiments, the temporal coding model is a temporal Transformer coding model; and the spatiotemporal modeling module is specifically configured to:

[0038] Arrange the visual feature vectors of the same satellite component in each spatial dependency result in chronological order to form a relationship sequence, generate a time position code according to the time position information corresponding to each frame image, and fuse the time position code into the relationship sequence;

[0039] The multi-head self-attention mechanism of the temporal Transformer encoding model is used to model the time dimension of the relationship sequence fused with the time position encoding to obtain the spatiotemporal relationship dependency result.

[0040] In some possible embodiments, the state generation module is specifically configured to:

[0041] Based on the spatiotemporal relationship dependency result, predicting a behavior state of the target satellite, the behavior state comprising at least one of an attitude adjustment state, a component deployment state, and an orbit change state;

[0042] A behavior state diagram for the target satellite is generated according to the behavior state prediction result.

[0043] In some possible embodiments, the target detection result also includes the category of the detected satellite component; the behavioral state diagram includes a multi-frame scene diagram, each frame of the scene diagram includes nodes and connecting edges between nodes, the nodes are used to indicate the category of the satellite component and the state attribute information of the satellite component, and the connecting edges are used to indicate the spatial connection relationship between different components.

[0044] In some possible embodiments, the behavior state diagram further includes a time edge, where the time edge is used to indicate a state change of the same satellite component between adjacent scene graphs.

[0045] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the satellite behavior state diagram generation method described in the first aspect is executed.

[0046] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for generating a satellite behavior state diagram described in the first aspect is executed.

[0047] The satellite behavior state diagram generation method, device, electronic device and storage medium provided in the present application, because the image sequence for the target satellite is acquired, and component target detection and feature extraction are performed on each frame image in the image sequence, the visual feature vectors of the different components of each frame image in the image sequence can be obtained, and then the spatiotemporal modeling model can be used to construct the spatial and temporal dependency relationship of each visual feature vector, and the spatiotemporal dependency result can be obtained, thereby generating a behavior state diagram for the target satellite. Here, the spatiotemporal modeling model can be used to deduce the spatial correlation and temporal evolution law between the various satellite components of the target satellite, which can not only realize the analysis of the target satellite's behavior state, but also effectively solve the problem of behavior continuity that cannot be captured by the single-frame detection method, which is conducive to improving the analysis accuracy of the target satellite's behavior state.

[0048] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0050] Figure 1 is a flow chart of a method for generating a satellite behavior state diagram according to an exemplary embodiment of the present application;

[0051] Figure 2This is a flow chart of a method for determining spatiotemporal relationship dependency results using a spatiotemporal modeling model, as shown in an exemplary embodiment of the present application;

[0052] Figure 3 is a schematic diagram of a behavior state diagram of a target satellite shown in an exemplary embodiment of the present application;

[0053] Figure 4 is a schematic diagram of a process for generating a behavior state diagram of a satellite, shown in an exemplary embodiment of the present application;

[0054] Figure 5 This is a functional module diagram of a satellite behavior state diagram generating device shown in an exemplary embodiment of the present application;

[0055] Figure 6 It is a structural diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0057] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0058] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0059] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0060] A satellite refers to a natural or artificial celestial body that orbits a planet. In the embodiments of the present application, the satellite mainly refers to an artificial satellite. It can be understood that a satellite can also be called a spacecraft.

[0061] Satellites have a wide range of applications, including communications, remote sensing, weather monitoring, scientific experiments, and navigation and positioning. Specifically, communications satellites act as radio relay stations, enabling global communications and television signal transmission. Navigation satellites transmit signals that allow ground receivers to calculate their position. Meteorological satellites capture images of the Earth's surface, monitoring atmospheric clouds, climate, and meteorological changes, providing data support for weather forecasting, meteorological research, and disaster warning.

[0062] Currently, satellites are required to operate within their designated orbits and maintain a functioning state. A satellite failure can affect its functionality; a satellite deviating from its designated orbit can pose a threat to other satellites. Therefore, analyzing satellite behavior is crucial.

[0063] Research has found that current satellite recognition and behavior analysis primarily relies on static reasoning based on single-frame images. This approach is limited in its recognition and prediction capabilities when faced with complex target shapes, large variations in viewing angles, and missing data. This is particularly true when dealing with unknown targets or heavily obscured scenes, where this approach results in lower accuracy in analyzing satellite behavior.

[0064] Based on the above research, the present application provides a method for generating a satellite behavior state diagram, which first obtains an image sequence for the target satellite; then, for each frame of the image in the image sequence, performs component target detection on the image to obtain a target detection result, and the target detection result includes a detection box; then, each frame of the image is cropped according to the area indicated by each detection box to obtain a component image, and an image coding network is used to extract features of the component image to obtain a visual feature vector; then, a spatiotemporal modeling model is used to construct a spatiotemporal dependency relationship for each of the visual feature vectors to obtain a spatiotemporal relationship dependency result; the spatiotemporal relationship dependency result includes the spatial dependency relationship between each component in the same frame image and the temporal dependency relationship between the same component in different frame images; finally, based on the spatiotemporal relationship dependency result, a behavior state diagram for the target satellite is generated.

[0065] The satellite behavior state diagram generation method provided by the present application obtains an image sequence for the target satellite, and performs component target detection and feature extraction on each frame of the image sequence. In this way, the visual feature vectors of the different components of each frame of the image sequence can be obtained. Then, the spatiotemporal modeling model can be used to construct the spatial and temporal dependency relationship of each visual feature vector, and the spatiotemporal dependency result can be obtained, thereby generating a behavior state diagram for the target satellite. Here, the spatiotemporal modeling model can be used to deduce the spatial correlation and temporal evolution law between the various satellite components of the target satellite, which not only realizes the analysis of the target satellite's behavior state, but also effectively solves the problem of behavior continuity that cannot be captured by the single-frame detection method, which is conducive to improving the analysis accuracy of the target satellite's behavior state.

[0066] To facilitate understanding of this embodiment, the satellite behavior state diagram generation method provided in the embodiment of the present application is first introduced in detail. The execution subject of the satellite behavior state diagram generation method provided in the embodiment of the present application is generally an electronic device. The electronic device may include a server or a terminal device. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms, without specific limitation. Terminal devices may include mobile devices, user terminals, terminals, vehicle-mounted devices, computing devices and wearable devices, etc. In addition, the satellite behavior state diagram generation method can also be implemented by a processor calling computer-readable instructions stored in a memory.

[0067] The satellite behavior state diagram generation method is described in detail below with reference to the accompanying drawings.

[0068] See also Figure 1FIG. 1 is a flow chart of a method for generating a satellite behavior state diagram according to an embodiment of the present application. The method for generating a satellite behavior state diagram may include the following steps S101 to S105:

[0069] S101: Acquire an image sequence of a target satellite.

[0070] The image sequence is obtained by observing the target satellite during a preset time period. For example, the image sequence can be obtained by continuously observing the target satellite during the preset time period, or by observing the target satellite at intervals during the preset time period, or by extracting frames from a continuous original image sequence, without limitation.

[0071] For example, the image sequence can be obtained by observing a specified airspace using an onboard optical sensor (e.g., a visible light camera or a multispectral camera) or ground-based remote sensing equipment. The time interval between any two adjacent frames in the image sequence can be the same or different. In other words, the interval between adjacent frames can be adjustable to meet different temporal resolution requirements.

[0072] Among them, onboard optical sensors are instruments installed on satellites that observe and collect data on other celestial bodies (such as satellites) based on optical principles.

[0073] S102 : For each frame of the image in the image sequence, perform component target detection on the image to obtain a target detection result, where the target detection result includes a detection frame.

[0074] After obtaining the image sequence, component target detection can be performed on each frame in the image sequence to obtain target detection results. Component target detection refers to target detection of different components of the target satellite. The detection boxes in the detection results indicate the bounding box of each component, and the component categories indicate the names of different components (e.g., body, solar panel, antenna, payload, etc.).

[0075] It can be understood that the detection frame has coordinate information, and the size and position of the area where the component is located can be determined based on the coordinates of the detection frame.

[0076] For example, a pre-trained target detection network can be used to perform component-level detection and semantic labeling of target satellites, and output the bounding box coordinates and component category of each component. Specifically, the present application can improve existing target detection networks. For example, the anchor box size and loss function can be optimized for satellite components to obtain the pre-trained target detection network.

[0077] Optionally, the target detection result may also include a confidence level, which can be used to measure the accuracy of the detection result. Furthermore, the detection accuracy and robustness of the target detection network in the embodiments of the present application meet preset requirements. For example, it is robust to complex viewing angles and partial occlusions, and can be applied to various satellite configurations.

[0078] S103 , cropping the image according to the area indicated by each detection frame to obtain component images, and performing feature extraction on the component images using an image coding network to obtain a visual feature vector.

[0079] For example, after component object detection is performed on each frame, each frame can be cropped based on the detection results. Specifically, the image can be cropped according to the boundaries of each detection box to obtain the component images in each frame. The cropped component images can then be fed into a pre-trained image encoding network (such as ResNet-50) to extract the component's visual feature vector. This visual feature vector can then serve as the initial embedding representation for subsequent scene graph nodes.

[0080] S104, using the spatiotemporal modeling model to construct the spatiotemporal dependency relationship of each of the visual feature vectors to obtain a spatiotemporal dependency result; the spatiotemporal dependency result includes the spatial dependency relationship between each component in the same frame image and the temporal dependency relationship between the same components between different frame images.

[0081] The dependency relationship refers to the association relationship between elements. The spatiotemporal modeling model can be a neural network. Specifically, the spatiotemporal modeling model can include a spatial coding model and a temporal coding model.

[0082] See also Figure 2 As shown, for step S104, when the spatiotemporal dependency relationship of each visual feature vector is constructed using the spatiotemporal modeling model to obtain the spatiotemporal dependency result, the following steps S1041 to S1042 may be included:

[0083] S1041: Use the spatial coding model to perform spatial relationship modeling on each visual feature vector in the same frame image to obtain a spatial dependency result.

[0084] Specifically, the spatiotemporal modeling model is a model based on the Transformer architecture, and the spatial encoding model is a spatial Transformer encoding model. When using the spatial encoding model to model the spatial relationship of various visual feature vectors within the same frame image, the method may include: based on the corresponding spatial position encoding information of each visual feature vector, fusing the spatial position encoding information with the corresponding visual feature vector to generate a position-aware feature representation; and inputting the position-aware feature representation into the spatial Transformer encoding model, performing modeling in the spatial dimension through a multi-head self-attention mechanism to obtain the spatial dependency result.

[0085] The spatial position coding information corresponding to each visual feature vector can be obtained through the coordinates of the detection frame of the corresponding component image.

[0086] Here, the spatial Transformer encoding model receives each visual feature vector and its corresponding spatial position encoding information (such as the coordinates of the bounding box center point). It then uses an attention mechanism to automatically mine the interdependencies between satellite components (such as the satellite body, solar panels, communication antennas, payloads, etc.) within each frame, thereby obtaining a spatial structural representation. In other words, it can obtain node feature representations that incorporate spatial dependency information. For example, the solar panels and the satellite body are connected. For another example, relationships between different satellite components can be "attached," "coordinated motion," or "symmetrical structures." Here, the use of the spatial Transformer encoding model effectively enhances structural perception within the same image, providing a more spatially consistent representational foundation for subsequent temporal modeling and graph construction.

[0087] S1042: Use the time coding model to perform time relationship modeling on the same component in the spatial dependency results corresponding to different frame images to obtain the spatiotemporal dependency results.

[0088] Exemplarily, the temporal encoding model may be a temporal Transformer encoding model. Therefore, when using the temporal encoding model to perform temporal relationship modeling on the same component in the spatial dependency results corresponding to different images, the temporal and spatial dependency results obtained may include the following (I) to (II):

[0089] (I) Arranging the visual feature vectors of the same satellite component in each spatial dependency result in time order to form a relational sequence, generating a time position code according to the time position information corresponding to each frame image, and fusing the time position code into the relational sequence.

[0090] (II) Utilizing the multi-head self-attention mechanism of the temporal Transformer encoding model, the temporal dimension of the relational sequence fused with the temporal position encoding is modeled to obtain the spatiotemporal relationship dependency result.

[0091] Specifically, for each target component on the satellite, its corresponding visual feature vector in each frame image can be extracted and arranged in chronological order to form a temporal feature sequence. The visual feature vector of each frame is fused with its corresponding time position encoding information and sent as input to the temporal Transformer encoding model. The temporal Transformer encoding model models the temporal feature sequence through its multi-head self-attention mechanism, thereby obtaining spatiotemporal dependency results across time dimensions.

[0092] Here, the spatial dependency results output by the spatial Transformer encoding model can be temporally rearranged. Specifically, all visual feature vectors appearing in each spatial dependency result (one spatial dependency result corresponds to one image frame) for the same satellite component are combined in chronological order to form a relational sequence, which is then input into the temporal Transformer encoding model. This temporal Transformer encoding model incorporates temporal positional embedding (TPE), which encodes the temporal relationship of each image frame. It utilizes a self-attention mechanism to model the dynamic changes of each visual feature vector in the temporal dimension, thereby generating a fused feature representation with global temporal semantics and obtaining a spatiotemporal dependency result. This temporal Transformer encoding model can capture the state changes and interrelationship evolution of satellite components over time, providing critical temporal semantic support for the subsequent generation of behavioral state diagrams.

[0093] S105: Generate a behavior state diagram for the target satellite based on the spatiotemporal relationship dependency result.

[0094] After obtaining the spatiotemporal dependency results, a behavior state diagram for the target satellite can be generated based on the spatiotemporal dependency results. The behavior state diagram may include attitude adjustments, component deployments, orbit changes, etc. For example, the behavior state diagram for the target satellite can be generated based on the spatiotemporal dependency results corresponding to the current frame and multiple frames preceding the current frame.

[0095] It is understandable that based on the spatiotemporal dependency result, not only the current behavior state diagram of the target satellite can be generated, but also the future behavior state diagram of the target satellite can be predicted. That is, in some embodiments, generating the behavior state diagram for the target satellite based on the spatiotemporal dependency result may include:

[0096] Based on the spatiotemporal relationship dependency result, the behavior state of the target satellite is predicted, and according to the behavior state prediction result, a behavior state diagram for the target satellite is generated; wherein the behavior state includes at least one of an attitude adjustment state, a component deployment state, and an orbit change state.

[0097] For example, if the current spatiotemporal dependency result corresponds to the spatiotemporal dependency result for three frames of imagery, the state of the target satellite after a preset time (e.g., 5 seconds) can be predicted based on this spatiotemporal dependency result. Specifically, the prediction head can predict the target satellite's future behavior state based on the spatiotemporal dependency result, and a behavior state diagram of the target satellite can be obtained based on the behavior state prediction results. For example, this behavior state may include orbital maneuvers, component deployment, attitude adjustments, and abnormal collision risks. Orbital maneuvers are technical means by which a satellite actively changes its trajectory through its propulsion system, primarily including orbital transitions, parameter corrections, and rendezvous and docking.

[0098] In some embodiments, the behavioral state graph can be generated from several frames in an image sequence, where each frame corresponds to a scene graph. Preferably, to improve processing efficiency and model key behavioral changes, several key frames can be selected for scene graph construction. For example, when an image sequence contains N frames, the behavioral state graph can contain M scene graphs (M ≤ N). Specifically, each frame of the scene graph includes nodes and connecting edges between nodes. The nodes are used to indicate the category of the satellite component and the state attribute information of the satellite component, and the connecting edges are used to indicate the connection relationship between different components.

[0099] Optionally, the behavioral state graph also includes a time edge, which connects nodes of the same satellite component in different frames. This time edge describes the evolution of the satellite component's state over time. Specifically, it can be used to indicate the state change of the same satellite component between adjacent scene graphs, such as "increasing the solar panel deployment angle" or "extending the camera payload," supporting temporal modeling and behavior recognition.

[0100] Here, the behavior decoder can be used to predict the status of each satellite component (such as attitude angle changes, whether it is deployed / folded, etc.), and at the same time identify the structural and functional relationship between adjacent frame images of the same satellite component, and then generate a behavior state diagram for the target satellite.

[0101] See also Figure 3 As shown, Figure 3This is a schematic diagram of a target satellite's behavior state diagram, provided in an embodiment of the present application. This diagram illustrates the temporal evolution of the target satellite's structural state and component relationships at different observation times. The diagram consists of three consecutive scene graphs corresponding to each moment, each representing the target satellite's dynamic structural characteristics at three key moments in the observation sequence.

[0102] Specifically, each frame of the scene graph is derived from the image frame after target detection and is constructed by combining the spatial dependency results output by the spatial Transformer encoding model. The nodes in each scene graph represent the category of the target satellite component and the status attribute information of the component, and the connecting edges represent the spatial dependency and functional coordination relationship between the components. In the embodiment of the present application, the nodes include the satellite body, the left solar panel, the right solar panel and the camera payload component, and the connecting edges reflect relationship attributes such as "attachment", "cooperative movement" and "symmetrical structure".

[0103] The first frame, scene graph a, reflects the initial structural layout, with the left and right solar panels deployed at a 35° angle, not yet fully deployed. By the time we reach the second frame, scene graph b, the left and right solar panels have fully deployed, reaching a 42° angle. The "Deployment Angle" feature in the corresponding node's state attributes has been significantly improved. The third frame, scene graph c, shows the target satellite extending its external camera payload. A new "Camera Payload" node is added to the scene graph, and an edge indicating an "Attachment" relationship is established with the satellite. A new "Operation Status" attribute is added to the node's state attributes to reflect its current activation status.

[0104] The embodiment of the present application generates a behavioral state diagram of the target satellite, which can not only intuitively display the evolution process of the key structures of the target satellite during the mission execution, but also provide a reference for subsequent behavioral understanding and mission prediction, which is conducive to improving the real-time and accuracy of space situational awareness, and provides reliable technical support for application scenarios such as on-orbit services and space traffic management.

[0105] The specific implementation process of the satellite behavior state diagram generation method provided in the embodiment of the present application is described below with reference to the accompanying drawings.

[0106] See also Figure 4 The figure shows a flow chart of a method for generating a satellite behavior state diagram provided by an embodiment of the present application. First, a continuous visible light image sequence of the target airspace is acquired by an optical imaging payload carried by the satellite platform. The observation time is 30 seconds, the frame rate is 10 Hz, and the image resolution is set to 2048×2048 pixels.

[0107] Next, a pre-trained object detection model optimized for orbital target characteristics is used to detect and identify satellite components frame by frame in the enhanced image sequence. The detection model uses a 1024×1024 pixel input and outputs bounding box locations, category labels, and detection confidence scores for components such as the satellite itself, solar panels, and communication antennas. For example, the detection results include a 98.2% confidence score for the satellite itself, and 92.7% and 89.3% confidence scores for the left and right panels, respectively.

[0108] Subsequently, each detected target region is used as a local perception unit and fed into an image encoding network (such as ResNet-50) to extract the image semantic features of each component and form node embeddings. After obtaining the node features, the inter-component connectivity graph is initialized to construct the spatial adjacency relationships between components. This graph structure, along with the node embedding representations, is then fed into a spatial Transformer encoding model. This spatial Transformer encoding model uses a multi-head self-attention mechanism to model the spatial dependencies and structural coupling between components, thereby generating node feature representations that incorporate spatial semantics (i.e., spatial dependency results).

[0109] Furthermore, the spatial dependency results at different time frames are serialized into time series embeddings and input into a temporal Transformer encoding model to learn the dynamic changes in component states over time. By incorporating temporal positional encoding information, the Transformer encoding model can extract temporal behavior patterns such as sailboard deployment and attitude adjustment trends, such as right sailboard deployment, ultimately yielding spatiotemporal dependency results.

[0110] After obtaining the spatiotemporal dependency results, a behavior decoder (such as an MLP) can be used to predict the target satellite's dynamic behavior indicators within the next 5 seconds based on the node evolution state at the current moment and within the past 3 seconds (30 frames). For example, the prediction results may include an attitude angle change of Δθ = 1.2° ± 0.3°, an orbit radius change of Δr, and an increase in the probability of the right sail panel deploying to 96.8%.

[0111] Based on the above prediction results, the behavior state diagram can be updated, annotated with timestamp information (for example, t=2023-05-20T08:15:30.125Z), and the component node properties can be modified synchronously, such as updating the left sailboard deployment angle from 35° to 42°, thereby reflecting the current state of the target satellite and its evolution trend.

[0112] It should be noted that the various network structures, model parameters, and input and output formats in this embodiment can be adjusted and replaced according to specific task requirements, such as changing the target detection algorithm, adjusting the number of layers of the Transformer network structure, and introducing multi-source sensor data.

[0113] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0114] Based on the same technical concept, the embodiment of the present disclosure also provides a satellite behavior state diagram generation device corresponding to the satellite behavior state diagram generation method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned satellite behavior state diagram generation method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0115] Reference Figure 5 FIG. 1 is a schematic diagram of a satellite behavior state diagram generating device provided by an embodiment of the present disclosure. The satellite behavior state diagram generating device 500 includes:

[0116] An image acquisition module 501 is used to acquire an image sequence for a target satellite;

[0117] An object detection module 502 is configured to perform component object detection on each frame of the image in the image sequence to obtain an object detection result, wherein the object detection result includes a detection frame;

[0118] A feature extraction module 503 is configured to crop each frame of the image according to the area indicated by each detection frame to obtain component images, and perform feature extraction on the component images using an image coding network to obtain a visual feature vector;

[0119] The spatiotemporal modeling module 504 is configured to construct a spatiotemporal dependency relationship for each of the visual feature vectors using a spatiotemporal modeling model to obtain a spatiotemporal dependency result; the spatiotemporal dependency result includes a spatial dependency relationship between components within the same frame image and a temporal dependency relationship between the same components in different frame images;

[0120] The state generating module 505 is configured to generate a behavior state diagram for the target satellite based on the spatiotemporal relationship dependency result.

[0121] In some possible embodiments, the spatiotemporal modeling model includes a spatial coding model and a temporal coding model; the spatiotemporal modeling module 504 is specifically configured to:

[0122] Using the spatial coding model to model the spatial relationship of each visual feature vector in the same frame image, to obtain a spatial dependency result;

[0123] The time coding model is used to perform time relationship modeling on the same component in the spatial dependency results corresponding to different frame images to obtain the time-space dependency results.

[0124] In some possible embodiments, the spatiotemporal modeling model is a model based on the Transformer architecture, and the spatial coding model is a spatial Transformer coding model; the spatiotemporal modeling module 504 is specifically configured to:

[0125] Based on the spatial position coding information corresponding to each visual feature vector, the spatial position coding information is fused with the corresponding visual feature vector to generate a position-aware feature representation;

[0126] The position-aware feature representation is input into the spatial Transformer encoding model, and modeling is performed in the spatial dimension through a multi-head self-attention mechanism to obtain the spatial dependency result.

[0127] In some possible embodiments, the temporal coding model is a temporal Transformer coding model; and the spatiotemporal modeling module 504 is specifically configured to:

[0128] Arrange the visual feature vectors of the same satellite component in each spatial dependency result in chronological order to form a relationship sequence, generate a time position code according to the time position information corresponding to each frame image, and fuse the time position code into the relationship sequence;

[0129] The multi-head self-attention mechanism of the temporal Transformer encoding model is used to model the time dimension of the relationship sequence fused with the time position encoding to obtain the spatiotemporal relationship dependency result.

[0130] In some possible embodiments, the state generation module 505 is specifically configured to:

[0131] Based on the spatiotemporal relationship dependency result, the behavior state of the target satellite is predicted, and a behavior state diagram for the target satellite is generated.

[0132] In some possible embodiments, the behavioral state diagram includes a multi-frame scene diagram, each frame of the scene diagram includes nodes and connecting edges between nodes, the nodes are used to indicate the category of the satellite component and the state attribute information of the satellite component, and the connecting edges are used to indicate the spatial connection relationship between different components.

[0133] In some possible embodiments, the behavior state diagram further includes a time edge, where the time edge is used to indicate a state change of the same satellite component between adjacent scene graphs.

[0134] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0135] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 6 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present disclosure, including a processor 601, a memory 602, and a bus 603. The memory 602 is used to store execution instructions and includes a memory 6021 and an external memory 6022. The memory 6021 is also referred to as internal memory and is used to temporarily store operation data in the processor 601 and data exchanged with an external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the memory 6021.

[0136] In the embodiment of the present application, the memory 602 is specifically used to store application code for executing the solution of the present application, and the execution is controlled by the processor 601. That is, when the electronic device 600 is running, the processor 601 communicates with the memory 602 via the bus 603, so that the processor 601 executes the application code stored in the memory 602, thereby performing the method described in any of the aforementioned embodiments.

[0137] The memory 602 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0138] Processor 601 may be an integrated circuit chip with signal processing capabilities. Such processors may be general-purpose processors, including central processing units (CPUs) and network processors (NPs). They may also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. These processors may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor.

[0139] It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device 600. In other embodiments of the present application, the electronic device 600 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0140] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the satellite behavior state diagram generation method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0141] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the satellite behavior state diagram generation method in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0142] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0143] In addition, embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

[0144] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0145] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such mass storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0146] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0147] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0148] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0149] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0150] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating a satellite behavior state diagram, characterized in that: include: Acquire an image sequence of a target satellite; For each frame of the image in the image sequence, performing component target detection on the image to obtain a target detection result, wherein the target detection result includes a detection frame; Cropping each frame of the image according to the area indicated by each detection frame to obtain component images, and extracting features from the component images using an image coding network to obtain a visual feature vector; Using a spatiotemporal modeling model to construct a spatiotemporal dependency relationship for each of the visual feature vectors, thereby obtaining a spatiotemporal dependency result; the spatiotemporal dependency result includes a spatial dependency relationship between components within the same frame image and a temporal dependency relationship between the same components in different frame images; Based on the spatiotemporal relationship dependency result, a behavior state diagram for the target satellite is generated.

2. The method according to claim 1, characterized in that The spatiotemporal modeling model includes a spatial coding model and a temporal coding model; the spatiotemporal dependency relationship is constructed for each of the visual feature vectors using the spatiotemporal modeling model to obtain a spatiotemporal dependency result, including: Using the spatial coding model to model the spatial relationship of each visual feature vector in the same frame image, to obtain a spatial dependency result; The time coding model is used to perform time relationship modeling on the same component in the spatial dependency results corresponding to different frame images to obtain the time-space dependency results.

3. The method according to claim 2, characterized in that The spatiotemporal modeling model is a model based on the Transformer architecture, and the spatial coding model is a spatial Transformer coding model; The spatial coding model is used to model the spatial relationship of each visual feature vector in the same frame image to obtain a spatial dependency result, including: Based on the spatial position coding information corresponding to each visual feature vector, the spatial position coding information is fused with the corresponding visual feature vector to generate a position-aware feature representation; The position-aware feature representation is input into the spatial Transformer encoding model, and modeling is performed in the spatial dimension through a multi-head self-attention mechanism to obtain the spatial dependency result.

4. The method according to claim 2 or 3, characterized in that The temporal coding model is a temporal Transformer coding model; and using the temporal coding model to perform temporal relationship modeling on the same component in the spatial dependency results corresponding to different images to obtain the temporal and spatial dependency results includes: Arrange the visual feature vectors of the same satellite component in each spatial dependency result in chronological order to form a relationship sequence, generate a time position code according to the time position information corresponding to each frame image, and fuse the time position code into the relationship sequence; The multi-head self-attention mechanism of the temporal Transformer encoding model is used to model the time dimension of the relationship sequence fused with the time position encoding to obtain the spatiotemporal relationship dependency result.

5. The method according to claim 1, characterized in that Generating a behavior state diagram for the target satellite based on the spatiotemporal relationship dependency result includes: Based on the spatiotemporal relationship dependency result, a behavior state of the target satellite is predicted; the behavior state includes at least one of an attitude adjustment state, a component deployment state, and an orbit change state; A behavior state diagram for the target satellite is generated according to the behavior state prediction result.

6. The method according to claim 1 or 5, characterized in that The target detection result also includes the category of the detected satellite component; The behavior state diagram includes a multi-frame scene diagram, each frame of the scene diagram includes nodes and connecting edges between nodes, the nodes are used to indicate the category of the satellite component and the state attribute information of the satellite component, and the connecting edges are used to indicate the spatial connection relationship between different components.

7. The method according to claim 6, characterized in that The behavior state graph further includes a time edge, and the time edge is used to indicate a state change of the same satellite component between adjacent scene graphs.

8. A satellite behavior state diagram generating device, characterized in that: The device comprises: An image acquisition module, used to acquire an image sequence of a target satellite; An object detection module is configured to perform component object detection on each frame of the image in the image sequence to obtain an object detection result, wherein the object detection result includes a detection frame; a feature extraction module, configured to crop each frame of the image according to the area indicated by each detection frame to obtain a component image, and perform feature extraction on the component image using an image coding network to obtain a visual feature vector; a spatiotemporal modeling module for constructing a spatiotemporal dependency relationship for each of the visual feature vectors using a spatiotemporal modeling model to obtain a spatiotemporal dependency result; the spatiotemporal dependency result includes a spatial dependency relationship between components within the same frame image and a temporal dependency relationship between the same components in different frame images; A state generation module is used to generate a behavior state diagram for the target satellite based on the spatiotemporal relationship dependency result.

9. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the satellite behavior state diagram generation method according to any one of claims 1 to 7 is performed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a satellite behavior state diagram according to any one of claims 1 to 7 is executed.

Images