Satellite behavior state diagram generation method and device, electronic equipment and storage medium

Abstract

The invention provides a satellite behavior state diagram generation method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining an image sequence for a target satellite; for each frame of image in the image sequence, component target detection is carried out on the image to obtain a target detection result, and the target detection result comprises a detection frame; cutting each frame of image according to an area indicated by each detection frame to obtain a component image, and performing feature extraction on the component image by using an image coding network to obtain a visual feature vector; performing space-time dependency relationship construction on each visual feature vector by using a space-time modeling model to obtain a space-time relationship dependency result; the space-time relationship dependency result comprises a space dependency relationship between components in the same frame of image and a time dependency relationship between the same components in different frames of images; and generating a behavior state diagram for the target satellite based on the time-space relationship dependency result. Therefore, the analysis precision of the behavior state of the target satellite can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of satellite state analysis, and particularly to a method, apparatus, electronic device and storage medium for generating a satellite behavior state diagram. Background Art

[0002] With the development of aerospace technology, not only the number of satellites is gradually increasing, but also the application fields of satellites are becoming more and more extensive. For example, they can be applied to fields such as communication, remote sensing, meteorological monitoring, scientific experiments, and navigation and positioning. At present, each satellite needs to operate in a predetermined orbit and be in an effective state. If a satellite fails, its function implementation will be affected. If a satellite deviates from its predetermined orbit, it will pose a threat to other satellites. Therefore, it is particularly important to analyze the behavior state of satellites. Summary of the Invention

[0003] In view of this, the present application provides a method, apparatus, device and storage medium for generating a satellite behavior state diagram, which can not only monitor the behavior state of satellites, but also improve the analysis accuracy of satellite behavior states.

[0004] According to a first aspect of the present application, there is provided a method for generating a satellite behavior state diagram, including: Obtaining an image sequence of a target satellite; For each frame of image in the image sequence, performing component object detection on the image to obtain a target detection result, where the target detection result includes detection frames; Cropping the image according to the regions indicated by the respective detection frames to obtain component images, and using an image encoding network to extract features from the component images to obtain visual feature vectors; Using a spatio-temporal modeling model to construct spatio-temporal dependence relationships among the respective visual feature vectors to obtain a spatio-temporal relationship dependence result; the spatio-temporal relationship dependence result includes spatial dependence relationships among components within the same frame of image and temporal dependence relationships among the same component in different frames of image; Generating a behavior state diagram for the target satellite based on the spatio-temporal relationship dependence result.

[0005] In some possible embodiments, the spatio-temporal modeling model includes a spatial encoding model and a temporal encoding model; the using the spatio-temporal modeling model to construct spatio-temporal dependence relationships among the respective visual feature vectors to obtain a spatio-temporal relationship dependence result includes: Using the spatial encoding model to model spatial relationships among the respective visual feature vectors within the same frame of image to obtain a spatial dependence result; Using the temporal encoding model to model temporal relationships among the same component in the spatial dependence results respectively corresponding to different frames of image to obtain the spatio-temporal relationship dependence result.

[0006] In some possible embodiments, the spatio-temporal modeling model is a model based on the Transformer architecture, and the spatial encoding model is a spatial Transformer encoding model; The spatial relationship modeling of each visual feature vector within the same frame of image by using the spatial encoding model to obtain the spatial dependence result includes: Based on the corresponding spatial position encoding information of each visual feature vector, fusing the spatial position encoding information with the corresponding visual feature vector to generate a position-aware feature representation; Inputting the position-aware feature representation into the spatial Transformer encoding model, and performing modeling in the spatial dimension through the multi-head self-attention mechanism to obtain the spatial dependence result.

[0007] In some possible embodiments, the temporal encoding model is a temporal Transformer encoding model; the temporal relationship modeling of the same component in the spatial dependence results respectively corresponding to different images by using the temporal encoding model to obtain the spatio-temporal relationship dependence result includes: Arranging the visual feature vectors of the same satellite component in each spatial dependence result in chronological order to form a relationship sequence, generating temporal position encoding according to the temporal position information corresponding to each frame of image, and fusing the temporal position encoding into the relationship sequence; Using the multi-head self-attention mechanism of the temporal Transformer encoding model to perform modeling in the temporal dimension on the relationship sequence fused with the temporal position encoding to obtain the spatio-temporal relationship dependence result.

[0008] In some possible embodiments, generating a behavior state graph for the target satellite based on the spatio-temporal relationship dependence result includes: Predicting the behavior state of the target satellite based on the spatio-temporal relationship dependence result, where the behavior state includes at least one of an attitude adjustment state, a component deployment state, and an orbit change state; Generating a behavior state graph for the target satellite according to the behavior state prediction result.

[0009] In some possible embodiments, the target detection result further includes the category of the detected satellite component; the behavior state graph includes multiple frames of scene graphs, and each frame of the scene graph includes nodes and connection edges between the nodes. The nodes are used to indicate the category of the satellite component and the state attribute information of the satellite component, and the connection edges are used to indicate the spatial connection relationship between different components.

[0010] In some possible embodiments, the behavior state diagram further includes a time edge, which is used to indicate the state change of the same satellite component between adjacent scenario diagrams.

[0011] According to a second aspect of the present application, there is provided a satellite behavior state diagram generation device, including: An image acquisition module, configured to acquire an image sequence of a target satellite; A target detection module, configured to perform component target detection on each frame of the image sequence to obtain a target detection result, where the target detection result includes a detection box; A feature extraction module, configured to crop each frame of the image according to the area indicated by each detection box to obtain a component image, and use an image encoding network to extract features of the component image to obtain a visual feature vector; A spatio-temporal modeling module, configured to construct a spatio-temporal dependence relationship of each visual feature vector by using a spatio-temporal modeling model to obtain a spatio-temporal relationship dependence result; the spatio-temporal relationship dependence result includes a spatial dependence relationship between components in the same frame of image and a temporal dependence relationship between the same component in different frames of image; A state generation module, configured to generate a behavior state diagram for the target satellite based on the spatio-temporal relationship dependence result.

[0012] In some possible embodiments, the spatio-temporal modeling model includes a spatial encoding model and a temporal encoding model; specifically, the spatio-temporal modeling module is configured to: Use the spatial encoding model to model the spatial relationship of each visual feature vector in the same frame of image to obtain a spatial dependence result; Use the temporal encoding model to model the temporal relationship of the same component in the spatial dependence results corresponding to different frames of image to obtain the spatio-temporal relationship dependence result.

[0013] In some possible embodiments, the spatio-temporal modeling model is a model based on the Transformer architecture, and the spatial encoding model is a spatial Transformer encoding model; specifically, the spatio-temporal modeling module is configured to: Based on the corresponding spatial position encoding information of each visual feature vector, fuse the spatial position encoding information with the corresponding visual feature vector to generate a position-aware feature representation; Input the position-aware feature representation into the spatial Transformer encoding model, and perform modeling in the spatial dimension through a multi-head self-attention mechanism to obtain the spatial dependence result.

[0014] In some possible embodiments, the time encoding model is a time Transformer encoding model; the spatio-temporal modeling module is specifically configured to: Arrange the visual feature vectors of the same satellite component in the spatial dependence results in chronological order to form a relationship sequence, generate time position encoding according to the time position information corresponding to each frame of image, and fuse the time position encoding into the relationship sequence; Use the multi-head self-attention mechanism of the time Transformer encoding model to perform spatio-temporal dimension modeling on the relationship sequence fused with the time position encoding to obtain the spatio-temporal relationship dependence result.

[0015] In some possible embodiments, the state generation module is specifically configured to: Predict the behavior state of the target satellite based on the spatio-temporal relationship dependence result, where the behavior state includes at least one of an attitude adjustment state, a component deployment state, and an orbit change state; Generate a behavior state map for the target satellite according to the behavior state prediction result.

[0016] In some possible embodiments, the target detection result further includes the category of the detected satellite component; the behavior state map includes multiple frames of scene graphs, each frame of the scene graph includes nodes and connection edges between the nodes, the nodes are used to indicate the category of the satellite component and the state attribute information of the satellite component, and the connection edges are used to indicate the spatial connection relationship between different components.

[0017] In some possible embodiments, the behavior state map further includes time edges, and the time edges are used to indicate the state change of the same satellite component between adjacent scene graphs.

[0018] According to a third aspect of the present application, there is provided an electronic device, including: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, when the electronic device runs, the processor communicates with the memory through the bus, and when the machine-readable instructions are executed by the processor, the satellite behavior state map generation method described in the first aspect above is executed.

[0019] According to a fourth aspect of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the satellite behavior state map generation method described in the first aspect above is executed.

[0020] The method, apparatus, electronic device, and storage medium for generating a satellite behavior state diagram provided by this application obtain an image sequence of a target satellite and perform component target detection and feature extraction on each frame of the image sequence. In this way, visual feature vectors of different components in each frame of the image sequence can be obtained, and then a spatio-temporal modeling model can be used to construct the dependency relationships in space and time for each visual feature vector to obtain a spatio-temporal relationship dependency result, thereby generating a behavior state diagram for the target satellite. Here, through the spatio-temporal modeling model, it is possible to deduce the spatial correlation and temporal evolution law between the various satellite components of the target satellite, which can not only analyze the behavior state of the target satellite but also effectively solve the problem of behavior continuity that cannot be captured by single-frame detection methods, facilitating the improvement of the analysis accuracy of the behavior state of the target satellite.

[0021] To make the above objects, features, and advantages of the present disclosure more obvious and understandable, the following specifically gives preferred embodiments and, in conjunction with the accompanying drawings, the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the accompanying drawings required for use in the embodiments. Here, the accompanying drawings are incorporated into the specification and constitute a part of this specification. These accompanying drawings show embodiments that conform to the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following accompanying drawings only show some embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related accompanying drawings can also be obtained based on these accompanying drawings.

[0023] Figure 1 is a flowchart of a method for generating a satellite behavior state diagram shown in an exemplary embodiment of this application; Figure 2 is a flowchart of a method for determining a spatio-temporal relationship dependency result using a spatio-temporal modeling model shown in an exemplary embodiment of this application; Figure 3 is a schematic diagram of a behavior state diagram of a target satellite shown in an exemplary embodiment of this application; Figure 4 is a schematic diagram of the generation process of a behavior state diagram of a satellite shown in an exemplary embodiment of this application; Figure 5 is a functional module diagram of a satellite behavior state diagram generation apparatus shown in an exemplary embodiment of this application; Figure 6 is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "the", and "said" used in this application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0026] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0027] The term "and / or" herein merely describes an association relationship and indicates that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set composed of A, B, and C.

[0028] A satellite refers to a natural or artificial celestial body orbiting a planet. The satellites in the embodiments of the present application mainly refer to artificial satellites. It can be understood that a satellite can also be called a spacecraft.

[0029] The application fields of satellites are relatively wide, and they can be applied to fields such as communication, remote sensing, meteorological monitoring, scientific experiments, and navigation and positioning. Specifically, communication satellites, as radio communication relay stations, can be used to achieve global communication and television signal transmission, etc.; navigation satellites can send signals to allow ground receivers to calculate their own positions; meteorological satellites can take pictures of the Earth's surface, monitor clouds, climate, and meteorological changes in the atmosphere, and provide data support for weather forecasting, meteorological research, and disaster warning, etc.

[0030] Currently, each satellite needs to operate in a predetermined orbit and be in an effective state. If a satellite fails, its function cannot be realized; if a satellite deviates from its predetermined orbit, it will pose a threat to other satellites. Therefore, it is particularly important to analyze the behavior state of satellites.

[0031] It has been found through research that the current satellite identification and behavior analysis mainly rely on single-frame images for static reasoning. However, when facing situations such as complex target shapes, large perspective changes, and data loss, the identification and prediction capabilities of this method are limited. Especially in scenarios where unknown targets or severe occlusions are involved, using this method will result in a relatively low accuracy in analyzing the behavior state of satellites.

[0032] Based on the above research, the present application provides a method for generating a satellite behavior state map. First, an image sequence of a target satellite is obtained; then, for each frame image in the image sequence, component object detection is performed on the image to obtain a target detection result, where the target detection result includes detection frames; then each frame of the image is cropped according to the regions indicated by the respective detection frames to obtain component images, and an image coding network is used to extract features from the component images to obtain visual feature vectors; then a spatio-temporal modeling model is used to construct spatio-temporal dependency relationships for the respective visual feature vectors to obtain a spatio-temporal relationship dependency result, where the spatio-temporal relationship dependency result includes the spatial dependency relationships between components within the same frame image and the temporal dependency relationships between the same component in different frame images; finally, based on the spatio-temporal relationship dependency result, a behavior state map for the target satellite is generated.

[0033] In the method for generating a satellite behavior state map provided by the present application, since an image sequence of a target satellite is obtained and component object detection and feature extraction are performed on each frame image in the image sequence, visual feature vectors of different components of each frame image in the image sequence can be obtained. Furthermore, a spatio-temporal modeling model can be used to construct spatial and temporal dependency relationships for the respective visual feature vectors to obtain a spatio-temporal relationship dependency result, thereby generating a behavior state map for the target satellite. Here, through the spatio-temporal modeling model, it is possible to deduce the spatial correlation and temporal evolution law between the various satellite components of the target satellite, not only realizing the analysis of the behavior state of the target satellite, but also effectively solving the problem of behavioral continuity that cannot be captured by single-frame detection methods, which is beneficial to improving the accuracy of analyzing the behavior state of the target satellite.

[0034] To facilitate the understanding of this embodiment, first, a detailed introduction to the method for generating a satellite behavior state diagram provided by the embodiments of the present application will be given. The execution subject of the method for generating a satellite behavior state diagram provided by the embodiments of the present application is generally an electronic device. This electronic device may include a server or a terminal device. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms, and specific limitations are not made. The terminal device may include mobile devices, user terminals, terminals, in-vehicle devices, computing devices, wearable devices, etc. In addition, the method for generating a satellite behavior state diagram may also be implemented by a processor calling computer-readable instructions stored in a memory.

[0035] The method for generating a satellite behavior state diagram will be described in detail below with reference to the accompanying drawings.

[0036] See Figure 1 As shown, it is a flowchart of a method for generating a satellite behavior state diagram provided by an embodiment of the present application. The method for generating a satellite behavior state diagram may include the following S101 to S105: S101, obtain an image sequence for a target satellite.

[0037] Among them, the image sequence is obtained by observing the target satellite within a preset time period. Exemplarily, an image sequence may be obtained by continuously observing the target satellite within a preset time period, or by observing the target satellite at intervals within a preset time period, or by extracting frames from a continuous original image sequence, and specific limitations are not made.

[0038] Exemplarily, an on-board optical sensor (such as a visible light camera, a multispectral camera) or a ground remote sensing device may be used to observe a specified airspace to obtain the image sequence. The time interval between any two adjacent frames in the image sequence may be the same or different. That is, the interval between adjacent frames is adjustable to meet different time resolution requirements.

[0039] Among them, an on-board optical sensor is an instrument device installed on a satellite that observes and collects data on other celestial bodies (such as satellites) based on optical principles.

[0040] S102, for each frame of the image sequence, perform component target detection on the image to obtain a target detection result, and the target detection result includes a detection frame.

[0041] After obtaining the image sequence, each frame of the image sequence can be subjected to component object detection to obtain object detection results. Among them, component object detection refers to performing object detection on different components of the target satellite. The detection boxes in the detection results are used to indicate the bounding boxes of each component, and the component categories are used to indicate the names of different components (such as the main body, solar panels, antennas, payloads, etc.).

[0042] It can be understood that the detection box has coordinate information, and thus the size and position of the area where the component is located can be determined according to the coordinates of the detection box.

[0043] Exemplarily, a pre-trained object detection network can be used to implement component-level detection and semantic annotation for the target satellite, and output the bounding box coordinates and component categories of each component. Specifically, in this application, the existing object detection network can be improved. For example, the anchor box size and loss function can be optimized for satellite components to obtain the pre-trained object detection network.

[0044] Optionally, the object detection result may further include a confidence level, and the accuracy of the detection result can be measured through the confidence level. In addition, the detection accuracy and robustness of the object detection network in the embodiments of this application both meet the preset requirements. For example, it can be robust to complex perspectives and partial occlusions and can be applicable to various satellite forms.

[0045] S103, crop the image according to the regions indicated by each of the detection boxes to obtain component images, and use an image encoding network to extract feature vectors of the component images to obtain visual feature vectors.

[0046] Exemplarily, after performing component object detection on each frame of the image, the image can be cropped according to the detection results. Specifically, the image can be cropped according to the boundaries of each detection box to obtain the component images in each frame of the image. Then, the cropped component images can be fed into a pre-trained image encoding network (such as ResNet-50) to extract the visual feature vectors of the components, and the visual feature vectors can be used as the initial embedding representations of the subsequent scene graph nodes.

[0047] S104, use a spatio-temporal modeling model to construct spatio-temporal dependency relationships for each of the visual feature vectors to obtain spatio-temporal relationship dependency results; the spatio-temporal relationship dependency results include the spatial dependency relationships between components within the same frame of the image and the temporal dependency relationships between the same components between different frames of the image.

[0048] Among them, the dependency relationship refers to the association relationship between elements. The spatio-temporal modeling model can be a neural network. Specifically, the spatio-temporal modeling model can include a spatial encoding model and a temporal encoding model.

[0049] See Figure 2 As shown, for step S104, when constructing the spatio-temporal dependence relationship of each of the visual feature vectors using the spatio-temporal modeling model to obtain the spatio-temporal relationship dependence result, the following S1041~S1042 may be included: S1041, use the spatial encoding model to model the spatial relationship of each visual feature vector within the same frame of image to obtain the spatial dependence result.

[0050] Specifically, the spatio-temporal modeling model is a model based on the Transformer architecture, and the spatial encoding model is a spatial Transformer encoding model. When using the spatial encoding model to model the spatial relationship of each visual feature vector within the same frame of image, it may include: based on the corresponding spatial position encoding information of each visual feature vector, fusing the spatial position encoding information with the corresponding visual feature vector to generate a position-aware feature representation; inputting the position-aware feature representation into the spatial Transformer encoding model, and performing modeling in the spatial dimension through the multi-head self-attention mechanism to obtain the spatial dependence result.

[0051] Among them, the corresponding spatial position encoding information of each visual feature vector can be obtained through the coordinates of the detection box of the component image corresponding to it.

[0052] Here, the spatial Transformer encoding model can receive each visual feature vector and its corresponding spatial position encoding information (such as the center point coordinates of the bounding box), and then automatically mine the interdependence relationship between each satellite component (such as the satellite body, solar panel, communication antenna, payload, etc.) within each frame of image through the attention mechanism, so as to obtain the structural representation at the spatial level, that is, the node feature representation integrating spatial dependence information can be obtained. For example, there is a connection relationship between the solar panel and the satellite body. Another example is that the relationship between other different satellite components can be "attachment", "coordinated movement", "symmetric structure", etc. Here, using the spatial Transformer encoding model can effectively enhance the structural perception ability within the same image, and provide a more spatially consistent representation basis for subsequent time modeling and graph construction.

[0053] S1042, use the time encoding model to model the time relationship of the same component in the spatial dependence results corresponding to different frames of images to obtain the spatio-temporal relationship dependence result.

[0054] Exemplarily, the time encoding model can be a time Transformer encoding model. Therefore, when using the time encoding model to model the temporal relationship of the same component in the spatial dependence results corresponding to different images to obtain the spatio-temporal relationship dependence result, the following (I) to (II) can be included: (I) Arrange the visual feature vectors of the same satellite component in each spatial dependence result in chronological order to form a relationship sequence, generate temporal position encoding according to the temporal position information corresponding to each frame of image, and fuse the temporal position encoding into the relationship sequence.

[0055] (II) Use the multi-head self-attention mechanism of the time Transformer encoding model to model the relationship sequence fused with the temporal position encoding in the temporal dimension to obtain the spatio-temporal relationship dependence result.

[0056] Specifically, for each target component on the satellite, the visual feature vectors corresponding to it in each frame of image can be extracted, arranged in chronological order to form a temporal feature sequence, and after fusing each frame of visual feature vector with its corresponding temporal position encoding information, it is sent as input to the time Transformer encoding model. The time Transformer encoding model models the temporal feature sequence through its multi-head self-attention mechanism, so as to obtain the spatio-temporal dependence result across the temporal dimension.

[0057] Here, the spatial dependence results output by the spatial Transformer encoding model can be rearranged in time. That is, all the visual feature vectors of the same satellite component that appear in each spatial dependence result (one spatial dependence result corresponds to one frame of image) are combined in chronological order to form a relationship sequence, and then input to the time Transformer encoding model. The time Transformer encoding model encodes by introducing temporal position encoding, that is, combines the temporal relationship of each frame of image, and uses the self-attention mechanism to model the dynamic changes of each visual feature vector in the temporal dimension, so as to generate a fused feature representation with global temporal semantics and obtain the spatio-temporal relationship dependence result. Here, through this time Transformer encoding model, the state changes and the evolution of the mutual relationship of the satellite components over time can be captured, and thus key temporal semantic support can be provided for the generation of the subsequent behavior state diagram.

[0058] S105. Generate a behavior state diagram for the target satellite based on the spatio-temporal relationship dependence result.

[0059] After obtaining the spatio-temporal relationship dependency result, a behavior state diagram for the target satellite can be generated based on the spatio-temporal relationship dependency result. Among them, the behavior state can include attitude adjustment, component deployment, orbit change, etc. For example, a behavior state diagram for the target satellite can be generated based on the spatio-temporal relationship dependency result corresponding to the current frame and multiple frames of images before the current frame.

[0060] It can be understood that based on this spatio-temporal relationship dependency result, not only the current behavior state diagram of the target satellite can be generated, but also the future behavior state diagram of the target satellite can be predicted. That is, in some embodiments, the generating a behavior state diagram for the target satellite based on the spatio-temporal relationship dependency result may include: Predict the behavior state of the target satellite based on the spatio-temporal relationship dependency result, and generate a behavior state diagram for the target satellite according to the behavior state prediction result; wherein, the behavior state includes at least one of an attitude adjustment state, a component deployment state, and an orbit change state.

[0061] For example, if the current spatio-temporal relationship dependency result is the spatio-temporal relationship dependency result corresponding to three frames of images, the state of the target satellite after a preset time (such as 5 seconds) can be predicted based on this spatio-temporal relationship dependency result. Specifically, the prediction head can predict the future behavior state of the target satellite according to the spatio-temporal relationship dependency result, and obtain the behavior state diagram of the target satellite according to the behavior state prediction result. Exemplarily, the behavior state can include orbit maneuver, component deployment, attitude adjustment, abnormal collision risk, etc. Among them, orbit maneuver is a technical means for a satellite to actively change its motion trajectory through a propulsion system, mainly including orbit transition, parameter correction, and rendezvous and docking, etc.

[0062] In some embodiments, the behavior state diagram can be generated by several frames of images in an image sequence, and each frame of image corresponds to a scene graph. Preferably, to improve processing efficiency and model key behavior changes, several key frames can be selected for scene graph construction. For example, when the image sequence contains N frames of images, the behavior state diagram may contain M scene graphs (M ≤ N). Specifically, each frame of the scene graph includes nodes and connection edges between the nodes. The nodes are used to indicate the category of the satellite components and the state attribute information of the satellite components, and the connection edges are used to indicate the connection relationship between different components.

[0063] Optionally, the behavior state diagram further includes time edges, and the time edges are used to connect the nodes of the same satellite component in different frames of images. The time edges are used to describe the evolution of the satellite component state in the time dimension, and can specifically be used to indicate the state change of the same satellite component between adjacent scene graphs, such as indicating "the deployment angle of the solar panel increases" or "extending the camera payload", for supporting temporal modeling and behavior recognition.

[0064] Here, the state of each satellite component can be predicted through a behavior decoder (such as attitude angle changes, whether to deploy / retract, etc.), while the structural and functional relationships of the same satellite component between adjacent frame images are identified, and then a behavior state diagram for the target satellite is generated.

[0065] See Figure 3 as shown Figure 3 FIG. is a schematic diagram of a behavior state diagram of a target satellite provided by an embodiment of the present application. The behavior state diagram is used to show the evolution process of the structural state and component relationship of the target satellite over time at different observation times. The behavior state diagram is composed of scene diagrams corresponding to three consecutive moments, and the three scene diagrams respectively represent the dynamic structural features of the target satellite at three key node moments in the observation sequence.

[0066] Specifically, each frame of the scene diagram is constructed by combining the image frame after target detection with the spatial dependence result output by the spatial Transformer encoding model. In each scene diagram, the nodes represent the category of the target satellite component and the state attribute information of the component, and the connecting edges represent the spatial dependence relationship and functional cooperation relationship between the components. In the embodiment of the present application, the nodes include the satellite body, the left solar panel, the right solar panel, and the camera payload component, and the connecting edges reflect relationship attributes such as "attachment", "coordinated movement", and "symmetric structure".

[0067] Among them, the first frame of the scene diagram a reflects the structural layout in the initial state, where the deployment angles of the left and right solar panels are 35°, and they are not fully deployed. When entering the moment shown in the second frame of the scene diagram b, the left and right solar panels have been fully deployed, and the deployment angle is 42°, and the "deployment angle" feature in the state attributes of the corresponding nodes is significantly improved. The third frame of the scene diagram c represents that the target satellite extends the external camera payload. At this time, a new "camera payload" node is added to the scene diagram, and a connecting edge indicating an "attachment" relationship is established with the satellite body. A new "operation state" attribute is added to the state attributes of the nodes to reflect its current active state.

[0068] In the embodiment of the present application, by generating the behavior state diagram of the target satellite, not only can the evolution process of the key structures of the target satellite during the task execution be intuitively displayed, but also it can provide a reference for subsequent behavior understanding and task prediction, which is beneficial to improving the real-time performance and accuracy of space situation awareness and provides reliable technical support for application scenarios such as on-orbit service and space traffic management.

[0069] The following will describe the specific implementation process of the satellite behavior state diagram generation method provided by the embodiment of the present application with reference to the accompanying drawings.

[0070] See Figure 4As shown in the figure, it is a schematic flowchart of a method for generating a satellite behavior state diagram provided by an embodiment of the present application. First, a continuous visible light image sequence of the target airspace is obtained through an optical imaging payload carried on the satellite platform. The observation duration is 30 seconds, the frame rate is 10 Hz, and the image resolution is set to 2048×2048 pixels.

[0071] Next, a target detection model that is pre-trained and optimized for the characteristics of orbital targets is used to detect and identify satellite components frame by frame in the enhanced image sequence. The input size of the detection model is set to 1024×1024 pixels, and the output includes the bounding box positions, class labels, and detection confidence levels of components such as the satellite body, solar panels, and communication antennas. Exemplarily, the detection results include: the confidence level of the satellite body is 98.2%, and the confidence levels of the left and right solar panels are 92.7% and 89.3% respectively.

[0072] Subsequently, each detected target area is used as a local perception unit and input into an image encoding network (such as ResNet-50) to extract the image semantic features of each component to form node embeddings. After obtaining the node features, the connection graph between components can be initialized, the spatial adjacency relationship between components can be constructed, and the graph structure together with the node embedding representation is input into the spatial Transformer encoding model. This spatial Transformer encoding model models the spatial dependence relationship and structural coupling characteristics between components through the multi-head self-attention mechanism, thereby generating a node feature representation containing spatial semantics (i.e., the spatial dependence result).

[0073] Furthermore, the spatial dependence results in different time frames are serialized into time series embeddings and input into the time Transformer encoding model to learn the dynamic change trend of component states in the time dimension. By introducing time position encoding information, this Transformer encoding model can extract temporal behavior patterns such as the deployment of solar panels and the trend of attitude adjustment, such as: the right solar panel is deployed, and finally the spatio-temporal relationship dependence result is obtained.

[0074] After obtaining the spatio-temporal relationship dependence result, a behavior decoder (such as an MLP) can be used to predict the dynamic behavior indicators of the target satellite within the next 5 seconds based on the node evolution state at the current moment and within the past 3 seconds (30 frames). Exemplarily, the prediction results can include the attitude angle change Δθ = 1.2° ± 0.3°, the orbital radius change Δr, the probability of the right solar panel deployment increasing to 96.8%, etc.

[0075] Based on the above prediction results, the behavior state diagram can be updated, timestamp information (such as t = 2023-05-20T08:15:30.125Z) can be marked, and the component node attributes can be synchronously modified, such as the deployment angle of the left solar panel is updated from 35° to 42°, so as to reflect the current state and its evolution trend of the target satellite.

[0076] It should be noted that each network structure, model parameter, input and output format in this embodiment can be adjusted and replaced according to specific task requirements, such as replacing the target detection algorithm, adjusting the number of layers of the Transformer network structure, introducing multi-source sensor data, etc.

[0077] Those skilled in the art can understand that in the above method of the specific implementation manner, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined according to its function and possible internal logic.

[0078] Based on the same inventive concept, an apparatus for generating a satellite behavior state diagram corresponding to the method for generating a satellite behavior state diagram is further provided in the embodiments of the present disclosure. Since the principle of solving problems by the apparatus in the embodiments of the present disclosure is similar to the above method for generating a satellite behavior state diagram in the embodiments of the present disclosure, the implementation of the apparatus can refer to the implementation of the method, and the repeated parts will not be described again.

[0079] Referring to Figure 5 As shown, it is a schematic diagram of an apparatus for generating a satellite behavior state diagram provided by an embodiment of the present disclosure. The satellite behavior state diagram generating apparatus 500 includes: An image acquisition module 501, configured to acquire an image sequence of a target satellite; A target detection module 502, configured to perform component target detection on each frame of the image sequence to obtain a target detection result, where the target detection result includes a detection frame; A feature extraction module 503, configured to crop each frame of the image according to the regions indicated by each detection frame to obtain a component image, and use an image encoding network to extract features from the component image to obtain a visual feature vector; A spatio-temporal modeling module 504, configured to construct a spatio-temporal dependence relationship between each visual feature vector by using a spatio-temporal modeling model to obtain a spatio-temporal relationship dependence result; the spatio-temporal relationship dependence result includes a spatial dependence relationship between components within the same frame of image and a temporal dependence relationship between the same component in different frames of image; A state generation module 505, configured to generate a behavior state diagram of the target satellite based on the spatio-temporal relationship dependence result.

[0080] In some possible embodiments, the spatio-temporal modeling model includes a spatial encoding model and a temporal encoding model; specifically, the spatio-temporal modeling module 504 is configured to: Use the spatial encoding model to model the spatial relationship between each visual feature vector within the same frame of image to obtain a spatial dependence result; Model the temporal relationship of the same component in the spatial dependence results respectively corresponding to different frame images by using the temporal encoding model, so as to obtain the spatio-temporal relationship dependence result.

[0081] In some possible embodiments, the spatio-temporal modeling model is a model based on the Transformer architecture, and the spatial encoding model is a spatial Transformer encoding model; specifically, the spatio-temporal modeling module 504 is configured to: Based on the spatial position encoding information corresponding to each visual feature vector, fuse the spatial position encoding information with the corresponding visual feature vector to generate a position-aware feature representation. Input the position-aware feature representation into the spatial Transformer encoding model, and perform modeling in the spatial dimension through the multi-head self-attention mechanism to obtain the spatial dependence result.

[0082] In some possible embodiments, the temporal encoding model is a temporal Transformer encoding model; specifically, the spatio-temporal modeling module 504 is configured to: Arrange the visual feature vectors of the same satellite component in the spatial dependence results in chronological order to form a relationship sequence, generate temporal position encoding according to the temporal position information corresponding to each frame image, and fuse the temporal position encoding into the relationship sequence. Use the multi-head self-attention mechanism of the temporal Transformer encoding model to perform modeling in the temporal dimension on the relationship sequence fused with the temporal position encoding, so as to obtain the spatio-temporal relationship dependence result.

[0083] In some possible embodiments, the state generation module 505 is specifically configured to: Predict the behavior state of the target satellite based on the spatio-temporal relationship dependence result, and generate a behavior state map for the target satellite.

[0084] In some possible embodiments, the behavior state map includes multiple frames of scene graphs, and each frame of the scene graph includes nodes and connection edges between the nodes. The nodes are used to indicate the category of the satellite component and the state attribute information of the satellite component, and the connection edges are used to indicate the spatial connection relationship between different components.

[0085] In some possible embodiments, the behavior state map further includes temporal edges, and the temporal edges are used to indicate the state changes of the same satellite component between adjacent scene graphs.

[0086] The description of the processing flow of each module in the device and the interaction flow between modules can refer to the relevant description in the above method embodiments, and will not be elaborated here.

[0087] Based on the same inventive concept, embodiments of the present disclosure also provide an electronic device. Referring to Figure 6 As shown, it is a schematic structural diagram of an electronic device 600 provided by an embodiment of the present disclosure, including a processor 601, a memory 602, and a bus 603. Among them, the memory 602 is used to store execution instructions, including an internal memory 6021 and an external memory 6022; the internal memory 6021 here is also called the main memory, which is used to temporarily store the operation data in the processor 601 and the data exchanged with the external memory 6022 such as a hard disk, and the processor 601 exchanges data with the external memory 6022 through the internal memory 6021.

[0088] In an embodiment of the present application, the memory 602 is specifically used to store the application program code for implementing the solution of the present application, and is controlled by the processor 601 to execute. That is, when the electronic device 600 runs, the processor 601 communicates with the memory 602 through the bus 603, so that the processor 601 executes the application program code stored in the memory 602, and further executes the method described in any of the foregoing embodiments.

[0089] Among them, the memory 602 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read - only memory (PROM), an erasable programmable read - only memory (EPROM), an electrically erasable programmable read - only memory (EEPROM), etc.

[0090] The processor 601 may be an integrated circuit chip with the ability to process signals. The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0091] It can be understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the electronic device 600. In other embodiments of this application, the electronic device 600 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0092] The embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the satellite behavior state diagram generation method in the above method embodiments. Among them, the storage medium may be a volatile or non-volatile computer-readable storage medium.

[0093] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the satellite behavior state diagram generation method in the above method embodiments. For details, please refer to the above method embodiments and will not be elaborated here.

[0094] Among them, the above computer program product can be specifically implemented in a manner of hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0095] In addition, embodiments of the subject matter and the functional operations described in this specification can be implemented in: digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or one or more combinations of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier to be executed by, or to control the operation of, a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode and transmit information to the appropriate receiver apparatus for execution by the data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0096] The processes and logical flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logical flows can also be performed by, or the apparatus can be implemented as, special purpose logic circuitry, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0097] Suitable computers for executing computer programs include, by way of example, general and / or special purpose microprocessors, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory and / or a random access memory. Basic elements of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, etc., or the computer will be operatively coupled to such mass storage devices to receive data therefrom or to transfer data thereto, or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive, to name just a few.

[0098] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memories may be supplemented by, or incorporated in, special purpose logic circuitry.

[0099] Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of what is claimed, but rather as mainly describing the features of specific embodiments of particular inventions. Certain features that are described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Additionally, although features may operate in certain combinations as described above and even be claimed as such initially, one or more features from a claimed combination may in some cases be removed from the combination, and the claimed combination may be directed to a sub-combination or a variation of a sub-combination.

[0100] Similarly, although operations are depicted in the drawings in a particular order, this should not be understood to require that the operations be performed in the particular order shown or sequentially, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system modules and components in the above embodiments should not be understood to require such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0101] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the acts recited in the claims may be performed in a different order and still achieve the desired result. Additionally, the processes depicted in the drawings are not necessarily in the particular order or sequential order shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0102] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for generating a satellite behavior state diagram, characterized in that Including: Obtain an image sequence for a target satellite; For each frame of the image sequence, perform component target detection on the image to obtain a target detection result, where the target detection result includes detection frames; Crop each frame of the image according to the regions indicated by the respective detection frames to obtain component images, and use an image encoding network to extract features from the component images to obtain visual feature vectors; Use a spatio-temporal modeling model to construct spatio-temporal dependence relationships for the respective visual feature vectors to obtain a spatio-temporal relationship dependence result; the spatio-temporal relationship dependence result includes the spatial dependence relationship between components within the same frame of image and the temporal dependence relationship between the same component in different frames of images; Generate a behavior state map for the target satellite based on the spatio-temporal relationship dependence result.

2. The method according to claim 1, characterized in that, The spatio-temporal modeling model includes a spatial encoding model and a temporal encoding model; the use of the spatio-temporal modeling model to construct spatio-temporal dependence relationships for the respective visual feature vectors to obtain a spatio-temporal relationship dependence result includes: Use the spatial encoding model to model the spatial relationships of the respective visual feature vectors within the same frame of image to obtain a spatial dependence result; Use the temporal encoding model to model the temporal relationships of the same component in the spatial dependence results corresponding to different frames of images to obtain the spatio-temporal relationship dependence result.

3. The method according to claim 2, wherein The spatio-temporal modeling model is a model based on the Transformer architecture, and the spatial encoding model is a spatial Transformer encoding model; The use of the spatial encoding model to model the spatial relationships of the respective visual feature vectors within the same frame of image to obtain a spatial dependence result includes: Based on the corresponding spatial position encoding information of each visual feature vector, fuse the spatial position encoding information with the corresponding visual feature vector to generate a position-aware feature representation; Input the position-aware feature representation into the spatial Transformer encoding model, and perform modeling in the spatial dimension through the multi-head self-attention mechanism to obtain the spatial dependence result.

4. The method according to claim 2 or 3, characterized in that, The temporal encoding model is a temporal Transformer encoding model; the use of the temporal encoding model to model the temporal relationships of the same component in the spatial dependence results corresponding to different images to obtain the spatio-temporal relationship dependence result includes: Arrange the visual feature vectors of the same satellite component in the respective spatial dependence results in chronological order to form a relationship sequence, generate temporal position encoding according to the temporal position information corresponding to each frame of image, and fuse the temporal position encoding into the relationship sequence; Use the multi-head self-attention mechanism of the temporal Transformer encoding model to perform modeling in the temporal dimension on the relationship sequence fused with the temporal position encoding to obtain the spatio-temporal relationship dependence result.

5. The method according to claim 1, characterized in that The generating of a behavior state map for the target satellite based on the spatio-temporal relationship dependence result includes: Predict the behavior state of the target satellite based on the spatio-temporal relationship dependence result; the behavior state includes at least one of an attitude adjustment state, a component deployment state, and an orbit change state; Generate a behavior status graph for the target satellite according to the behavior status prediction result.

6. The method according to claim 1 or 5, characterized in that, The target detection result also includes the categories of the detected satellite components. The behavior status graph includes multiple frames of scene graphs. Each frame of the scene graph includes nodes and connection edges between the nodes. The nodes are used to indicate the categories of the satellite components and the status attribute information of the satellite components, and the connection edges are used to indicate the spatial connection relationships between different components.

7. The method according to claim 6, characterized in that, The behavior status graph further includes time edges, which are used to indicate the status changes of the same satellite component between adjacent scene graphs.

8. A satellite behavior state diagram generation device, characterized in that The device includes: An image acquisition module, configured to acquire an image sequence of a target satellite. A target detection module, configured to perform component target detection on each frame of the image sequence to obtain a target detection result, where the target detection result includes detection frames. A feature extraction module, configured to crop each frame of the image according to the regions indicated by the respective detection frames to obtain component images, and use an image encoding network to extract features from the component images to obtain visual feature vectors. A spatio-temporal modeling module, configured to construct spatio-temporal dependency relationships for the respective visual feature vectors by using a spatio-temporal modeling model to obtain a spatio-temporal relationship dependency result; the spatio-temporal relationship dependency result includes the spatial dependency relationships between the components within the same frame of image and the temporal dependency relationships between the same component in different frames of image. A status generation module, configured to generate a behavior status graph for the target satellite based on the spatio-temporal relationship dependency result.

9. An electronic device, characterized in that, Includes: A processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the satellite behavior status graph generation method according to any one of claims 1-7 is executed.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is run by the processor, the satellite behavior status graph generation method according to any one of claims 1-7 is executed.

Images