Satellite backplane component identification method based on time sequence relationship

Abstract

The application discloses a satellite backboard component recognition method based on a time sequence relation and belongs to the technical field of space. The application comprises the following steps: performing sequential processing on collected images, constructing an image feature model and a time sequence feature model of the sequential images, and training the image feature model and the time sequence feature model, so as to finally realize the satellite backboard component recognition method based on the time sequence relation. Compared with a general satellite backboard component recognition method based on a convolutional neural network, the method of the application improves the robustness of the extracted features to a changeable environment and a changeable threshold, is helpful to improve the recognition rate of a model to satellite backboard images collected by a hand-eye camera in a motion process, and has practical engineering significance.

Description

TECHNICAL FIELD

[0001] The present application relates to a satellite backboard component identification method based on a time sequence relationship, and belongs to the technical field of space. BACKGROUND

[0002] The task background of the present application is image perception for intelligent on-orbit service operation. On-orbit service operation refers to the process of using a mechanical arm carried by a service spacecraft to perform final stage operation on a target spacecraft after the service spacecraft and the target spacecraft complete rendezvous and docking. It includes on-orbit fault repair, module replacement, fuel refueling, and auxiliary de-orbiting of spacecraft at the end of their life or in failure. On-orbit service operation plays an important role in reducing space transportation costs and mission risks, and has great significance for improving the efficiency of space assets and ensuring the safe and reliable operation of spacecraft in orbit.

[0003] On-orbit service operation is changing towards a fully autonomous space mode, and in the future, spacecraft will autonomously and intelligently complete on-orbit service operation tasks. For future intelligent on-orbit service operation, the image perception system needs to autonomously perceive important targets in complex space environments. Traditional image perception methods rely on expert-designed features and have low intelligence. Although image perception methods based on convolutional neural networks can autonomously perceive targets with complex shapes and have certain robustness and generalization, they rely on a large number of effective training samples and lack logical reasoning ability.

[0004] The targets in the images collected during on-orbit service operation are partially occluded, the scenes in which the targets are located are changing, and the targets are imaged at different angles. The target recognition method based on convolutional neural networks does not work well in the above-mentioned scenarios. SUMMARY

[0005] The technical problem solved by the present application is to overcome the shortcomings of the prior art and provide a satellite backboard component identification method based on a time sequence relationship. The method achieves satellite backboard component identification based on a time sequence relationship by sequentially processing collected images, constructing, training, and testing image feature models and time sequence feature models.

[0006] The technical solution of the present application is a satellite backboard component identification method based on a time sequence relationship, comprising:

[0007] A hand-eye camera is used to continuously collect satellite backboard image data during on-orbit service operation, and the collected images are sequentially processed to obtain processed labeled sequence test images and real-time collected unlabeled sequence test images;

[0008] An image feature model and a time sequence feature model are constructed;

[0009] The image feature model and the time sequence feature model are trained.

[0010] The processed labeled sequence test image or the unlabeled sequence test image collected in real time is sent into the defined image feature model and the timing feature model trained to realize end-to-end satellite backboard component identification.

[0011] Further, the satellite backboard image data in the on-orbit service operation is continuously collected by using the hand-eye camera, and the collected image is sequentially processed, specifically including:

[0012] Satellite backboard image data in the on-orbit service operation is continuously collected;

[0013] The collected image is sequentially processed, specifically including: according to the hyperparameter T, the collected image set is divided into N sequence image sets with a size of T+1 in time sequence, T represents the previous T frames of images of the current frame; for the training data set, the divided image set is Wherein, N represents the number of sequences of the training sample, N l represents the number of the lth sample, k represents the number of sample categories, is the ith training sequence, contains T+1 frames of images and the category label y i of the current frame image, and the current frame image is while is the previous T frames of images; for the test data set, the divided image set is Wherein, M represents the number of sequences of the test sample, M l represents the number of the lth sample, k represents the number of sample categories, is the ith training sequence, contains T+1 frames of images and the category label y i of the current frame image, and the current frame image is while is the previous T frames of images, contains the same number of images as .

[0014] Further, the sequential processing of the collected image specifically includes:

[0015] Setting the hyperparameter T;

[0016] According to the forward time sequence, the image set is divided according to T+1 intervals, and each divided sequence subset contains the current frame image and the previous T frames of images T+1 frames of images, and the category y i of the target in the current frame image, that is

[0017] Further, the constructing the image feature model and the time sequence feature model specifically comprises:

[0018] constructing the image feature extraction model based on the convolutional neural network based on the ResNet50 neural network architecture;

[0019] constructing the time sequence graph based on the sequence images;

[0020] constructing the time sequence feature extraction model based on the graph convolutional neural network based on the time sequence graph and the graph convolutional neural network.

[0021] Further, a 2048-dimensional feature vector is used to represent the image features of the image.

[0022] Further, the constructing the time sequence graph based on the sequence images specifically comprises:

[0023] constructing the time sequence graph containing T+1 nodes based on the current frame image and the previous T frame images;

[0024] initializing the node features of the time sequence graph by using the image features of the sequence images extracted by the ResNet50 neural network architecture;

[0025] constructing the edges of the time sequence graph according to the time sequence relationship of the current frame image and the previous T frame images.

[0026] Further, the constructing the time sequence feature extraction model based on the graph convolutional neural network based on the time sequence graph and the graph convolutional neural network comprises two graph convolutional layers and a series of hierarchically connected fully connected layers, batch normalization layers and nonlinear activation layers.

[0027] Further, the constructing the image feature model and the time sequence feature model constructs the image feature model and the time sequence feature model in series connection.

[0028] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to realize the steps of the satellite backboard component identification method based on the time sequence relationship.

[0029] A satellite backboard component identification device based on the time sequence relationship, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor executes the computer program to realize the steps of the satellite backboard component identification method based on the time sequence relationship.

[0030] Compared with the prior art, the present application has the following advantages:

[0031] (1) In the case of occlusion, multi-angle imaging and changing scenes, the features of the current frame image extracted by the image feature model and the time sequence feature model have stronger robustness.

[0032] (2) When modeling the book sequence feature, the prior sequence knowledge is introduced in the form of a time sequence graph, improving the efficiency of modeling the time sequence feature.

[0033] (3) The recognition effect of the method constructed by the present application is not affected by the threshold value, and at a high threshold value, there is still a high recognition effect. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is the principle diagram of the method of the present application;

[0035] Figure 2 is the time sequence graph constructed in step 2.2 of the present application;

[0036] Figure 3 is the time sequence feature extraction model based on the graph convolutional neural network constructed in step 2.3 of the present application. DETAILED DESCRIPTION

[0037] In order to better understand the above technical solutions, the technical solutions of the present application will be described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solutions of the present application, and are not limitations of the technical solutions of the present application. In the case of no conflict, the technical features in the embodiments of the present application and the embodiments can be combined with each other.

[0038] The satellite backboard component recognition method based on time sequence relationship provided by the embodiments of the present application will be described in further detail below in combination with the accompanying drawings of the specification. The specific implementation manner can include (as shown in Figures 1-3 ).

[0039] Step 1, a hand-eye camera is used to continuously collect satellite backboard image data in on-orbit service operation, and the collected images are sequentially processed. The hand-eye camera is configured at the end of the mechanical arm, and moves together with the mechanical arm in the on-orbit service operation, and real-time collects the satellite backboard image of the current scene; the collected images meet the space-time continuity condition between adjacent frames, that is, meet certain space-time constraints, and there will be no large jump; the collected images are sequentially processed according to time sequence, which means that according to the hyperparameter T, the collected image set is divided into N sequence image sets with a size of T+1 according to time sequence. Overall, the purpose of this step is to provide data meeting the requirements for training and testing of the image feature model and the time sequence feature model.

[0040] Step 2, constructing image feature model and temporal feature model. The image feature model is used to extract the image features of the sequence images, while the temporal feature model is used to extract the temporal features of the sequence images. The purpose of this step is to construct reasonable feature models to extract the image features and temporal features in the images, and make the extracted features still robust in the changing environment.

[0041] Step 3, training the image feature model and the temporal feature model. The labeled sequence training images in step 1 are sent into the network defined in step 2, and the image feature model and the temporal feature model are optimized according to the gradient descent method. The purpose of this step is to make the model learn the optimal parameters of the image feature and temporal feature extraction model from the sequence training data set.

[0042] Step 4, testing the satellite backboard component recognition method based on temporal relationship. The labeled sequence test images in step 1 or the unlabeled sequence test images collected in real time are sent into the image feature model and the temporal feature model trained in step 3, and the end-to-end satellite backboard component recognition is realized. The purpose of this step is to test or use the optimized model to recognize the satellite backboard components.

[0043] Further, in one possible implementation, step 1 uses a hand-eye camera to continuously collect satellite backboard image data in on-orbit service operation, and the collected images are sequentially processed including the following steps:

[0044] Step 1.1, continuously collecting satellite backboard image data in on-orbit service operation. A hand-eye camera configured at the end of a mechanical arm is used to collect real-time satellite backboard images of the current scene, wherein the hand-eye camera is continuously moving with the mechanical arm, and the collected images meet the spatiotemporal continuity condition between adjacent frames, i.e. meet certain spatiotemporal constraints, and there is no large jump.

[0045] Step 1.2, sequentially processing the collected images. Referring to the hyperparameter T, the collected image set is divided into N sequence image sets with a size of T+1 in chronological order, T represents the previous T frame images of the current frame. For the training data set, the divided image set is wherein N represents the number of sequences of the training samples, N l represents the number of the l-th class of samples, k represents the number of sample categories, is the i-th training sequence, contains T+1 frame images and the category label y i of the current frame image, and are the previous T frame images. For the test data set, the divided image set is Wherein, M represents the number of sequences of test samples, M l represents the number of the first sample, k represents the number of sample categories, is the i-th training sequence, contains the category label y i of the T+1 frame images and the current frame image, and the current frame image is while is the previous T frame image, contains the same number of images as .

[0046] In one possible implementation, step 1.2 performs sequence processing on the collected images, including the following steps:

[0047] Step 1.2.1, set the hyperparameter T;

[0048] Step 1.2.2, according to the forward time sequence, divide the image set according to T+1 intervals, and each segmented sequence subset contains the current frame image and the previous T frame image T+1 frame images, and the category y i of the target in the current frame image, that is

[0049] Further, step 2 constructs an image feature model and a time sequence feature model, including the following steps:

[0050] Step 2.1, construct an image feature extraction model based on a convolutional neural network based on ResNet50;

[0051] Step 2.2, construct a time sequence graph based on sequence images;

[0052] Step 2.3, construct a time sequence feature extraction model based on a graph convolutional neural network based on the time sequence graph and the graph convolutional neural network.

[0053] In one possible implementation, the image feature extraction model based on the convolutional neural network is constructed based on ResNet50, and a 2048-dimensional feature vector is used to abstractly represent the image features of the image.

[0054] In one possible implementation, step 2.2 constructs a time sequence graph based on sequence images, including the following steps:

[0055] 2.2.1, construct a time sequence graph containing T+1 nodes with the current frame image and the previous T frame image;

[0056] 2.2.2, initialize the node features of the time sequence graph using the image features of the sequence image extracted by step 2.1 ResNet50;

[0057] 2.2.3, constructing edges of the temporal graph according to the temporal relationship between the current frame image and the previous T frame images.

[0058] In a possible implementation manner, the step 2.3 constructs the graph convolutional neural network-based temporal feature extraction model based on the temporal graph and the graph convolutional neural network, and the graph convolutional neural network-based temporal feature extraction model comprises two graph convolutional layers and a series of hierarchically connected fully connected layers, batch normalization layers and nonlinear activation layers.

[0059] In a possible implementation manner, the step 2 constructs the image feature model and the temporal feature model, and the image feature model and the temporal feature model are serially connected.

[0060] In a possible implementation manner, the step 2 constructs the image feature model and the temporal feature model, and the temporal feature model can correct the feature of the current frame image by using the image feature and the temporal feature of the previous T frame images, so that the corrected feature is more robust to the transformed environment.

[0061] In a possible implementation manner, the step 3 trains the image feature model and the temporal feature model, and the sequence image is input into the model when the image feature model and the temporal feature model are trained.

[0062] In a possible implementation manner, the step 3 trains the image feature model and the temporal feature model, and the image feature model and the temporal feature model are end-to-end simultaneously trained.

[0063] In a possible implementation manner, the step 3 trains the image feature model and the temporal feature model, and the output loss of the current frame graph node is used when the network parameter optimization based on the gradient descent method is performed.

[0064] In a possible implementation manner, the step 4 tests the satellite backboard component recognition method based on the temporal relationship, and the image feature extracted by the image feature extraction model is serialized, and the serialized image feature

[0065] In a possible implementation manner, the step 4 tests the satellite backboard component recognition method based on the temporal relationship, and the feature extracted by the final network contains the information of the current frame image and the historical frame image.

[0066] In the scheme provided in the embodiments of the present application, as shown in Figure 1 a spatial fine operation offline reinforcement learning method comprises the following steps:

[0067] Step 1, continuously collect satellite backplane image data in on-orbit service operation by using a hand-eye camera, and sequentially process the collected images. The hand-eye camera is configured at the end of the mechanical arm and moves with the mechanical arm in on-orbit service operation, and real-time satellite backplane images of the current scene are collected; the collected images meet the spatiotemporal continuity condition between adjacent frames, that is, they meet certain spatiotemporal constraints and do not jump greatly; the collected images are sequentially processed in time sequence, that is, the collected image set is divided into N sequence image sets with a size of T+1 in time sequence according to the hyperparameter T. Overall, the purpose of this step is to provide data meeting the requirements for training and testing of image feature models and time sequence feature models.

[0068] Further, the specific steps of step 1 are as follows:

[0069] Step 1.1, continuously collect satellite backplane image data in on-orbit service operation. Real-time satellite backplane images of the current scene are collected by using a hand-eye camera configured at the end of the mechanical arm, wherein the hand-eye camera continuously moves with the mechanical arm, and the collected images meet the spatiotemporal continuity condition between adjacent frames, that is, they meet certain spatiotemporal constraints and do not jump greatly.

[0070] Step 1.2, sequentially process the collected images. That is, the collected image set is divided into N sequence image sets with a size of T+1 in time sequence according to the hyperparameter T, and T represents the previous T frame images of the current frame. For the training data set, the divided image set is wherein N represents the number of sequence of the training sample, N l represents the number of the l-th sample, k represents the number of sample categories, is the i-th training sequence, the category label y i of the T+1 frame images and the current frame image, the current frame image is and the previous T frame images. For the test data set, the divided image set is wherein M represents the number of sequence of the test sample, M l represents the number of the l-th sample, k represents the number of sample categories, is the i-th training sequence, the category label y i of the T+1 frame images and the current frame image, the current frame image is and the previous T frame images, the number of images contained is the same as .

[0071] Further, step 1.2 serializes the collected images, and the specific steps are as follows:

[0072] Step 1.2.1, set the hyperparameter T;

[0073] Step 1.2.2, according to the forward time sequence, divide the image set into T+1 intervals, and each segmented sequence subset contains the current frame image and the previous T frame images T+1 frame images, and the category y of the target in the current frame image i That is

[0074] Step 2, construct the image feature model and the time sequence feature model. The image feature model is used to extract the image features of the sequence image, and the time sequence feature model is used to extract the time sequence features of the sequence image. The purpose of this step is to construct a reasonable feature model to extract the image features and time sequence features in the image, and make the extracted features still robust in the changing environment.

[0075] Further, the specific steps of step 2 are as follows:

[0076] Step 2.1, construct a convolutional neural network-based image feature extraction model based on ResNet50;

[0077] Step 2.2, construct a time sequence graph based on the sequence image;

[0078] Step 2.3, construct a graph convolutional neural network-based time sequence feature extraction model based on the time sequence graph and the graph convolutional neural network.

[0079] Further, step 2.2 constructs a time sequence graph based on the sequence image, and the specific steps are as follows:

[0080] 2.2.1, construct a time sequence graph containing T+1 nodes with the current frame image and the previous T frame images;

[0081] 2.2.2, initialize the node features of the time sequence graph using the image features of the sequence image extracted by step 2.1 ResNet50;

[0082] 2.2.3, construct the edges of the time sequence graph according to the time sequence relationship of the current frame image and the previous T frame images.

[0083] Step 3, train the image feature model and the time sequence feature model. The labeled sequence training images in step 1 are sent into the network defined in step 2, and the image feature model and the time sequence feature model are optimized according to the gradient descent method. The purpose of this step is to make the model learn the optimal parameters of the image feature and time sequence feature extraction model from the sequence training data set.

[0084] Step 4, test the satellite backboard component recognition method based on the time sequence relationship. The labeled sequence test image in step 1 or the unlabeled sequence test image collected in real time is input into the image feature model and the time sequence feature model defined in step 2 trained in step 3 to realize end-to-end satellite backboard component recognition. The purpose of this step is to test or use the optimized model to recognize the satellite backboard components.

[0085] As shown in Figure 2 , when the hyperparameter T is 3, the time sequence graph is constructed. The nodes of the time sequence graph are the current frame image and the previous three frame images, and the initial features of the nodes are the image features extracted by the image feature extraction model based on the convolutional neural network, as shown in Figure 1 . The previous three frame images are all related to the current frame image in the time sequence relationship.

[0086] As shown in Figure 3 , the time sequence feature extraction model based on the graph convolutional neural network constructed in step 2.3 of the present application corresponds to the GCN module in Figure 1 . The model includes two graph convolutional layers and a series of hierarchically connected fully connected layers, batch normalization layers and nonlinear activation layers. The input of the model is the time sequence graph, as shown in Figure 2 . The model can extract new features of the current frame node containing image features and time sequence features , as shown in Figure 1 .

[0087] The present application improves the robustness of the features extracted by the feature model to the changing environment and the changing threshold, which helps to improve the recognition rate of the satellite backboard image collected by the hand-eye camera during the movement.

[0088] The present application provides a computer readable storage medium, which stores computer instructions, when the computer instructions run on a computer, make the computer execute Figure 1 the method.

[0089] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program codes.

[0090] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0091] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0092] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks ​ one or more flow or blocks

[0093] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the

[0094] Those skilled in the art will appreciate that the application described herein is susceptible to variations and modifications other than those specifically described. It is to be understood that the application includes all such variations and modifications which fall within the spirit and scope of the present application.

Claims

1. A satellite backplane component identification method based on timing relationship, characterized in that, The application relates to a satellite backboard component recognition method based on a convolutional neural network. Satellite backboard image data in on-orbit service operation is continuously collected by a hand-eye camera, and the collected images are sequentially processed to obtain processed labeled sequence test images and real-time collected unlabeled sequence test images; An image feature model and a time sequence feature model are constructed; The image feature model and the time sequence feature model are trained; The processed labeled sequence test images or the real-time collected unlabeled sequence test images are sent into the trained image feature model and the time sequence feature model to realize end-to-end satellite backboard component recognition. The satellite backboard image data in on-orbit service operation is continuously collected by the hand-eye camera, and the collected images are sequentially processed, and the specific process comprises the following steps. Satellite backboard image data in on-orbit service operation is continuously collected. The collected images are sequentially processed, specifically including: according to the hyperparameter T, the collected image set is divided into N sequence image sets with a size of T+1 in time sequence, T represents the previous T frame images of the current frame; for the training data set, the segmented image set is Wherein, N represents the sequence number of the training sample, N l represents the number of the lth class sample, k represents the sample category number, is the ith training sequence, the category label y i of the T+1 frame images and the current frame image, the current frame image is and the previous T frame images; for the test data set, the segmented image set is Wherein, M represents the sequence number of the test sample, M l represents the number of the lth class sample, k represents the sample category number, is the ith test sequence, the category label y i of the T+1 frame images and the current frame image, the current frame image is and the previous T frame images, the number of images contained is the same as . The image feature model and the time sequence feature model are constructed, and the specific process comprises the following steps. An image feature extraction model based on a convolutional neural network is constructed based on a ResNet50 neural network architecture. A time sequence graph is constructed based on sequence images. A time sequence feature extraction model based on a graph convolutional neural network is constructed based on the time sequence graph and the graph convolutional neural network. The time sequence graph is constructed based on sequence images, and the specific process comprises the following steps. A time sequence graph containing T+1 nodes is constructed by using a current frame image and T previous frame images. The node features of the time sequence graph are initialized by using the image features of the sequence images extracted by the ResNet50 neural network architecture. The edges of the time sequence graph are constructed according to the time sequence relationship of the current frame image and the T previous frame images. The image feature model and the time sequence feature model are constructed in series.

2. The method of claim 1, wherein: The collected images are sequentially processed, and the specific process comprises the following steps. A hyperparameter T is set. According to the forward time sequence, the image set is divided according to T+1 intervals, and each segmented sequence subset contains the current frame image and the previous T frame images T+1 frame images, and the category y in which the target is located in the current frame image i That is 3. The method of claim 1, wherein: A 2048-dimensional feature vector is used to represent the image features of the images.

4. The method of claim 1, wherein, The time sequence feature extraction model based on the graph convolutional neural network is constructed based on the time sequence graph and the graph convolutional neural network, and contains two graph convolutional layers and a series of hierarchically connected fully connected layers, batch normalization layers and nonlinear activation layers.

5. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method according to any one of claims 1-4. 6.A satellite backplane component identification device based on timing relationship, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to realize the steps of the method according to any one of claims 1-4.

Images