A spacecraft debris search and rescue identification and tracking method and system based on neural network

Abstract

The present invention discloses a spacecraft wreckage search and rescue identification and tracking method based on a neural network, comprising: determining the theoretical landing point of the spacecraft wreckage, planning a search area, and controlling an aircraft to move to the search area; controlling the aircraft to move in the direction of the actual landing point of the spacecraft wreckage and capturing images in real time during the movement; using a multi-scale image neural network model to perform real-time identification and detection on the acquired images to determine whether there is spacecraft wreckage in the image; if so, adjusting the direction of the aircraft in real time according to the position of the detection target in the image, and controlling the aircraft to move above the detection target; determining whether the confidence level of the current detection result is greater than a preset threshold, and if so, confirming whether the detection target is spacecraft wreckage; if not or the detection target is not spacecraft wreckage, leveling the aircraft's camera and continuing to control the aircraft's movement. The present invention can achieve rapid search for spacecraft wreckage, effectively narrow the search range, and improve search efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of spacecraft identification and positioning, and in particular to a spacecraft wreckage search and rescue identification and tracking method and system based on a neural network. Background Art

[0002] In recent years, with the continuous development of aerospace technology, the number of spacecraft launches has increased. During the launch process of the spacecraft, some structures of the spacecraft will separate and fall. Therefore, how to achieve the accurate recovery of the spacecraft debris after the fall is an important issue that is inevitable in the development of the current aerospace industry.

[0003] To prevent spacecraft crashes from posing a threat to human life and property, launch sites are typically built in densely populated areas, and the spacecraft's trajectory during launch is often also set over densely populated areas. Since the spacecraft's detached bodies fall in free fall after launch, the actual landing point of the spacecraft debris is difficult to determine. Search operations primarily rely on local search personnel, and when spacecraft debris falls in uninhabited areas, the search relies solely on the experience and intuition of search and rescue commanders, making it difficult to locate the search target and severely impacting the search and recovery of spacecraft debris. Furthermore, some spacecraft debris may contain hazardous gases, and if the debris is not found for recovery over a prolonged period, serious consequences could result. Summary of the Invention

[0004] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a spacecraft wreckage search, rescue, identification and tracking method and system based on neural network.

[0005] The technical solutions of the present invention are as follows:

[0006] In a first aspect, a method for searching, identifying, and tracking spacecraft wreckage based on a neural network is provided, the method comprising:

[0007] Determine the theoretical landing point of the spacecraft debris, plan the search area, and control the aircraft to the search area;

[0008] After the spacecraft wreckage falls, control the aircraft to move towards the actual landing point of the spacecraft wreckage and take real-time images during the movement;

[0009] Use a multi-scale image neural network model to perform real-time recognition and detection on the acquired images to determine whether there is spacecraft debris in the images;

[0010] If there is spacecraft debris in the image, the aircraft's direction is adjusted in real time based on the position of the detection target in the image so that the center point of the detection target is located at the center of the aircraft's camera, and the aircraft is controlled to move above the detection target;

[0011] Determine whether the confidence level of the detection result of the current captured image is greater than a preset threshold, and if so, confirm whether the detection target is spacecraft debris;

[0012] If the confidence level is below the preset threshold or the detection target is not spacecraft debris, the camera of the aircraft is leveled, and the aircraft is continued to be controlled to move in the direction of the actual landing point of the spacecraft debris and take images in real time until the spacecraft debris is found.

[0013] In some possible implementations, the multi-scale image neural network model includes a data enhancement module, a feature extraction module, a multi-scale output module, and a maximum suppression network model connected in sequence;

[0014] The data enhancement module is used to perform image enhancement processing on the input original image;

[0015] The feature extraction module is used to extract features from the feature matrix output by the data enhancement module in different sampling modes;

[0016] The multi-scale output module is used to perform correlation modeling on the feature matrix output by the feature extraction module according to different flight altitudes of the aircraft;

[0017] The maximum value suppression network module is used to select and output the matrix with the largest confidence among the matrices output by the multi-scale output module.

[0018] In some possible implementations, performing image enhancement processing on the input original image includes:

[0019] A grayscale operation is performed on the original image to convert the three-channel image into a single-channel image. The single-channel image is binarized to obtain an image model containing the characteristics of the spacecraft debris. The image model is sequentially eroded and expanded to remove surrounding noise, resulting in a feature matrix Δx with one channel. The image containing the characteristics of the spacecraft debris and the original image are merged along the channel direction to form a new feature vector set e = (x, Δx), where x represents the original image.

[0020] In some possible implementations, extracting features from the feature matrix output by the data enhancement module using different sampling methods includes:

[0021] The matrix output by the data enhancement module is downsampled 8 times to obtain the feature matrix The feature e1 is input into the DBL module composed of three layers of convolution, BN and Leaky ReLU to obtain the feature matrix e′1=φe1. Based on the feature matrix e′1, the softmax activation function is used to obtain the prediction matrix e″1=δe′1={O1,O2…O 21};

[0022] The matrix output by the data enhancement module is downsampled by 16 times to obtain the feature matrix The feature e2 is input into the DBL module composed of three layers of convolution, BN and Leaky ReLU to obtain the feature matrix e′2=φe2. Based on the feature matrix e′2, the softmax activation function is used to obtain the prediction matrix e″2=δe′2={P1,P2…P 21};

[0023] The matrix output by the data enhancement module is downsampled by 32 times to obtain the feature matrix The feature e3 is input into the DBL module composed of three layers of convolution, BN and Leaky ReLU to obtain the feature matrix e′3=φe3. Based on the feature matrix e′3, the softmax activation function is used to obtain the prediction matrix e″3=δe′3={Q1,Q2…Q 21};

[0024] in, represents three-times average pooling, represents four-time average pooling, represents five-time average pooling, φ represents the DBL module algorithm, δ represents the softmax activation function, O n (n=1,2,...,21) represents the feature matrix of size 52×52, P n (n=1,2,...,21) represents the feature matrix of size 26×26, Q n (n=1,2,...,21) indicates that the feature matrix size is 13×13 and the number of channels is 21.

[0025] In some possible implementations, the correlation modeling performed on the feature matrix output by the feature extraction module is expressed as:

[0026]

[0027] Among them, out represents the matrix output by the multi-scale output module, and w represents the height coefficient matrix corresponding to different flight altitudes of the aircraft.

[0028] In some possible implementations, the maximum suppression network module processes the matrix output by the multi-scale output module through the NMS algorithm, selects the matrix with the largest confidence in the matrix, and outputs the selected matrix as the final predicted target image.

[0029] In some possible implementations, the method further includes:

[0030] Obtain historical images with spacecraft debris and mark the locations of the spacecraft debris in the images to form a dataset;

[0031] The multi-scale image neural network model is iteratively trained using the data set in a back-propagation manner to update the parameters of the multi-scale image neural network model.

[0032] In some possible implementations, when performing iterative training of the multi-scale image neural network model, the loss functions include: a position loss function, a shape loss function, a confidence loss function of including the target result, a confidence loss function of not including the target result, and a category loss function;

[0033] Among them, the position loss function is expressed as:

[0034]

[0035] The shape loss function is expressed as:

[0036]

[0037] The confidence loss function including the target result is expressed as:

[0038]

[0039] The confidence loss function without the target result is expressed as:

[0040]

[0041] The category loss function is expressed as:

[0042]

[0043] Among them, λ coord Indicates the loss weight caused by the failure of coordinate information prediction, S indicates the size of the feature map, B indicates the number of categories of the detection box, Indicates the e″ i The jth feature map is responsible for predicting the detection box of the spacecraft debris. Indicates the predicted horizontal coordinate of the detection box, b xi Indicates the true horizontal coordinate of the detection frame, Indicates the predicted vertical coordinate of the detection box, byi Indicates the true vertical coordinate of the detection frame, Indicates the predicted width of the detection box, b wi Indicates the actual width of the detection frame, Indicates the predicted height of the detection box, b hi Indicates the true height of the detection box, C i Indicates the predicted probability value of the i-th target, represents the true probability value of the i-th target, λ noobj Indicates the loss weight that does not include the object detection frame, Indicates the e″ i The jth detection box in the feature map is not responsible for predicting spacecraft debris, Indicates that the center of the target falls on the current feature map, Indicates that the center of the target does not fall on the current feature map, classes indicates the number of predicted categories, p i (c) represents the predicted probability value of the i-th category, Represents the average probability value of the i-th category.

[0044] In some possible implementations, the center point of the detection target in the image is selected as the center point of the detection box output by the multi-scale image neural network model, and the direction of the aircraft is adjusted according to the difference between the center point of the detection box and the center point of the camera pixel of the aircraft.

[0045] In a second aspect, a spacecraft debris search, rescue, identification and tracking system based on a neural network is provided, the system comprising:

[0046] The planning module is used to determine the theoretical landing point of the spacecraft debris and plan the search area;

[0047] An aircraft control module, configured to control the movement of the aircraft and capture images according to instructions from the control module, and to transmit aircraft status information and captured image information to the control module in real time;

[0048] The control module is configured to transmit instructions to the aircraft control module and transmit image information sent by the aircraft control module to the prediction module;

[0049] The prediction module is used to perform real-time recognition and detection on the captured image using a multi-scale image neural network model, and obtain and output the detection results to the control module;

[0050] The confirmation module is used to display the captured image and detection target information in real time to complete the confirmation of the detection target.

[0051] The main advantages of the technical solution of the present invention are as follows:

[0052] The neural network-based spacecraft wreckage search and identification and tracking method of the present invention controls the aircraft to perform mobile search and capture images in real time according to the actual landing direction of the spacecraft wreckage, and uses a neural network model to identify and detect the real-time captured images to adjust the movement direction of the aircraft in real time. It can realize rapid search of spacecraft wreckage, effectively narrow the search range, reduce search time, significantly improve search efficiency, and reduce search costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0054] Figure 1 This is a flowchart of a spacecraft wreckage search, rescue, identification and tracking method based on a neural network according to an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a motion trajectory of an aircraft according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the structure of a multi-scale image neural network model according to one embodiment of the present invention;

[0057] Figure 4 The figure is a schematic structural diagram of a spacecraft debris search, rescue, identification and tracking system based on a neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0059] The technical solutions provided by the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0060] First, see Figure 1 An embodiment of the present invention provides a spacecraft wreckage search, rescue, identification and tracking method based on a neural network, the method comprising the following steps:

[0061] S1, determine the theoretical landing point of the spacecraft debris, plan the search area, and control the aircraft to the search area;

[0062] S2, after the spacecraft wreckage falls, controls the aircraft to move towards the actual landing point of the spacecraft wreckage and takes real-time images during the movement;

[0063] S3, uses a multi-scale image neural network model to perform real-time recognition and detection on the acquired image to determine whether there is spacecraft debris in the image;

[0064] S4, if there is spacecraft debris in the image, adjust the direction of the aircraft in real time according to the position of the detection target in the image so that the center point of the detection target is located at the center of the aircraft's camera, and control the aircraft to move above the detection target;

[0065] S5, determining whether the confidence level of the detection result of the current captured image is greater than a preset threshold, and if so, confirming whether the detection target is spacecraft debris;

[0066] S6: If the confidence level is below the preset threshold or the detection target is not spacecraft debris, the camera of the aircraft is leveled, and the aircraft is continued to be controlled to move toward the actual landing point of the spacecraft debris and take images in real time, and then return to step S3 until the spacecraft debris is found.

[0067] A neural network-based spacecraft wreckage search, rescue, identification and tracking method provided in one embodiment of the present invention controls the aircraft to perform mobile search and capture images in real time according to the actual landing direction of the spacecraft wreckage, and uses a neural network model to identify and detect the real-time captured images to adjust the movement direction of the aircraft in real time. This method can achieve rapid search of spacecraft wreckage, effectively narrow the search range, reduce search time, significantly improve search efficiency, and reduce search costs.

[0068] The following specifically describes the steps and principles of a neural network-based spacecraft wreckage search, rescue, identification, and tracking method provided in one embodiment of the present invention.

[0069] Step S1, determine the theoretical landing point of the spacecraft wreckage, plan the search area, and control the aircraft to go to the search area.

[0070] See also Figure 2 In one embodiment of the present invention, the theoretical landing point of the spacecraft debris can be determined based on the motion trajectory of the spacecraft and the separation time of the spacecraft, and the search area is planned based on the theoretical landing point of the spacecraft debris.

[0071] Furthermore, in order to facilitate the subsequent recovery of spacecraft debris, different recovery devices can be deployed at different locations in the search area.

[0072] Step S2: After the spacecraft wreckage falls, the aircraft is controlled to move toward the actual landing point of the spacecraft wreckage, and images are captured in real time during the movement.

[0073] Specifically, after the spacecraft wreckage falls, the actual landing direction of the spacecraft wreckage can be observed and determined, and the aircraft can be controlled to move from the theoretical landing point of the spacecraft wreckage to the actual landing direction, and real-time images can be captured during the movement of the aircraft. In order to facilitate real-time image capture, the aircraft is equipped with a camera.

[0074] Optionally, in order to improve the search efficiency, the aircraft can adopt a high-endurance, high-load UAV.

[0075] Step S3: Use a multi-scale image neural network model to perform real-time recognition and detection on the acquired image to determine whether there is spacecraft debris in the image.

[0076] In order to improve the efficiency of image recognition and detection, in one embodiment of the present invention, an image transmission unit can be provided on the aircraft. The image transmission unit is used to transmit the images taken by the aircraft in real time to the ground terminal through the OBS streaming server. The ground terminal uses a multi-scale image neural network model to perform real-time recognition and detection on the acquired images.

[0077] Optionally, the ground terminal may be a portable computer.

[0078] See also Figure 3 ,The multi-scale image neural network model includes: a data enhancement module, a feature extraction module, a multi-scale output module and a maximum ,suppression network model that are connected in sequence;

[0079] The data enhancement module is used to perform image enhancement processing on the input original image. The feature extraction module is used to extract features from the feature matrix output by the data enhancement module using different sampling methods. The multi-scale output module is used to perform correlation modeling on the feature matrix output by the feature extraction module according to the different flight altitudes of the aircraft. The maximum suppression network module is used to select and output the matrix with the highest confidence among the matrices output by the multi-scale output module.

[0080] In one embodiment of the present invention, by utilizing a pre-set multi-scale image neural network model to identify and detect the acquired image, a target detection frame in the image is obtained to predict the target in the image, thereby improving the image processing efficiency and thus improving the search efficiency of spacecraft wreckage.

[0081] Specifically, the input original image is subjected to image enhancement processing, including:

[0082] A grayscale operation is performed on the original image to convert the three-channel image into a single-channel image. The single-channel image is binarized to obtain an image model containing the characteristics of the spacecraft debris. The image model is sequentially subjected to erosion and dilation operations to remove peripheral noise, resulting in a feature matrix Δx = σ(x) with a channel number of 1. The image containing the characteristics of the spacecraft debris and the original image are merged along the channel direction to form a new feature vector set e = (x, Δx), where σ represents the image processing operation and x represents the original image.

[0083] Furthermore, the feature extraction module performs feature extraction on the matrix output by the data enhancement module through three target sampling channels, wherein the three target sampling channels include a small target sampling channel, a medium target sampling channel and a large target sampling channel, and the three target sampling channels are connected through a convolution layer, a downsampling layer and an upsampling layer.

[0084] Among them, the matrix output by the data enhancement module is subjected to feature extraction through the small target sampling channel, including:

[0085] The matrix output by the data enhancement module is downsampled 8 times to obtain the feature matrix The feature e1 is input into the DBL module composed of three layers: convolution, BN (BatchNorm) and Leaky ReLU, and the feature matrix e′1 = φe1 is obtained. Based on the feature matrix e′1, the softmax activation function is used to obtain the prediction matrix e″1 = δe′1 = {O1, O2…O 21},in, represents three-times average pooling, φ represents the DBL module algorithm, δ represents the softmax activation function, O n (n=1,2,...,21) indicates that the feature matrix size is 52×52 and the number of channels is 21.

[0086] Feature extraction is performed on the matrix output by the data enhancement module through the target sampling channel, including:

[0087] The matrix output by the data enhancement module is downsampled by 16 times to obtain the feature matrix The feature e2 is input into the DBL module composed of three layers: convolution, BN (BatchNorm) and Leaky ReLU, and the feature matrix e′2 = φe2 is obtained. Based on the feature matrix e′2, the softmax activation function is used to obtain the prediction matrix e″2 = δe′2 = {P1, P2…P 21},in, represents four-time average pooling, φ represents the DBL module algorithm, δ represents the softmax activation function, P n (n=1,2,...,21) indicates that the feature matrix size is 26×26 and the number of channels is 21.

[0088] The matrix output by the data enhancement module is subjected to feature extraction through the large target sampling channel, including:

[0089] The matrix output by the data enhancement module is downsampled by 32 times to obtain the feature matrix The feature e3 is input into the DBL module composed of three layers: convolution, BN (BatchNorm) and Leaky ReLU, and the feature matrix e′3 = φe3 is obtained. Based on the feature matrix e′3, the softmax activation function is used to obtain the prediction matrix e″3 = δe′3 = {Q1, Q2…Q 21},in, represents five-time average pooling, φ represents the DBL module algorithm, δ represents the softmax activation function, Q n (n=1,2,...,21) indicates that the feature matrix size is 13×13 and the number of channels is 21.

[0090] Furthermore, the prediction result output by the feature extraction module can be expressed as 3×(b x ,b y ,b w ,b h ,c,p1(c),p2(c)), where b x represents the offset of the predicted target detection box with respect to the real target image along the horizontal coordinate, b y represents the offset of the predicted target detection box along the vertical coordinate with respect to the real target image, b w represents the scaling factor of the predicted target detection box with respect to the width direction of the real target image, b h represents the scaling factor of the predicted target detection box with respect to the height direction of the real target image, c is the confidence of the predicted target detection box, p1(c) represents the probability that the predicted target is spacecraft debris, and p1(c) represents the probability that the predicted target is background.

[0091] In one embodiment of the present invention, by setting three target sampling channels to perform feature extraction on the matrix output by the data enhancement module, it is possible to realize the recognition and detection of large targets, medium targets and small targets in the image.

[0092] Furthermore, correlation modeling is performed on the feature matrix output by the feature extraction module according to different flight altitudes of the aircraft, including:

[0093] According to the altitude coefficient matrices corresponding to different flight altitudes of the aircraft, the altitude coefficient matrices are allocated to the three feature matrices output by the feature extraction module, so as to perform correlation modeling on the three feature matrices output by the feature extraction module.

[0094] Specifically, the correlation modeling of the three feature matrices output by the feature extraction module can be expressed as:

[0095]

[0096] Among them, out represents the matrix output by the multi-scale output module, that is, the modeling result, and w represents the height coefficient matrix.

[0097] Furthermore, in one embodiment of the present invention, the maximum suppression network module processes the matrix out output by the multi-scale output module through the NMS (NonMaximumSuppression) algorithm, and selects the matrix e″ with the largest confidence in the matrix m , m=1,2,3, and select the matrix e″ m As the final predicted target image output.

[0098] The NMS algorithm selects the rectangular box M with the highest confidence and the corresponding confidence T from the output detection boxes of the same type. It then calculates the Interconnection of Union (IOU) value between all rectangular boxes and M. If the IOU value is greater than the threshold t, the two boxes are similar and the similar rectangles are removed. Otherwise, no operation is performed. The IOU value represents the overlap between two rectangular boxes, that is, the ratio of the area of ​​the overlapping area of ​​the two rectangles to the total area.

[0099] Furthermore, in order to ensure the recognition accuracy of the multi-scale image neural network model, it is necessary to train the multi-scale image neural network model before using it to recognize and detect images. To this end, in one embodiment of the present invention, the method further includes:

[0100] Obtain historical images with spacecraft debris and mark the locations of the spacecraft debris in the images to form a dataset;

[0101] The multi-scale image neural network model is iteratively trained using the dataset in a back-propagation manner to update the parameters of the feature extraction module in the multi-scale image neural network model.

[0102] The data set can be represented as x={x1,x2,…,x n}, x represents n images, x n Represents the nth image, and the location information of the spacecraft debris can be expressed as is the horizontal coordinate of the pixel of the marked spacecraft debris in the image, is the vertical coordinate of the pixel in the image of the marked spacecraft debris, is the width of the marked spacecraft debris in the image in pixels. is the height value of the annotated spacecraft debris in the image with respect to the pixel.

[0103] Furthermore, in one embodiment of the present invention, when performing iterative training of a multi-scale image neural network model, the loss function includes: a position loss function, a shape loss function, a confidence loss function including the target result, a confidence loss function not including the target result, and a category loss function.

[0104] Specifically, the position loss function is expressed as:

[0105]

[0106] The shape loss function is expressed as:

[0107]

[0108] The confidence loss function including the target result is expressed as:

[0109]

[0110] The confidence loss function without the target result is expressed as:

[0111]

[0112] The category loss function is expressed as:

[0113]

[0114] Among them, λ coord Indicates the loss weight caused by the failure of coordinate information prediction, S indicates the size of the feature map, B indicates the number of categories of the detection box, Indicates the e″ i The jth feature map is responsible for predicting the detection box of the spacecraft debris. Indicates the predicted horizontal coordinate of the detection box, b xi Indicates the true horizontal coordinate of the detection frame, Indicates the predicted vertical coordinate of the detection box, b yi Indicates the true vertical coordinate of the detection frame, Indicates the predicted width of the detection box, b wi Indicates the actual width of the detection frame, Indicates the predicted height of the detection box, b hi Indicates the true height of the detection box, C i Indicates the predicted probability value of the i-th target, represents the true probability value of the i-th target, λ noobj Indicates the loss weight that does not include the object detection frame, Indicates the e″ i The jth detection box in the feature map is not responsible for predicting spacecraft debris, Indicates that the center of the target falls on the current feature map, Indicates that the center of the target does not fall on the current feature map, classes indicates the number of predicted categories, p i (c) represents the predicted probability value of the i-th category, Represents the average probability value of the i-th category.

[0115] According to the calculated loss value, the gradient descent method is used to gradually optimize the loss value to reduce it, and the neural network model is further updated through back propagation.

[0116] Step S4: If there is spacecraft debris in the image, the direction of the aircraft is adjusted in real time according to the position of the detection target in the image so that the center point of the detection target is located at the center of the aircraft's camera, and the aircraft is controlled to move above the detection target.

[0117] Specifically, in one embodiment of the present invention, the center point of the detection target in the image is selected as the center point of the detection box output by the maximum suppression network module, which can be expressed as:

[0118]

[0119] The direction of the aircraft is adjusted by correcting the difference between the Center value and the center point of the camera pixel.

[0120] In the process of controlling the aircraft to move above the detection target, the camera is still used to capture images in real time and perform real-time recognition and detection on the images.

[0121] Step S5: determine whether the confidence level of the detection result of the current captured image is greater than a preset threshold. If so, confirm whether the detection target is spacecraft debris.

[0122] Specifically, if the confidence level of the detection result is greater than a preset threshold, it is further confirmed whether the detection target is spacecraft debris.

[0123] Optionally, the ground terminal may display the obtained detection results in real time, and manually confirm whether the detection target is spacecraft debris based on the detection results displayed by the ground terminal.

[0124] Step S6: If the confidence level is below the preset threshold or the detection target is not spacecraft debris, the camera of the aircraft is leveled, and the aircraft is continued to be controlled to move toward the actual landing point of the spacecraft debris and take images in real time, and then return to step S3 until the spacecraft debris is found.

[0125] Specifically, if the confidence level of the detection result of the current captured image is below a preset threshold, indicating that the detection target is not spacecraft wreckage, or the confidence level is greater than the preset threshold, but further confirmation shows that the detection target is not spacecraft wreckage, the camera of the aircraft is leveled, and the aircraft is continued to be controlled to move toward the actual landing point of the spacecraft wreckage and take images in real time, and return to step S3 until the spacecraft wreckage is found.

[0126] Furthermore, in one embodiment of the present invention, in order to facilitate the recovery of spacecraft debris, the method may further include:

[0127] When it is confirmed that the detection target is spacecraft debris, the current position coordinates of the aircraft are determined, and the recovery device is controlled to go to the location of the aircraft to recover the spacecraft debris.

[0128] Second, see Figure 4 An embodiment of the present invention further provides a spacecraft debris search, rescue, identification and tracking system based on a neural network, the system comprising:

[0129] The planning module is used to determine the theoretical landing point of the spacecraft debris and plan the search area;

[0130] An aircraft control module, used to control the movement of the aircraft and capture images according to the instructions of the control module, and to transmit the status information of the aircraft and the captured image information to the control module in real time;

[0131] a control module, configured to transmit instructions to the aircraft control module and transmit image information sent by the aircraft control module to the prediction module;

[0132] The prediction module is used to use a multi-scale image neural network model to perform real-time recognition and detection on the captured images, and obtain and output the detection results to the control module;

[0133] The confirmation module is used to display the captured image and detection target information in real time to complete the confirmation of the detection target.

[0134] Specifically, the planning module determines the theoretical landing point of the spacecraft wreckage and plans the search area according to the theoretical landing point of the spacecraft wreckage. After the spacecraft wreckage falls, the control module generates a control instruction according to the actual landing direction of the spacecraft wreckage and sends it to the aircraft control module. The aircraft control module controls the movement of the aircraft according to the control instruction, takes images in real time, and transmits the image information to the prediction module in real time through the control module. The prediction module uses a multi-scale image neural network model to perform real-time recognition and detection on the taken images, obtains and outputs the detection results to the control module, and the control module determines whether there is spacecraft wreckage in the image. If so, a control instruction is generated according to the detection target position in the image and sent to the aircraft control module. The aircraft control module controls the aircraft to move above the detection target according to the control instruction, takes images in real time, and according to the confidence level of the detection result of the current captured image, the confirmation module displays the captured image and detection target information in real time to complete the confirmation of the detection target.

[0135] Furthermore, in one embodiment of the present invention, the system also includes a training module, which is used to construct and train a multi-scale image neural network model, and to transfer the trained multi-scale image neural network model to the prediction module.

[0136] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0137] It should be noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements that are inherent to such process, method, article or apparatus.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the module division is merely a logical functional division. In actual implementation, other division methods may be used, such as combining multiple modules or ignoring or not performing certain features.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A spacecraft wreckage search and rescue identification and tracking method based on a neural network, characterized in that: include: Determine the theoretical landing point of the spacecraft debris, plan the search area, and control the aircraft to the search area; After the spacecraft wreckage falls, control the aircraft to move towards the actual landing point of the spacecraft wreckage and take real-time images during the movement; Use a multi-scale image neural network model to perform real-time recognition and detection on the acquired images to determine whether there is spacecraft debris in the images; If there is spacecraft debris in the image, the aircraft's direction is adjusted in real time based on the position of the detection target in the image so that the center point of the detection target is located at the center of the aircraft's camera, and the aircraft is controlled to move above the detection target; Determine whether the confidence level of the detection result of the current captured image is greater than a preset threshold, and if so, confirm whether the detection target is spacecraft debris; If the confidence level is below the preset threshold or the detected target is not spacecraft debris, the aircraft's camera is leveled and the aircraft continues to move toward the actual landing point of the spacecraft debris while capturing images in real time until the spacecraft debris is found. The multi-scale image neural network model includes a data enhancement module, a feature extraction module, a multi-scale output module and a maximum suppression network model connected in sequence; The data enhancement module is used to perform image enhancement processing on the input original image; The feature extraction module is used to extract features from the feature matrix output by the data enhancement module in different sampling modes; The multi-scale output module is used to perform correlation modeling on the feature matrix output by the feature extraction module according to different flight altitudes of the aircraft; The maximum value suppression network module is used to select and output the matrix with the largest confidence among the matrices output by the multi-scale output module; The image enhancement processing is performed on the input original image, comprising: Perform a grayscale operation on the original image to convert the three-channel image into a single-channel image. Perform a binarization operation on the single-channel image to obtain an image model containing the spacecraft debris features. Perform erosion and dilation operations on the image model to remove surrounding noise, obtaining a feature matrix Δx with one channel. The image containing the spacecraft debris features and the original image are merged along the channel direction to form a new feature vector set e = (x, Δx), where x represents the original image. The extracting features from the feature matrix output by the data enhancement module in different sampling modes includes: The feature matrix output by the data enhancement module is downsampled by 8 times, 16 times, and 32 times to obtain three feature matrices, and the three feature matrices are input into the DBL module composed of three layers of convolution, BN, and Leaky ReLU to obtain three updated feature matrices. Based on the updated feature matrices, the corresponding prediction matrix is ​​obtained using the softmax activation function; The correlation modeling of the feature matrix output by the feature extraction module is expressed as: Wherein, out represents the matrix output by the multi-scale output module, w represents the height coefficient matrix corresponding to different flight altitudes of the aircraft, e1″, e2″, and e3″ represent the prediction matrices obtained by extracting features from the feature matrix output by the data enhancement module by downsampling by 8 times, 16 times, and 32 times, respectively; The maximum suppression network module processes the matrix output by the multi-scale output module through the NMS algorithm, selects the matrix with the largest confidence in the matrix, and outputs the selected matrix as the final predicted target image.

2. The spacecraft wreckage search, rescue, identification and tracking method based on neural network according to claim 1, characterized in that: The method further comprises: Obtain historical images with spacecraft debris and mark the locations of the spacecraft debris in the images to form a dataset; The multi-scale image neural network model is iteratively trained using the data set in a back-propagation manner to update the parameters of the multi-scale image neural network model.

3. The spacecraft wreckage search, rescue, identification and tracking method based on neural network according to claim 2, characterized in that: When performing iterative training of the multi-scale image neural network model, the loss functions include: a position loss function, a shape loss function, a confidence loss function including the target result, a confidence loss function not including the target result, and a category loss function; Among them, the position loss function is expressed as: The shape loss function is expressed as: The confidence loss function including the target result is expressed as: The confidence loss function without the target result is expressed as: The category loss function is expressed as: Among them, λ coord Indicates the loss weight caused by the failure of coordinate information prediction, S indicates the size of the feature map, B indicates the number of categories of the detection box, Indicates the e i The jth feature map in the feature map is responsible for predicting the detection box of the spacecraft debris. Indicates the predicted horizontal coordinate of the detection box, b xi Indicates the true horizontal coordinate of the detection frame, Indicates the predicted vertical coordinate of the detection box, b yi Indicates the true vertical coordinate of the detection frame, Indicates the predicted width of the detection box, b wi Indicates the actual width of the detection frame, Indicates the predicted height of the detection box, b hi Indicates the true height of the detection box, C i Indicates the predicted probability value of the i-th target, represents the true probability value of the i-th target, λ noobj Indicates the loss weight that does not include the object detection frame, Indicates the e i The jth detection box in the feature map is not responsible for predicting spacecraft debris, Indicates that the center of the target falls on the current feature map, Indicates that the center of the target does not fall on the current feature map, classes indicates the number of predicted categories, p i (c) represents the predicted probability value of the i-th category, Represents the average probability value of the i-th category.

4. The spacecraft wreckage search, rescue, identification and tracking method based on a neural network according to any one of claims 1 to 3, characterized in that: The center point of the detected target in the image is selected as the center point of the detection box output by the multi-scale image neural network model, and the direction of the aircraft is adjusted according to the difference between the center point of the detection box and the center point of the aircraft's camera pixel.

5. A spacecraft wreckage search, rescue, identification and tracking system based on a neural network using the spacecraft wreckage search, rescue, identification and tracking method based on a neural network as claimed in claim 1, characterized in that: The system comprises: The planning module is used to determine the theoretical landing point of the spacecraft debris and plan the search area; An aircraft control module, configured to control the movement of the aircraft and capture images according to instructions from the control module, and to transmit aircraft status information and captured image information to the control module in real time; The control module is configured to transmit instructions to the aircraft control module and transmit image information sent by the aircraft control module to the prediction module; The prediction module is used to perform real-time recognition and detection on the captured image using a multi-scale image neural network model, and obtain and output the detection results to the control module; The confirmation module is used to display the captured image and detection target information in real time to complete the confirmation of the detection target.

Images