Space small target extraction method based on multi-camera multi-frame images

Abstract

The application discloses a kind of space small target extraction methods based on multi-camera multi-frame image, comprising: obtaining the multiple frames first image containing star corresponding to each camera;According to the multiple frames first image of multiple cameras, the relative transformation relationship between multiple camera images is determined;Space target is simultaneously imaged using multiple cameras, and the multiple frames second image corresponding to each camera is obtained;According to the relative transformation relationship between multiple camera images, the second image of multiple cameras under the same time is superimposed, and multiple frames superimposed image is obtained;From each frame superimposed image, extract the suspected space target, determine and remove the part belonging to star in suspected space target;Based on the extracted suspected space target, target data correlation, target trajectory prediction update and target trajectory management are carried out on multiple frames superimposed image, and space target image trajectory is obtained.The method of the application can realize sufficient detection of space small target, and significantly improve the detection accuracy of space small target.

Description

Technical Field

[0001] This invention relates to the field of space target detection technology, specifically to a method for extracting weak spatial targets based on multi-camera, multi-frame images. Background Technology

[0002] There are many important space targets in space, such as communication, navigation, and remote sensing satellites, and manned spacecraft such as space stations. Therefore, space target detection is of great significance.

[0003] Space cameras are currently one of the main methods used in space target detection. Images are captured by space cameras mounted on satellites or spacecraft, and then processed to extract space targets. However, due to the low signal-to-noise ratio, blurred edges, and weak signal strength of small space targets, they occupy few pixels in camera images and lack structural information such as texture, shape, and size. When using images acquired by a single space camera to extract small space targets, these targets are easily obscured by stars and noise backgrounds, making effective and sufficient detection difficult and prone to missed or false detections. Summary of the Invention

[0004] To address some or all of the technical problems existing in the prior art, the present invention provides a method for extracting spatially weak targets based on multi-camera, multi-frame images.

[0005] The technical solution of the present invention is as follows:

[0006] A method for extracting spatially weak targets based on multi-camera, multi-frame images is provided, the method comprising:

[0007] Acquire the first multi-frame image containing stars for each camera;

[0008] Based on multiple frames of first images from multiple cameras, determine the relative transformation relationship between the multiple camera images;

[0009] Multiple cameras are used to simultaneously image spatial targets, and multiple frames of second images corresponding to each camera are obtained.

[0010] Based on the relative transformation relationship between multiple camera images, the second images of multiple cameras at the same time are superimposed to obtain a multi-frame superimposed image.

[0011] Suspected space targets are extracted from each frame of overlaid image, and the parts of the suspected space targets that belong to stars are identified and removed.

[0012] Based on the extracted suspected spatial targets, target data association, target trajectory prediction and update, and target trajectory management are performed on multi-frame superimposed images to obtain the spatial target image trajectory.

[0013] In some possible implementations, determining the relative transformation relationship between multiple camera images based on multiple frames of first images from multiple cameras further includes:

[0014] Extract suspected star regions from the first image of each frame, and use the centroid of the suspected star regions as detection points to obtain a set of detection points;

[0015] Select a camera as the reference camera, use the detection point set of the reference camera as the reference, and match the detection point sets of multiple cameras to obtain matching point pairs;

[0016] The matching point pairs of the first images of multiple frames from each camera are merged, and the relative transformation relationship between multiple camera images is estimated using a polynomial relationship.

[0017] In some possible implementations, the suspected stellar region is extracted from the first image in the following way:

[0018] The first image is binarized using local thresholding to obtain a binary image;

[0019] Morphological processing was used to label connected components in binary images, remove singular pixel sets, and extract suspected stellar regions.

[0020] In some possible implementations, matching the detection point sets of multiple cameras to obtain matching point pairs further includes:

[0021] The global nearest neighbor algorithm is used to match the detection point sets of multiple cameras to determine the matching point pairs with the same name;

[0022] Based on translation consistency, the random sample consistency algorithm is used to filter and purify the matching point pairs with the same name to obtain matching point pairs.

[0023] In some possible implementations, suspected spatial targets are extracted from overlaid images in the following manner:

[0024] Local thresholding is used to binarize the superimposed images to obtain a binary image.

[0025] Morphological processing is used to label connected components in binary images, remove singular pixel sets, and extract suspected spatial targets.

[0026] In some possible implementations, the portion of a suspected space target that belongs to a star is identified and eliminated in the following way:

[0027] Based on the attitude information of each camera, determine the star region to be captured by the camera;

[0028] Select stars from the corresponding star region in the star map and map the selected stars onto the image side of a pre-selected reference camera;

[0029] The mapped stars are matched with the suspected space targets using point set matching to identify the stellar portion of the suspected space targets and then remove the stellar portion of the suspected space targets.

[0030] In some possible implementations, target data association, target trajectory prediction and update, and target trajectory management are performed on multi-frame overlay images to obtain the spatial target image-side trajectory, further including:

[0031] Spatial target data association between multiple overlaid images is performed using a data association algorithm.

[0032] Based on the data association results, the Kalman filter algorithm is used to predict and update the trajectory of space targets.

[0033] Based on the trajectory prediction and update results, target trajectory management is performed to obtain the spatial target image trajectory.

[0034] In some possible implementations, target trajectory management is performed in the following way:

[0035] For each spatial target, determine whether there are corresponding consecutive preset number of target state measurement values. If so, construct the image-side trajectory of the current spatial target using the consecutive preset number of target state measurement values.

[0036] For each spatial target, if there is no corresponding target state measurement value for a consecutive preset number of frames, the current spatial target is considered lost, and the trajectory prediction and update of the current spatial target are terminated.

[0037] If a spatial target has a corresponding target state measurement value at a certain moment, the target state measurement value is used as the target state value at that moment to update the target trajectory. If a spatial target does not have a corresponding target state measurement value at a certain moment, the target state prediction value is used as the target state value at that moment to update the target trajectory.

[0038] The main advantages of the technical solution of this invention are as follows:

[0039] The spatial weak target extraction method based on multi-camera multi-frame images of the present invention solves the transformation relationship between multi-camera images, registers and superimposes the multi-camera images according to the transformation relationship, and extracts and tracks spatial targets based on the superimposed image. This method can achieve full detection of spatial weak targets and significantly improve the detection accuracy of spatial weak targets. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of a spatial weak target extraction method based on multi-camera multi-frame images according to an embodiment of the present invention;

[0042] Figures 2a-2d This is a schematic diagram of star extraction results from four single-frame images from four cameras, as given in an embodiment of the present invention.

[0043] Figures 3a-3c The corresponding embodiment of the present invention is given below. Figures 2a-2d A schematic diagram of point set matching results for images from different cameras;

[0044] Figure 4 This is a schematic diagram of target tracking results when using single-camera images for space target detection.

[0045] Figure 5 This diagram illustrates the target tracking results when using a spatial target detection method based on multi-camera, multi-frame images according to an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] The technical solutions provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0048] See Figure 1 An embodiment of the present invention provides a method for extracting spatially weak targets based on multi-camera, multi-frame images, the method comprising the following steps S1-S6:

[0049] Step S1: Obtain the first multi-frame image containing stars for each camera.

[0050] When the relative installation relationships between multiple cameras are fixed, images from multiple cameras operating simultaneously within the same field of view can be directly registered and superimposed. However, in practical applications, due to factors such as camera installation errors and post-launch deformation, the initially determined relative installation relationships between the cameras may change, resulting in new transformation relationships between the images from each camera. Therefore, it is necessary to determine the actual relative transformation relationships between the images from each camera.

[0051] Considering that images captured from space scenes lack significant texture information and have low grayscale values ​​in most areas, traditional feature point extraction operators are insufficient for extracting feature points. Furthermore, due to the abundance of stars in space, images acquired by cameras during astronomical observations will contain multiple high-magnitude stars. These stars can be used as reference points to determine the transformation relationships between camera images. In one embodiment of this invention, multiple cameras are used simultaneously to image space at preset exposure times and preset time intervals, acquiring multiple first images containing stars for each camera.

[0052] The preset exposure time and preset time interval can be set according to the actual situation.

[0053] Step S2: Determine the relative transformation relationship between the multiple camera images based on the multiple frames of the first images from multiple cameras.

[0054] Specifically, in one embodiment of the present invention, determining the relative transformation relationship between multiple camera images based on multiple frames of first images from multiple cameras further includes the following steps S21-S23:

[0055] Step S21: Extract the suspected star region from the first image of each frame, and use the centroid position of the suspected star region as the detection point to obtain the detection point set.

[0056] Considering that stars typically appear as bright spots in images when the camera operates at a suitable exposure time, but due to factors such as light pollution, some images may exhibit uneven brightness distribution. In one embodiment of the present invention, the suspected stellar region is extracted from the first image using the following method:

[0057] The first image is binarized using local thresholding to obtain a binary image;

[0058] Morphological processing was used to label connected components in binary images, remove singular pixel sets, and extract suspected stellar regions.

[0059] The threshold used for segmenting the first image can be set according to the actual situation.

[0060] In one embodiment of the present invention, the centroid of the suspected stellar region is located at the center of the suspected stellar region.

[0061] Step S22: Select a camera as a reference camera, use the detection point set of the reference camera as a reference, match the detection point sets of multiple cameras, and obtain matching point pairs.

[0062] In one embodiment of the present invention, when determining the relative transformation relationship between multiple camera images, a camera can be selected as a reference camera. The detection point set of the reference camera image is used as a reference, and point set matching is performed on other camera images to obtain the transformation relationship between other camera images and the reference camera image, thereby determining the relative transformation relationship between multiple camera images.

[0063] Furthermore, considering the great distance between stars and Earth, the transformation relationships between images are primarily translational. In one embodiment of the present invention, matching the detection point sets of multiple cameras to obtain matching point pairs further includes:

[0064] The Global Nearest Neighbor (GNN) algorithm is used to match the detection point sets of multiple cameras to determine the matching point pairs with the same name;

[0065] Based on translation consistency, the Random Sample Consistency (RANSAC) algorithm is used to filter and purify the matching point pairs with the same name to obtain matching point pairs.

[0066] By performing point set matching in the above manner, the number of erroneous matches and low-precision matching point pairs can be significantly reduced.

[0067] Step S23: Merge the matching point pairs of the first images of multiple frames from each camera, construct and estimate the relative transformation relationship between multiple camera images using polynomial relations.

[0068] Specifically, the matching point pairs of the first images of multiple frames from each camera are merged, and the transformation relationship between other camera images and the reference camera image is estimated using a polynomial relationship, thereby determining the relative transformation relationship between multiple camera images.

[0069] Step S3: Simultaneously image the spatial target using multiple cameras to acquire multiple frames of second images corresponding to each camera.

[0070] In one embodiment of the present invention, multiple cameras are used simultaneously to image space targets in space with preset exposure times and preset time intervals, thereby acquiring multiple frames of second images corresponding to each camera.

[0071] The preset exposure time and preset time interval can be set according to the actual situation.

[0072] Step S4: Based on the relative transformation relationship between multiple camera images, the second images of multiple cameras at the same time are superimposed to obtain a multi-frame superimposed image.

[0073] Specifically, using the second image of a pre-selected reference camera as a reference, the second images of other cameras are registered according to the relative transformation relationship between multiple camera images determined in step S2. Based on the registered second images of multiple cameras, one frame of the second image of multiple cameras at the same time is superimposed to obtain the corresponding multi-frame superimposed image.

[0074] By registering and superimposing multiple camera images based on their relative transformation relationships, the signal-to-noise ratio of the overall image can be significantly improved, facilitating the subsequent extraction of spatially weak targets and enhancing the detection accuracy of such targets.

[0075] Step S5: Extract suspected space targets from each frame of superimposed image, identify and remove the parts of the suspected space targets that belong to stars.

[0076] Considering that spatial targets move at high speeds and are far away from the camera, they have low signal-to-noise ratios and occupy fewer pixels in camera images. In one embodiment of the present invention, suspected spatial targets are extracted from overlaid images using the following method:

[0077] Local thresholding is used to binarize the superimposed images to obtain a binary image.

[0078] Morphological processing is used to label connected components in binary images, remove singular pixel sets, and extract suspected spatial targets.

[0079] The threshold used for segmenting the superimposed image can be set according to the specific circumstances.

[0080] Furthermore, since the extracted suspected space targets may contain some stars, to avoid these stars interfering with subsequent target trajectory extraction and tracking, in one embodiment of the present invention, the portion of the suspected space target that belongs to stars is also identified and removed from the list of suspected space targets that belongs to stars.

[0081] Specifically, in one embodiment of the present invention, the portion of a suspected space target that belongs to a star is determined and eliminated in the following manner:

[0082] Based on the attitude information of each camera, determine the star region to be captured by the camera;

[0083] Select stars from the corresponding star region in the star map and map the selected stars onto the image side of a pre-selected reference camera;

[0084] The mapped stars are matched with the suspected space targets using point set matching to identify the stellar portion of the suspected space targets and then remove the stellar portion of the suspected space targets.

[0085] Step S6: Based on the extracted suspected spatial targets, perform target data association, target trajectory prediction and update, and target trajectory management on the multi-frame superimposed images to obtain the spatial target image trajectory.

[0086] Considering that the number of detected spatial targets in superimposed images of different frames is dynamically changing and may also contain some false targets, in one embodiment of the present invention, based on the extracted suspected spatial targets, target data association, target trajectory prediction and updating, and target trajectory management are performed on the superimposed images of multiple frames to obtain the image-side trajectory of the spatial targets, so as to further improve the detection accuracy of spatial targets.

[0087] Specifically, in one embodiment of the present invention, target data association, target trajectory prediction and update, and target trajectory management are performed on multiple superimposed images to obtain the spatial target image trajectory, further including:

[0088] Spatial target data association between multiple overlaid images is performed using a data association algorithm.

[0089] Based on the data association results, the Kalman filter algorithm is used to predict and update the trajectory of space targets.

[0090] Based on the trajectory prediction and update results, target trajectory management is performed to obtain the spatial target image trajectory.

[0091] By using data association algorithms to correlate spatial target data between multiple overlaid images, false targets can be further removed, improving the detection accuracy of spatial targets. These data association algorithms can include, for example, Global Nearest Neighbor (GNN), Joint Probabilistic Data Interconnection (JPDA), or Multiple Hypothesis Tracking (MHT).

[0092] Furthermore, in one embodiment of the present invention, target trajectory management is performed in the following manner:

[0093] For each spatial target, determine whether there are corresponding consecutive preset number of target state measurement values. If so, construct the image-side trajectory of the current spatial target using the consecutive preset number of target state measurement values.

[0094] For each spatial target, if there is no corresponding target state measurement value for a consecutive preset number of frames, the current spatial target is considered lost, and the trajectory prediction and update of the current spatial target are terminated.

[0095] If a spatial target has a corresponding target state measurement value at a certain moment, the target state measurement value is used as the target state value at that moment to update the target trajectory. If a spatial target does not have a corresponding target state measurement value at a certain moment, the target state prediction value is used as the target state value at that moment to update the target trajectory.

[0096] In one embodiment of the present invention, the target state measurement value represents the weighted average of the true target state value and the predicted target state value. The true target state value is the target information obtained from the camera image, and the predicted target state value is the target information predicted using the Kalman filter algorithm.

[0097] In one embodiment of the present invention, the preset number of frames can be set according to the actual situation, for example, it can be 3 frames.

[0098] An embodiment of the present invention provides a spatial weak target extraction method based on multi-camera multi-frame images. By solving the transformation relationship between multi-camera images, registering and superimposing the multi-camera images according to the transformation relationship, and extracting and tracking spatial targets based on the superimposed images, the method can achieve sufficient detection of spatial weak targets and significantly improve the detection accuracy of spatial weak targets.

[0099] The following describes the beneficial effects of a spatial weak target extraction method based on multi-camera, multi-frame images provided by an embodiment of the present invention, with reference to specific embodiments:

[0100] In this embodiment, images captured in orbit by four cameras of the same type on a specific satellite are selected for space target extraction.

[0101] Figures 2a-2d The image shows the star extraction results from single-frame images taken at the same time by four cameras (Camera 1, Camera 2, Camera 3, and Camera 4). The circled areas represent the extracted stars. It can be seen that the extracted star positions are shifted across the images from each camera, but the number of extracted stars is inconsistent, requiring point pair matching and further refinement.

[0102] Using camera 1 as the reference camera, Figures 3a-3c It shows the corresponding Figures 2a-2d The point set matching results of different camera images, where, Figure 3a This represents the point set matching result between camera 2 and camera 1. Figure 3b This represents the point set matching result between camera 3 and camera 1. Figure 3c The line represents the point set matching result between camera 4 and camera 1, and the line connecting them represents the matched point pairs. It can be seen that the point set matching effect is very good, and the images from each camera approximately satisfy a translation relationship, which can be used for spatial alignment of multi-camera images.

[0103] Furthermore, based on the selected image, spatial target extraction and tracking are performed using existing methods for spatial target extraction from images acquired by a single camera, and spatial target extraction and tracking are performed using the multi-camera, multi-frame image-based weak spatial target extraction method provided in an embodiment of the present invention, resulting in the following: Figures 4-5The diagram shows the spatial target tracking results. The circled areas represent extracted suspected spatial targets, and the curved areas (approximately straight lines) represent the tracked spatial target image trajectories. It can be seen that many false targets were present before tracking. Tracking through multiple frames not only effectively suppressed false targets but also obtained the target's motion trajectory. Compared to existing methods using single-camera images for spatial target extraction, the method provided in this embodiment can effectively extract more small spatial targets in the field of view, resulting in superior detection performance and higher detection accuracy.

[0104] The method provided in one embodiment of the present invention is used for spatial target extraction and tracking.

[0105] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Additionally, the terms "front," "back," "left," "right," "upper," and "lower" in this document refer to the placement shown in the accompanying drawings.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting a spatial small target based on multi-camera multi-frame images, characterized in that, The method comprises the following steps: acquiring a plurality of first images containing stars corresponding to each camera; determining the relative transformation relationship between the images of the plurality of cameras according to the plurality of first images of the plurality of cameras; acquiring a plurality of second images corresponding to each camera by simultaneously imaging the space target by the plurality of cameras; superimposing the second images of the plurality of cameras at the same time according to the relative transformation relationship between the images of the plurality of cameras to obtain a plurality of superimposed images; extracting a suspected space target from each superimposed image, determining and removing the part of the suspected space target belonging to stars; based on the extracted suspected space target, performing target data association, target trajectory prediction update and target trajectory management on the plurality of superimposed images to obtain the image plane trajectory of the space target; the step of determining the relative transformation relationship between the images of the plurality of cameras according to the plurality of first images of the plurality of cameras comprises the following steps: extracting a suspected star region from each first image, and taking the centroid position of the suspected star region as a detection point to obtain a set of detection points; selecting one camera as a reference camera, taking the set of detection points of the reference camera as a reference, and matching the sets of detection points of the plurality of cameras to obtain a matching point pair; merging the matching point pairs of the plurality of first images of each camera to construct and estimate the relative transformation relationship between the images of the plurality of cameras by using a polynomial relationship; wherein the suspected star region is extracted from the first image in the following manner: using local threshold segmentation to binarize the first image to obtain a binary image; using morphological processing to label the connected domain of the binary image, remove the singular pixel set, and extract the suspected star region; wherein the matching of the sets of detection points of the plurality of cameras to obtain the matching point pair comprises the following steps: using a global nearest neighbor algorithm to match the sets of detection points of the plurality of cameras to determine the same-name matching point pair; using a random sample consistency algorithm to screen and purify the same-name matching point pair according to the translational consistency to obtain the matching point pair; wherein the merging of the matching point pairs of the plurality of first images of each camera to construct and estimate the transformation relationship between the images of the other cameras relative to the image of the reference camera to determine the relative transformation relationship between the images of the plurality of cameras.

2. The multi-camera multi-frame image-based spatial small target extraction method according to claim 1, characterized in that, The suspected space target is extracted from the superimposed image in the following manner: using local threshold segmentation to binarize the superimposed image to obtain a binary image; using morphological processing to label the connected domain of the binary image, remove the singular pixel set, and extract the suspected space target.

3. The multi-camera multi-frame image-based spatial weak small target extraction method according to claim 1 or 2, characterized in that, The part of the suspected space target belonging to stars is determined and removed in the following manner: determining the star region photographed by the camera according to the attitude information of each camera; selecting the stars corresponding to the star region from the star map, and mapping the selected stars in the image plane of the preselected reference camera; performing point set matching between the mapped stars and the suspected space target to determine the part of the suspected space target belonging to stars, and removing the part of the suspected space target belonging to stars.

4. The multi-camera multi-frame image-based spatial small target extraction method according to claim 1, characterized in that, The target data association, target trajectory prediction update and target trajectory management on the plurality of superimposed images to obtain the image plane trajectory of the space target further comprise the following steps: performing spatial target data association between the plurality of superimposed images by using a data association algorithm; According to the data association result, a Kalman filtering algorithm is used for trajectory prediction and update of the space target; According to the trajectory prediction and update result, target trajectory management is performed to obtain the image-side trajectory of the space target.

5. The multi-camera multi-frame image-based spatial small target extraction method according to claim 4, characterized in that, The target trajectory management is performed in the following manner: For each space target, it is determined whether there is a corresponding target state measurement value of a continuous preset number of frames, if yes, the image-side trajectory of the current space target is constructed by using the target state measurement value of the continuous preset number of frames; For each space target, if there is no corresponding target state measurement value of the continuous preset number of frames, it is determined that the current space target is lost, and the trajectory prediction and update of the current space target are terminated; If there is a corresponding target state measurement value of the space target at a certain time, the target state measurement value is used as the target state value at the corresponding time for trajectory update, and if there is no corresponding target state measurement value of the space target at a certain time, the target state prediction value is used as the target state value at the corresponding time for trajectory update.

Images