Target recognition and approach method in complex environments based on dual-band imaging on satellite platform

Abstract

This invention belongs to the field of aerospace on-orbit servicing technology, specifically relating to a method for target identification and approach in complex environments based on dual-band imaging on a satellite platform. It utilizes three commonly used sensors: an infrared camera, a visible light camera, and a laser rangefinder. The image information source employs multispectral fusion of infrared and visible light. Synchronization methods include software time synchronization and image registration. A fast infrared mean judgment suitable for onboard embedding is introduced, along with Earth background suppression technology and a target information enhancement scheme. It employs fusion tracking and ranging coordination between vision and laser rangefinders, and a laser rangefinder start-stop strategy is designed based on onboard power consumption. This invention overcomes the shortcomings of insufficient robustness of a single sensor in observing targets against complex Earth backgrounds during changes in the field of view on a satellite platform.

Description

Technical Field

[0001] This invention belongs to the field of aerospace on-orbit service technology, specifically relating to a method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform. Background Technology

[0002] Currently, the number of spacecraft in orbit is increasing rapidly year by year. At the same time, the generation of space debris has also brought many safety hazards. Being able to identify, capture and track space targets in the space environment, and promptly detect and judge the intentions of space targets, can reduce the probability of spacecraft collisions in orbit.

[0003] During spacecraft rendezvous and docking missions, optical sensors are often required to operate in favorable imaging environments alongside the target spacecraft. However, in current satellite platforms, single optical sensors struggle to adapt to the changing space environment. Infrared cameras inherently have low resolution, while visible light cameras are overly reliant on lighting conditions. In low or high light, their imaging performance significantly deteriorates, leading to image blurring and loss of feature information. Motion blur also affects the final image quality during spacecraft tracking, complicating subsequent target recognition and tracking algorithms. Furthermore, in space, the camera's field of view can change, potentially resulting in images facing Earth or partially obscuring the Earth against the deep space background. In such cases, the target may be lost in the complex background, making effective detection or tracking impossible. Therefore, combining the advantages of different sensors to obtain more robust target recognition results and richer target information is crucial for current spacecraft safety.

[0004] In summary, to address the issue that current space target recognition processes are susceptible to the influence of the space environment, resulting in poor robustness, a dual-band imaging space target recognition and acquisition method based on infrared and visible light is proposed, which utilizes multiple data sources to improve the robustness of target recognition. Summary of the Invention

[0005] To address the issue that the inherent characteristics of sensors on satellite platforms are affected by the environment, making it difficult for a single sensor to accurately and robustly acquire observation information of space targets in complex and ever-changing space environments, this invention provides a method for identifying and approaching targets in complex environments based on dual-band imaging on satellite platforms. This method overcomes the shortcomings of single sensors in terms of robustness when observing targets against the complex background of Earth during changes in the field of view on satellite platforms. It establishes a method for identifying targets through dual-band observation and uses a laser rangefinder to supplement and enrich the observation information.

[0006] This invention is implemented as follows: it provides a method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform, comprising the following steps:

[0007] Step 1: Build an integrated device for a visible light camera, an infrared camera, and a laser measuring instrument. Perform coordinate calibration, time synchronization processing, and image sampling transformation on the visible light camera and the infrared camera. Perform optical axis deviation calibration on the laser rangefinder.

[0008] Step 2: Based on the integrated equipment and time synchronization processing method built in Step 1, the visible light camera and infrared camera collect image data to obtain visible light and infrared images of space targets, and perform Earth background determination and suppression on the visible light and infrared images in response to the space environment.

[0009] Step 3: Use a dynamic weighting and multi-scale fusion algorithm to fuse visible light and infrared images to obtain the line-of-sight angle observation information of spatial targets in the fused image;

[0010] Step 4: After obtaining the line-of-sight angle observation information of the space target, set the laser rangefinder on / off strategy, and turn on the laser rangefinder to measure distances when needed, and obtain the laser rangefinder measurement information:

[0011] Step 5: Continuous unscented Kalman filter algorithm state estimation of the spatial target using line-of-sight observation information and laser rangefinder measurement information: By establishing a state model of the spatial target relative to the observed target, and assuming that the spatial target moves at a constant speed in the short term as a state transition process, the motion state of the spatial target is predicted using the time interval between observations of two adjacent fused images and the process noise covariance; and a corresponding measurement model is constructed based on the relative position state of the spatial target and the observed target. When there is no effective laser rangefinder measurement information at a certain moment, a large variance method is used to extrapolate and predict the state of the spatial target using only line-of-sight observation information.

[0012] Step 6: By reading the line-of-sight angle observation information of each frame of the fused image, and if there is laser rangefinder measurement information, it is also read. The information is input into the state model and measurement model in Step 5 to predict the state of the space target and update the state model and measurement model. Finally, the position and velocity information of the space target are output to approach the space target.

[0013] Preferably, step 1, which involves calibrating the coordinates of the visible light camera and the infrared camera, and calibrating the optical axis deviation of the laser rangefinder, specifically includes the following steps:

[0014] Step 1.1: The intrinsic parameter calibration of the visible light camera and infrared camera adopts the checkerboard calibration method, and the focal length, principal point and distortion information of the visible light camera and infrared camera are obtained by using the corner detection method.

[0015] Step 1.2: After the intrinsic parameter calibration is completed, fix the visible light camera and the infrared camera in the integrated device, then calibrate the central line of sight of the visible light camera and the infrared camera to make the central line of sight of the two cameras parallel, and then use the checkerboard calibration method to calibrate the extrinsic parameters to obtain the coordinate transformation matrix between the visible light camera and the infrared camera and the coordinate transformation matrix between the visible light camera, the infrared camera and the reference coordinate system of the integrated device.

[0016] Step 1.3: Turn on the visible light camera and the laser rangefinder, so that the laser rangefinder spot appears in the field of view of the visible light camera. The visible light camera takes multiple frames of images, calculates the deviation of the laser rangefinder spot from the principal pixel of the visible light camera, and completes the optical axis deviation calibration of the laser rangefinder.

[0017] Further preferably, in step 1, the time synchronization processing of the visible light camera and the infrared camera is performed by selecting the nearest frame through "coarse synchronization" plus "fine synchronization" to achieve soft time synchronization, specifically including the following steps:

[0018] "Coarse synchronization" calculates candidate frames:

[0019] Step 1.4: When the integrated device starts running, it marks each frame of image received by the visible light camera and infrared camera with a receiving timestamp and frame number. Within n seconds of running, it reads the [frame number]. k Received timestamps of visible light images Select the receiving timestamp and... in the infrared image buffer. The closest j A frame of infrared image, the timestamp of which was received is... , making ,in This is a preset time tolerance threshold. R Represents a visible light image. I Represents an infrared image;

[0020] Step 1.5: Combine each pair of image data obtained in Step 1.4 Saved to the paired image data dynamic cache, where Indicates the received timestamp is Given the infrared image frame number, after collecting n seconds of paired data, a least squares fitting method is used to obtain the following linear model of the mapping relationship:

[0021] , ;

[0022] in, The average timestamps of visible light images within the paired image data dynamic buffer. The average frame number of the infrared images within the paired image data dynamic buffer. a , b Representing the coefficients, we obtain the final linear model of the mapping relationship: ;

[0023] Step 1.6: After obtaining the final mapping linear model in Step 1.5, use this linear model to coarsely predict the corresponding infrared image frame number. When each visible light image frame is received, input the visible light image timestamp. The corresponding infrared image frame number is calculated. The coarsely matched infrared image frame number The corresponding image, and the frame number is -1、 The infrared image of +1 is extracted from the paired image data dynamic buffer;

[0024] "Precise synchronization" matches synchronization frames:

[0025] Step 1.7: Perform precise matching between the visible light image extracted in Step 1.6 and the three candidate infrared images. First, extract the gradient map of the visible light image. Similarly, gradient maps of the three candidate infrared images were extracted. ,calculate With three candidate infrared images The normalized similarity is used to determine the most similar infrared image frame as the most synchronized frame with the visible light image, thus completing the "precise synchronization" between the visible light and infrared images.

[0026] Further preferably, in step 1, the image sampling transformation of the visible light camera and the infrared camera specifically involves:

[0027] Step 1.8: After obtaining the visible light image and infrared image after time synchronization of the visible light camera and infrared camera, since the two are different in size, the infrared image is upsampled and stretched to make it consistent with the size of the visible light image.

[0028] Preferably, step 2, which involves determining and suppressing the Earth background in the space environment, includes the following steps:

[0029] Step 2.1: Based on the analysis of typical characteristics of the Earth, in both visible light and infrared images, the deep space region is completely black, while the Earth region is a region with a significant increase in bright gray value. The brightness / gray level and proportion method is used to quickly determine whether the Earth region appears: During the initial operation of the satellite, the pre-set empirical threshold T is used as the indicator to determine whether the Earth region appears in the image. The average gray value of the entire image is taken and compared with the threshold T. If the average gray value of the entire image is greater than the threshold T, it is considered that the Earth background appears in the field of view of the visible light camera and the infrared camera. Subsequently, during the operation, the average gray value of the entire image is calculated over a period of time. The lowest average gray value of the deep space segment and the highest average gray value of the Earth segment during actual on-orbit operation are selected to dynamically update the threshold T.

[0030] Step 2.2: After thresholding, the infrared image is binarized using the OTSU algorithm to obtain the high grayscale pixel segmentation threshold of the infrared image. Based on the segmentation, it is divided into two levels: one is partial Earth background, and the other is full Earth background.

[0031] Step 2.3: After the judgment in Step 2.2, further processing is performed based on the proportion of the Earth's background. The processing method is as follows:

[0032] The Earth's background percentage is less than Skip Earth background suppression;

[0033] The Earth's background accounts for a certain percentage of the total area. Between: Light feathering mask, specifically:

[0034] First, the visible light image after suppressing the Earth background. With infrared images Each was subjected to a light feathering masking process. x, y This represents the specific coordinates of a pixel in a visible light image or infrared image.

[0035] Based on the thresholding method in step 2.1, the Earth background region is determined from the visible light image and the infrared image, and a mask is constructed. ,in It is a binary mask image;

[0036] Secondly, to avoid obvious abrupt changes in the edges between the mask and the target area, Gaussian convolution is used to smooth the mask edges, resulting in feathering.

[0037] ;

[0038] in, For feathering weight, A two-dimensional Gaussian kernel with empirically derived values;

[0039] Finally, the obtained feathering weights are used to process the visible light image and the infrared image respectively:

[0040] ;

[0041] in, These are the processed visible light image and infrared image, respectively;

[0042] The Earth's background accounts for a certain percentage of the total area. Between: Moderate suppression of the Earth's background is achieved through threshold masking and local contrast enhancement. Specifically:

[0043] By analyzing the dynamic threshold T using histograms, the overall background pixels of infrared and visible light images are darkened, while non-background areas are contrast-stretched, making the dark areas of the overall image darker and the local brightness of the target area brighter.

[0044] The Earth's background ratio is greater than The infrared and visible light images undergo whole-image histogram compression and local entropy saliency enhancement processing. Specifically:

[0045] Histogram compression is applied to the entire infrared and visible light images to reduce overall brightness. Then, the salience of spatial targets is enhanced by local entropy. The weight is increased in regions with low entropy and decreased in regions with high entropy. The salience of bright spots of spatial targets is improved by entropy map, while the background region of the Earth is suppressed.

[0046] The preset percentage threshold, the specific value and the range of the endpoint values ​​are set according to different scenarios based on the ground test.

[0047] More preferably, step 3 includes the following steps:

[0048] Step 3.1: First, perform dynamic weighted fusion of the visible light image and the infrared image:

[0049] ;

[0050] in, For coarsely blended images, the blending weights are... Adaptive calculations are performed based on the actual conditions in the space, and the calculation method is as follows:

[0051] ;

[0052] in, These are the local variances of the infrared image and the visible light image, respectively. The local entropy of the infrared image. These are the weighting coefficients. To prevent division by zero; higher local variance and local entropy of infrared images indicate a larger weighting of infrared images. Therefore, the weighting coefficients for fusing infrared and visible light images are dynamically determined based on the actual imaging situation.

[0053] Step 3.2: Use infrared images for coarse detection of spatial target regions, introducing local entropy, local standard deviation, and adjustment coefficients to identify local hotspots or areas of significant motion. The specific method is as follows:

[0054] ;

[0055] in, DOG It involves performing differential Gaussian processing on the image to remove large areas of smooth background and significantly enhance the high-frequency regions of the spatial target area. and For adjustment coefficients, Represents the maximum local standard deviation in an infrared image;

[0056] Step 3.3: Based on the real-time situation on the satellite, dynamically change the adjustment coefficients in Step 3.2 according to the background mean. When the current image imaging result shows weak thermal targets, a large proportion of the Earth background, and abundant cloud texture information, improve... DOG weighting percentage Reduce the weight of fine-texture features while increasing the weight of features resembling spatial targets; when the thermal features of the spatial target are strong and the background noise is high, increase... This allows variance to suppress noise and simultaneously reduce DOG Operator weights should be adjusted to avoid treating excessive noise as spatial targets.

[0057] Step 3.4: After acquiring the local space target region through infrared images, in order to reduce the computing power burden on the satellite, only the space target region is subjected to multi-scale Laplacian pyramid fusion. The final fused image is the space target region with the highest accuracy to ensure infrared thermal characteristics and visible light texture information, while the non-space target region retains the coarse fused image.

[0058] Step 3.5: After extracting the spatial target, determine the displacement of the spatial target within multiple frames of images based on motion consistency analysis, accurately identify the spatial target, and output the spatial target information based on the confidence level;

[0059] Step 3.6: Extract the centroid pixel coordinates (u,v) of the spatial target in the fused image using Step 3.5, and obtain the image plane line of sight by combining the camera intrinsic and extrinsic parameters as follows:

[0060] ;

[0061] ;

[0062] ;

[0063] Small angle approximate line-of-sight pitch angle Azimuth as follows:

[0064] ;

[0065] in, and These are the coordinates of the principal point of the visible light camera's intrinsic parameters. and It is the pixel focal length of a visible light camera. These are normalized lateral image plane coordinates. These are the normalized vertical image plane coordinates. It is the normalized direction of the unit line-of-sight vector in the visible light camera coordinate system.

[0066] Further preferred, the laser rangefinder switching strategy in step 4 is specifically as follows:

[0067] Step 4.1: The judgment is performed sequentially according to three parts: confidence gating, optical axis deviation gating, and count gating. The confidence gating judgment method is as follows: ,in, This represents the confidence level of spatial targets in the fused image of the current frame. The confidence threshold is used to determine whether the optical axis deviation is entered into the judgment process when the confidence of the spatial target in the fused image of the current frame is greater than or equal to the confidence threshold. The judgment is based on the following criteria:

[0068] ;

[0069] in, , The coordinates of the centroid pixels of the spatial target in the fused image of the current frame. , It is the deviation of the optical axis between the ground-calibrated laser rangefinder and the visible light camera, ultimately obtaining the instantaneous deviation of the centroid of the spatial target relative to the optical axis of the laser rangefinder. and ;

[0070] Step 4.2: The deviation judgment condition is obtained by converting the ground calibration pixel threshold and spatial error to obtain the allowable threshold for spatial target-spot size deviation. px, when and If the threshold is within the specified range, the counting gate judgment is entered. In the counting gate, if the fused images of M consecutive frames stably meet the above conditions, the laser rangefinder is confirmed to be turned on.

[0071] Further preferably, step 5 specifically includes:

[0072] Step 5.1: Construct the state vector X , is defined as the relative state of a space target in the reference coordinate system of the observed target, where the state vector is... Represents the position coordinates of the space target relative to the observed target. With the corresponding three velocity states ;

[0073] Step 5.2: Establish the state transition model F(·) of the space target relative to the observed target. Assuming that the space target is in uniform motion for a short period of time, the process model is expressed as:

[0074] ;

[0075] Where Δt is the time interval between observations of two adjacent frames of the fused image. This is process noise;

[0076] Step 5.3: Based on the process noise covariance Q, predict the maximum possible maneuvering acceleration of the space target according to the time interval Δt between the observations of two adjacent fused images and update the process model to adapt to the dynamic motion changes of the space target in real time.

[0077] Step 5.4: Establish the measurement model h(·), the expression of which is:

[0078] ;

[0079] In the formula, D The relative distance between the spatial target and the observed target as measured by the laser rangefinder;

[0080] Step 5.5: If the laser rangefinder does not obtain a valid range echo at a certain moment, then only the line-of-sight angle observation information is used to predict the spatial target state. At this time, the relative distance in the measurement model will be... D The corresponding measurement variance is set to an ultra-large value, that is, a variance that is at least two orders of magnitude greater than the normal valid measurement value, thereby automatically ignoring the weight of invalid distance measurement information and extrapolating the spatial target state only through line-of-sight angle observation information.

[0081] Compared with the prior art, the advantages of the present invention are as follows:

[0082] (1) The image information source adopts multi-spectral fusion (infrared + visible light), which has stronger anti-interference ability compared with single visible light and single infrared light;

[0083] (2) The synchronization method introduces software time synchronization + image registration, which reduces the hardware deployment and avoids serious frame matching misalignment caused by inaccurate synchronization.

[0084] (3) Introducing a fast infrared mean judgment suitable for satellite embedding, which solves the problem of high false recognition rate caused by the lack of background detection in similar satellites;

[0085] (4) Incorporate Earth background suppression technology and target information enhancement scheme to reduce the target tracking loss rate and address the problem of weak or abandoned target observation in the field of view of similar technologies.

[0086] (5) Adopting a fusion tracking and ranging collaboration of vision + laser rangefinder to mitigate on-board tracking drift, and designing rangefinder start-stop strategy based on on-board power consumption;

[0087] In summary, this solution is suitable for near-Earth orbit space missions, such as docking, formation flying, and rendezvous observation; it can still stably detect and track spacecraft even under strong background interference from Earth; it enhances the ability to resist obstruction and changes in lighting conditions, improving the robustness of the navigation system; it reduces the risk of sensor failure and enhances redundant observation capabilities; the overall navigation and observation system can be adapted to low-power embedded platforms, better meeting the needs of commercial space deployment. Attached Figure Description

[0088] Figure 1 For different visible light images;

[0089] Figure 2 for Figure 1 Thresholded images and dynamic thresholding segmentation maps of various visible light images;

[0090] Figure 3 for Figure 1 (a) Image comparison after moderate brightness suppression;

[0091] Figure 4 for Figure 1 (a) Image after background suppression;

[0092] Figure 5 for Figure 1 (a) High-frequency information analysis diagram of the image before background suppression;

[0093] Figure 6 for Figure 1 (a) High-frequency information analysis diagram of the image after background suppression;

[0094] Figure 7 (a) is a real-shot visible light image, and (b) is a real-shot infrared image;

[0095] Figure 8 In the middle (a) and (b) respectively Figure 7 Local entropy calculation and analysis diagram for visible light and infrared images;

[0096] Figure 9 In the middle (a) and (b) respectively Figure 7 Weighted analysis diagram of visible light and infrared images;

[0097] Figure 10 (a), (b), and (c) are respectively Figure 7 The image shows the fusion of visible light and infrared images, the local entropy fusion effect, and the weighted fusion effect. Detailed Implementation

[0098] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0099] This invention provides a method for target identification and approach in complex environments based on dual-band imaging on a satellite platform, comprising the following steps:

[0100] Step 1: Build an integrated device for a visible light camera, an infrared camera, and a laser rangefinder. Perform coordinate calibration, time synchronization processing, and image sampling transformation on the visible light and infrared cameras. Perform optical axis deviation calibration on the laser rangefinder. Specifically:

[0101] The specific steps for coordinate calibration of visible light cameras and infrared cameras, and optical axis deviation calibration of laser rangefinders, include the following:

[0102] Step 1.1: The intrinsic parameter calibration of the visible light camera and infrared camera adopts the checkerboard calibration method, and the focal length, principal point and distortion information of the visible light camera and infrared camera are obtained by using the corner detection method.

[0103] Step 1.2: After the intrinsic parameter calibration is completed, fix the visible light camera and the infrared camera in the integrated device, then calibrate the central line of sight of the visible light camera and the infrared camera to make the central line of sight of the two cameras parallel, and then use the checkerboard calibration method to calibrate the extrinsic parameters to obtain the coordinate transformation matrix between the visible light camera and the infrared camera and the coordinate transformation matrix between the visible light camera, the infrared camera and the reference coordinate system of the integrated device.

[0104] Step 1.3: Turn on the visible light camera and the laser rangefinder, so that the laser rangefinder spot appears in the field of view of the visible light camera. The visible light camera takes multiple frames of images, calculates the deviation of the laser rangefinder spot from the principal pixel of the visible light camera, and completes the optical axis deviation calibration of the laser rangefinder.

[0105] For time synchronization of visible light and infrared cameras, soft time synchronization is achieved by combining "coarse synchronization" and "fine synchronization" to find the nearest frame. The specific steps include:

[0106] "Coarse synchronization" calculates candidate frames:

[0107] Step 1.4: When the integrated device starts running, it marks each frame of image received by the visible light camera and infrared camera with a receiving timestamp and frame number. Within n seconds of running, it reads the [frame number]. k Received timestamps of visible light images Select the receiving timestamp and... in the infrared image buffer. The closest j A frame of infrared image, the timestamp of which was received is... , making ,in This is a preset time tolerance threshold. R Represents a visible light image. I Represents an infrared image;

[0108] Step 1.5: Combine each pair of image data obtained in Step 1.4 Saved to the paired image data dynamic cache, where Indicates the received timestamp is Given the infrared image frame number, after collecting n seconds of paired data, a least squares fitting method is used to obtain the following linear model of the mapping relationship:

[0109] , ;

[0110] in, The average timestamps of visible light images within the paired image data dynamic buffer. The average frame number of the infrared images within the paired image data dynamic buffer. a , b Representing the coefficients, we obtain the final linear model of the mapping relationship: ;

[0111] Step 1.6: After obtaining the final mapping linear model in Step 1.5, use this linear model to coarsely predict the corresponding infrared image frame number. When each visible light image frame is received, input the visible light image timestamp. The corresponding infrared image frame number is calculated. The coarsely matched infrared image frame number The corresponding image, and the frame number is -1、 The infrared image of +1 is extracted from the paired image data dynamic buffer;

[0112] "Precise synchronization" matches synchronization frames:

[0113] Step 1.7: Perform precise matching between the visible light image extracted in Step 1.6 and the three candidate infrared images. First, extract the gradient map of the visible light image. Similarly, gradient maps of the three candidate infrared images were extracted. ,calculate With three candidate infrared images The normalized similarity is used to determine the most similar infrared image frame as the most synchronized frame with the visible light image, thus completing the "precise synchronization" between the visible light and infrared images.

[0114] The specific steps for image sampling transformation of visible light cameras and infrared cameras are as follows:

[0115] Step 1.8: After obtaining the visible light image and infrared image after time synchronization of the visible light camera and infrared camera, since the two are different in size, the infrared image is upsampled and stretched to make it consistent with the size of the visible light image.

[0116] Step 2: Based on the integrated equipment and time synchronization processing method established in Step 1, the visible light camera and infrared camera acquire image data to obtain visible light and infrared images of the space target. Then, the visible light and infrared images are processed to determine and suppress the Earth background in the space environment. Specifically, the Earth background determination and suppression in the space environment includes the following steps:

[0117] Step 2.1: Based on the analysis of typical characteristics of the Earth, in both visible light and infrared images, the deep space region is completely black, while the Earth region is a region with a significant increase in bright gray value. The brightness / gray level and proportion method is used to quickly determine whether the Earth region appears: During the initial operation of the satellite, the pre-set empirical threshold T is used as the indicator to determine whether the Earth region appears in the image. The average gray value of the entire image is taken and compared with the threshold T. If the average gray value of the entire image is greater than the threshold T, it is considered that the Earth background appears in the field of view of the visible light camera and the infrared camera. Subsequently, during the operation, the average gray value of the entire image is calculated over a period of time. The lowest average gray value of the deep space segment and the highest average gray value of the Earth segment during actual on-orbit operation are selected to dynamically update the threshold T.

[0118] Step 2.2: After thresholding, the infrared image is binarized using the OTSU algorithm to obtain the high grayscale pixel segmentation threshold of the infrared image. Based on the segmentation, it is divided into two levels: one is partial Earth background, and the other is full Earth background.

[0119] Step 2.3: After the judgment in Step 2.2, further processing is performed based on the proportion of the Earth's background. The processing method is as follows:

[0120] The Earth's background percentage is less than Skip Earth background suppression;

[0121] The Earth's background accounts for a certain percentage of the total area. Between: Light feathering mask, specifically:

[0122] First, the visible light image after suppressing the Earth background. With infrared images Each was subjected to a light feathering masking process. x, y This represents the specific coordinates of a pixel in a visible light image or infrared image.

[0123] Based on the thresholding method in step 2.1, the Earth background region is determined from the visible light image and the infrared image, and a mask is constructed. ,in It is a binary mask image;

[0124] Secondly, to avoid obvious abrupt changes in the edges between the mask and the target area, Gaussian convolution is used to smooth the mask edges, resulting in feathering.

[0125] ;

[0126] in, For feathering weight, A two-dimensional Gaussian kernel with empirically derived values;

[0127] Finally, the obtained feathering weights are used to process the visible light image and the infrared image respectively:

[0128] ;

[0129] in, These are the processed visible light image and infrared image, respectively;

[0130] The Earth's background accounts for a certain percentage of the total area. Between: Moderate suppression of the Earth's background is achieved through threshold masking and local contrast enhancement. Specifically:

[0131] By analyzing the dynamic threshold T using histograms, the overall background pixels of infrared and visible light images are darkened, while non-background areas are contrast-stretched, making the dark areas of the overall image darker and the local brightness of the target area brighter.

[0132] The Earth's background ratio is greater than The infrared and visible light images undergo whole-image histogram compression and local entropy saliency enhancement processing. Specifically:

[0133] Histogram compression is applied to the entire infrared and visible light images to reduce overall brightness. Then, the salience of spatial targets is enhanced by local entropy. The weight is increased in regions with low entropy and decreased in regions with high entropy. The salience of bright spots of spatial targets is improved by entropy map, while the background region of the Earth is suppressed.

[0134] The preset percentage threshold, the specific value and the range of the endpoint values ​​are set according to different scenarios based on the ground test.

[0135] Using actual visible light images as an example, this explains how to determine and suppress the Earth's background:

[0136] refer to Figure 1 (a), (b), (c), and (d) are different visible light images. As the satellite's attitude and orbital state change, different background regions of the Earth will appear in the visible light field of view.

[0137] First, thresholding segmentation is performed on the visible light image to obtain... Figure 1 The proportions of Earth's background in the various visible light images are 62.53%, 34.33%, 55.39%, and 44.98%, respectively. Their thresholded images and dynamic thresholding segmentation maps are shown below. Figure 2 As shown in (a), (b), (c), (d), (e), (f), (g), and (h).

[0138] The Earth background branch is set to four levels: less than 30%, 30-50%, 50-70%, and greater than 70%. Figure 1 Taking Figure (a) as an example, the Earth occupies 62.53% of the image, requiring threshold masking and local contrast enhancement to moderately suppress the Earth background.

[0139] Histogram analysis yielded a threshold of 97, which darkens the overall background pixels of the image while contrast-stretching non-background areas. This makes dark areas darker and bright areas brighter, ensuring that high-frequency information of the target point is not lost. (See [link to relevant documentation]). Figure 3 (a), (b), (c), (d);

[0140] Secondly, feathering is used to suppress and eliminate the Earth background. See [link / reference] Figure 4 Through high-frequency information analysis, it can be seen that Figure 5 The high-frequency information of the target in the original image was not affected. Figure 6 The bright areas of the Earth after processing are significantly suppressed, and the target features are unaffected and even more prominent.

[0141] Step 3: Visible light and infrared images are fused using a dynamic weighting and multi-scale fusion algorithm to obtain the line-of-sight angle observation information of spatial targets in the fused image. This specifically includes the following steps:

[0142] Step 3.1: First, perform dynamic weighted fusion of the visible light image and the infrared image:

[0143] ;

[0144] in, For coarsely blended images, the blending weights are... Adaptive calculations are performed based on the actual conditions in the space, and the calculation method is as follows:

[0145] ;

[0146] in, These are the local variances of the infrared image and the visible light image, respectively. The local entropy of the infrared image. These are the weighting coefficients. To prevent division by zero; higher local variance and local entropy of infrared images indicate a larger weighting of infrared images. Therefore, the weighting coefficients for fusing infrared and visible light images are dynamically determined based on the actual imaging situation.

[0147] Step 3.2: Use infrared images for coarse detection of spatial target regions, introducing local entropy, local standard deviation, and adjustment coefficients to identify local hotspots or areas of significant motion. The specific method is as follows:

[0148] ;

[0149] in, DOG It involves performing differential Gaussian processing on the image to remove large areas of smooth background and significantly enhance the high-frequency regions of the spatial target area. and For adjustment coefficients, Represents the maximum local standard deviation in an infrared image;

[0150] Step 3.3: Based on the real-time situation on the satellite, dynamically change the adjustment coefficients in Step 3.2 according to the background mean. When the current image imaging result shows weak thermal targets, a large proportion of the Earth background, and abundant cloud texture information, increase the adjustment coefficients. DOG weighting percentage Reduce the weight of fine-texture features while increasing the weight of features resembling spatial targets; when the thermal features of the spatial target are strong and the background noise is high, increase... This allows variance to suppress noise and simultaneously reduce DOG Operator weights should be adjusted to avoid treating excessive noise as spatial targets.

[0151] Step 3.4: After acquiring the local space target region through infrared images, in order to reduce the computing power burden on the satellite, only the space target region is subjected to multi-scale Laplacian pyramid fusion. The final fused image is the space target region with the highest accuracy to ensure infrared thermal characteristics and visible light texture information, while the non-space target region retains the coarse fused image.

[0152] Step 3.5: After extracting the spatial target, determine the displacement of the spatial target within multiple frames of images based on motion consistency analysis, accurately identify the spatial target, and output the spatial target information based on the confidence level;

[0153] Step 3.6: Extract the centroid pixel coordinates (u,v) of the spatial target in the fused image using Step 3.5, and obtain the image plane line of sight by combining the camera intrinsic and extrinsic parameters as follows:

[0154] ;

[0155] ;

[0156] ;

[0157] Small angle approximate line-of-sight pitch angle Azimuth as follows:

[0158] ;

[0159] in, and These are the coordinates of the principal point of the visible light camera's intrinsic parameters. and It is the pixel focal length of a visible light camera. These are normalized lateral image plane coordinates. These are the normalized vertical image plane coordinates. It is the normalized direction of the unit line-of-sight vector in the visible light camera coordinate system.

[0160] Real photos of the satellite's actual propulsion system Figure 7 Taking an example, the dynamic weighted fusion process of infrared and visible light images is explained:

[0161] First, a dynamic weighted fusion analysis is performed on the visible light image and the infrared image. Figure 8 and Figure 9The analysis results show that the partial entropy values ​​of the visible light images are 3.04, 5.74, and 3.54, while the corresponding local entropy values ​​of the infrared images are 3.26, 5.98, and 5.32, respectively. The analysis shows that the local variance and local entropy of the infrared images are higher, which also indicates that the infrared images have a greater weight in the fusion process.

[0162] In this embodiment, the local entropy radius is set to 9, the Gaussian smoothing kernel is set to 5, and fusion is performed after dynamic weighted analysis. Figure 10 (a), (b), and (c) are Figure 7 The image shows the fusion results of visible light and infrared images, local entropy fusion, and weighted fusion, demonstrating the data fusion of infrared and visible light images.

[0163] Step 4: After obtaining the line-of-sight angle observation information of the space target, set the laser rangefinder switching strategy, and turn on the laser rangefinder to measure distances when needed, and obtain the laser rangefinder measurement information. The specific laser rangefinder switching strategy is as follows:

[0164] Step 4.1: The judgment is performed sequentially according to three parts: confidence gating, optical axis deviation gating, and count gating. The confidence gating judgment method is as follows: ,in, This represents the confidence level of spatial targets in the fused image of the current frame. The confidence threshold is used to determine whether the optical axis deviation is entered into the judgment process when the confidence of the spatial target in the fused image of the current frame is greater than or equal to the confidence threshold. The judgment is based on the following criteria:

[0165] ;

[0166] in, , The coordinates of the centroid pixels of the spatial target in the fused image of the current frame. , It is the deviation of the optical axis between the ground-calibrated laser rangefinder and the visible light camera, ultimately obtaining the instantaneous deviation of the centroid of the spatial target relative to the optical axis of the laser rangefinder. and ;

[0167] Step 4.2: The deviation judgment condition is obtained by converting the ground calibration pixel threshold and spatial error to obtain the allowable threshold for spatial target-spot size deviation. px, when and If the threshold is within the specified range, the counting gate judgment is entered. In the counting gate, if the fused images of M consecutive frames stably meet the above conditions, the laser rangefinder is confirmed to be turned on.

[0168] Step 5: Continuous unscented Kalman filter algorithm state estimation of the spatial target using line-of-sight observation information and laser rangefinder measurement information: By establishing a state model of the spatial target relative to the observed target, and assuming that the spatial target moves at a constant speed in the short term as a state transition process, the motion state of the spatial target is predicted using the time interval between observations of two adjacent fused image frames and the process noise covariance; and a corresponding measurement model is constructed based on the relative position state of the spatial target and the observed target. When there is no effective laser rangefinder measurement information at a certain moment, a large variance method is used to extrapolate and predict the state of the spatial target using only line-of-sight observation information, specifically including:

[0169] Step 5.1: Construct the state vector X , is defined as the relative state of a space target in the reference coordinate system of the observed target, where the state vector is... Represents the position coordinates of the space target relative to the observed target. With the corresponding three velocity states ;

[0170] Step 5.2: Establish the state transition model F(·) of the space target relative to the observed target. Assuming that the space target is in uniform motion for a short period of time, the process model is expressed as:

[0171] ;

[0172] Where Δt is the time interval between observations of two adjacent frames of the fused image. This is process noise;

[0173] Step 5.3: Based on the process noise covariance Q, predict the maximum possible maneuvering acceleration of the space target according to the time interval Δt between the observations of two adjacent fused images and update the process model to adapt to the dynamic motion changes of the space target in real time.

[0174] Step 5.4: Establish the measurement model h(·), the expression of which is:

[0175] ;

[0176] In the formula, D The relative distance between the spatial target and the observed target as measured by the laser rangefinder;

[0177] Step 5.5: If the laser rangefinder does not obtain a valid range echo at a certain moment, then only the line-of-sight angle observation information is used to predict the spatial target state. At this time, the relative distance in the measurement model will be... D The corresponding measurement variance is set to an ultra-large value, that is, a variance that is at least two orders of magnitude greater than the normal valid measurement value, thereby automatically ignoring the weight of invalid distance measurement information and extrapolating the spatial target state only through line-of-sight angle observation information.

[0178] Step 6: By reading the line-of-sight angle observation information of each frame of the fused image, and if there is laser rangefinder measurement information, it is also read. The information is input into the state model and measurement model in Step 5 to predict the state of the space target and update the state model and measurement model. Finally, the position and velocity information of the space target are output to approach the space target.

Claims

1. A method for target identification and approach in complex environments based on dual-band imaging on a satellite platform, characterized in that, The steps include: Step 1: Build an integrated device for a visible light camera, an infrared camera, and a laser measuring instrument. Perform coordinate calibration, time synchronization processing, and image sampling transformation on the visible light camera and the infrared camera. Perform optical axis deviation calibration on the laser rangefinder. Step 2: Based on the integrated equipment and time synchronization processing method built in Step 1, the visible light camera and infrared camera collect image data to obtain visible light and infrared images of space targets, and perform Earth background determination and suppression on the visible light and infrared images in response to the space environment. Step 3: Use a dynamic weighting and multi-scale fusion algorithm to fuse visible light and infrared images to obtain the line-of-sight angle observation information of spatial targets in the fused image; Step 4: After obtaining the line-of-sight angle observation information of the space target, set the laser rangefinder on / off strategy, and turn on the laser rangefinder to measure distances when needed, and obtain the laser rangefinder measurement information: Step 5: Continuous unscented Kalman filter algorithm state estimation of the spatial target using line-of-sight observation information and laser rangefinder measurement information: By establishing a state model of the spatial target relative to the observed target, and assuming that the spatial target moves at a constant speed in the short term as a state transition process, the motion state of the spatial target is predicted using the time interval between observations of two adjacent fused images and the process noise covariance; and a corresponding measurement model is constructed based on the relative position state of the spatial target and the observed target. When there is no effective laser rangefinder measurement information at a certain moment, a large variance method is used to extrapolate and predict the state of the spatial target using only line-of-sight observation information. Step 6: By reading the line-of-sight angle observation information of each frame of the fused image, and if there is laser rangefinder measurement information, it is also read. The information is input into the state model and measurement model in Step 5 to predict the state of the space target and update the state model and measurement model. Finally, the position and velocity information of the space target are output to approach the space target.

2. The method for target identification and approach in complex environments based on dual-band imaging on a satellite platform according to claim 1, characterized in that, Step 1, which involves coordinate calibration of the visible light camera and infrared camera, and optical axis deviation calibration of the laser rangefinder, specifically includes the following steps: Step 1.1: The intrinsic parameter calibration of the visible light camera and infrared camera adopts the checkerboard calibration method, and the focal length, principal point and distortion information of the visible light camera and infrared camera are obtained by using the corner detection method. Step 1.2: After the intrinsic parameter calibration is completed, fix the visible light camera and the infrared camera in the integrated device, then calibrate the central line of sight of the visible light camera and the infrared camera to make the central line of sight of the two cameras parallel, and then use the checkerboard calibration method to calibrate the extrinsic parameters to obtain the coordinate transformation matrix between the visible light camera and the infrared camera and the coordinate transformation matrix between the visible light camera, the infrared camera and the reference coordinate system of the integrated device. Step 1.3: Turn on the visible light camera and the laser rangefinder, so that the laser rangefinder spot appears in the field of view of the visible light camera. The visible light camera takes multiple frames of images, calculates the deviation of the laser rangefinder spot from the principal pixel of the visible light camera, and completes the optical axis deviation calibration of the laser rangefinder.

3. The method for target identification and approach in complex environments based on dual-band imaging on a satellite platform according to claim 2, characterized in that, In step 1, time synchronization processing for the visible light camera and infrared camera involves selecting a combination of "coarse synchronization" and "fine synchronization" to find the nearest frame and achieve soft time synchronization. This specifically includes the following steps: "Coarse synchronization" calculates candidate frames: Step 1.4: When the integrated device starts running, it marks each frame of image received by the visible light camera and infrared camera with a receiving timestamp and frame number. Within n seconds of running, it reads the [frame number]. k Received timestamps of visible light images Select the receiving timestamp and... in the infrared image buffer. The closest j A frame of infrared image, the timestamp of which was received is... , making ,in This is a preset time tolerance threshold. R Represents a visible light image. I Represents an infrared image; Step 1.5: Combine each pair of image data obtained in Step 1.4 Saved to the paired image data dynamic cache, where Indicates the received timestamp is Given the infrared image frame number, after collecting n seconds of paired data, a least squares fitting method is used to obtain the following linear model of the mapping relationship: ， ； in, The average timestamps of visible light images within the paired image data dynamic buffer. The average frame number of the infrared images within the paired image data dynamic buffer. a , b Representing the coefficients, we obtain the final linear model of the mapping relationship: ; Step 1.6: After obtaining the final mapping linear model in Step 1.5, use this linear model to coarsely predict the corresponding infrared image frame number. When each visible light image frame is received, input the visible light image timestamp. The corresponding infrared image frame number is calculated. The coarsely matched infrared image frame number The corresponding image, and the frame number is -1、 The infrared image of +1 is extracted from the paired image data dynamic buffer; "Precise synchronization" matches synchronization frames: Step 1.7: Perform precise matching between the visible light image extracted in Step 1.6 and the three candidate infrared images. First, extract the gradient map of the visible light image. Similarly, gradient maps of the three candidate infrared images were extracted. ,calculate With three candidate infrared images The normalized similarity is used to determine the most similar infrared image frame as the most synchronized frame with the visible light image, thus completing the "precise synchronization" between the visible light and infrared images.

4. The method for target identification and approach in complex environments based on dual-band imaging on a satellite platform according to claim 3, characterized in that, In step 1, the image sampling transformation of the visible light camera and the infrared camera specifically involves: Step 1.8: After obtaining the visible light image and infrared image after time synchronization of the visible light camera and infrared camera, since the two are different in size, the infrared image is upsampled and stretched to make it consistent with the size of the visible light image.

5. The method for target identification and approach in complex environments based on dual-band imaging on a satellite platform according to claim 1, characterized in that, Step 2, which involves determining and suppressing the Earth background in the space environment, includes the following steps: Step 2.1: Based on the analysis of typical characteristics of the Earth, in both visible light and infrared images, the deep space region is completely black, while the Earth region is a region with a significant increase in bright gray value. The brightness / gray level and proportion method is used to quickly determine whether the Earth region appears: During the initial operation of the satellite, the pre-set empirical threshold T is used as the indicator to determine whether the Earth region appears in the image. The average gray value of the entire image is taken and compared with the threshold T. If the average gray value of the entire image is greater than the threshold T, it is considered that the Earth background appears in the field of view of the visible light camera and the infrared camera. Subsequently, during the operation, the average gray value of the entire image is calculated over a period of time. The lowest average gray value of the deep space segment and the highest average gray value of the Earth segment during actual on-orbit operation are selected to dynamically update the threshold T. Step 2.2: After thresholding, the infrared image is binarized using the OTSU algorithm to obtain the high grayscale pixel segmentation threshold of the infrared image. Based on the segmentation, it is divided into two levels: one is partial Earth background, and the other is full Earth background. Step 2.3: After the judgment in Step 2.2, further processing is performed based on the proportion of the Earth's background. The processing method is as follows: The Earth's background percentage is less than Skip Earth background suppression; The Earth's background accounts for a certain percentage of the total area. Between: Light feathering mask, specifically: First, the visible light image after suppressing the Earth background. With infrared images Each was subjected to a light feathering masking process. x, y This represents the specific coordinates of a pixel in a visible light image or infrared image. Based on the thresholding method in step 2.1, the Earth background region is determined from the visible light image and the infrared image, and a mask is constructed. ,in It is a binary mask image; Secondly, to avoid obvious abrupt changes in the edges between the mask and the target area, Gaussian convolution is used to smooth the mask edges, resulting in feathering. ； in, For feathering weight, A two-dimensional Gaussian kernel with empirically derived values; Finally, the obtained feathering weights are used to process the visible light image and the infrared image respectively: ； in, These are the processed visible light image and infrared image, respectively; The Earth's background accounts for a certain percentage of the total area. Between: Moderate suppression of the Earth's background is achieved through threshold masking and local contrast enhancement. Specifically: By analyzing the dynamic threshold T using histograms, the overall background pixels of infrared and visible light images are darkened, while non-background areas are contrast-stretched, making the dark areas of the overall image darker and the local brightness of the target area brighter. The Earth's background ratio is greater than The infrared and visible light images undergo whole-image histogram compression and local entropy saliency enhancement processing. Specifically: Histogram compression is applied to the entire infrared and visible light images to reduce overall brightness. Then, the salience of spatial targets is enhanced by local entropy. The weight is increased in regions with low entropy and decreased in regions with high entropy. The salience of bright spots of spatial targets is improved by entropy map, while the background region of the Earth is suppressed. The preset percentage threshold, the specific value and the range of the endpoint values ​​are set according to different scenarios based on the ground test.

6. The method for target identification and approach in complex environments based on dual-band imaging on a satellite platform according to claim 5, characterized in that, Step 3 includes the following steps: Step 3.1: First, perform dynamic weighted fusion of the visible light image and the infrared image: ； in, For coarsely blended images, the blending weights are... Adaptive calculations are performed based on the actual conditions in the space, and the calculation method is as follows: ； in, These are the local variances of the infrared image and the visible light image, respectively. The local entropy of the infrared image. These are the weighting coefficients. To prevent division by zero; higher local variance and local entropy of infrared images indicate a larger weighting of infrared images. Therefore, the weighting coefficients for fusing infrared and visible light images are dynamically determined based on the actual imaging situation. Step 3.2: Use infrared images for coarse detection of spatial target regions, introducing local entropy, local standard deviation, and adjustment coefficients to identify local hotspots or areas of significant motion. The specific method is as follows: ； in, DOG It involves performing differential Gaussian processing on the image to remove large areas of smooth background and significantly enhance the high-frequency regions of the spatial target area. and For adjustment coefficients, Represents the maximum local standard deviation in an infrared image; Step 3.3: Based on the real-time situation on the satellite, dynamically change the adjustment coefficients in Step 3.2 according to the background mean. When the current image imaging result shows weak thermal targets, a large proportion of the Earth background, and abundant cloud texture information, improve... DOG weighting percentage Reduce the weight of fine-texture features while increasing the weight of features resembling spatial targets; when the thermal features of the spatial target are strong and the background noise is high, increase... This allows variance to suppress noise and simultaneously reduce DOG Operator weights should be adjusted to avoid treating excessive noise as spatial targets. Step 3.4: After acquiring the local space target region through infrared images, in order to reduce the computing power burden on the satellite, only the space target region is subjected to multi-scale Laplacian pyramid fusion. The final fused image is the space target region with the highest accuracy to ensure infrared thermal characteristics and visible light texture information, while the non-space target region retains the coarse fused image. Step 3.5: After extracting the spatial target, determine the displacement of the spatial target within multiple frames of images based on motion consistency analysis, accurately identify the spatial target, and output the spatial target information based on the confidence level; Step 3.6: Extract the centroid pixel coordinates (u,v) of the spatial target in the fused image using Step 3.5, and obtain the image plane line of sight by combining the camera intrinsic and extrinsic parameters as follows: ； ； ； Small angle approximate line-of-sight pitch angle Azimuth as follows: ； in, and These are the coordinates of the principal point of the visible light camera's intrinsic parameters. and It is the pixel focal length of a visible light camera. These are normalized lateral image plane coordinates. These are the normalized vertical image plane coordinates. It is the normalized direction of the unit line-of-sight vector in the visible light camera coordinate system.

7. The method for target identification and approach in complex environments based on dual-band imaging on a satellite platform according to claim 6, characterized in that, The laser rangefinder switching strategy in step 4 is as follows: Step 4.1: The judgment is performed sequentially according to three parts: confidence gating, optical axis deviation gating, and count gating. The confidence gating judgment method is as follows: ,in, This represents the confidence level of spatial targets in the fused image of the current frame. The confidence threshold is used to determine whether the optical axis deviation is entered into the judgment process when the confidence of the spatial target in the fused image of the current frame is greater than or equal to the confidence threshold. The judgment is based on the following criteria: ； in, , The coordinates of the centroid pixels of the spatial target in the fused image of the current frame. , It is the deviation of the optical axis between the ground-calibrated laser rangefinder and the visible light camera, ultimately obtaining the instantaneous deviation of the centroid of the spatial target relative to the optical axis of the laser rangefinder. and ; Step 4.2: The deviation judgment condition is obtained by converting the ground calibration pixel threshold and spatial error to obtain the allowable threshold for spatial target-spot size deviation. px, when and If the threshold is within the specified range, the counting gate judgment is entered. In the counting gate, if the fused images of M consecutive frames stably meet the above conditions, the laser rangefinder is confirmed to be turned on.

8. The method for target identification and approach in complex environments based on dual-band imaging on a satellite platform according to claim 6, characterized in that, Step 5 specifically includes: Step 5.1: Construct the state vector X , is defined as the relative state of a space target in the reference coordinate system of the observed target, where the state vector is... Represents the position coordinates of the space target relative to the observed target. With the corresponding three velocity states ; Step 5.2: Establish the state transition model F(·) of the space target relative to the observed target. Assuming that the space target is in uniform motion for a short period of time, the process model is expressed as: ； Where Δt is the time interval between observations of two adjacent frames of the fused image. This is process noise; Step 5.3: Based on the process noise covariance Q, predict the maximum possible maneuvering acceleration of the space target according to the time interval Δt between the observations of two adjacent fused images and update the process model to adapt to the dynamic motion changes of the space target in real time. Step 5.4: Establish the measurement model h(·), the expression of which is: ； In the formula, D The relative distance between the spatial target and the observed target as measured by the laser rangefinder; Step 5.5: If the laser rangefinder does not obtain a valid range echo at a certain moment, then only the line-of-sight angle observation information is used to predict the spatial target state. At this time, the relative distance in the measurement model will be... D The corresponding measurement variance is set to an ultra-large value, that is, a variance that is at least two orders of magnitude greater than the normal valid measurement value, thereby automatically ignoring the weight of invalid distance measurement information and extrapolating the spatial target state only through line-of-sight angle observation information.

Images