Complex environment target identification approaching method based on dual-band imaging under satellite platform

Abstract

The invention belongs to the technical field of spaceflight on-orbit services, and particularly relates to a complex environment target recognition approaching method based on dual-band imaging under a satellite platform, which is based on three common sensors, namely an infrared camera, a visible light camera and a laser range finder, and is characterized in that an image information source adopts infrared and visible light multi-spectrum fusion; software time synchronization and an image registration mode are introduced into a synchronization mode, infrared fast mean value judgment suitable for on-satellite embedding is introduced, an earth background suppression technology and a target information enhancement scheme are added, fusion tracking and ranging cooperation of vision and a laser range finder is adopted, and meanwhile, a laser range finder start-stop strategy is designed according to on-satellite power consumption. According to the invention, the defect of insufficient robustness of a single sensor facing an earth complex background observation target in the view field change process of a satellite platform is overcome.

Description

Technical Field

[0001] The present invention belongs to the field of aerospace on-orbit service technology, and in particular relates to a complex environment target recognition and approach method based on dual-band imaging on a satellite platform. Background Art

[0002] At present, the number of spacecraft in orbit is increasing rapidly year by year. At the same time, the generation of space debris has also brought many safety hazards. Being able to identify, capture and track space targets in a space environment, and timely discover and judge the intentions of space targets can reduce the probability of spacecraft collisions in orbit.

[0003] During spacecraft rendezvous and docking missions, optical sensors often require good imaging conditions with the target spacecraft. Current satellite platforms struggle to adapt to the changing environment of space with single optical sensors. Infrared cameras inherently have low resolution, while visible light cameras are overly dependent on lighting conditions. In low or bright light conditions, the imaging quality of the target spacecraft degrades significantly, resulting in image blur and loss of feature information. Motion blur also impacts the camera's image quality when tracking spacecraft motion, complicating subsequent target recognition and tracking algorithms. Furthermore, in space, the camera's field of view can shift, potentially resulting in images facing Earth or split between Earth and deep space. In these cases, the target can be lost against Earth's complex background, making it difficult to detect or track it. Therefore, combining the strengths of different sensors to achieve more robust target recognition and richer target information is crucial for current spacecraft safety.

[0004] In summary, in view of the fact that the current space target recognition process is easily affected by the space environment, resulting in poor target recognition robustness, a dual-band imaging space target recognition and capture method based on infrared and visible light is proposed, which uses multiple data sources to improve the robustness of target recognition. Summary of the Invention

[0005] In order to solve the problem that the characteristics of sensors on satellite platforms are affected by the environment, and a single sensor cannot accurately and robustly obtain observation information of space targets in complex and changeable space environments, the present invention is based on three commonly used sensors: infrared camera, visible light camera and laser rangefinder. It provides a method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform. It overcomes the defect that a single sensor on a satellite platform is insufficiently robust in observing targets against the complex background of the earth when the field of view changes. It establishes a method for dual-band observation and identification of targets, and supplements and enriches the observation information with a laser rangefinder.

[0006] The present invention is implemented by providing a method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform, comprising the following steps: Step 1: Build an integrated device consisting of a visible light camera, an infrared camera, and a laser tester. Perform coordinate calibration, time synchronization processing, and image sampling conversion on the visible light camera and infrared camera, and perform optical axis deviation calibration on the laser rangefinder. Step 2: Based on the integrated equipment built in Step 1 and the time synchronization processing method, the visible light camera and infrared camera collect image data to obtain visible light images and infrared images of the space target. The visible light images and infrared images are then used to determine and suppress the Earth background based on the space environment. Step 3: Use dynamic weighting and multi-scale fusion algorithm to fuse visible light image and infrared image to obtain the viewing angle observation information of space target in the fused image; Step 4: After obtaining the viewing angle observation information of the space target, set the laser rangefinder switching strategy, turn on the laser rangefinder for ranging when needed, and obtain the laser rangefinder measurement information: Step 5: Use the line-of-sight angle observation information and laser rangefinder measurement information to perform continuous unscented Kalman filter algorithm state estimation on the space target: By establishing a state model of the space target relative to the observed target and assuming that the space target moves at a uniform speed in a short period of time as the state transition process, the motion state of the space target is predicted using the time interval between two adjacent frames of fused image observations and the process noise covariance; and based on the relative position state of the space target and the observed target, a corresponding measurement model is constructed. When there is no valid laser rangefinder measurement information at a certain moment, the large variance method is used to extrapolate the space target state using only the line-of-sight angle observation information; Step 6: By reading the viewing angle observation information of each frame of the fused image, and reading the laser rangefinder measurement information if available, the information is input into the state model and measurement model in step 5 to predict the state of the space target and update the state model and measurement model. Finally, the position and velocity information of the space target is output to approach the space target.

[0007] Preferably, in step 1, calibrating the coordinates of the visible light camera and the infrared camera and calibrating the optical axis deviation of the laser rangefinder specifically include the following steps: Step 1.1: The intrinsic calibration of the visible light camera and infrared camera is carried out using the checkerboard calibration method. The focal length, principal point, and distortion information of the visible light camera and infrared camera are obtained by using the corner point detection method. Step 1.2: After the intrinsic parameter calibration is completed, the visible light camera and the infrared camera are fixed in the integrated device. The central viewing axes of the visible light camera and the infrared camera are calibrated to make the central viewing axes of the two cameras parallel. Then, the extrinsic parameters are calibrated using the checkerboard calibration method to obtain the coordinate transformation matrix between the visible light camera and the infrared camera, as well as the coordinate transformation matrix between the visible light camera, the infrared camera and the reference coordinate system of the integrated device. Step 1.3: Turn on the visible light camera and laser rangefinder so that the laser rangefinder's light spot appears in the visible light camera's field of view. The visible light camera captures multiple frames of images and calculates the deviation between the laser rangefinder's light spot and the visible light camera's principal point pixel to complete the laser rangefinder optical axis deviation calibration.

[0008] Further preferably, in step 1, the time synchronization processing for the visible light camera and the infrared camera is performed by selecting "coarse synchronization" plus "fine synchronization" to find the nearest frame to achieve soft time synchronization, which specifically includes the following steps: "Coarse synchronization" calculates candidate frames: Step 1.4: When the integrated device starts running, it marks the receiving timestamp and frame number of each frame of image received by the visible light camera and the infrared camera, and reads the first frame within n seconds. k Receiving timestamp of the frame visible light image , select the receiving timestamp and The closest j Frame infrared image, the receiving timestamp of the infrared image is , making ,in is the preset time tolerance threshold, R represents a visible light image, I represents infrared image; Step 1.5: Match each pair of image data obtained in step 1.4 Save to the paired image data dynamic buffer area, where Indicates that the receiving timestamp is After collecting n seconds of paired data, the least squares method is used to fit the infrared image frame number and the following mapping relationship linear model is obtained: , ; in, is the timestamp mean of the visible light image in the dynamic buffer of paired image data, is the average frame number of the infrared image in the dynamic buffer of paired image data, a 、 b Represents the coefficients, and the final mapping relationship linear model is obtained: ; Step 1.6: After obtaining the final mapping relationship linear model in step 1.5, use this linear model to roughly predict the corresponding infrared image frame number. When receiving each frame of visible light image, input the visible light image timestamp , calculate the corresponding infrared image frame number , the roughly matched infrared image frame number The corresponding image and frame number are -1. +1 infrared image is extracted from the paired image data dynamic buffer; "Fine Sync" matches the synchronization frame: Step 1.7: Accurately match the visible light image extracted in step 1.6 with the three candidate infrared images. First, extract the gradient map of the visible light image. , and also extract the gradient maps of the three candidate infrared images respectively ,calculate and three frames of candidate infrared images The normalized similarity of the infrared image is used to determine the most similar frame, and the most synchronized frame with the visible light image is considered. This completes the “precise synchronization” of the visible light image and the infrared image.

[0009] More preferably, in step 1, the image sampling transformation of the visible light camera and the infrared camera is specifically performed as follows: Step 1.8: After obtaining the time-synchronized visible light and infrared images from the visible light and infrared cameras, upsample the infrared image to match the size of the visible light image because of their different sizes.

[0010] Preferably, in step 2, determining and suppressing the earth background for the space environment includes the following steps: Step 2.1: Based on the typical characteristics of the Earth, in both visible light and infrared images, deep space areas appear completely black, while Earth areas appear as areas with significantly increased bright grayscale values. A brightness / grayscale level and ratio method is used to quickly determine whether Earth areas appear. During the initial operation of the satellite, a pre-set empirical threshold T on the ground is used as the criterion for determining whether Earth areas appear in the image. The grayscale mean of the entire image is taken and compared with the threshold T. If the grayscale mean of the entire image is greater than the threshold T, it is considered that the Earth background appears in the field of view of the visible light camera and the infrared camera. Subsequently, during the operation, the average grayscale mean of the entire image is calculated over a period of time. The lowest grayscale mean in the deep space segment and the highest grayscale mean in the Earth segment during the actual on-orbit operation are selected to dynamically update the threshold T. Step 2.2: After threshold judgment, the infrared image is binarized using the OTSU algorithm to obtain the high grayscale pixel segmentation threshold of the infrared image. The image is divided into two levels according to the segmentation situation: one is the partial earth background and the other is the full earth background. Step 2.3: After judging in step 2.2, perform subsequent processing based on the earth background ratio. The processing method is as follows: The earth's background is less than : Skip earth background suppression; The earth's background accounts for Between: Lightly feathered mask, specifically: First, the visible light image after suppressing the background of the earth With infrared images Perform light feathering mask processing respectively, ( x,y ) represents the specific coordinates of a pixel point in the visible light image and infrared image: According to the threshold method in step 2.1, the background area of ​​the earth is determined from the visible light image and the infrared image, and a mask is constructed. ,in is a binary mask image; Secondly, to avoid obvious edge mutations between the mask and the spatial target area, the mask edge is smoothed and feathered by Gaussian convolution: ; in, is the feather weight, is a two-dimensional Gaussian kernel with empirical values; Finally, the obtained feathering weights are used to process the visible light image and infrared image respectively, as follows: ; in, They are the processed visible light image and infrared image respectively; The earth's background accounts for Between: Moderately suppress the earth background through threshold mask + local contrast enhancement, specifically: By using the dynamic threshold T of histogram analysis, the overall background pixels of the infrared and visible light images are darkened, and the non-background areas are contrast-stretched, making the dark areas of the overall image darker and the local brightness of the bright areas brighter. The earth's background accounts for more than :Perform whole-image histogram compression and local entropy clamp saliency enhancement on infrared and visible light images. Specifically: Histogram compression is used on the entire infrared image and visible light image to reduce the overall brightness. The significance of the space target is then enhanced through local entropy. The weight is increased in areas with low entropy and decreased in areas with high entropy. The entropy map is used to enhance the significance of the bright spots of the space target and suppress the background area of ​​the earth. The preset percentage threshold value, and the specific value and endpoint value range are set for different scenarios based on ground tests.

[0011] More preferably, step 3 includes the following steps: Step 3.1: First, perform dynamic weighted fusion of the visible light image and the infrared image: ; in, is the coarse fusion image, fusion weight Adaptive calculation is performed based on the actual situation in the space. The calculation method is as follows: ; in, are the local variances of infrared images and visible light images respectively, is the local entropy of the infrared image, is the weight coefficient, It is division by zero protection; higher local variance and local entropy of infrared images indicate that the infrared image has a larger weight. Therefore, the weight coefficient of the fusion of infrared image and visible light image is dynamically determined according to the actual imaging situation. Step 3.2: Use infrared images to perform rough detection of spatial target areas. Introduce local entropy, local standard deviation, and adjustment coefficient to find local hot spots or areas with obvious motion. The specific method is as follows: ; in, DOG It is a differential Gaussian processing of the image, which removes large blocks of smooth background and significantly improves the high-frequency area of ​​the spatial target area. and is the adjustment coefficient, Represents the local standard deviation maximum in the infrared image; Step 3.3: According to the real-time situation on the satellite, dynamically change the adjustment coefficient in step 3.2 according to the background mean. When the current image imaging result shows that the thermal target is weak and the earth background accounts for a large proportion and there is a lot of cloud texture information, improve DOG The weight ratio , reduce the weight of fine texture features, and increase the weight of features similar to space targets; when the thermal features of the space target are strong and the background noise is high, increase , allowing the variance to suppress noise while reducing DOGThe weight of the operator is used to avoid treating too many noise points as spatial targets; Step 3.4: After the local space target area is acquired through infrared images, in order to reduce the computing power burden on the satellite, multi-scale Laplacian pyramid fusion is performed only on the space target area. The final fused image is the space target area to ensure the infrared thermal characteristics and visible light texture information to the greatest extent possible, while the non-space target area remains a coarse fused image; Step 3.5: After extracting the spatial target, determine the displacement of the spatial target within the multi-frame image based on motion consistency analysis, accurately identify the spatial target, and output the spatial target information based on the confidence level; Step 3.6: Extract the centroid pixel coordinates (u, v) of the spatial target in the fused image through step 3.5, and combine the camera intrinsic and extrinsic parameters to obtain the image plane line of sight as follows: ; ; ; Approximate line-of-sight pitch angle at small angles , azimuth as follows: ; in, and is the coordinate of the principal point of the visible light camera, and is the pixel focal length of the visible light camera, is the normalized transverse image plane coordinate, is the normalized longitudinal image plane coordinate, is the normalized direction of the unit line of sight vector in the visible light camera coordinate system.

[0012] Further preferably, the laser rangefinder switching strategy in step 4 is specifically as follows: Step 4.1: Make judgments in three parts: confidence gate, optical axis deviation gate, and count gate. The confidence gate judgment method is: ,in, is the confidence of the spatial target in the fused image of the current frame, is the confidence gating threshold. When the confidence of the spatial target in the current frame fusion image is greater than or equal to the confidence gating threshold, the optical axis deviation gating judgment is entered. The judgment is based on the following: ; in, , is the pixel coordinate of the centroid of the spatial target in the fused image of the current frame, , It is the optical axis deviation of the laser rangefinder and the visible light camera calibrated on the ground, and finally the instantaneous deviation of the center of mass of the space target relative to the optical axis of the laser rangefinder is obtained. and ; Step 4.2: The deviation judgment condition is converted by the ground calibration pixel threshold and the spatial error to obtain the space target-spot size deviation allowable threshold px, when and Within the threshold, the counting gate judgment is entered; in the counting gate, if the continuous M-frame fusion image stably meets the above conditions, it is confirmed that the laser rangefinder is turned on.

[0013] More preferably, the step 5 specifically includes: Step 5.1: Construct the state vector X , is defined as the relative state of the space target in the observation target reference coordinate system, where the state vector Represents the position coordinates of the space target relative to the observed target Corresponding to the three speed states ; Step 5.2: Establish the state transition model F(·) of the space target relative to the observation target. Assuming that the space target is in a uniform motion state within a short period of time, the process model can be expressed as: ; Among them, Δt is the time interval between the observations of two adjacent fusion images. is the process noise; Step 5.3: Based on the process noise covariance Q and the time interval Δt between two adjacent fused image frames, the maximum possible maneuvering acceleration of the space target is predicted and the process model is updated to adapt to the dynamic motion changes of the space target in real time. Step 5.4: Establish the measurement model h(·). The measurement model expression is: ; Where, D The relative distance between the space target and the observation target measured by the laser rangefinder; Step 5.5: When the laser rangefinder does not obtain a valid ranging echo at a certain moment, only the line-of-sight angle observation information is used to predict the state of the space target. At this time, the relative distance in the measurement model is D The corresponding measurement variance is set to an ultra-large value, that is, at least two orders of magnitude greater than the variance of the normal valid measurement value, so that the weight of the invalid distance measurement information is automatically ignored, and the space target state is extrapolated only through the line-of-sight angle observation information.

[0014] Compared with the prior art, the advantages of the present invention are: (1) The image information source adopts multi-spectral fusion (infrared + visible light), which has stronger anti-interference ability compared with single visible light and single infrared light; (2) The synchronization method introduces software time synchronization + image registration to reduce hardware deployment while avoiding the serious frame matching misalignment caused by lack of precise synchronization; (3) Introducing infrared fast mean judgment suitable for onboard embedding, compared with similar satellites that do not perform background detection and have a high false recognition rate; (4) Adding earth background suppression technology and target information enhancement scheme to reduce the target tracking loss rate and solve the problem of weak observation or abandonment of target observation in the field of view facing the earth in current similar technologies; (5) Adopt the fusion tracking + ranging coordination of vision + laser rangefinder to reduce the drift of on-board tracking, and design the rangefinder start-stop strategy according to the on-board power consumption; In summary, this solution is suitable for low-Earth orbit space missions, such as docking, formation flying, and companion observation. It can stably detect and track spacecraft even under strong background interference from the Earth. It enhances the ability to resist occlusion and illumination changes, and improves the robustness of the navigation system. It reduces the risk of sensor failure and enhances redundant observation capabilities. The overall navigation and observation system can be adapted to low-power embedded platforms to better meet the needs of commercial aerospace deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 For different visible light images; Figure 2 for Figure 1 Thresholded images and dynamic threshold segmentation maps of each visible light image in ; Figure 3 for Figure 1 Middle (a) is the comparison image after moderate suppression of bright parts; Figure 4 for Figure 1 (a) The result of image background suppression; Figure 5 for Figure 1 Middle (a) High-frequency information analysis diagram before image background suppression; Figure 6 for Figure 1 Middle (a) High-frequency information analysis diagram after image background suppression; Figure 7 (a) is a real visible light image, and (b) is a real infrared image; Figure 8 (a) and (b) are Figure 7 Local entropy calculation and analysis diagram of visible light image and infrared image; Figure 9 (a) and (b) are Figure 7 Weight calculation analysis diagram of visible light image and infrared image; Figure 10 (a), (b), and (c) are Figure 7 Fusion diagram of visible light image and infrared image, local entropy fusion effect diagram and weight fusion effect diagram. DETAILED DESCRIPTION

[0016] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0017] The present invention provides a method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform, comprising the following steps: Step 1: Build an integrated device consisting of a visible light camera, an infrared camera, and a laser tester. Perform coordinate calibration, time synchronization processing, and image sampling conversion on the visible light camera and infrared camera, and perform optical axis deviation calibration on the laser rangefinder. Specifically: The coordinate calibration of the visible light camera and infrared camera and the optical axis deviation calibration of the laser rangefinder specifically include the following steps: Step 1.1: The intrinsic calibration of the visible light camera and infrared camera is carried out using the checkerboard calibration method. The focal length, principal point, and distortion information of the visible light camera and infrared camera are obtained by using the corner point detection method. Step 1.2: After the intrinsic parameter calibration is completed, the visible light camera and the infrared camera are fixed in the integrated device. The central viewing axes of the visible light camera and the infrared camera are calibrated to make the central viewing axes of the two cameras parallel. Then, the extrinsic parameters are calibrated using the checkerboard calibration method to obtain the coordinate transformation matrix between the visible light camera and the infrared camera, as well as the coordinate transformation matrix between the visible light camera, the infrared camera and the reference coordinate system of the integrated device. Step 1.3: Turn on the visible light camera and laser rangefinder so that the laser rangefinder's light spot appears in the visible light camera's field of view. The visible light camera captures multiple frames of images and calculates the deviation between the laser rangefinder's light spot and the visible light camera's principal point pixel to complete the laser rangefinder optical axis deviation calibration.

[0018] For time synchronization of visible light cameras and infrared cameras, select "coarse synchronization" plus "fine synchronization" to find the nearest frame to achieve soft time synchronization. The specific steps include the following: "Coarse synchronization" calculates candidate frames: Step 1.4: When the integrated device starts running, it marks the receiving timestamp and frame number of each frame of image received by the visible light camera and the infrared camera, and reads the first frame within n seconds. k Receiving timestamp of the frame visible light image , select the receiving timestamp and The closest j Frame infrared image, the receiving timestamp of the infrared image is , making ,in is the preset time tolerance threshold, R represents a visible light image, I represents infrared image; Step 1.5: Match each pair of image data obtained in step 1.4 Save to the paired image data dynamic buffer area, where Indicates that the receiving timestamp is After collecting n seconds of paired data, the least squares method is used to fit the infrared image frame number and the following mapping relationship linear model is obtained: , ; in, is the timestamp mean of the visible light image in the dynamic buffer of paired image data, is the average frame number of the infrared image in the dynamic buffer of paired image data, a 、 b Represents the coefficients, and the final mapping relationship linear model is obtained: ; Step 1.6: After obtaining the final mapping relationship linear model in step 1.5, use this linear model to roughly predict the corresponding infrared image frame number. When receiving each frame of visible light image, input the visible light image timestamp , calculate the corresponding infrared image frame number , the roughly matched infrared image frame number The corresponding image and frame number are -1. +1 infrared image is extracted from the paired image data dynamic buffer; "Fine Sync" matches the synchronization frame: Step 1.7: Accurately match the visible light image extracted in step 1.6 with the three candidate infrared images. First, extract the gradient map of the visible light image. , and also extract the gradient maps of the three candidate infrared images respectively ,calculate and three frames of candidate infrared images The normalized similarity of the infrared image is used to determine the most similar frame, and the most synchronized frame with the visible light image is considered. This completes the “precise synchronization” of the visible light image and the infrared image.

[0019] The image sampling transformation for visible light cameras and infrared cameras is as follows: Step 1.8: After obtaining the time-synchronized visible light and infrared images from the visible light and infrared cameras, upsample the infrared image to match the size of the visible light image because of their different sizes.

[0020] Step 2: Based on the integrated equipment and time synchronization processing method built in Step 1, the visible light camera and infrared camera collect image data to obtain visible light images and infrared images of the space target. The visible light images and infrared images are then used to determine and suppress the Earth background in the space environment. Specifically, determining and suppressing the Earth background in the space environment includes the following steps: Step 2.1: Based on the typical characteristics of the Earth, in both visible light and infrared images, deep space areas appear completely black, while Earth areas appear as areas with significantly increased bright grayscale values. A brightness / grayscale level and ratio method is used to quickly determine whether Earth areas appear. During the initial operation of the satellite, a pre-set empirical threshold T on the ground is used as the criterion for determining whether Earth areas appear in the image. The grayscale mean of the entire image is taken and compared with the threshold T. If the grayscale mean of the entire image is greater than the threshold T, it is considered that the Earth background appears in the field of view of the visible light camera and the infrared camera. Subsequently, during the operation, the average grayscale mean of the entire image is calculated over a period of time. The lowest grayscale mean in the deep space segment and the highest grayscale mean in the Earth segment during the actual on-orbit operation are selected to dynamically update the threshold T. Step 2.2: After threshold judgment, the infrared image is binarized using the OTSU algorithm to obtain the high grayscale pixel segmentation threshold of the infrared image. The image is divided into two levels according to the segmentation situation: one is the partial earth background and the other is the full earth background. Step 2.3: After judging in step 2.2, perform subsequent processing based on the earth background ratio. The processing method is as follows: The earth's background is less than : Skip earth background suppression; The earth's background accounts for Between: Lightly feathered mask, specifically: First, the visible light image after suppressing the background of the earth With infrared images Perform light feathering mask processing respectively, ( x,y ) represents the specific coordinates of a pixel point in the visible light image and infrared image: According to the threshold method in step 2.1, the background area of ​​the earth is determined from the visible light image and the infrared image, and a mask is constructed. ,in is a binary mask image; Secondly, to avoid obvious edge mutations between the mask and the spatial target area, the mask edge is smoothed and feathered by Gaussian convolution: ; in, is the feather weight, is a two-dimensional Gaussian kernel with empirical values; Finally, the obtained feathering weights are used to process the visible light image and infrared image respectively, as follows: ; in, They are the processed visible light image and infrared image respectively; The earth's background accounts for Between: Moderately suppress the earth background through threshold mask + local contrast enhancement, specifically: By using the dynamic threshold T of histogram analysis, the overall background pixels of the infrared and visible light images are darkened, and the non-background areas are contrast-stretched, making the dark areas of the overall image darker and the local brightness of the bright areas brighter. The earth's background accounts for more than :Perform whole-image histogram compression and local entropy clamp saliency enhancement on infrared and visible light images. Specifically: Histogram compression is used on the entire infrared image and visible light image to reduce the overall brightness. The significance of the space target is then enhanced through local entropy. The weight is increased in areas with low entropy and decreased in areas with high entropy. The entropy map is used to enhance the significance of the bright spots of the space target and suppress the background area of ​​the earth. The preset percentage threshold value, and the specific value and endpoint value range are set for different scenarios based on ground tests.

[0021] Taking an actual visible light image as an example, this article explains how to determine and suppress the Earth background: refer to Figure 1 (a), (b), (c), and (d) are different visible light images. As the satellite's attitude and orbital state change, different Earth background areas will appear in the visible light field of view.

[0022] First, threshold segmentation is performed on the visible light image to obtain Figure 1The earth background accounts for 62.53%, 34.33%, 55.39% and 44.98% of the visible light images respectively. The thresholded images and dynamic threshold segmentation maps are shown in Figure 2. Figure 2 As shown in (a), (b), (c), (d), (e), (f), (g), and (h).

[0023] Set the Earth background branch to four levels: less than 30%, 30-50%, 50-70%, and greater than 70%. Figure 1 Taking the middle image (a) as an example, the earth accounts for 62.53%, and threshold masking + local contrast enhancement are needed to moderately suppress the earth background: The threshold value of 97 obtained through histogram analysis can darken the background pixels of the entire image and perform contrast stretching on the non-background area, making the dark areas of the entire image darker and the local brightness of the target area brighter, so that the high-frequency information of the target point is not lost. Figure 3 (a), (b), (c), (d); Then use feathering to suppress and eliminate the earth background, see Figure 4 , through high-frequency information analysis, we can see that Figure 5 The target high-frequency information in the original image is not affected. Figure 6 After medium processing, the highlight areas of the earth are significantly suppressed, while the target features are not affected and are more clearly highlighted.

[0024] Step 3: Use dynamic weighting and multi-scale fusion algorithm to fuse visible light image and infrared image to obtain the viewing angle observation information of space target in the fused image. Specifically, it includes the following steps: Step 3.1: First, perform dynamic weighted fusion of the visible light image and the infrared image: ; in, is the coarse fusion image, fusion weight Adaptive calculation is performed based on the actual situation in the space. The calculation method is as follows: ; in, are the local variances of infrared images and visible light images respectively, is the local entropy of the infrared image, is the weight coefficient, It is division by zero protection; higher local variance and local entropy of infrared images indicate that the infrared image has a larger weight. Therefore, the weight coefficient of the fusion of infrared image and visible light image is dynamically determined according to the actual imaging situation. Step 3.2: Use infrared images to perform rough detection of spatial target areas. Introduce local entropy, local standard deviation, and adjustment coefficient to find local hot spots or areas with obvious motion. The specific method is as follows: ; in, DOG It is a differential Gaussian processing of the image, which removes large blocks of smooth background and significantly improves the high-frequency area of ​​the spatial target area. and is the adjustment coefficient, Represents the local standard deviation maximum in the infrared image; Step 3.3: According to the real-time situation on the satellite, dynamically change the adjustment coefficient in step 3.2 according to the background mean. When the current image imaging result shows that the thermal target is weak and the earth background accounts for a large proportion and there is a lot of cloud texture information, improve DOG The weight ratio , reduce the weight of fine texture features, and increase the weight of features similar to space targets; when the thermal features of the space target are strong and the background noise is high, increase , allowing the variance to suppress noise while reducing DOG The weight of the operator is used to avoid treating too many noise points as spatial targets; Step 3.4: After the local space target area is acquired through infrared images, in order to reduce the computing power burden on the satellite, multi-scale Laplacian pyramid fusion is performed only on the space target area. The final fused image is the space target area to ensure the infrared thermal characteristics and visible light texture information to the greatest extent possible, while the non-space target area remains a coarse fused image; Step 3.5: After extracting the spatial target, determine the displacement of the spatial target within the multi-frame image based on motion consistency analysis, accurately identify the spatial target, and output the spatial target information based on the confidence level; Step 3.6: Extract the centroid pixel coordinates (u, v) of the spatial target in the fused image through step 3.5, and combine the camera intrinsic and extrinsic parameters to obtain the image plane line of sight as follows: ; ; ; Approximate line-of-sight pitch angle at small angles , azimuth as follows: ; in, and is the coordinate of the principal point of the visible light camera, and is the pixel focal length of the visible light camera, is the normalized transverse image plane coordinate, is the normalized longitudinal image plane coordinate, is the normalized direction of the unit line of sight vector in the visible light camera coordinate system.

[0025] Actual photos of the propulsion system actually carried by the satellite Figure 7 Take the following as an example to illustrate the dynamic weighted fusion process of infrared image and visible light image: Firstly, the visible light image and infrared image are dynamically weighted and fused. Figure 8 and Figure 9 The analysis results show that the partial entropy values ​​of the visible light image are 3.04, 5.74, and 3.54. In the infrared image, the local entropy values ​​of the corresponding information are 3.26, 5.98, and 5.32 respectively. The analysis shows that the local variance and local entropy of the infrared image are higher, which also means that the infrared image has a greater weight in the fusion process. In this embodiment, the local entropy radius is set to 9, the Gaussian smoothing kernel is set to 5, and fusion is performed after dynamic weighted analysis, where Figure 10 (a), (b), and (c) are Figure 7 The fusion diagram of visible light image and infrared image, local entropy fusion and weight fusion effect diagram realize the data fusion of infrared image and visible light image.

[0026] Step 4: After obtaining the viewing angle observation information of the space target, set the laser rangefinder switching strategy, turn on the laser rangefinder to measure distance when needed, and obtain the laser rangefinder measurement information. The laser rangefinder switching strategy is as follows: Step 4.1: Make judgments in three parts: confidence gate, optical axis deviation gate, and count gate. The confidence gate judgment method is: ,in, is the confidence of the spatial target in the fused image of the current frame, is the confidence gating threshold. When the confidence of the spatial target in the current frame fusion image is greater than or equal to the confidence gating threshold, the optical axis deviation gating judgment is entered. The judgment is based on the following: ; in, , is the pixel coordinate of the centroid of the spatial target in the fused image of the current frame, , It is the optical axis deviation of the laser rangefinder and the visible light camera calibrated on the ground, and finally the instantaneous deviation of the center of mass of the space target relative to the optical axis of the laser rangefinder is obtained. and ; Step 4.2: The deviation judgment condition is converted by the ground calibration pixel threshold and the spatial error to obtain the space target-spot size deviation allowable threshold px, when and Within the threshold, the counting gate judgment is entered; in the counting gate, if the continuous M-frame fusion image stably meets the above conditions, it is confirmed that the laser rangefinder is turned on.

[0027] Step 5: Use the line-of-sight angle observation information and laser rangefinder measurement information to perform continuous unscented Kalman filter algorithm state estimation on the space target: By establishing a state model of the space target relative to the observed target and assuming that the space target moves at a uniform speed in a short period of time as the state transition process, the motion state of the space target is predicted using the time interval between two adjacent frames of fused image observations and the process noise covariance; and construct a corresponding measurement model based on the relative position state of the space target and the observed target. When there is no valid laser rangefinder measurement information at a certain moment, the large variance method is used to extrapolate the space target state using only the line-of-sight angle observation information. Specifically, the following steps are performed: Step 5.1: Construct the state vector X , is defined as the relative state of the space target in the observation target reference coordinate system, where the state vector Represents the position coordinates of the space target relative to the observed target Corresponding to the three speed states ; Step 5.2: Establish the state transition model F(·) of the space target relative to the observation target. Assuming that the space target is in a uniform motion state within a short period of time, the process model can be expressed as: ; Among them, Δt is the time interval between the observations of two adjacent fusion images. is the process noise; Step 5.3: Based on the process noise covariance Q and the time interval Δt between two adjacent fused image frames, the maximum possible maneuvering acceleration of the space target is predicted and the process model is updated to adapt to the dynamic motion changes of the space target in real time. Step 5.4: Establish the measurement model h(·). The measurement model expression is: ; Where, D The relative distance between the space target and the observation target measured by the laser rangefinder; Step 5.5: When the laser rangefinder does not obtain a valid ranging echo at a certain moment, only the line-of-sight angle observation information is used to predict the state of the space target. At this time, the relative distance in the measurement model is DThe corresponding measurement variance is set to an ultra-large value, that is, at least two orders of magnitude greater than the variance of the normal valid measurement value, so that the weight of the invalid distance measurement information is automatically ignored, and the space target state is extrapolated only through the line-of-sight angle observation information.

[0028] Step 6: By reading the viewing angle observation information of each frame of the fused image, and reading the laser rangefinder measurement information if available, the information is input into the state model and measurement model in step 5 to predict the state of the space target and update the state model and measurement model. Finally, the position and velocity information of the space target is output to approach the space target.

Claims

1. A target recognition and approach method in complex environments based on dual-band imaging on a satellite platform, characterized by: The steps include: Step 1: Build an integrated device consisting of a visible light camera, an infrared camera, and a laser tester. Perform coordinate calibration, time synchronization processing, and image sampling conversion on the visible light camera and infrared camera, and perform optical axis deviation calibration on the laser rangefinder. Step 2: Based on the integrated equipment built in Step 1 and the time synchronization processing method, the visible light camera and infrared camera collect image data to obtain visible light images and infrared images of the space target. The visible light images and infrared images are then used to determine and suppress the Earth background based on the space environment. Step 3: Use dynamic weighting and multi-scale fusion algorithm to fuse visible light image and infrared image to obtain the viewing angle observation information of space target in the fused image; Step 4: After obtaining the viewing angle observation information of the space target, set the laser rangefinder switching strategy, turn on the laser rangefinder for ranging when needed, and obtain the laser rangefinder measurement information: Step 5: Use the line-of-sight angle observation information and laser rangefinder measurement information to perform continuous unscented Kalman filter algorithm state estimation on the space target: By establishing a state model of the space target relative to the observed target and assuming that the space target moves at a uniform speed in a short period of time as the state transition process, the motion state of the space target is predicted using the time interval between two adjacent frames of fused image observations and the process noise covariance; and based on the relative position state of the space target and the observed target, a corresponding measurement model is constructed. When there is no valid laser rangefinder measurement information at a certain moment, the large variance method is used to extrapolate the space target state using only the line-of-sight angle observation information; Step 6: By reading the viewing angle observation information of each frame of the fused image, and reading the laser rangefinder measurement information if available, the information is input into the state model and measurement model in step 5 to predict the state of the space target and update the state model and measurement model. Finally, the position and velocity information of the space target is output to approach the space target.

2. The method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform according to claim 1, characterized in that: In step 1, coordinate calibration of the visible light camera and the infrared camera and optical axis deviation calibration of the laser rangefinder specifically include the following steps: Step 1.1: The intrinsic calibration of the visible light camera and infrared camera is carried out using the checkerboard calibration method. The focal length, principal point, and distortion information of the visible light camera and infrared camera are obtained by using the corner point detection method. Step 1.2: After the intrinsic parameter calibration is completed, the visible light camera and the infrared camera are fixed in the integrated device. The central viewing axes of the visible light camera and the infrared camera are calibrated to make the central viewing axes of the two cameras parallel. Then, the extrinsic parameters are calibrated using the checkerboard calibration method to obtain the coordinate transformation matrix between the visible light camera and the infrared camera, as well as the coordinate transformation matrix between the visible light camera, the infrared camera and the reference coordinate system of the integrated device. Step 1.3: Turn on the visible light camera and laser rangefinder so that the laser rangefinder's light spot appears in the visible light camera's field of view. The visible light camera captures multiple frames of images and calculates the deviation between the laser rangefinder's light spot and the visible light camera's principal point pixel to complete the laser rangefinder optical axis deviation calibration.

3. The method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform according to claim 2, characterized in that: In step 1, the time synchronization processing of the visible light camera and the infrared camera is performed by selecting "coarse synchronization" plus "fine synchronization" to find the nearest frame to achieve soft time synchronization, which specifically includes the following steps: "Coarse synchronization" calculates candidate frames: Step 1.4: When the integrated device starts running, it marks the receiving timestamp and frame number of each frame of image received by the visible light camera and the infrared camera, and reads the first frame within n seconds. k Receiving timestamp of the frame visible light image , select the receiving timestamp and The closest j Frame infrared image, the receiving timestamp of the infrared image is , making ,in is the preset time tolerance threshold, R represents a visible light image, I represents infrared image; Step 1.5: Match each pair of image data obtained in step 1.4 Save to the paired image data dynamic buffer area, where Indicates that the receiving timestamp is After collecting n seconds of paired data, the least squares method is used to fit the infrared image frame number and the following mapping relationship linear model is obtained: ， ； in, is the timestamp mean of the visible light image in the dynamic buffer of paired image data, is the average frame number of the infrared image in the dynamic buffer of paired image data, a 、 b Represents the coefficients, and the final mapping relationship linear model is obtained: ; Step 1.6: After obtaining the final mapping relationship linear model in step 1.5, use this linear model to roughly predict the corresponding infrared image frame number. When receiving each frame of visible light image, input the visible light image timestamp , calculate the corresponding infrared image frame number , the roughly matched infrared image frame number The corresponding image and frame number are -1. +1 infrared image is extracted from the paired image data dynamic buffer; "Fine Sync" matches the synchronization frame: Step 1.7: Accurately match the visible light image extracted in step 1.6 with the three candidate infrared images. First, extract the gradient map of the visible light image. , and also extract the gradient maps of the three candidate infrared images respectively ,calculate and three frames of candidate infrared images The normalized similarity of the infrared image is calculated, and the most similar frame of the infrared image is regarded as the frame most synchronized with the visible light image. Thus, the "precise synchronization" of the visible light image and the infrared image is completed.

4. The method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform according to claim 3, characterized in that: In step 1, the image sampling transformation of the visible light camera and the infrared camera is specifically performed as follows: Step 1.8: After obtaining the time-synchronized visible light and infrared images from the visible light and infrared cameras, upsample the infrared image to match the size of the visible light image because of their different sizes.

5. The method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform according to claim 1, characterized in that: In step 2, determining and suppressing the Earth background in the space environment includes the following steps: Step 2.1: Based on the typical characteristics of the Earth, in both visible light and infrared images, deep space areas appear completely black, while Earth areas appear as areas with significantly increased bright grayscale values. A brightness / grayscale level and ratio method is used to quickly determine whether Earth areas appear. During the initial operation of the satellite, a pre-set empirical threshold T on the ground is used as the criterion for determining whether Earth areas appear in the image. The grayscale mean of the entire image is taken and compared with the threshold T. If the grayscale mean of the entire image is greater than the threshold T, it is considered that the Earth background appears in the field of view of the visible light camera and the infrared camera. Subsequently, during the operation, the average grayscale mean of the entire image is calculated over a period of time. The lowest grayscale mean in the deep space segment and the highest grayscale mean in the Earth segment during the actual on-orbit operation are selected to dynamically update the threshold T. Step 2.2: After threshold judgment, the infrared image is binarized using the OTSU algorithm to obtain the high grayscale pixel segmentation threshold of the infrared image. The image is divided into two levels according to the segmentation situation: one is the partial earth background and the other is the full earth background. Step 2.3: After judging in step 2.2, perform subsequent processing based on the earth background ratio. The processing method is as follows: The earth's background is less than : Skip earth background suppression; The earth's background accounts for Between: Lightly feathered mask, specifically: First, the visible light image after suppressing the background of the earth With infrared images Perform light feathering mask processing respectively, ( x,y ) represents the specific coordinates of a pixel point in the visible light image and infrared image: According to the threshold method in step 2.1, the background area of ​​the earth is determined from the visible light image and the infrared image, and a mask is constructed. ,in is a binary mask image; Secondly, to avoid obvious edge mutations between the mask and the spatial target area, the mask edge is smoothed and feathered by Gaussian convolution: ； in, is the feather weight, is a two-dimensional Gaussian kernel with empirical values; Finally, the obtained feathering weights are used to process the visible light image and infrared image respectively, as follows: ； in, They are the processed visible light image and infrared image respectively; The earth's background accounts for Between: Moderately suppress the earth background through threshold mask + local contrast enhancement, specifically: By using the dynamic threshold T of histogram analysis, the overall background pixels of the infrared and visible light images are darkened, and the non-background areas are contrast-stretched, making the dark areas of the overall image darker and the local brightness of the bright areas brighter. The earth's background accounts for more than :Perform whole-image histogram compression and local entropy clamp saliency enhancement on infrared and visible light images. Specifically: Histogram compression is used on the entire infrared image and visible light image to reduce the overall brightness. The significance of the space target is then enhanced through local entropy. The weight is increased in areas with low entropy and decreased in areas with high entropy. The entropy map is used to enhance the significance of the bright spots of the space target and suppress the background area of ​​the earth. The preset percentage threshold value, and the specific value and endpoint value range are set for different scenarios based on ground tests.

6. The method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform according to claim 5, characterized in that: The step 3 comprises the following steps: Step 3.1: First, perform dynamic weighted fusion of the visible light image and the infrared image: ； in, is the coarse fusion image, fusion weight Adaptive calculation is performed based on the actual situation in the space. The calculation method is as follows: ； in, are the local variances of infrared images and visible light images respectively, is the local entropy of the infrared image, is the weight coefficient, It is division by zero protection; higher local variance and local entropy of infrared images indicate that the infrared image has a larger weight. Therefore, the weight coefficient of the fusion of infrared image and visible light image is dynamically determined according to the actual imaging situation. Step 3.2: Use infrared images to perform rough detection of spatial target areas. Introduce local entropy, local standard deviation, and adjustment coefficient to find local hot spots or areas with obvious motion. The specific method is as follows: ； in, DOG It is a differential Gaussian processing of the image, which removes large blocks of smooth background and significantly improves the high-frequency area of ​​the spatial target area. and is the adjustment coefficient, Represents the local standard deviation maximum in the infrared image; Step 3.3: According to the real-time situation on the satellite, dynamically change the adjustment coefficient in step 3.2 according to the background mean. When the current image imaging result shows that the thermal target is weak and the earth background accounts for a large proportion and there is a lot of cloud texture information, improve DOG The weight ratio , reduce the weight of fine texture features, and increase the weight of features similar to space targets; when the thermal features of the space target are strong and the background noise is high, increase , allowing the variance to suppress noise while reducing DOG The weight of the operator is used to avoid treating too many noise points as spatial targets; Step 3.4: After the local space target area is acquired through infrared images, in order to reduce the computing power burden on the satellite, multi-scale Laplacian pyramid fusion is performed only on the space target area. The final fused image is the space target area to ensure the infrared thermal characteristics and visible light texture information to the greatest extent possible, while the non-space target area remains a coarse fused image; Step 3.5: After extracting the spatial target, determine the displacement of the spatial target within the multi-frame image based on motion consistency analysis, accurately identify the spatial target, and output the spatial target information based on the confidence level; Step 3.6: Extract the centroid pixel coordinates (u, v) of the spatial target in the fused image through step 3.5, and combine the camera intrinsic and extrinsic parameters to obtain the image plane line of sight as follows: ； ； ； Approximate line-of-sight pitch angle at small angles , azimuth as follows: ； in, and is the coordinate of the principal point of the visible light camera, and is the pixel focal length of the visible light camera, is the normalized transverse image plane coordinate, is the normalized longitudinal image plane coordinate, is the normalized direction of the unit line of sight vector in the visible light camera coordinate system.

7. The method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform according to claim 6, characterized in that: The laser rangefinder switching strategy in step 4 is specifically as follows: Step 4.1: Make judgments in three parts: confidence gate, optical axis deviation gate, and count gate. The confidence gate judgment method is: ,in, is the confidence of the spatial target in the fused image of the current frame, is the confidence gating threshold. When the confidence of the spatial target in the current frame fusion image is greater than or equal to the confidence gating threshold, the optical axis deviation gating judgment is entered. The judgment is based on the following: ； in, , is the pixel coordinate of the centroid of the spatial target in the fused image of the current frame, , It is the optical axis deviation of the laser rangefinder and the visible light camera calibrated on the ground, and finally the instantaneous deviation of the center of mass of the space target relative to the optical axis of the laser rangefinder is obtained. and ; Step 4.2: The deviation judgment condition is converted by the ground calibration pixel threshold and the spatial error to obtain the space target-spot size deviation allowable threshold px, when and Within the threshold, the counting gate judgment is entered; in the counting gate, if the continuous M-frame fusion image stably meets the above conditions, it is confirmed that the laser rangefinder is turned on.

8. The method for identifying and approaching targets in complex environments based on dual-band imaging on a satellite platform according to claim 6, characterized in that: The step 5 specifically includes: Step 5.1: Construct the state vector X , is defined as the relative state of the space target in the observation target reference coordinate system, where the state vector Represents the position coordinates of the space target relative to the observed target Corresponding to the three speed states ; Step 5.2: Establish the state transition model F(·) of the space target relative to the observation target. Assuming that the space target is in a uniform motion state within a short period of time, the process model can be expressed as: ； Among them, Δt is the time interval between the observations of two adjacent fusion images. is the process noise; Step 5.3: Based on the process noise covariance Q and the time interval Δt between two adjacent fused image frames, the maximum possible maneuvering acceleration of the space target is predicted and the process model is updated to adapt to the dynamic motion changes of the space target in real time. Step 5.4: Establish the measurement model h(·). The measurement model expression is: ； Where, D The relative distance between the space target and the observation target measured by the laser rangefinder; Step 5.5: When the laser rangefinder does not obtain a valid ranging echo at a certain moment, only the line-of-sight angle observation information is used to predict the state of the space target. At this time, the relative distance in the measurement model is D The corresponding measurement variance is set to an ultra-large value, that is, at least two orders of magnitude greater than the variance of the normal valid measurement value, so that the weight of the invalid distance measurement information is automatically ignored, and the space target state is extrapolated only through the line-of-sight angle observation information.

Images