Space object detection method adaptive to different distances and light conditions

Abstract

The application discloses a space target detection method suitable for different distances and light conditions, which comprises the following steps: judging whether a space target is in a backlight environment and a distance; detecting the space target in the backlight environment by using an event camera; realizing space target detection initial position by using a traditional camera, combining the event camera to obtain the space target; detecting the motion state of the space target by using the event camera, and simultaneously acquiring the contour and texture information of the space target by using the traditional camera; and fusing the output information of the traditional camera and the event camera to realize space target detection under different distances and light conditions. The space target detection method suitable for different distances and light conditions has the advantages that the event camera is used to realize space target detection capability under the backlight condition, and the event camera and the traditional camera are combined to solve the problem of low recognition rate caused by the slow detection response and motion blur of the traditional camera used in the existing space detection.

Description

Technical Field

[0001] This application relates to a spatial target detection method that adapts to different distances and lighting conditions. Background Technology

[0002] Accurate detection of space targets is a crucial foundation for wide-area situational awareness. However, existing optical imaging techniques for space target detection based on traditional cameras suffer from drawbacks such as slow imaging rates, limited dynamic range, high power consumption, and high bandwidth requirements, making it difficult to achieve high real-time target detection and long-term operation under backlight conditions. Furthermore, in backlight conditions, interference from sunlight causes overexposure of the imaging sensor, hindering target detection. Simultaneously, when space targets are moving relatively quickly, the slow imaging rate and long integration time of traditional cameras lead to motion blur, making accurate target identification difficult.

[0003] The emergence of event cameras can address some of the shortcomings of traditional cameras. Event cameras offer advantages such as fast detection response, large dynamic range, low average power consumption, and small bandwidth usage. They can generate target imaging data with microsecond-level temporal resolution, thus solving the motion blur problem and achieving clear target contours and textures. Furthermore, their small bandwidth usage enables high-speed data transmission and processing, resulting in high-real-time and high-precision target recognition capabilities. The large dynamic range of event cameras allows them to adapt to extreme lighting conditions, providing superior target detection capabilities even in backlight, and solving the overexposure problem of traditional cameras. In addition, the average power consumption of event cameras can be maintained at the mW level, ensuring low-power long-term operation on space platforms.

[0004] However, how to allocate event cameras and traditional cameras to explore spatial targets still needs further research. Summary of the Invention

[0005] The purpose of this invention is to provide a spatial target detection method that adapts to different distances and lighting conditions, and has the advantage of being able to identify spatial targets under different lighting conditions and at different distances from the spatial target.

[0006] To achieve the above objectives, the present invention provides a spatial target detection method adaptable to different distances and lighting conditions, comprising:

[0007] S10. Determine whether the spatial target is in a backlit environment. If yes, proceed to S30; otherwise, proceed to S20.

[0008] S20. Determine whether the distance to the spatial target is greater than a distance threshold. If it is greater than the distance threshold, proceed to step S40; if it is less than the distance threshold, proceed to step S50.

[0009] S30. Use the event camera to detect the space target in the backlight environment, and then proceed to S60.

[0010] S40. Use a conventional camera to detect the initial position of the spatial target, and combine the event camera to detect the spatial target to obtain the spatial target. After completion, proceed to S60.

[0011] S50: The event camera detects the motion state of the spatial target and simultaneously uses the conventional camera to acquire the contour and texture information of the spatial target. After completion, proceed to S60.

[0012] S60. The output information of the traditional camera and the event camera are fused to achieve spatial target detection under different distances and lighting conditions.

[0013] Preferably, step S40 includes:

[0014] S401. Use a conventional camera to acquire the space target and background stars in the field of view, and realize the initial position selection of the space target;

[0015] S402. Based on the position and attitude of the space platform, match the field of view with the star map template, and extract the space targets in the non-star map template as the preferred effective space targets;

[0016] S403. The event camera is used to detect the field of view that the conventional camera has detected, and the relatively moving targets in the field of view are obtained. The targets are compared with the effective spatial targets extracted by the conventional camera to evaluate the candidate effective spatial targets.

[0017] S404. If the confidence level of the evaluation result of the effective space target is higher than the threshold, the effective space target is continuously tracked, and long-term observation of the effective space target is carried out. The distance is judged based on the change in the shape and size of the effective space target.

[0018] Preferably, step S50 includes: the sparse asynchronous event information acquired by the event camera is transmitted to the spiking neural network to complete real-time target recognition, while the conventional camera obtains the contour and texture of the spatial target, and high-definition image reconstruction is completed in combination with the event camera, and then the spatial target is identified according to template matching or deep neural network target recognition algorithm.

[0019] Preferably, the conventional camera is a high-sensitivity CMOS camera or CCD camera; the event camera is a high-temporal-resolution neuromorphic imaging sensor, and the field of view of the event camera is consistent with that of the conventional camera.

[0020] Preferably, in step S20, determining whether the distance to the spatial target is greater than a distance threshold includes: determining whether the spatial target is greater than the distance threshold by measuring the target size output by the event camera and the conventional camera.

[0021] Preferably, step S60, which fuses the output information of the conventional camera and the event camera, includes:

[0022] By collecting motion characteristics, contours, and texture features of typical space maneuvering targets in advance, a knowledge base of typical space targets is formed, and then the output data of the event camera and the traditional camera are used to achieve fusion.

[0023] Preferably, step S60, which fuses the output information of the conventional camera and the event camera, includes fusion at the data level, feature level, and decision level.

[0024] Preferably, the field of view of the event camera is the same as that of the conventional camera, and the output data of the conventional camera and the event camera are fused at the data level, fusing the high temporal resolution features of the event camera and the high spatial resolution features of the conventional camera to complete the fusion and reconstruction of high frame rate and high spatial resolution image data.

[0025] Preferably, the field of view of the event camera is the same as that of the conventional camera, and the feature-level fusion of the output data of the conventional camera and the event camera includes: obtaining multi-scale features from the event stream data of the event camera, extracting image features from the conventional camera using digital image processing operators or deep neural networks, and achieving feature-level fusion using a feature fusion algorithm.

[0026] Preferably, the field of view of the event camera is the same as that of the conventional camera, and the fusion of the output data of the conventional camera and the event camera at the decision level includes: performing target recognition separately on the event camera and the conventional camera, and then achieving fusion at the decision level.

[0027] In summary, compared with the prior art, the spatial target detection method adapted to different distances and lighting conditions provided by the present invention has the following beneficial effects:

[0028] This application presents a spatial target detection method adaptable to different distances and lighting conditions, and a spatial target detection method that integrates traditional cameras and event cameras to adapt to different distances and lighting conditions. It can utilize event cameras to solve the problem of blinding due to overexposure in sunlight caused by traditional optical cameras used in existing space exploration, and achieve the ability to detect spatial targets under backlight conditions. By combining event cameras and traditional cameras, it solves the problems of untimely detection response and low recognition rate caused by motion blur in traditional cameras used in existing space exploration. Attached Figure Description

[0029] Figure 1 A flowchart of a spatial target detection method that integrates traditional cameras and event cameras to adapt to different distances and lighting conditions.

[0030] Figure 2 This is a flowchart illustrating the target detection process using a fusion camera and a traditional camera under backlighting conditions.

[0031] Figure 3 This is a flowchart illustrating the spatial target detection method adapted to different distances and lighting conditions in this application. Detailed Implementation

[0032] The following will be combined with the appendix in the embodiments of the present invention. Figure 1 ~Attached Figure 3 The technical solutions, structural features, objectives and effects achieved in the embodiments of the present invention will be described in detail.

[0033] It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions. They are only used to facilitate and clarify the purpose of illustrating the embodiments of the present invention, and are not intended to limit the implementation conditions of the present invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationship, or adjustments to the size should still fall within the scope of the technical content disclosed in the present invention, provided that they do not affect the effects and objectives that the present invention can produce.

[0034] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only the expressly listed elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0035] like Figure 3 As shown, the present invention provides a spatial target detection method adaptable to different distances and lighting conditions, comprising the following steps:

[0036] S10. Determine whether the spatial target is in a backlit environment. If yes, proceed to S30; otherwise, proceed to S20.

[0037] S20. Determine whether the distance to the spatial target is greater than the distance threshold. If it is greater than the distance threshold, proceed to step S40; if it is less than the distance threshold, proceed to step S50.

[0038] In this embodiment, in step S20, determining whether the distance to the spatial target is greater than the distance threshold includes: determining whether the spatial target is greater than the distance threshold by measuring the target size output by the event camera and the traditional camera.

[0039] like Figure 2 As shown, in step S30, the event camera is used to detect spatial targets in a backlit environment, and after completion, the process proceeds to step S60.

[0040] In a space environment, when a target is positioned between a traditional camera and the sun, sunlight introduces significant variations in light intensity within the camera's field of view. Due to the limited dynamic range of traditional cameras (typically 60-70 dB), the contrast between the background light and the target becomes excessively high under backlighting conditions, making the target difficult to detect within the camera's field of view. This can lead to overexposure and failure to detect the target, resulting in blinding. In contrast, event cameras employ differential photoelectric sampling, which boasts a large dynamic range (typically 90-110 dB). This significantly mitigates the overexposure blinding problem of traditional cameras, ensuring effective target detection under special lighting conditions. Therefore, under backlighting conditions, the large dynamic range of event cameras can be leveraged to achieve space target detection.

[0041] Under backlighting conditions, the data quality of the traditional camera is first evaluated. If the data quality of the traditional camera is poor, the weight of the event camera data in the fusion algorithm is increased. If the traditional camera is completely unable to acquire target data, target detection and identification are achieved solely by relying on event stream data.

[0042] like Figure 1 As shown, in step S40, the initial position of the spatial target is detected using a traditional camera, and the spatial target is obtained by combining the detection of the spatial target with the detection of the event camera. After completion, the process proceeds to step S60.

[0043] Traditional cameras use high-sensitivity CMOS or CCD cameras; event cameras use high temporal resolution neuromorphic imaging sensors, and the field of view of event cameras is the same as that of traditional cameras.

[0044] In this embodiment, step S40 specifically includes:

[0045] S401. Use a traditional camera to acquire spatial targets and background celestial bodies in the field of view, and realize the initial position selection of spatial targets;

[0046] S402. Based on the position and attitude of the space platform, match the field of view with the star map template, and extract space targets in the non-star map template as the preferred effective space targets;

[0047] S403. Use the event camera to detect the field of view that the traditional camera has detected, obtain the relatively moving targets in the field of view, compare them with the effective spatial targets extracted by the traditional camera, and evaluate the candidate effective spatial targets.

[0048] S404. If the confidence level of the evaluation result based on the effective space target is higher than the threshold, the effective space target will be continuously tracked, and long-term observation of the effective space target will be carried out. The distance will be judged based on the change in the shape and size of the effective space target.

[0049] In step S40, both the event camera and the conventional camera are under non-backlight conditions. Under non-backlight conditions, both the event camera and the conventional camera operate normally, and both simultaneously detect distant targets.

[0050] Traditional cameras have good detection sensitivity and can acquire faint space targets and background stars in the field of view. Therefore, the positions of candidate space targets are acquired by traditional cameras, these positions are recorded, and the brightness changes of the candidate space targets are continuously monitored. If the brightness change is large, it is considered that the target is more likely to be a valid space target.

[0051] The target brightness change range is calculated using the following two formulas:

[0052]

[0053]

[0054] ΔL is used to characterize the average brightness change during the time period from t1 to t2. t1_eva L represents the average brightness during the time period t1. t2_eva The value represents the average brightness during the time period t2. The time periods t1 and t2 are the same, both being Δt.

[0055] S 2 Used to characterize the variance of brightness over a period of time, where L is the observed brightness value during that period. Let L be the average value, and n be the number of brightness samples observed during that time period.

[0056] ΔL and S 2 If either of these factors exceeds a set threshold, the brightness change is considered significant. Simultaneously, referencing the space platform's position and attitude, the conventional optical image acquired from the detection field of view is matched with a star map template corresponding to that position and direction. If a candidate target does not have a corresponding star in the star map template, it is considered highly likely to be a valid space target. The space platform's position and attitude refer to the location and attitude of the space platform itself, which carries the event camera and the conventional camera.

[0057] Furthermore, an event camera is used to detect targets within the same field of view. The event camera works by recording brightness changes in individual pixels, making it highly sensitive to moving targets and capable of acquiring relatively moving targets within the field of view. This target is recorded and compared with candidate space targets extracted by a traditional camera. If a candidate target simultaneously meets the criteria of relative motion, large brightness changes, and no matching celestial body, it is highly likely to be a valid detection target. If the candidate target evaluation result has a high confidence level, it is continuously tracked. Kalman filtering or particle filtering can be used for long-term observation, and the distance can be determined by changes in its external dimensions, or by using other ranging sensors on the space payload for distance assessment.

[0058] S50: The event camera detects the motion state of the spatial target, while a traditional camera acquires the contour and texture information of the spatial target. After completion, proceed to S60. In this embodiment, step S50 includes: transmitting the sparse asynchronous event information acquired by the event camera to a spiking neural network to complete real-time target recognition; simultaneously, the traditional camera acquires the contour and texture of the spatial target; combining the event camera data with high-definition image reconstruction; and then identifying the spatial target based on template matching or a deep neural network target recognition algorithm. The event camera performs rapid imaging detection of the spatial target's motion state. By utilizing the event camera's pixel differential detection mechanism, it performs differential detection on the illumination changes within a single pixel. If the illumination intensity change reaches a set threshold, the pixel generates a pulse signal, representing the polarity information of the illumination change within that pixel. The relative motion of the target will cause some pixel brightness changes, thereby realizing the acquisition of the target's motion state information.

[0059] The outline and texture information of a spatial target are obtained by using a traditional camera to fully image the spatial target.

[0060] Step S50 is used to handle situations where the space target is in non-backlight conditions and the distance between the space target and the space platform is less than a distance threshold. Under non-backlight conditions, if the long-distance space target is confirmed and its distance is assessed to be less than a set threshold, the event camera and a traditional camera are used to observe it simultaneously, ultimately achieving high-speed target recognition.

[0061] Target distance is determined by the target's image size in a traditional camera.

[0062] Size = N1 × N2

[0063] Size is the image size of the target in a traditional camera, N1 is the maximum horizontal length of the target, and N2 is the maximum vertical length of the target.

[0064] When the target distance is less than a set threshold, the event camera can acquire the motion state characteristics, partial texture and contour information of the spatial target, i.e. the event stream data of the target, which is the raw data of the event camera.

[0065] When the target distance is less than a set threshold, traditional cameras can acquire complete texture and contour information of the spatial target, i.e., the target's optical image data. However, due to the imaging principle of traditional cameras, their detection response speed is slow, and each frame of image requires a certain integration time, which will produce motion blur problems, seriously interfering with the target imaging effect and recognition efficiency.

[0066] Event cameras, with their high temporal resolution and high real-time response, can effectively alleviate the motion blur and slow response problems of traditional cameras. Therefore, this solution achieves data fusion at the data level, feature level, and decision level, thereby enabling efficient target recognition tasks.

[0067] S60. Integrate the output information from traditional cameras and event cameras to achieve spatial target detection under different distances and lighting conditions. Step S60, integrating the output information from traditional cameras and event cameras, includes: collecting motion features, contours, and texture features of typical maneuvering spatial targets in advance to form a typical spatial target knowledge base, and then using the output data from event cameras and traditional cameras to achieve fusion.

[0068] Step S60 involves fusing the output information from traditional cameras and event cameras, including fusion at the data level, feature level, and decision level.

[0069] The output data of traditional cameras and event cameras are fused at the data level, fusing the high temporal resolution features of event cameras with the high spatial resolution features of traditional cameras to complete the fusion and reconstruction of high frame rate, high spatial resolution image data.

[0070] Data-level fusion can leverage the high temporal resolution of event cameras and the high spatial resolution of traditional cameras to reconstruct high frame rate, high spatial resolution image data, thereby enabling the use of the reconstructed high frame rate, high spatial resolution images for target recognition tasks.

[0071] Data-level fusion can be achieved using deep learning networks or traditional super-resolution reconstruction algorithms. Deep learning methods can employ EvIntSR-Net, which uses event stream data to guide image super-resolution reconstruction. The process consists of two steps: first, latent frames are generated based on a given image acquired by a traditional camera and the event stream data preceding and following that image; second, the generated latent frames and images acquired by the traditional camera are used to perform multi-image super-resolution reconstruction to obtain the final super-resolution image. Traditional super-resolution reconstruction algorithms first convert images obtained from event cameras into high-frame-rate event frame images, and then fuse the two through super-resolution fusion.

[0072] The specific process of traditional super-resolution reconstruction algorithms is as follows:

[0073] First, accumulate event stream data X over a period of time.<x,y,t,p> x, y, t, and p represent the horizontal coordinate, vertical coordinate, time, and polarity information, respectively. Subsequently, an event frame image is formed based on the accumulated coordinate information.

[0074] Next, iterative back-projection spatial domain super-resolution reconstruction is performed on the event frame image and the traditional frame image. First, an initial high-resolution image is obtained by interpolation of the reference frame or non-uniform interpolation of the image sequence. Then, the initial image is degraded to obtain a simulated low-resolution image, which is compared with the actual observed image. The difference is called the simulation error. The information of the current estimated image is continuously updated based on the simulation error. This process is repeated until convergence is obtained to obtain a high-resolution image. The mathematical expression is as follows:

[0075]

[0076] In the formula, g k [m, n] represents the actual low-resolution image. Represents the high-resolution image after the nth iteration. The resulting simulated low-resolution image, h BP It is the analog error back projection operator.

[0077] The feature-level fusion of output data from traditional cameras and event cameras includes: obtaining multi-scale features from event stream data from event cameras, extracting image features from traditional cameras using digital image processing operators or deep neural networks, and achieving feature-level fusion using feature fusion algorithms.

[0078] Feature-level fusion utilizes event stream data output by event cameras to extract multi-scale features, and uses digital image processing operators or deep neural networks to extract features from traditional camera optical images. Subsequently, feature fusion algorithms are used to achieve feature-level fusion.

[0079] Event stream data can be extracted using multi-scale feature extraction methods, which utilize different receptive fields and multi-scale, multi-directional three-dimensional operators to extract event stream data features.

[0080] Image feature extraction from traditional cameras can be performed using traditional feature extraction operators or convolution operators commonly used in deep learning networks.

[0081] Finally, feature fusion is achieved through feature transformation (such as popular learning methods, linear methods, kernel function methods, etc.), feature selection (such as global search methods, random search methods, etc.), or artificial neural networks.

[0082] The fusion of output data from traditional cameras and event cameras at the decision level includes: performing target recognition separately on the event cameras and the traditional cameras, and then fusing them at the decision level.

[0083] Decision-level fusion refers to using the output signals of event cameras and traditional cameras to perform target recognition work separately, and then achieving fusion at the decision-making level.

[0084] The output signal from the event camera can be input into a pre-trained target recognition spiking neural network. This network has been pre-trained with samples based on target categories and states to form its network structure and parameters, and then outputs the recognition result. The training samples can consist of event stream data obtained from the optical images of the target through simulation software. The output signal from a traditional camera can be used to identify the imaging results through traditional machine learning methods or deep learning methods.

[0085] Subsequently, a decision-level fusion method was used to evaluate the confidence level of information acquired by the event camera and the traditional camera. Then, by combining decision-level fusion methods such as voting, weighted averaging and Dempster-Shafer theory, the sensor output results of the two cameras were fused.

[0086] This embodiment uses Dempster-Shafer theory as an example to present a method for fusing event cameras and traditional cameras.

[0087] Assume there are q categories to be identified, where m is the category of the current sample.

[0088] m∈M{m1, m2, ..., m} q}

[0089] The basic assignment probabilities are given by the classification algorithms of the event camera and the traditional camera, respectively.

[0090] p tra (m)=p tra {p tra (m1), p tra (m2), ..., p tra (m q )}

[0091] p eυent (m)=p eυent {p tra (m1), p eυent (m2), ..., p eυent (m q )}

[0092] Where, p tra ( mi ) is the m=m obtained by a traditional camera i The probability, p eυent (m i) is the m=m obtained from the event camera. i The probability of.

[0093] Calculate the joint probability for each hypothetical category.

[0094]

[0095] Finally, the category with the highest joint probability is selected as the decision-level fusion output category.

[0096] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for detecting space objects under different distances and illumination conditions, characterized in that, The space target detection method comprises: S10, judging whether the space target is in a backlight environment, if yes, entering S30; if not, entering S20; S20, judging whether the distance of the space target is greater than a distance threshold, if yes, entering step S40; if not, entering step S50; S30, detecting the space target in the backlight environment by using an event camera, and entering S60 after completion; S40, realizing space target detection initial position by using a traditional camera, combining the event camera to detect the space target to obtain the space target, and entering S60 after completion; the step S40 comprises: S401, acquiring the space target and background stars in the field of view by using a traditional camera, and realizing initial position selection of the space target; S402, matching the field of view with a star map template based on the position and attitude of a space platform, extracting the space target in the non-star map template as the preferred effective space target; S403, detecting the field of view detected by the traditional camera by using the event camera, obtaining the relative motion target in the field of view, comparing the effective space target extracted by the traditional camera, and evaluating the candidate effective space target; S404, based on the evaluation result of the effective space target, if the confidence is higher than a threshold, continuously tracking the effective space target, performing long-time sequence observation on the effective space target, and judging the distance nearness according to the shape size change of the effective space target; S50, detecting the motion state of the space target by using the event camera, and simultaneously acquiring the contour and texture information of the space target by using the traditional camera, and entering S60 after completion; S60, fusing the output information of the traditional camera and the event camera, and realizing space target detection under different distances and illumination conditions. 2.The method of claim 1, wherein, The step S50 comprises: transmitting the sparse asynchronous event information acquired by the event camera to a pulse neural network to complete real-time target recognition, simultaneously acquiring the contour and texture of the space target by using the traditional camera, combining the event camera to complete high-definition image reconstruction, and then identifying the space target according to a template matching or a deep neural network target recognition algorithm. 3.The method of claim 1, wherein, The traditional camera adopts a high-sensitivity CMOS camera or a CCD camera; the event camera adopts a high-time resolution neuro-morphic imaging sensor, and the field of view of the event camera is consistent with that of the traditional camera. 4.The method of claim 1, wherein, In step S20, judging whether the distance of the space target is greater than a distance threshold comprises: judging whether the space target is greater than the distance threshold by the size of the target size output by the event camera and the traditional camera. 5.The method of claim 1, wherein, Fusing the output information of the traditional camera and the event camera in step S60 comprises: forming a typical space target knowledge base by collecting the motion characteristics, contour and texture characteristics of typical space maneuvering targets in advance, and then realizing fusion by using the output data of the event camera and the traditional camera. 6.The method for detecting spatial objects under different distances and illumination conditions according to claim 1, wherein, Fusing the output information of the traditional camera and the event camera in step S60 comprises realizing fusion at the data level, the feature level and the decision level.

7. The method of claim 6, wherein the method is characterized by, The event camera is consistent with the field of view of the traditional camera, and output data of the traditional camera and the event camera are fused at a data level, high time resolution features of the event camera are fused with high spatial resolution features of the traditional camera, and fusion and reconstruction of high frame rate high spatial resolution image data are completed.

8. The method of claim 6, wherein the method is characterized by, The event camera is consistent with the field of view of the traditional camera, and output data of the traditional camera and the event camera are fused at a feature level, including: obtaining event stream data of the event camera, extracting multi-scale features, using a digital image processing operator or a deep neural network to realize traditional camera image feature extraction, and using a feature fusion algorithm to realize feature feature level fusion.

9. The method of claim 6, wherein the method is characterized by, The event camera is consistent with the field of view of the traditional camera, and output data of the traditional camera and the event camera are fused at a decision level, including: separately implementing target identification on the event camera and the traditional camera, and realizing fusion at a decision level.

Images