Fresnel optical image target detection and identification method and system based on event stream data enhancement

Abstract

The invention provides a Fresnel optical image target detection and identification method based on event stream data enhancement, and the method comprises the steps: carrying out the data preprocessing of collected Fresnel optical image event data through employing a denoising algorithm based on time-space continuity, and screening the original event stream data; converting the screened event stream data of the target area into pixel data under an image coordinate system by adopting a dual threshold limiting strategy of event quantity and accumulation time; existing image features are enhanced, and pixel-level fusion, feature-level fusion and decision-level fusion are carried out on event stream data of an event camera and RGB camera data; and the target detection module is used for performing target detection and recognition on the fused data by using a YOLO target detection model. According to the method, the detection of the Fresnel optical image target is realized, so that the detection and recognition capability of the Fresnel optical image target in a complex environment is improved, and the method is particularly suitable for scenes such as seaborne landing guidance and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, more particularly, to a Fresnel optical image target detection and recognition method and system based on event stream data enhancement. BACKGROUND

[0002] The Fresnel optical landing aid system is a key landing guidance device, which is often applied to sea landing. The system emits a specific shaped light beam, i.e. a Fresnel optical image signal, to indicate the correct glide path and help the pilot find the appropriate glide angle, thereby improving the safety and success rate of carrier-based aircraft landing in complex environments.

[0003] Currently, the algorithm for Fresnel optical target detection only processes the classical algorithm, and the recognition performance is often very limited, which cannot process Fresnel optical image target recognition in complex environments. Moreover, there is no public data set for the Fresnel optical target reference light and "meatball" target. Since the Fresnel optical landing aid system is applied to the sea, the image target obtained by the airborne camera is greatly affected by the complex environmental weather. For target detection in such a complex environment, the current mainstream deep framework and data set cannot quickly and accurately detect the target, thereby failing to meet the real-time and accuracy requirements in flight landing guidance.

[0004] The event camera is a new type of non-contact visual sensor, which has high temporal resolution and high dynamic range. However, there is no overall solution that can well adapt to the collection, conversion and processing of Fresnel optical targets based on event data. Moreover, the related process is not perfect, and the engineering practical value is not high.

[0005] In summary, the existing Fresnel optical target detection has many defects, and the Fresnel optical target detection method based on the event camera is not perfect, and the overall effect is not good. The performance and precision of the detection model have a lot of room for improvement. SUMMARY

[0006] To solve the above technical problems, the present application provides a Fresnel optical image target detection and recognition method and system based on event stream data enhancement. The event stream output data of the event camera and the traditional RGB camera data are fused to obtain enhanced Fresnel optical target features, which are input into the subsequent feature fusion extraction network, and finally the target detection task is realized.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0008] A Fresnel optical image target detection and recognition method based on event stream data enhancement, comprising the following steps:

[0009] Step 1, Fresnel optical image event data denoising, for the collected Fresnel optical image event data, a spatiotemporal continuity based denoising algorithm is used for data preprocessing, and the original event stream data is screened;

[0010] Step 2, spatial alignment of Fresnel optical image event data and RGB camera data, the event stream data of the target region after screening is converted into pixel data in the image coordinate system using the event number and accumulated time double threshold limit strategy;

[0011] Step 3, Fresnel optical image multi-modal data fusion, the existing image features are enhanced, and the event stream data of the event camera and the RGB camera data are fused at the pixel level, the feature level and the decision level;

[0012] Step 4, Fresnel optical image target detection and recognition, which is used for target detection and recognition using YOLO target detection model on the fused data.

[0013] Further, in step 1, for the Fresnel optical image event data obtained by collection, an event camera denoising algorithm based on spatiotemporal continuity is used to retain only the events occurring in the target region and eliminate other background events, specifically including:

[0014] Assuming that the event stream caused by object motion has the characteristics of spatiotemporal continuity, the formula is obtained:

[0015]

[0016] Wherein, is the core-removed neighborhood to be calculated, event i is the ith event, L is the threshold in space, Δt is the threshold in time, t is the time of the core-removed neighborhood, t i is the time of the ith event, x is the horizontal coordinate of the core-removed neighborhood, x i is the horizontal coordinate of the ith event, y is the vertical coordinate of the core-removed neighborhood, y i is the vertical coordinate of the ith event;

[0017] In the program, the threshold in space and the threshold in time are input, and two thresholds L and Δt are obtained:

[0018]

[0019] Wherein, event event is the valid point, event nosie is the noise point, is the core-removed neighborhood of event i;

[0020] According to the double threshold method, if it is a noise point, the point is removed, and if it is an effective point, the point is recorded as an event efn ; wherein the event efn is an effective point.

[0021] Finally, the cumulative events of the output are obtained wherein n is the number of current events, and N' is the total number of events.

[0022] Further, in step 3, the pixel-level fusion-based scheme design is to process at the image input level, first perform alpha fusion on the event stream data and the RGB camera data, then mark the data set, and then input the network for training.

[0023] Further, alpha fusion is to directly add the pixels corresponding to the same position of the two images, and alpha is the proportion of each frame of image participating in fusion, that is, the weight of different images:

[0024] I out =αI1+(1-α)I2;

[0025] wherein I out is the output image after fusion, alpha is the fusion ratio, I1 is the first image, and I2 is the second image.

[0026] Further, the feature-level fusion-based scheme design combines different types of image inputs by designing a neural network, and uses different background features to improve the accuracy and robustness of the Fresnel optical detection task, specifically including:

[0027] First, the RGB camera data and event stream data of the target to be detected are obtained, and then the event frame and discrete event polarity integral are obtained according to the output event stream data, wherein the speed-invariant frame is calculated according to the output quantity of the RGB camera, and then the speed-invariant frame of the RGB camera data and the event camera are input into the feature extraction module of the improved neural network for feature extraction, to obtain the RGB feature and the optimized event feature, and then the RGB feature and the optimized event feature are input into the feature fusion network for feature fusion, to obtain the fused feature, and then the fused feature is input into the decoder of the YOLO network for decoding, to obtain the detection result of the target to be detected.

[0028] Further, the decision-level fusion-based scheme design adopts the WBF algorithm, based on the idea of weighted fusion, uses the RGB camera data to train model one in the process of screening the prediction frame, uses the event camera data to train model two, and finally combines the confidence of the prediction of the two models to perform weighted fusion, thereby obtaining an optimal frame.

[0029] A Fresnel optical image target detection and recognition system based on event stream data augmentation includes:

[0030] Fresnel optical image event data denoising module: It uses an event camera denoising algorithm based on spatiotemporal continuity algorithm to preprocess the data and filter the raw event stream data;

[0031] Fresnel optical image event data spatial alignment module: Converts the filtered target area event stream data into pixel data in the image coordinate system using a dual threshold limitation strategy of event quantity and accumulation time;

[0032] Fresnel optical image multimodal data fusion module: used to enhance existing image features and perform pixel-level fusion, feature-level fusion, and decision-level fusion of event stream data from event cameras and RGB camera data;

[0033] Fresnel optical image target detection and recognition module: used to perform target detection and recognition on the fused data using the YOLO target detection model.

[0034] The beneficial effects of this invention compared to the prior art are as follows:

[0035] This invention proposes a method and system for Fresnel optical image target detection and recognition based on event stream data enhancement. After acquiring Fresnel optical image event data using an event camera, the proposed Fresnel optical image event data denoising algorithm based on spatiotemporal continuity effectively reduces noise and improves the signal-to-noise ratio. A dual constraint strategy based on the maximum number of events per frame and an event accumulation time threshold transforms the Fresnel optical image event data stream into pixel data in the image coordinate system. Finally, multimodal data fusion is used to enhance the Fresnel optical image event data frames, thus successfully detecting Fresnel optical targets using the event data. This invention employs a unique multimodal data fusion method, combining the high dynamic range and high spatiotemporal resolution of the event camera with the rich texture information of RGB camera data. This significantly improves the detection performance and robustness of Fresnel optical image targets, further enhancing the accuracy and generalization performance of Fresnel optical target detection. Attached Figure Description

[0036] Figure 1 This is a flowchart of the Fresnel optical image target detection and recognition method based on event stream data enhancement of the present invention.

[0037] Figure 2 This is a flowchart of the dual-stream feature fusion process of the present invention.

[0038] Figure 3 This is a diagram of the improved dual-stream network architecture of the present invention.

[0039] Figure 4 This is a schematic diagram of the decision-level fusion algorithm of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] This invention addresses the limitations of single RGB visual image target detection models in Fresnel optical target detection, which suffer from insufficient information representation and incomplete target detection, particularly in high-dynamic environments. It employs a combination of an optical camera and an event camera to perform Fresnel optical image visual detection. Specifically, for Fresnel visual navigation light group targets, simultaneous data acquisition by the event camera and a visible light camera is performed. Based on this, the event stream and RGB image data are fused using pixel-level, feature-level, and decision-level multimodal methods to achieve Fresnel optical image target detection. This improves the detection and recognition capabilities of Fresnel optical image targets in complex environments, making it particularly suitable for scenarios such as maritime landing guidance.

[0042] refer to Figure 1 This embodiment provides a Fresnel optical image target detection and recognition method based on event stream data augmentation, including the following steps:

[0043] Step 1: Denoising Fresnel optical image event data. The acquired Fresnel optical image event data is preprocessed using a denoising algorithm based on spatiotemporal continuity, and the original event stream data is filtered.

[0044] For the acquired Fresnel optical image event data, an event camera denoising algorithm based on spatiotemporal continuity is used to retain only events occurring in the target area and remove other background events. Specifically, this includes:

[0045] The algorithm is based on the assumption that the event stream caused by the motion of an object has spatiotemporal continuity, resulting in the formula:

[0046]

[0047] in, For the deden neighborhood that needs to be computed, event i Let L be the spatial threshold for the i-th event, Δt be the temporal threshold, and t be the time for removing the psychic neighborhood. iLet x be the time of the i-th event, and x be the x-coordinate of the denoted neighborhood. i Let y be the x-coordinate of the i-th event and y be the y-coordinate of the de-centered neighborhood. i Let be the ordinate of the i-th event;

[0048] In an abstract sense, the program takes two thresholds, L and Δt, as inputs in spatial and temporal dimensions, and obtains:

[0049]

[0050] Among them, event event For valid points, event nosie For noise points, The de-centered neighborhood of event i;

[0051] Based on the above double threshold method, if a point is noise, it is removed; if it is a valid point, it is recorded as an event. efn Among them, event efn For valid points;

[0052] Finally, the cumulative events are output. Then, visualization research is implemented; where n is the current number of events and N' is the total number of events.

[0053] Step 2: Spatial alignment of Fresnel optical image event data and RGB camera data; the filtered target area event stream data is converted into pixel data in the image coordinate system using a dual threshold limitation strategy of event quantity and accumulation time.

[0054] Step 3: Fresnel optical image multimodal data fusion, which enhances existing image features and performs pixel-level fusion, feature-level fusion, and decision-level fusion of event stream data from event cameras and RGB camera data.

[0055] Step 4, Fresnel optical image target detection and recognition, is used to perform target detection and recognition on the fused data using the YOLO target detection model.

[0056] The data acquired by the event camera is denoised using a spatiotemporal continuity algorithm. The data space is aligned by using a dual threshold limit strategy of event quantity and accumulation time. The acquired event data is then fused with RGB camera data in a multimodal manner, enabling the target detection and recognition module to complete the detection task of the fused data.

[0057] In step 3, the pixel-level fusion-based scheme is designed to process the image input at the image input level. First, the event stream data and RGB camera data are alpha-fused, then the dataset is labeled, and then fed into the network for training.

[0058] The specific method for step 3 is as follows:

[0059] 3.1 The pixel-level fusion-based scheme is designed to process images at the input level. First, the event stream image is fused with the RGB camera image. Then, the dataset is labeled and fed into the network for training.

[0060] Alpha fusion involves directly adding the pixels at corresponding positions in two images. Alpha represents the proportion of each frame participating in the fusion, i.e., the weight of different images. It is suitable for stitching tasks where the image types are similar.

[0061] I out =αI1+(1-α)I2

[0062] Among them, I out The image is the output image after fusion, where α is the fusion ratio, I1 is the first image, and I2 is the second image.

[0063] 3.2. Feature-level fusion-based scheme design: By designing a neural network to combine different types of image inputs, different background features are merged.

[0064] like Figure 2 As shown, firstly, RGB camera data and event stream data of the target to be detected are acquired. Then, event frames and discrete event polarity integrals are obtained based on the output event stream. It is required that velocity-invariant frames be calculated according to the number of outputs from the RGB cameras. Then, the RGB camera data and the velocity-invariant frames of the event cameras are respectively input into the feature extraction module of the improved neural network for feature extraction to obtain RGB features and optimized event features. Then, the RGB features and optimized event features are input into the feature fusion network for feature fusion to obtain fused features. Finally, the fused features are input into the decoder of the YOLO network for decoding to obtain the detection result of the target to be detected.

[0065] Given an input RGB image and an event camera imaging EVENT, feed them into a series of convolutional blocks to extract their local features.

[0066]

[0067] Among them, I RGB For the original RGB image, I EVENT The original event image, The feature map generated from the original RGB image. φ is a feature map generated from the original event image. i For the first stage of RGB convolutional blocks, This is the convolutional block for the EVENT branch in the first stage.

[0068] Network architecture such as Figure 3 As shown, its detection backbone includes a two-stream feature extraction network and three image feature fusion modules, while the detection network includes neck and head sections for cross-modal target detection. The feature extraction network helps extract local features from RGB camera data and event stream data, denoted as follows: and Then, these two features are input into the fusion module. By associating cross-modal features in the hidden state space, the differences between cross-modal features are reduced and the representational consistency of the fused features is enhanced.

[0069] 3.3 Scheme Design Based on Decision-Level Fusion: The WBF (weighted box fusion) algorithm is adopted. Based on the idea of ​​weighted fusion, RGB images are used to train Model 1 during the selection of prediction boxes, and event camera data is used to train Model 2. Finally, the confidence scores of the two models are combined for weighted fusion to obtain an optimal frame.

[0070] like Figure 4 As shown, using two detection boxes and their respective confidence levels to jointly predict new detection boxes and confidence levels results in smaller variance and residuals.

[0071]

[0072] Where x1 is the x-coordinate of the top left corner of the first detection box, y1 is the y-coordinate of the top left corner of the first detection box, x2 is the x-coordinate of the top left corner of the second detection box, y2 is the y-coordinate of the top left corner of the second detection box, x1′ is the x-coordinate of the bottom right corner of the first detection box, y1′ is the y-coordinate of the bottom right corner of the first detection box, x′2 is the x-coordinate of the bottom right corner of the second detection box, y′2 is the y-coordinate of the bottom right corner of the second detection box, s1 is the confidence score of the first detection box, s2 is the confidence score of the second detection box, x3 is the x-coordinate of the top left corner of the fused detection box, y3 is the y-coordinate of the top left corner of the fused detection box, x′3 is the x-coordinate of the bottom right corner of the fused detection box, y′3 is the y-coordinate of the bottom right corner of the fused detection box, and s3 is the confidence score of the fused detection box.

[0073] In another embodiment, a Fresnel optical image target detection and recognition system based on event stream data enhancement is provided, comprising:

[0074] Fresnel optical image event data denoising module: It uses an event camera denoising algorithm based on spatiotemporal continuity algorithm to preprocess the data and filter the raw event stream data;

[0075] Fresnel optical image event data spatial alignment module: Converts the filtered target area event stream data into pixel data in the image coordinate system using a dual threshold limitation strategy of event quantity and accumulation time;

[0076] Fresnel optical image multimodal data fusion module: used to enhance existing image features and perform pixel-level fusion, feature-level fusion, and decision-level fusion of event stream data from event cameras and RGB camera data;

[0077] Fresnel optical image target detection and recognition module: used to perform target detection and recognition on the fused data using the YOLO target detection model.

[0078] This invention proposes a method and system for Fresnel optical image target detection and recognition based on event stream data enhancement. After acquiring Fresnel optical image event data using an event camera, the proposed Fresnel optical image event data denoising algorithm based on spatiotemporal continuity effectively reduces noise and improves the signal-to-noise ratio. A dual constraint strategy based on the maximum number of events per frame and an event accumulation time threshold transforms the Fresnel optical image event data stream into pixel data in the image coordinate system. Finally, multimodal data fusion is used to enhance the Fresnel optical image event data frames, thus successfully detecting Fresnel optical targets using the event data. This invention employs a unique multimodal data fusion method, combining the high dynamic range and high spatiotemporal resolution of the event camera with the rich texture information of RGB camera data. This significantly improves the detection performance and robustness of Fresnel optical image targets, further enhancing the accuracy and generalization performance of Fresnel optical target detection.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Fresnel optical image target detection and recognition method based on event stream data augmentation, characterized in that, Includes the following steps: Step 1: Denoising Fresnel optical image event data. The acquired Fresnel optical image event data is preprocessed using a denoising algorithm based on spatiotemporal continuity, and the original event stream data is filtered. Step 2: Spatial alignment of Fresnel optical image event data and RGB camera data; the filtered target area event stream data is converted into pixel data in the image coordinate system using a dual threshold limitation strategy of event quantity and accumulation time. Step 3: Fresnel optical image multimodal data fusion, which enhances existing image features and performs pixel-level fusion, feature-level fusion, and decision-level fusion of event stream data from the event camera and RGB camera data; Step 4, Fresnel optical image target detection and recognition, is used to perform target detection and recognition on the fused data using the YOLO target detection model.

2. The Fresnel optical image target detection and recognition method based on event stream data augmentation according to claim 1, characterized in that, In step 1, for the acquired Fresnel optical image event data, an event camera denoising algorithm based on spatiotemporal continuity is used to retain only the events occurring in the target area and remove other background events. Specifically, this includes: Assuming that the flow of events caused by the motion of an object has the characteristic of spatiotemporal continuity, we obtain the formula: in, For the dedone neighborhood that needs to be computed, event i Let L be the spatial threshold for the i-th event, Δt be the temporal threshold, and t be the time for removing the psychic neighbor. i Let x be the time of the i-th event, and x be the x-coordinate of the denoted neighborhood. i Let y be the x-coordinate of the i-th event and y be the y-coordinate of the de-centered neighborhood. i Let be the ordinate of the i-th event; In an abstract sense, the program takes two thresholds, L and Δt, as inputs in spatial and temporal dimensions, and obtains: Among them, event event For valid points, event nosie For noise points, The de-centered neighborhood of event i; Based on the above double threshold method, if a point is noise, it is removed; if it is a valid point, it is recorded as an event. efn Among them, event efn For valid points; Finally, the cumulative events are output. Where n is the current number of events, and N' is the total number of events.

3. The Fresnel optical image target detection and recognition method based on event stream data augmentation according to claim 1, characterized in that, In step 3, the pixel-level fusion-based scheme design is performed at the image input level. First, the event stream data and RGB camera data are alpha-fused, then the dataset is labeled, and then fed into the network for training.

4. The Fresnel optical image target detection and recognition method based on event stream data augmentation according to claim 1, characterized in that, Alpha fusion involves directly adding the pixels at corresponding locations in two images. alpha represents the proportion of each frame participating in the fusion process, i.e., the weight of the different images. I out =αI1+(1-α)I2; Among them, I out I1 represents the first image and I2 represents the second image. α is the fusion ratio, I1 is the first image, and I2 is the second image.

5. The Fresnel optical image target detection and recognition method based on event stream data augmentation according to claim 1, characterized in that, The scheme design based on feature-level fusion improves the accuracy and robustness of Fresnel optical detection tasks by designing a neural network that combines different types of image inputs and merges different background features. Specifically, it includes: First, acquire RGB camera data and event stream data of the target to be detected. Then, obtain event frames and discrete event polarity integrals based on the output event stream data. It is required to calculate the velocity-invariant frames according to the number of outputs of the RGB cameras. Then, input the RGB camera data and the velocity-invariant frames of the event cameras into the feature extraction module of the improved neural network for feature extraction to obtain RGB features and optimized event features. Then, input the RGB features and optimized event features into the feature fusion network for feature fusion to obtain the fused features. Finally, input the fused features into the decoder of the YOLO network for decoding to obtain the detection result of the target to be detected.

6. The Fresnel optical image target detection and recognition method based on event stream data augmentation according to claim 5, characterized in that, The scheme design based on decision-level fusion adopts the WBF algorithm. Based on the idea of ​​weighted fusion, RGB camera data is used to train model one in the process of selecting prediction boxes, and event camera data is used to train model two. Finally, the confidence scores of the two models are combined for weighted fusion to obtain an optimal frame.

7. A Fresnel optical image target detection and recognition system based on event stream data augmentation, implementing the method of any one of claims 1-6, characterized in that, include: Fresnel optical image event data denoising module: It uses an event camera denoising algorithm based on spatiotemporal continuity algorithm to preprocess the data and filter the raw event stream data; Fresnel optical image event data spatial alignment module: Converts the filtered target area event stream data into pixel data in the image coordinate system using a dual threshold limitation strategy of event quantity and accumulation time; Fresnel optical image multimodal data fusion module: used to enhance existing image features and perform pixel-level fusion, feature-level fusion, and decision-level fusion of event stream data from event cameras and RGB camera data; Fresnel optical image target detection and recognition module: used to perform target detection and recognition on the fused data using the YOLO target detection model.

Images