Demosaicing and motion detection method and system based on hybrid event camera

Abstract

The application provides a demosaicing and motion detection method and system based on a hybrid event camera, comprising: acquiring RAW images and event stream data; performing demosaicing processing on the RAW images based on a pyramid squeeze attention to obtain RGB images and shared features; performing motion detection on the shared features and the event stream data based on a lightweight recursive residual network to obtain a detection result; and integrating the RGB images and the detection result to output target image data. The demosaicing processing by the pyramid squeeze attention can effectively improve the edge features of dynamic targets; the motion detection on the shared features by the lightweight recursive residual network can not only improve the feature utilization rate, avoid repeated feature extraction, improve the system synergy, but also effectively improve the detection confidence, adapt to various scenes, realize lightweight deployment, and solve the problems of poor synergy, low feature utilization rate, and insufficient lightweight of the existing demosaicing and motion detection method of the hybrid event camera.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for demosaicing and motion detection based on a hybrid event camera. Background Technology

[0002] Hybrid event cameras combine the high-resolution imaging capabilities of traditional CMOS (Complementary Metal-Oxide-Semiconductor) image sensors with the high temporal resolution event capture capabilities of EVS (Event-based Vision Sensor) sensors, enabling the simultaneous acquisition of scene texture details and dynamic change information, thus becoming a core technology direction in the field of dynamic scene imaging.

[0003] In practical applications, hybrid event cameras need to simultaneously achieve two core functions: first, to perform de-mosaic processing on the Bayer format Raw images output by the CMOS sensor and reconstruct high-fidelity RGB images; and second, to detect moving targets (such as people and vehicles) in the scene based on the event stream data output by the EVS sensor, in order to meet the needs of real-time monitoring and intelligent interaction.

[0004] However, existing "de-mosaic-motion detection" processing solutions for hybrid event cameras have the following key technical challenges: 1. Poor coordination between demosaicing and motion detection: Existing solutions mostly adopt a step-by-step independent architecture of "demosaicing first, then motion detection" - the demosaicing module does not consider the feature requirements of subsequent motion detection, and although the output RGB image meets the visual quality, it lacks the key features of dynamic targets; the motion detection module needs to extract features from the RGB image again, resulting in computational redundancy, making it difficult to adapt to scenarios with limited computing power such as mobile devices, and the processing latency generally exceeds 100ms, which cannot meet the real-time requirements; 2. Low utilization of demosaicing features: Although traditional demosaicing networks (such as U-Net) achieve resolution restoration through downsampling-upsampling, they do not differentiate the feature weights of "texture detail regions" and "smooth regions" in dynamic scenes, which makes color artifacts easy to appear at the edges of dynamic targets. Furthermore, the high-frequency features lost during downsampling cannot be effectively recovered, affecting the target localization accuracy of subsequent motion detection. 3. Insufficient lightweighting of motion detection networks: Existing motion detection systems mostly rely on Transformer architectures or complex CNN networks, which have a large number of parameters and high computational complexity, making it difficult to complete quantization deployment on mobile devices. At the same time, the fusion method of EVS event stream data and image features is simple (such as direct pixel overlay), which does not fully utilize the "dynamic change sensitivity" of EVS event stream data, resulting in a high rate of missed detection of moving targets under complex conditions such as low light and backlight.

[0005] In other words, existing hybrid event camera "de-mosaic-motion detection" processing schemes suffer from problems such as poor coordination, low feature utilization, and insufficient lightweight design, which severely limit the application scenarios of hybrid event cameras. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for demosaicing and motion detection based on a hybrid event camera, so as to solve the problems of poor coordination, low feature utilization and insufficient lightweighting of existing hybrid event camera demosaicing and motion detection methods.

[0007] To address the aforementioned technical problems, this invention provides a method for demosaicing and motion detection based on a hybrid event camera, comprising: Acquire raw image data, which includes RAW images and event stream data; A demosaic module with pyramid compression attention is used to demosaic the RAW image to obtain the RGB image and shared features. A motion detection module with a lightweight recursive residual network is used to perform motion detection on shared features and event stream data to obtain detection results; The RGB image is integrated with the detection results to output the target image data.

[0008] Optionally, in the hybrid event camera-based demosaic and motion detection method, after acquiring the original image data, the hybrid event camera-based demosaic and motion detection method further includes: The RAW image is quantized to a preset number of bits to obtain the quantized RAW image; Event stream data is accumulated into event images according to time windows, and the event images are downsampled to obtain downsampled event images.

[0009] Optionally, in the aforementioned method for demosaicing and motion detection based on a hybrid event camera, the time window is synchronized with the acquisition period of the RAW image.

[0010] Optionally, in the hybrid event camera-based demosaic and motion detection method, the resolution of the event image is the same as the resolution of the RAW image.

[0011] Optionally, in the aforementioned demosaicing and motion detection method based on a hybrid event camera, the method of using a demosaicing module with pyramid squeeze attention to perform demosaicing processing on the RAW image to obtain an RGB image and shared features includes: Construct a demosaicing module, including a U-Net architecture with pyramid squeezed attention; We utilize the U-Net architecture with pyramid squeeze attention to downsample RAW images in order to extract shared features; The shared features are upsampled using a U-Net architecture with pyramid squeeze attention to obtain RGB images; The first loss function is used to apply error constraints to the RGB image to obtain the enhanced RGB image.

[0012] Optionally, in the aforementioned method for demosaicing and motion detection based on hybrid event cameras, the U-Net architecture with pyramid squeeze attention includes a first convolutional unit, a pyramid squeeze attention unit, a second convolutional unit, and a third convolutional unit. The first convolutional unit is used to downsample the input RAW image and input the downsampled feature map into the pyramid squeeze attention unit; The pyramid squeezing attention unit is used to perform pyramid squeezing on the feature map obtained by downsampling the RAW image through pooling layers of different scales to obtain the first multi-scale channel attention weight. It is also used to multiply the first multi-scale channel attention weight with the feature map obtained by downsampling the RAW image channel by channel to output the first feature map. The first convolutional unit is also used to downsample the first feature map and input the downsampled feature map into the pyramid squeeze attention unit; The pyramid squeezing attention unit is also used to perform pyramid squeezing on the feature map obtained by downsampling the first feature map through pooling layers of different scales to obtain the second multi-scale channel attention weights. It is also used to multiply the second multi-scale channel attention weights with the feature map obtained by downsampling the first feature map channel by channel to output shared features. The second convolutional unit is used to perform transpose convolution on the shared features and concatenate them with the first feature map to obtain the first concatenated features. It is also used to input the first concatenated features into the pyramid squeeze attention unit to obtain the second feature map. The second convolutional unit is also used to perform transpose convolution on the second feature map and to perform feature concatenation with the feature map obtained by downsampling the RAW image to obtain the second concatenated feature. It is also used to input the second concatenated feature into the pyramid squeeze attention unit to obtain the third feature map. The third convolutional unit is used to restore the resolution of the third feature map to be consistent with that of the RAW image, so as to obtain an RGB image.

[0013] Optionally, in the aforementioned demosaicing and motion detection method based on a hybrid event camera, the method of using a first loss function to impose error constraints on the RGB image to obtain an enhanced RGB image includes: An error constraint is applied to the RGB image and the reference image using a first loss function, which is the L2 loss function, expressed as:

[0014] in, The resolution of the RAW image is represented by ; i and j represent the horizontal and vertical coordinates of the pixels in the image, respectively; c represents the RGB channel index, where c=1 represents the R channel, c=2 represents the G channel, and c=3 represents the B channel. This represents the pixel value of the c channel at coordinate (i,j) in the RGB image obtained using the U-Net architecture with pyramid squeeze attention. This represents the pixel value of the reference image at coordinates (i,j) in channel c.

[0015] Optionally, in the aforementioned method for demosaicing and motion detection based on hybrid event cameras, the method for using a motion detection module with a lightweight recurrent residual network to perform motion detection on shared features and event stream data to obtain detection results includes: Shared features are concatenated with event stream data to obtain fused features; Construct a motion detection module, including a lightweight recursive residual network; A lightweight recursive residual network is used to perform recursive residual processing on the fused features to obtain the target feature map; Target classification and detection are performed on the target feature map to obtain the detection results; The detection results are optimized using a second loss function to obtain optimized detection results.

[0016] Optionally, in the aforementioned method for demosaicing and motion detection based on a hybrid event camera, the lightweight recursive residual network includes a first residual unit, a second residual unit, and a third residual unit connected in sequence with recursive residuals; the first residual unit, the second residual unit, and the third residual unit each include two 3×3 convolutional layers, a BatchNorm layer, and a ReLU activation function.

[0017] Optionally, in the aforementioned method for demosaicing and motion detection based on hybrid event cameras, the method for concatenating shared features and event stream data to obtain fused features includes: The shared features are concatenated with the event stream data to obtain the initial fused features; By using a 1×1 convolutional layer, the number of channels in the initial fused features is reduced to obtain the fused features.

[0018] Optionally, in the aforementioned method for desacrifice and motion detection based on a hybrid event camera, the method for performing target classification and detection on the target feature map to obtain the detection result includes: Global average pooling is performed on the target feature map to obtain a multi-dimensional feature vector, and a fully connected layer is used to classify the multi-dimensional feature vector to obtain the target classification result. Perform a 1×1 convolution on the target feature map to obtain the detection box coordinates for each target; Based on the target classification results and the coordinates of the detection boxes, the confidence score of each detection box is calculated using the sigmoid function; Based on the confidence level, determine whether the target classification detection is effective in order to obtain the detection result.

[0019] Optionally, in the aforementioned demosaicing and motion detection method based on hybrid event cameras, the second loss function is the VFL loss function, expressed as:

[0020] Where N represents the number of detection boxes; This represents the prediction confidence of the k-th bounding box; Represents the true label, where the target exists. When the value is 1, the target does not exist. =0; This indicates the adjustment parameter.

[0021] Optionally, in the aforementioned method for demosaicing and motion detection based on a hybrid event camera, the method for integrating the RGB image with the detection results to output target image data includes: Non-maximum suppression is applied to the detection results to output the target image data.

[0022] To address the aforementioned technical problems, the present invention also provides a demosaic and motion detection system based on a hybrid event camera, used to implement the demosaic and motion detection method based on a hybrid event camera as described in any of the preceding claims. The demosaic and motion detection system based on a hybrid event camera includes: An image data acquisition module is used to acquire raw image data, which includes RAW images and event stream data. The demosaic module is used to perform demosaic processing on RAW images based on pyramid squeeze attention to obtain RGB images and shared features; The motion detection module is used to perform motion detection on shared features and event stream data based on a lightweight recursive residual network to obtain detection results; The results output module is used to integrate the RGB image with the detection results to output the target image data.

[0023] The present invention provides a method and system for demosaicing and motion detection based on a hybrid event camera, comprising: acquiring raw image data, wherein the raw image data includes RAW images and event stream data; performing demosaicing processing on the RAW images using a demosaicing module with pyramid squeeze attention to obtain RGB images and shared features; performing motion detection on the shared features and event stream data using a motion detection module with a lightweight recurrent residual network to obtain detection results; and integrating the RGB images with the detection results to output target image data. By utilizing a demosaic module with pyramid squeeze attention for demosaic processing, dynamic target edge features can be effectively enhanced, improving image quality. Furthermore, by employing a motion detection module with a lightweight recursive residual network to perform motion detection on shared features and event stream data, the motion detection process can leverage the shared features provided by the demosaic process. This not only improves feature utilization and avoids redundant feature extraction, enhancing system synergy, but also effectively increases detection confidence, adapting to various scenarios. Moreover, it achieves lightweight deployment, addressing the issues of poor synergy, low feature utilization, and insufficient lightweight design in existing demosaic and motion detection methods for hybrid event cameras. Attached Figure Description

[0024] Figure 1 This is a flowchart of the demosaic and motion detection method based on a hybrid event camera provided in this embodiment; Figure 2 This is a schematic diagram of the network architecture of the feature-sharing two-stage model provided in this embodiment; Figure 3 This is a block diagram of the demosaic and motion detection system based on a hybrid event camera provided in this embodiment. Detailed Implementation

[0025] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a more detailed account of the de-mosaic and motion detection method and system based on a hybrid event camera proposed in this invention. It should be noted that the drawings are all in a very simplified form and use non-precise scales, intended only to facilitate and clarify the illustration of the embodiments of the invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different figures may emphasize different aspects and may sometimes use different scales.

[0026] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects in order to describe embodiments of the invention, and are not used to describe a specific order or sequence. It should be understood that such uses of terminology are interchangeable where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] This embodiment provides a method for demosaicing and motion detection based on a hybrid event camera, such as... Figure 1 As shown, it includes: S1, acquire raw image data, which includes RAW images and event stream data; S2, using a demosaic module with pyramid squeeze attention to demosaic the RAW image to obtain the RGB image and shared features; S3 utilizes a motion detection module with a lightweight recursive residual network to perform motion detection on shared features and event stream data to obtain detection results; S4 integrates the RGB image with the detection results to output the target image data.

[0028] The demosaicing and motion detection method based on a hybrid event camera provided in this embodiment effectively enhances the edge features of dynamic targets and improves image quality by using a demosaicing module with pyramid squeeze attention for demosaicing. By utilizing a motion detection module with a lightweight recursive residual network to perform motion detection on shared features and event stream data, the motion detection process can utilize the shared features provided by the demosaicing process. This not only improves feature utilization and avoids redundant feature extraction, thus improving system synergy, but also effectively increases detection confidence, adapts to various different scenarios, and achieves lightweight deployment. It solves the problems of poor synergy, low feature utilization, and insufficient lightweightness in existing hybrid event camera demosaicing and motion detection methods.

[0029] Specifically, in this embodiment, step S1 involves acquiring raw image data, which includes RAW images and event stream data.

[0030] In practical applications, RAW images and their corresponding event stream data are acquired from hybrid event cameras.

[0031] Preferably, in this embodiment, after acquiring the original image data, the original image data is further preprocessed. Specifically, in this embodiment, the preprocessing includes: The RAW image is quantized to a preset number of bits to obtain a quantized RAW image. For example, the original RAW image (resolution w×h) provided by the mixed event camera is quantized to 10 bits to obtain a quantized RAW image, denoted as RAW_in, which is used as input for subsequent steps. The specific implementation of quantization is well known to those skilled in the art, and will not be described in detail here.

[0032] Furthermore, the event stream data is accumulated into an event image, denoted as EVS_in, according to a time window. The resolution of this event image is consistent with that of the RAW_in image, and the pixel value of the event image represents the frequency of event occurrence at that location. Then, the event image EVS_in is downsampled to obtain a downsampled event image, denoted as EVS_down. The resolution of the event image EVS_down can be w / 4×h / 4, consistent with the size of the subsequently obtained shared features. The time window is synchronized with the acquisition period of the RAW image (RAW_in), for example, 33ms.

[0033] Furthermore, in this embodiment, step S2 involves using a demosaic module with pyramid compression attention to perform demosaic processing on the RAW image to obtain an RGB image and shared features.

[0034] Specifically, in this embodiment, firstly, a demosaic module is constructed, which may include a U-Net architecture with Pyramid Squeeze Attention (PSA). Of course, in other embodiments, the demosaic module may also include a Transformer architecture with Pyramid Squeeze Attention (specifically, it may be DiT, Swin-Unet, CNN-Transformer, etc. with Pyramid Squeeze Attention), or KAN may be introduced to replace MLP in the U-Net architecture with Pyramid Squeeze Attention. Furthermore, the demosaic module may include FCN, SegNet, DeepLab, or Mask R-CNN, etc., with Pyramid Squeeze Attention.

[0035] This embodiment uses a U-Net architecture with pyramid squeeze attention as an example to illustrate how to perform demosaic processing on RAW images to obtain RGB images and shared features. Those skilled in the art can, based on the example of this embodiment and in conjunction with existing technologies, obtain other implementations of demosaic modules to obtain RGB images and shared features; these will not be elaborated upon further in this application.

[0036] In this embodiment, the U-Net architecture with pyramid squeeze attention includes a first convolutional unit, a pyramid squeeze attention unit, a second convolutional unit, and a third convolutional unit.

[0037] Then, the RAW images are downsampled using a U-Net architecture with pyramid squeeze attention to extract shared features.

[0038] In practical applications, a first-level downsampling is performed using a first convolutional unit and a pyramid squeezing attention unit. This includes downsampling the input RAW image (RAW_in) using the first convolutional unit and inputting the downsampled feature map into the pyramid squeezing attention unit. In one specific embodiment, the first convolutional unit includes a convolutional layer with a 3×3 kernel, a stride of 2, and padding of 1. After downsampling RAW_in using this convolutional layer, a feature map of w / 2×h / 2×64 is output, and this feature map is input into the pyramid squeezing attention unit.

[0039] Furthermore, a pyramid-squeezing attention unit is used to perform pyramid-style compression on the feature map obtained by downsampling the RAW image through pooling layers of different scales to obtain the first multi-scale channel attention weights. These first multi-scale channel attention weights are then multiplied channel-by-channel with the feature map obtained by downsampling the RAW image to output the first feature map. In one specific embodiment, the pyramid-squeezing attention unit may include four pooling layers of different scales, with pooling kernel sizes of 2×2, 4×4, 6×6, and 8×8, respectively. The first multi-scale channel attention weights are obtained by pyramid-style compression of the w / 2×h / 2×64 feature map through the four pooling layers. These first multi-scale channel attention weights are then multiplied channel-by-channel with the w / 2×h / 2×64 feature map to enhance key features such as dynamic target edges, outputting the first feature map, denoted as F1.

[0040] Then, a second level of downsampling is performed using the first convolutional unit and the pyramid squeezing attention unit. The process of the second level of downsampling is similar to that of the first level of downsampling, including: using the first convolutional unit to downsample the first feature map F1, and inputting the downsampled feature map into the pyramid squeezing attention unit; using the pyramid squeezing attention unit to perform pyramid squeezing on the feature map obtained by downsampling the first feature map through pooling layers of different scales to obtain the second multi-scale channel attention weights, and also using the second multi-scale channel attention weights to multiply the feature map obtained by downsampling the first feature map channel by channel to output a shared feature of w / 4×h / 4×128, denoted as F2.

[0041] Of course, in other embodiments, two identical sets of first convolutional units and pyramid squeeze attention units can be constructed in U-Net to perform first-level downsampling and second-level downsampling respectively, thereby improving image processing efficiency.

[0042] Finally, the shared features are upsampled using the U-Net architecture with pyramid squeeze attention to obtain the RGB image.

[0043] In practical applications, a first-level upsampling is performed using a second convolutional unit and a pyramid squeezing attention unit. This includes: transposing the shared features using the second convolutional unit and concatenating them with the first feature map to obtain a first concatenated feature; then inputting the first concatenated feature into the pyramid squeezing attention unit to obtain a second feature map. In one specific embodiment, the second convolutional unit includes a convolutional layer with a 4×4 kernel, a stride of 2, and padding of 1. This convolutional layer is used to transpose the shared feature F2 and concatenate it with the first feature map F1, merging the number of channels to 128. This merged feature map is then input into the pyramid squeezing attention unit to obtain the second feature map, denoted as F3.

[0044] Subsequently, a second level of upsampling is performed using the second convolutional unit and the pyramid squeezing attention unit. The second-level upsampling process is similar to the first-level upsampling process, including: transposing the second feature map F3 using the second convolutional unit and concatenating it with the feature map obtained from downsampling the RAW image to obtain the second concatenated feature. This second concatenated feature is then input into the pyramid squeezing attention unit to obtain the third feature map. In one specific embodiment, the second convolutional unit includes a convolutional layer with a 4×4 kernel, a stride of 2, and padding of 1. This convolutional layer is used to transpose the second feature map F3 and concatenate it with the w / 2×h / 2×64 feature map, merging the channel count to 96. This merged feature map is then input into the pyramid squeezing attention unit to obtain the third feature map.

[0045] Finally, the resolution of the third feature map is restored to match that of the RAW image using the third convolutional unit to obtain the RGB image. In one specific embodiment, the third convolutional unit includes a convolutional layer with a 3×3 kernel. This convolutional layer is used to restore the resolution of the third feature map to w×h, which matches that of the RAW image, to obtain the original-size RGB image, denoted as RGB_out.

[0046] Of course, in other embodiments, the pyramid squeezing attention units used in the downsampling and upsampling processes are set independently, thereby improving image processing efficiency.

[0047] Preferably, in this embodiment, the first loss function can also be used to apply error constraints to the RGB image to obtain an enhanced RGB image.

[0048] Specifically, an error constraint is applied to the RGB image and the reference image using a first loss function, which is the L2 loss function, expressed as:

[0049] in, The resolution of the RAW image is represented by ; i and j represent the horizontal and vertical coordinates of the pixels in the image, respectively; c represents the RGB channel index, where c=1 represents the R channel, c=2 represents the G channel, and c=3 represents the B channel. This represents the pixel value of the c channel at coordinate (i,j) in the RGB image obtained using the U-Net architecture with pyramid squeeze attention. This represents the pixel value of the reference image at coordinates (i,j) in channel c. This represents the loss value for the demosaicing task, used to measure the color error between the generated RGB image and the real image.

[0050] In practical applications, the reference image comes from a real reference image captured by a professional color calibration camera. The L2 loss function is used to uniformly constrain the color error of each pixel, especially enhancing the color accuracy of dynamic target edges.

[0051] The demosaicing and motion detection method based on a hybrid event camera provided in this embodiment utilizes a demosaicing module with pyramid squeeze attention to process RAW images. It leverages multi-scale pooling of pyramid squeeze attention to generate attention weights for dynamic scenes captured by the hybrid event camera. These attention weights are then used to assign high weights to moving target edges and texture detail regions (e.g., vehicle edges with a weight of 0.8), and low weights to smooth regions (e.g., sky regions with a weight of 0.2), solving the problem of indiscriminate feature allocation in traditional U-Net and enhancing key features of dynamic targets. Simultaneously, the precise weight allocation of pyramid squeeze attention ensures that the demosaicing process focuses on color restoration of dynamic regions, avoiding color artifacts caused by insufficient feature extraction in dynamic scenes and improving image quality. Furthermore, the shared features optimized by pyramid squeeze attention can retain core features of dynamic targets while meeting the image quality requirements of RGB images, providing high-quality input for subsequent motion detection, avoiding redundant feature extraction, and reducing computational redundancy.

[0052] Furthermore, in this embodiment, step S3 involves using a motion detection module containing a lightweight recursive residual network to perform motion detection on shared features and event stream data to obtain detection results.

[0053] Specifically, in this embodiment, firstly, the shared features and event stream data are concatenated by channels to obtain fused features. In practical applications, the shared features F2 (w / 4×h / 4) and event stream data EVS_down (w / 4×h / 4×1) are concatenated by channels to obtain the initial fused features, denoted as F_fuse (w / 4×h / 4×129). Then, a 1×1 convolutional layer is used to reduce the number of channels of the initial fused features F_fuse to 64, resulting in the fused features, denoted as F_fuse_opt, thereby achieving deep fusion of image features and EVS event dynamic information.

[0054] Then, a motion detection module is constructed, including a lightweight recurrent residual network. In this embodiment, the lightweight recurrent residual network includes a first residual unit, a second residual unit, and a third residual unit; each of the first, second, and third residual units includes two 3×3 convolutional layers (stance 1, padding 1), a BatchNorm layer, and a ReLU activation function. The first, second, and third residual units are sequentially connected through residuals, i.e., input features + output features, thereby mitigating gradient vanishing. In practical applications, the number of parameters is controlled within 10M to meet the requirements of edge-side quantization deployment.

[0055] Next, a lightweight recursive residual network is used to perform recursive residual processing on the fused features to obtain the target feature map. Specifically, in this embodiment, the fused feature F_fuse_opt is input into the first residual unit, and the feature map F_res1 is output; the feature map F_res1 and the fused feature F_fuse_opt are added element-wise, i.e., recursive feature enhancement processing is performed, and then input into the second residual unit, and the feature map F_res2 is output; the feature map F_res2 and the fused feature F_fuse_opt are added element-wise and then input into the third residual unit, and the target feature map F_res3 (w / 4×h / 4×64) is output.

[0056] Of course, in practical applications, more residual units can be set according to actual needs, and this application does not impose any restrictions on this.

[0057] Finally, target classification detection is performed on the target feature map to obtain the detection result. Specifically, in this embodiment, firstly, global average pooling is performed on the target feature map F_res3 to obtain a multi-dimensional feature vector, such as a 64-dimensional feature vector. Then, a fully connected layer is used to classify the multi-dimensional feature vector to obtain the target classification result. The output dimension of the fully connected layer is num_classes, representing the number of target categories detected, such as person-1, vehicle-2, background-0, etc., and the probability of each category is output as the target classification result. Then, a 1×1 convolution is performed on the target feature map F_res3, with an output dimension of 4×num_classes, where 4 corresponds to the x1, y1, x2, y2 coordinates of the detection box, thus obtaining the detection box coordinates of each target. Next, based on the target classification result and the detection box coordinates, the confidence score of each detection box is calculated using the sigmoid function, where the confidence score ranges from 0 to 1. Finally, based on the confidence score, it is determined whether the target classification detection is effective to obtain the detection result. In one specific embodiment, if the confidence level is greater than or equal to 0.5, the detection box is determined to be valid, and the output detection result may include the target category (num_classes), the coordinates of the detection box (x1, y1, x2, y2), and the confidence level.

[0058] Preferably, in this embodiment, a second loss function can also be used to optimize the detection results to obtain optimized detection results.

[0059] Specifically, the second loss function is the VFL loss function, expressed as:

[0060] Where N represents the number of detection boxes; This represents the prediction confidence of the k-th bounding box; Represents the true label, where the target exists. When the value is 1, the target does not exist. =0; This represents an adjustment parameter used for differentiated weight allocation to improve the detection accuracy of small and dense targets. In practical applications, its value can be 2. This represents the logarithmic loss term corresponding to the prediction confidence level, penalizing low-confidence predictions; This represents the log loss term corresponding to the true label, penalizing classification errors; The total loss value for motion detection tasks, taking into account both target classification and bounding box regression accuracy.

[0061] By using the VFL loss function, differentiated weights can be assigned to "high-confidence targets" and "low-confidence targets," thereby improving the detection confidence of small targets (such as pedestrians in the distance) and the bounding box localization accuracy of dense targets (such as congested vehicles), and improving the accuracy of motion detection results.

[0062] Furthermore, the combined loss of the demosaicing process and the motion detection process can be expressed as:

[0063] in, This represents a hyperparameter, with a value range of 0 to 1.

[0064] Furthermore, in this embodiment, step S4 involves integrating the RGB image with the detection results to output target image data.

[0065] Specifically, the detection results are processed by non-maximum suppression (NMS) to remove redundant detection boxes with an overlap rate exceeding a preset overlap rate threshold (e.g., 0.3), in order to output target image data. The target image data may include: Visual results: The RGB image output from the desacrifice process is overlaid with the valid detection boxes and category labels output from the motion detection process; Data results include a list of moving target categories, bounding box coordinates and confidence scores for each target, to facilitate subsequent intelligent analysis.

[0066] The following specific embodiment illustrates the technical effects of the demosaicing and motion detection method based on a hybrid event camera provided in this embodiment.

[0067] First, a feature-sharing two-stage model is constructed, including a demosaic module with pyramid squeeze attention and a motion detection module with a lightweight recurrent residual network. In a specific embodiment, such as... Figure 2 As shown, the demosaic module includes a U-Net architecture with pyramid squeezing attention and an RGB reconstruction loss function to enhance the color accuracy of dynamic target edges, while the motion detection module includes a lightweight recursive residual network and a loss function to optimize the output of the detection results.

[0068] Next, RAW images (4032×3024 resolution) are acquired from the CMOS sensor of the hybrid event camera, and corresponding event stream data are acquired from the EVS sensor. The RAW images are quantized to 10 bits to obtain RAW_in; the event stream data is accumulated in a 33ms time window to generate event images EVS_in (4032×3024 resolution, pixel value range 0~255). The event images EVS_in are downsampled to obtain EVS__down with a resolution of 1008×756.

[0069] Next, RAW_in is input into the U-Net architecture. After passing through a 3×3 convolutional layer (stride of 2, padding of 1, output channels of 64), a feature map F_1 of 2016×1512×64 is obtained. This feature map is then compressed using Pyramid Squeezing Attention (PSA) to calculate 64-dimensional channel attention weights. These weights are multiplied channel-by-channel by feature map F_1 to output the first feature map F1 (2016×1512×64). F1 is then input into a 3×3 convolutional layer (stride of 2, padding of 1, output channels of 128) to obtain a feature map F_2 of 1008×756×128. After passing through Pyramid Squeezing Attention (PSA), a shared feature map F2 (1008×756×128) is output.

[0070] Then, the shared feature F2 is input into a 4×4 transposed convolution (stride 2, padding 1) to restore the resolution to 2016×1512. This is then concatenated with the first feature map F1 (2016×1512×64) to form a 2016×1512×192 feature map F_3. After weight optimization using Pyramid Squeeze Attention (PSA), the second feature map F3 (2016×1512×192) is obtained. F3 is then input into a 4×4 transposed convolution (stride 2, padding 1) to restore the resolution to 4032×3024, resulting in Fout. This Fout is then concatenated with RAW_in after a 1×1 convolution (output channels 3), resulting in a 4032×3024×3 feature map, to form a 4032×3024×195 feature map. Finally, a 3×3 convolutional layer (output channels 3) outputs RGB_out (4032×3024×3).

[0071] Ideally, an RGB reconstruction module is used, which includes an L2 loss function. Using a real RGB reference image as a baseline, the Adam optimizer (learning rate 1e-4, 50 iterations) is employed to minimize the loss function. The loss resulted in the PSNR of the final output RGB_out stabilizing at 43.2dB.

[0072] Then, the shared feature F2 (1008×756×128) and EVS_down (1008×756×1) are concatenated to form an initial fused feature of 1008×756×129, which is then passed through a 1×1 convolutional layer (64 output channels) to obtain the fused feature F_fuse_opt (1008×756×64).

[0073] Next, the target feature map F_res3 (1008×756×64) is output using three layers of residual units. Among them, the first residual unit outputs the feature map F_res1 (1008×756×64).

[0074] Then, the target feature map is classified and detected using the detection output module to obtain the detection results.

[0075] Ideally, the detection results are optimized using the VFL loss function, where the actual labeled data (detection box coordinates, category labels) is used as a benchmark, and γ=2 is set to calculate... The loss is minimized by using the SGD optimizer (learning rate 5e-5, 30 iterations) to make the detection box localization error ≤2 pixels.

[0076] Finally, non-maximum suppression (NMS) is applied to the detection results to remove redundant detection boxes with an overlap rate exceeding a preset overlap rate threshold (e.g., 0.3), thus outputting the target image data. Ultimately, the RGB image with overlaid detection boxes and category labels, along with the processed data, can be visualized on the screen.

[0077] The demosaicing and motion detection method based on a hybrid event camera provided in this embodiment avoids redundant feature extraction by sharing the lowest-level feature (shared feature) F2 of U-Net in the demosaicing and motion detection processes, reducing computational redundancy by approximately 40%. The end-to-end processing latency on the experimental processor is reduced to 68ms, meeting the real-time requirements of mobile devices. The pyramid squeezing attention unit increases the weight of dynamic target edge features by approximately 30%, reducing the color artifact rate of RGB_out images to approximately 3% (a 12% reduction compared to traditional U-Net), and achieving a PSNR (Peak Signal-to-Noise Ratio) of 43.2dB (a 1.5dB improvement compared to traditional L1 loss methods). Deep fusion of EVS event stream data and image features reduces the false negative rate of moving targets in low-light (10 lux) and backlight (10000 lux) conditions to 5% (a 10% reduction compared to existing solutions). VariFocal Loss improves the detection confidence of small targets by approximately 25%, and reduces the overlap rate of dense target boxes to 8% (a 10% reduction compared to existing Focal Loss methods). Loss is reduced by 12%; the number of parameters in the recursive residual network is only 8.6M (82.8% less than the existing Transformer's 50M), and FLOPs are reduced to 85G (57.5% less than the existing Transformer). It supports INT8 quantization deployment, and the memory usage on mobile devices is ≤10MB, meeting the hardware constraints of consumer-grade devices. It can adapt to multi-class detection with up to 10 target categories (such as people, cars, trucks, bicycles, etc.), and the detection box positioning error is ≤2 pixels. The vehicle counting accuracy in intelligent traffic monitoring scenarios reaches 98.5%, and the pedestrian detection accuracy in security scenarios reaches 97.8%.

[0078] The de-mosaicing and motion detection method based on a hybrid event camera provided in this embodiment is applicable to scenarios such as consumer electronic devices (smartphones, portable cameras), intelligent traffic monitoring, and security early warning. It can output high-quality RGB images while detecting moving people, vehicles, and other targets in the scene in real time, achieving integrated "imaging-detection" processing.

[0079] This embodiment also provides a demosaic and motion detection system based on a hybrid event camera, used to implement the demosaic and motion detection method based on a hybrid event camera as described above, such as... Figure 3 As shown, the demosaicing and motion detection system based on a hybrid event camera includes: An image data acquisition module is used to acquire raw image data, which includes RAW images and event stream data. The demosaic module is used to perform demosaic processing on RAW images based on pyramid squeeze attention to obtain RGB images and shared features; The motion detection module is used to perform motion detection on shared features and event stream data based on a lightweight recursive residual network to obtain detection results; The results output module is used to integrate the RGB image with the detection results to output the target image data.

[0080] The demosaicing and motion detection system based on a hybrid event camera provided in this embodiment effectively enhances the edge features of dynamic targets and improves image quality by using a demosaicing module with pyramid squeeze attention for demosaicing processing. By utilizing a motion detection module with a lightweight recursive residual network to perform motion detection on shared features and event stream data, the motion detection process can be achieved using the shared features provided by the demosaicing process. This not only improves feature utilization and avoids redundant feature extraction, thus improving system synergy, but also effectively increases detection confidence, adapts to various different scenarios, and achieves lightweight deployment. It solves the problems of poor synergy, low feature utilization, and insufficient lightweightness in existing hybrid event camera demosaicing and motion detection methods.

[0081] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. In addition, different parts between embodiments can also be combined with each other, and this invention does not limit this.

[0082] The method and system for demosaicing and motion detection based on a hybrid event camera provided in this embodiment include: acquiring raw image data, which includes RAW images and event stream data; performing demosaicing processing on the RAW images using a demosaicing module with pyramid squeeze attention to obtain RGB images and shared features; performing motion detection on the shared features and event stream data using a motion detection module with a lightweight recurrent residual network to obtain detection results; and integrating the RGB images with the detection results to output target image data. By utilizing a demosaic module with pyramid squeeze attention for demosaic processing, dynamic target edge features can be effectively enhanced, improving image quality. Furthermore, by employing a motion detection module with a lightweight recursive residual network to perform motion detection on shared features and event stream data, the motion detection process can leverage the shared features provided by the demosaic process. This not only improves feature utilization and avoids redundant feature extraction, enhancing system synergy, but also effectively increases detection confidence, adapting to various scenarios. Moreover, it achieves lightweight deployment, addressing the issues of poor synergy, low feature utilization, and insufficient lightweight design in existing demosaic and motion detection methods for hybrid event cameras.

[0083] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A method for desacrifice and motion detection based on a hybrid event camera, characterized in that, include: Acquire raw image data, which includes RAW images and event stream data; A demosaic module with pyramid compression attention is used to demosaic the RAW image to obtain the RGB image and shared features. A motion detection module with a lightweight recursive residual network is used to perform motion detection on shared features and event stream data to obtain detection results; The RGB image is integrated with the detection results to output the target image data.

2. The method for desacrifice and motion detection based on a hybrid event camera according to claim 1, characterized in that, After acquiring the raw image data, the demosaicing and motion detection method based on a hybrid event camera further includes: The RAW image is quantized to a preset number of bits to obtain the quantized RAW image; Event stream data is accumulated into event images according to time windows, and the event images are downsampled to obtain downsampled event images.

3. The method for desacrifice and motion detection based on a hybrid event camera according to claim 2, characterized in that, The time window is synchronized with the acquisition period of the RAW image.

4. The method for desacrifice and motion detection based on a hybrid event camera according to claim 2, characterized in that, The resolution of the event image is the same as the resolution of the RAW image.

5. The method for desacrifice and motion detection based on a hybrid event camera according to claim 1, characterized in that, The method for demosaicing RAW images using a demosaic module with pyramid compression attention to obtain RGB images and shared features includes: Construct a demosaicing module, including a U-Net architecture with pyramid squeezed attention; We utilize the U-Net architecture with pyramid squeeze attention to downsample RAW images in order to extract shared features; The shared features are upsampled using a U-Net architecture with pyramid squeeze attention to obtain RGB images; The first loss function is used to apply error constraints to the RGB image to obtain the enhanced RGB image.

6. The method for desacrifice and motion detection based on a hybrid event camera according to claim 5, characterized in that, The U-Net architecture with pyramid squeeze attention includes a first convolutional unit, a pyramid squeeze attention unit, a second convolutional unit, and a third convolutional unit; The first convolutional unit is used to downsample the input RAW image and input the downsampled feature map into the pyramid squeeze attention unit; The pyramid squeezing attention unit is used to perform pyramid squeezing on the feature map obtained by downsampling the RAW image through pooling layers of different scales to obtain the first multi-scale channel attention weight. It is also used to multiply the first multi-scale channel attention weight with the feature map obtained by downsampling the RAW image channel by channel to output the first feature map. The first convolutional unit is also used to downsample the first feature map and input the downsampled feature map into the pyramid squeeze attention unit; The pyramid squeezing attention unit is also used to perform pyramid squeezing on the feature map obtained by downsampling the first feature map through pooling layers of different scales to obtain the second multi-scale channel attention weights. It is also used to multiply the second multi-scale channel attention weights with the feature map obtained by downsampling the first feature map channel by channel to output shared features. The second convolutional unit is used to perform transpose convolution on the shared features and concatenate them with the first feature map to obtain the first concatenated features. It is also used to input the first concatenated features into the pyramid squeeze attention unit to obtain the second feature map. The second convolutional unit is also used to perform transpose convolution on the second feature map and to perform feature concatenation with the feature map obtained by downsampling the RAW image to obtain the second concatenated feature. It is also used to input the second concatenated feature into the pyramid squeeze attention unit to obtain the third feature map. The third convolutional unit is used to restore the resolution of the third feature map to be consistent with that of the RAW image, so as to obtain an RGB image.

7. The method for desacrifice and motion detection based on a hybrid event camera according to claim 5, characterized in that, The method of applying error constraints to the RGB image using a first loss function to obtain an enhanced RGB image includes: An error constraint is applied to the RGB image and the reference image using a first loss function, which is the L2 loss function, expressed as: in, The resolution of the RAW image is represented by ; i and j represent the horizontal and vertical coordinates of the pixels in the image, respectively; c represents the RGB channel index, where c=1 represents the R channel, c=2 represents the G channel, and c=3 represents the B channel. This represents the pixel value of the c channel at coordinate (i,j) in the RGB image obtained using the U-Net architecture with pyramid squeeze attention. This represents the pixel value of the reference image at coordinates (i,j) in channel c.

8. The method for desacrifice and motion detection based on a hybrid event camera according to claim 1, characterized in that, The method for performing motion detection on shared features and event stream data using a motion detection module with a lightweight recurrent residual network to obtain detection results includes: Shared features are concatenated with event stream data to obtain fused features; Construct a motion detection module, including a lightweight recursive residual network; A lightweight recursive residual network is used to perform recursive residual processing on the fused features to obtain the target feature map; Target classification and detection are performed on the target feature map to obtain the detection results; The detection results are optimized using a second loss function to obtain optimized detection results.

9. The method for desacrifice and motion detection based on a hybrid event camera according to claim 8, characterized in that, The lightweight recursive residual network includes a first residual unit, a second residual unit, and a third residual unit that are recursively connected. Each of the first residual unit, the second residual unit, and the third residual unit includes two 3×3 convolutional layers, a BatchNorm layer, and a ReLU activation function.

10. The method for desacrifice and motion detection based on a hybrid event camera according to claim 8, characterized in that, The method for concatenating shared features with event stream data to obtain fused features includes: The shared features are concatenated with the event stream data to obtain the initial fused features; By using a 1×1 convolutional layer, the number of channels in the initial fused features is reduced to obtain the fused features.

11. The method for desacrifice and motion detection based on a hybrid event camera according to claim 8, characterized in that, The method for classifying and detecting targets in the target feature map to obtain detection results includes: Global average pooling is performed on the target feature map to obtain a multi-dimensional feature vector, and a fully connected layer is used to classify the multi-dimensional feature vector to obtain the target classification result. Perform a 1×1 convolution on the target feature map to obtain the detection box coordinates for each target; Based on the target classification results and the coordinates of the detection boxes, the confidence score of each detection box is calculated using the sigmoid function; Based on the confidence level, determine whether the target classification detection is effective in order to obtain the detection result.

12. The method for desacrifice and motion detection based on a hybrid event camera according to claim 8, characterized in that, The second loss function is the VFL loss function, expressed as: Where N represents the number of detection boxes; This represents the prediction confidence of the k-th bounding box; Represents the true label, where the target exists. When the value is 1, the target does not exist. =0; This indicates the adjustment parameter.

13. The method for desacrifice and motion detection based on a hybrid event camera according to claim 1, characterized in that, The method for integrating RGB images with detection results to output target image data includes: Non-maximum suppression is applied to the detection results to output the target image data.

14. A demosaic and motion detection system based on a hybrid event camera, used to implement the demosaic and motion detection method based on a hybrid event camera as described in any one of claims 1 to 13, characterized in that, The demosaic and motion detection system based on a hybrid event camera includes: An image data acquisition module is used to acquire raw image data, which includes RAW images and event stream data. The demosaic module is used to perform demosaic processing on RAW images based on pyramid squeeze attention to obtain RGB images and shared features; The motion detection module is used to perform motion detection on shared features and event stream data based on a lightweight recursive residual network to obtain detection results; The results output module is used to integrate the RGB image with the detection results to output the target image data.

Images