Design and imaging method of fovea centralis bionic mixed pixel dynamic vision sensor and sensor

Abstract

The invention discloses a fovea centralis bionic mixed pixel dynamic visual sensor design and imaging method and a sensor, and the method comprises the steps: designing a bionic central mixed pixel array, employing non-uniform Gaussian distribution to arrange RGB pixels, setting high-density RGB pixels in the central region of the sensor, gradually carrying out downsampling towards the periphery, and integrating high-dynamic event pixels. Simulating a human eye retina cone and rod cell division mechanism, and outputting central RGB data and a full-pixel event stream; through event flow optical flow estimation, optical flow prediction and center RGB dynamic compensation are combined, center semantics are diffused to the periphery, and a full-pixel fusion RGB image is recovered; a multi-modal data processing framework is constructed, semantic enhancement of color, shape, feature point and target detection is realized, and low power consumption and high-precision perception are both considered. According to the method, the core problem that detail and semantic information of a traditional event camera is deficient can be effectively solved, and the effects of several downstream tasks such as color and shape detection, feature point detection and target detection are remarkably improved.

Description

Technical Field

[0001] This invention relates to computer vision and intelligent sensor technology, and in particular to a concave biomimetic hybrid pixel dynamic vision sensor design, imaging method, and sensor. Background Technology

[0002] Event cameras, as a new generation of bio-inspired visual sensors, have shown great potential in the field of high-speed motion scene perception due to their high dynamic range (>120dB), microsecond-level latency, and low power consumption. Currently, event cameras have been successfully applied to dynamic environment perception scenarios such as autonomous driving, drone navigation, and robot vision. However, the working principle of traditional event cameras dictates that they can only generate asynchronous event streams in response to changes in brightness, resulting in two key deficiencies in their perception capabilities: first, the inability to directly acquire key visual features such as scene details, texture, and color leads to a severe lack of semantic information; second, in static scenes, the lack of brightness changes causes them to stop outputting events, making it difficult to continuously update the limited semantic information. This inherent deficiency in semantic acquisition limits the performance of traditional event cameras when facing complex applications requiring continuous and complete scene understanding, especially in tasks that simultaneously process dynamic targets and static scenes. The scarcity and unsustainability of semantic information have become major bottlenecks restricting their practical applications.

[0003] In terms of technologies to improve the semantic acquisition capabilities of event cameras, existing research mainly explores two directions: active light source control and optical hardware optimization. Active light source control utilizes controllable light sources (structured light, scanning lasers, strobe LEDs, etc.) to generate data in the spatiotemporal domain that is more likely to trigger events. For example, by using spatiotemporally encoded laser dot arrays to embed 3D geometric semantics into single-frame event data, it overcomes the limitation of traditional event cameras that can only acquire 2D motion information, achieving real-time 3D semantic reconstruction at 500Hz. Methods have developed spatiotemporal synchronization mechanisms between striped structured light and event cameras, transforming optically encoded material reflection characteristics into stable features in the event stream, solving the problem of semantic loss on static object surfaces in traditional event cameras. Other methods propose a multi-source frequency division multiplexing architecture, which uses orthogonal optical coding to enable multiple active illumination sources to form distinguishable semantic channels in event data, supporting multi-target semantic parsing in complex scenes. Optical hardware optimization directly improves the imaging chain (e.g., designing microlens arrays, stacked pixels, polarization / filters, etc.) to enhance the spatial continuity, temporal stability, and semantic integrity of the event stream. For example, by adding a rotating wedge prism to simulate the micro-scanning motion of the human eye, the direction of incident light is actively changed to continuously trigger the event stream. Combined with motion compensation algorithms, stable texture perception and high information output in static scenes are achieved. Furthermore, existing technologies have developed event-frame hybrid sensors such as the DVS346, which significantly improve the ability to capture texture features in static scenes and dynamic targets by simultaneously outputting event streams and traditional image frames.

[0004] While existing technologies have enhanced the semantic acquisition capabilities of event cameras through active light source modulation and optical hardware optimization, these methods still have significant limitations. Active light source modulation techniques, although able to improve event data quality through structured light and laser coding, are heavily dependent on ambient lighting conditions, showing a significant decline in effectiveness under strong ambient light interference, thus severely limiting their application scenarios. While optimizing event triggering mechanisms through optical hardware design can improve the signal-to-noise ratio and stability of event streams, it fundamentally fails to address the core problem of event cameras lacking semantic information, particularly the inability to acquire key semantic features such as the category and color of scene objects. Furthermore, existing event+frame fusion sensors (such as the DVS346), while attempting to combine the advantages of both modalities, introduce a large amount of redundant data due to their full-resolution parallel acquisition architecture, leading to decreased algorithm processing efficiency and significantly increasing system power consumption and bandwidth requirements. This makes these hybrid sensors less energy-efficient than solutions using event cameras or traditional cameras alone. These technological limitations mean that current event cameras still face significant challenges in complex application scenarios requiring continuous and complete semantic understanding.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a concave biomimetic hybrid pixel dynamic vision sensor design, imaging method, and sensor. Without significantly increasing power consumption and bandwidth, the dynamic vision sensor can continuously and stably acquire high-quality semantic information containing color, texture, and category information to support complex downstream vision tasks.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A design and imaging method for a concave biomimetic hybrid pixel dynamic vision sensor includes the following steps: S1. Bionic central hybrid pixel array design: It adopts a non-uniform Gaussian distribution of RGB pixels, arranges high-density RGB pixels in the central area of ​​the sensor, and gradually reduces the sampling rate towards the periphery. At the same time, it integrates high dynamic event pixels to simulate the division of labor mechanism between cone cells and rod cells in the human eye retina, and outputs central RGB data and full pixel event stream. S2. Dynamic semantic compensation based on event optical flow: The scene optical flow is estimated by using the high-frequency motion information of the event flow, and the semantic information of the central region is adaptively diffused to the outer event perception region through two modules: optical flow prediction and central RGB dynamic compensation, so as to restore the fused RGB image of the full pixel range. S3. Multimodal data processing and fusion for downstream tasks: Construct a standard algorithm framework that includes raw data preprocessing, sparse optical flow estimation, task-oriented feature extraction, and multimodal feature fusion to achieve semantic enhancement for color and shape detection, feature point detection, and object detection tasks.

[0008] Furthermore, the non-uniform Gaussian distribution of RGB pixel arrangement described in step S1 specifically includes: By controlling the sampling density using a Gaussian probability distribution function, the sampling rate is highest in the central area and decreases exponentially towards the periphery, thereby reducing the overall RGB data volume while ensuring color accuracy in key areas.

[0009] Furthermore, the optical flow prediction module in step S2 specifically includes: The continuous event stream is converted into a voxel grid representation and divided into the current event sequence and adjacent event sequences according to the time window; A dual-branch feature encoding network is used for processing. The first branch uses a CNN feature encoder with shared weights to calculate the spatial correlation of feature maps of adjacent event sequences and construct the event temporal correlation feature body. The second branch uses a context encoder with independent weights to extract the semantic context features of the current frame. Iterative optical flow optimization is performed using a GRU recurrent network, and initialization is performed by bilinear interpolation based on the optical flow prediction results of the previous frame to generate a pixel-level dense optical flow field.

[0010] Furthermore, the central RGB dynamic compensation module mentioned in step S2 specifically includes: Based on the current optical flow prediction results, the previous frame compensation image is subjected to reverse deformation. The deformation is achieved by offsetting the pixel position by optical flow vector and using bilinear interpolation to ensure accuracy. The semantic information of the central region is dynamically diffused by using the precise pixel values ​​of the current frame in the sparsely sampled central region and the deformation results of the previous frame image in the non-central region. The fused image is forward-deformed using the predicted optical flow field, and the compensation result of the current frame is updated progressively to recover the fused RGB image with a full pixel range.

[0011] Furthermore, the multimodal data processing and fusion framework described in step S3 specifically includes the following for the target detection task: An improved two-dimensional time surface model is used to construct a spatiotemporal feature representation of event data, and the temporal characteristics of the event stream are preserved by introducing a bipolar time decay function; The optical flow-compensated RGB images and event data were processed using object detection networks and spiking neural networks, respectively, and their respective modal features were extracted. A bidirectional LSTM timing coding module is used to extract and refine the temporal attributes of events; We use a cross-modal attention mechanism and an asymmetric weighting strategy to perform weighted fusion of RGB features and event features. The fused features are then input into the detection head to obtain the prediction results.

[0012] Furthermore, the two-dimensional time surface model is constructed as follows: A temporal surface tensor containing positive and negative event polarity channels is constructed, and the pixel values ​​are calculated using an exponential decay function to preserve the temporal characteristics of the event stream.

[0013] Furthermore, the asymmetric weight allocation strategy is specifically as follows: The contribution weights of RGB features and event features in the fusion process are adaptively adjusted based on scene motion characteristics and semantic complexity to achieve weighted fusion of RGB features and event features.

[0014] Furthermore, the method also includes: Through the collaborative design of the biomimetic central hybrid pixel array and the event optical flow compensation algorithm, semantic integrity enhancement is achieved across all scenarios while maintaining low power consumption, making it suitable for autonomous driving, drone navigation, or mobile robot visual perception tasks.

[0015] Furthermore, the method reduces the amount of RGB data and uses event streams to guide semantic diffusion, thereby reducing system power consumption and bandwidth requirements while maintaining the same semantic accuracy, and improving the deployment feasibility of edge devices.

[0016] A concave biomimetic hybrid pixel dynamic vision sensor, comprising: A biomimetic central hybrid pixel array is used to sense light signals and output data. The array adopts a non-uniform Gaussian distribution of RGB pixels. High-density RGB pixels are arranged in the central area of ​​the sensor, and the sampling rate is gradually reduced towards the periphery. At the same time, high dynamic event pixels are integrated to simulate the division of labor mechanism between cone cells and rod cells in the human eye retina, and output central RGB data and full pixel event stream. The processing circuit is configured to execute: Dynamic semantic compensation based on event optical flow: The scene optical flow is estimated by using the high-frequency motion information of the event flow, and the semantic information of the central region is adaptively diffused to the peripheral event perception region through optical flow prediction and central RGB dynamic compensation, so as to restore the fused RGB image of the full pixel range. Multimodal data processing and fusion for downstream tasks: Construct a processing flow that includes raw data preprocessing, sparse optical flow estimation, task-oriented feature extraction, and multimodal feature fusion to achieve semantic enhancement for color and shape detection, feature point detection, and object detection tasks.

[0017] The present invention has the following beneficial effects: This invention proposes a design and imaging method for a central foveal bionic hybrid pixel dynamic vision sensor. Essentially, it is a novel bionic event camera design method based on semantic enhancement. Inspired by the rod-cone cell dual-pathway photosensitive mechanism, this method combines hardware-level hybrid pixel array design with algorithm-level collaborative optimization to achieve high-precision, low-power semantic perception in driving scenarios.

[0018] In terms of technical performance improvement, this method, while maintaining the low-power characteristics of traditional event cameras, introduces biomimetic foveal RGB pixels and an event-guided optical flow compensation algorithm. This enables the camera to simultaneously acquire high-speed motion information and stable, rich semantic features in complex dynamic scenes, effectively solving the core problem of insufficient detail and semantic information in traditional event cameras. This significantly improves the performance of downstream tasks such as color and shape detection, feature point detection, and object detection. Simultaneously, it successfully addresses the inherent shortcomings of traditional event cameras in semantic perception, achieving semantic understanding of complex scenes while retaining the inherent advantages of low power consumption and high dynamic range. Specifically, this method, by simulating the dual-pathway mechanism of rod-cone cells in the human eye, employs a biomimetic foveal RGB pixel array and an event-guided optical flow compensation algorithm. This overcomes the limitations of existing event cameras in terms of insufficient and unstable semantic information, innovatively achieving the organic fusion of high-speed event flow and central RGB. It provides edge computing devices with a visual solution that combines low power consumption and rich semantic perception capabilities, which is of great significance for promoting the practical application of event cameras in fields such as autonomous driving and robot navigation. In terms of application scenarios, this invention can be widely used in fields such as dynamic scene perception and edge computing vision systems, and is especially suitable for real-time visual processing in UAV visual navigation, mobile robot environmental perception and intelligent monitoring systems.

[0019] Key innovations of this invention include: the construction of a hardware-software co-design framework. This framework, by simulating the division of labor between rod and cone cells in the human eye, fully utilizes the low-power characteristics of event data and the rich semantic information of RGB data. At the hardware level, it integrates high-dynamic event pixels with a centrally enhanced RGB array. At the algorithm level, it develops a dual-modal data dynamic fusion compensation algorithm, ultimately achieving high-precision, low-power visual perception in downstream tasks such as color and shape detection, feature point detection, and target detection. Specifically, at the hardware level, this invention proposes a non-uniform Gaussian distribution of RGB pixels. This method uses high-density RGB sampling in the central region of the sensor (bionic concave area) and gradually reduces the sampling rate towards the periphery. This ensures color accuracy in key areas (such as the target detection ROI) while effectively reducing the overall RGB data volume, thereby reducing transmission and computational power consumption. At the algorithm optimization level, this invention proposes a dynamic semantic compensation algorithm based on event optical flow. This algorithm estimates scene optical flow by utilizing the high-frequency motion information of the event flow, adaptively diffusing the semantic features of the central RGB region to the peripheral event perception region. While maintaining the low-power characteristics of event data, this invention effectively solves the problem of incomplete semantic information caused by sparse RGB sampling, achieving stable and persistent semantic enhancement across all scenarios. In addition, this invention proposes a standard data processing-algorithm design framework for downstream tasks of biomimetic event cameras. This framework constructs a complete benchmark algorithm flow for visual tasks such as color and shape detection, feature point detection, and object detection. The flow includes core modules such as raw data preprocessing, sparse optical flow estimation, task-oriented feature extraction, and multimodal feature adaptive fusion, providing standardized technical support for the efficient and accurate execution of downstream visual tasks.

[0020] The experimental results demonstrate the following significant effects of the present invention: (1) Breaking through the dependence of traditional active light source technology on lighting conditions, through the biomimetic hybrid pixel array design, stable semantic acquisition under high dynamic range lighting environment is achieved without the need for external auxiliary light source, thus solving the performance degradation problem under strong light interference.

[0021] (2) By adopting a dynamic compensation algorithm with central concave RGB arrangement and event guidance, while maintaining the low power consumption characteristics of the event camera, color and detail acquisition and continuous semantic updates are achieved, overcoming the shortcomings of traditional optical optimization methods that cannot acquire key semantics such as color and category.

[0022] (3) Through the collaborative design of non-uniform sampling and dynamic compensation, the amount of RGB data is reduced by more than 85% compared with full-resolution hybrid sensors such as DVS346. Under the same semantic accuracy, the system power consumption and bandwidth requirements are significantly reduced, and the deployment feasibility of edge devices is significantly improved.

[0023] (4) By designing a feature enhancement mechanism based on time-series modeling and a feature fusion strategy based on asymmetric weights, the detection accuracy in the target detection task is comparable to that of a full-resolution hybrid sensor, while significantly reducing system resource consumption.

[0024] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of an embodiment of the present invention.

[0026] Figure 2 This is a system overall structure diagram according to an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of a biomimetic central concave hybrid pixel array according to an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the dynamic semantic compensation algorithm based on event optical flow according to an embodiment of the present invention.

[0029] Figure 5 This is a data processing-algorithm design framework diagram for target detection tasks according to an embodiment of the present invention. Detailed Implementation

[0030] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0031] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0032] The present invention aims to propose a design and imaging method for a concave biomimetic hybrid pixel dynamic visual sensor. By designing a novel biomimetic event camera based on semantic enhancement, while maintaining the low power consumption characteristics of traditional event cameras, the method introduces RGB pixels with a biomimetic concave center and an event-guided optical flow compensation algorithm. This enables the camera to simultaneously acquire high-speed motion information and stable and rich semantic features in complex dynamic scenes, solving the core problem of the lack of detail and semantic information in traditional event cameras. This significantly improves the performance of several downstream tasks such as color and shape detection, feature point detection, and target detection.

[0033] See Figure 1This invention provides a design and imaging method for a centrally concave biomimetic hybrid pixel dynamic visual sensor, comprising the following steps: Step S1, Bionic Central Hybrid Pixel Array Design: A non-uniform Gaussian distribution of RGB pixels is adopted. High-density RGB pixels are arranged in the central area of ​​the sensor, and the sampling rate is gradually reduced towards the periphery. At the same time, high dynamic event pixels are integrated to simulate the division of labor mechanism between cone cells and rod cells in the human eye retina, and output the central RGB data and the full pixel event stream.

[0034] In some embodiments, the non-uniform Gaussian distribution of RGB pixels in step S1 specifically includes: controlling the sampling density through a Gaussian probability distribution function so that the sampling rate is highest in the central region and decreases exponentially towards the periphery, thereby reducing the overall RGB data volume while ensuring the color accuracy of key areas.

[0035] Step S2: Dynamic semantic compensation based on event optical flow: The scene optical flow is estimated by using the high-frequency motion information of the event flow, and the semantic information of the central region is adaptively diffused to the outer event perception region through two modules: optical flow prediction and central RGB dynamic compensation, so as to restore the fused RGB image of the full pixel range.

[0036] In some embodiments, the optical flow prediction module in step S2 specifically includes: converting the continuous event flow into a voxel grid representation, and dividing it into the current event sequence and adjacent event sequences according to a time window (the length of the time window can be adjusted according to the scene motion speed or computational overhead requirements); processing it using a dual-branch feature encoding network, wherein the first branch uses a CNN feature encoder with shared weights to calculate the spatial correlation of feature maps of adjacent event sequences and construct the event temporal correlation feature body, and the second branch uses a context encoder with independent weights to extract the semantic context features of the current frame; performing iterative optical flow optimization using a GRU recurrent network, and initializing it with bilinear interpolation in combination with the optical flow prediction results of the previous frame to generate a pixel-level dense optical flow field.

[0037] In some embodiments, the central RGB dynamic compensation module in step S2 specifically includes: performing reverse deformation on the previous frame compensation image based on the current optical flow prediction result, wherein the deformation is achieved by offsetting pixel positions using optical flow vectors and employing bilinear interpolation to ensure accuracy; using the precise pixel values ​​of the current frame in the central sparse sampling region, and using the deformation result of the previous frame image in the non-central region to dynamically diffuse the semantic information of the central region; using the predicted optical flow field to perform forward deformation on the fused image, progressively updating the current frame compensation result, thereby recovering the fused RGB image with a full pixel range. The progressive update can be triggered by the arrival of new RGB data or changes in the optical flow field to ensure the timeliness of the compensation result.

[0038] Step S3, Multimodal data processing and fusion for downstream tasks: Construct a standard algorithm framework that includes raw data preprocessing, sparse optical flow estimation, task-oriented feature extraction and multimodal feature fusion to achieve semantic enhancement for color and shape detection, feature point detection and object detection tasks.

[0039] In some embodiments, the multimodal data processing and fusion framework described in step S3 specifically includes the following for the target detection task: constructing a spatiotemporal feature representation of event data using an improved two-dimensional time surface model, and preserving the temporal characteristics of the event stream by introducing a bipolar time decay function; processing the optically flow-compensated RGB image and event data using a target detection network and a spiking neural network respectively, and extracting their respective modal features; extracting and refining the temporal attributes of the event using a bidirectional LSTM temporal coding module; and using a cross-modal attention mechanism and an asymmetric weight allocation strategy to perform weighted fusion of RGB features and event features, and then inputting the fused features into the detection head to obtain the prediction result.

[0040] In some embodiments, the two-dimensional time surface model is constructed by: establishing a time surface tensor containing positive and negative event polarity channels, and calculating each pixel value through an exponential decay function to preserve the temporal characteristics of the event stream.

[0041] In some embodiments, the asymmetric weight allocation strategy specifically involves adaptively adjusting the contribution weights of RGB features and event features during the fusion process based on scene motion characteristics and semantic complexity, thereby achieving weighted fusion of RGB features and event features. The weights can be determined by quantifying scene motion characteristics and semantic complexity, for example, by calculating based on indicators such as event occurrence rate or feature entropy value.

[0042] In some embodiments, the method further includes: through the collaborative design of the biomimetic central hybrid pixel array and the event optical flow compensation algorithm, semantic integrity enhancement is achieved across all scenarios while maintaining low power consumption characteristics, making it suitable for tasks such as autonomous driving, drone navigation, and mobile robot visual perception.

[0043] In some embodiments, the method significantly reduces system power consumption and bandwidth requirements while maintaining the same semantic accuracy by reducing the amount of RGB data and using event streams to guide semantic diffusion, thereby improving the deployment feasibility of edge devices.

[0044] This invention also provides a concave biomimetic hybrid pixel dynamic vision sensor, including a biomimetic central hybrid pixel array and a processing circuit. The biomimetic central hybrid pixel array is used to sense light signals and output data. The array adopts a non-uniform Gaussian distribution of RGB pixels, with high-density RGB pixels arranged in the central region of the sensor, gradually decreasing the sampling rate towards the periphery. It also integrates high-dynamic event pixels to simulate the division of labor between cone cells and rod cells in the human retina, outputting central RGB data and a full-pixel event stream. The processing circuit is configured to perform: dynamic semantic compensation based on event optical flow: estimating scene optical flow using the high-frequency motion information of the event stream, and adaptively diffusing the semantic information of the central region to the peripheral event sensing region through optical flow prediction and central RGB dynamic compensation to recover a fused RGB image across the entire pixel range; multimodal data processing and fusion for downstream tasks: constructing a processing flow including raw data preprocessing, sparse optical flow estimation, task-oriented feature extraction, and multimodal feature fusion to achieve semantic enhancement for color and shape detection, feature point detection, and target detection tasks.

[0045] This invention proposes a foveal-inspired bionic hybrid pixel dynamic vision sensor design and imaging method. Through a bionic central hybrid pixel array design, it employs a non-uniform Gaussian RGB distribution (high density in the central region, gradually downsampling towards the periphery) and integrates high-dynamic event pixels. This simulates the division of labor between human eye cones (mainly concentrated in the fovea, responsible for fine vision and color perception) and rod cells (mostly distributed in the periphery, responsible for motion perception and low-light adaptation). This reduces the amount of RGB data while ensuring the accuracy of information in key areas. Through dynamic semantic compensation based on event optical flow, it estimates optical flow using high-frequency motion information from the event flow, adaptively diffusing the central semantics to the periphery, restoring a fused RGB image across the entire pixel range, thus solving the problem of incomplete semantic information. Through a multimodal data processing and fusion framework, it achieves semantic enhancement for color and shape detection, feature point detection, and target detection tasks. Ultimately, while maintaining low power consumption, the sensor combines high-speed motion information acquisition with stable and rich semantic features, improving visual perception. This method effectively solves the problem of insufficient detail and semantic information in traditional event cameras, thereby significantly improving the performance of downstream visual tasks such as color and shape detection, feature point detection, and object detection.

[0046] The following further describes specific embodiments of the present invention and examples of its algorithms.

[0047] like Figure 2As shown, a design and imaging method for a centrally foveal biomimetic hybrid pixel dynamic visual sensor realizes the design of a biomimetic visual event camera. This includes three core modules: a biomimetic centrally hybrid pixel array arrangement design, a dynamic semantic compensation algorithm based on event optical flow, and a standard data processing-algorithm design framework for downstream tasks. The system input is ambient light signals, and the output is RGB data fused with highly dynamic event flow and rich semantic information. This system simulates the biological mechanism of multi-layered neural cells in the human retina, including rod cells, cone cells, bipolar cells, and retinal ganglion cells, collaboratively processing light signals. Through hardware and software co-design, it achieves human-eye-like intelligent visual perception.

[0048] The biomimetic concave hybrid pixel array is the foundational hardware module of the system. The RGB pixels are arranged using a non-uniform Gaussian distribution, with high-density RGB pixels positioned in the central region of the sensor. The sampling rate is gradually reduced outwards using an exponential decay model. This design simulates the distribution characteristics of cone cells in the human retina, significantly reducing power consumption while ensuring sufficient information in critical areas. This module's hybrid pixel array simultaneously outputs central RGB data and full-pixel high-resolution events, overcoming the limitations of traditional event cameras that suffer from insufficient and unstable semantic information.

[0049] The event-based optical flow dynamic semantic compensation algorithm further achieves the fusion and enhancement of event and RGB information. This algorithm comprises two modules: Event optical flow prediction and center RGB dynamic compensation. The optical flow prediction module models the event flow temporally, extracts spatiotemporal features using a CNN, and iteratively optimizes it using a GRU recurrent network. It combines historical optical flow information to achieve temporal consistency, generating high-precision dense optical flow field prediction results. The center RGB dynamic compensation module dynamically diffuses the semantic information of the center RGB region to the surrounding event-aware region. This module first aligns and fuses the current frame with the previous frame's optical flow deformation result through motion compensation, and then uses an iterative update mechanism to deform the fused image along the optical flow trajectory. This algorithm guides the dynamic diffusion of RGB information through event data, effectively overcoming the spatial discontinuity of semantic information caused by low-sampling-rate RGB arrays. While maintaining the system's low-power characteristics, it significantly improves the semantic integrity of the entire scene.

[0050] Building upon this foundation, this invention constructs a standardized data processing and algorithm framework for three visual tasks: color recognition, feature point detection, and object detection. This framework includes core modules such as raw data preprocessing, sparse optical flow estimation, task-oriented feature extraction, and multimodal feature fusion. Experimental results demonstrate that, compared to traditional event cameras, the novel biomimetic event camera design method proposed in this invention significantly improves the perceived detail and accuracy of downstream visual tasks while maintaining low power consumption.

[0051] Figure 3An example of a biomimetic concave hybrid pixel array is demonstrated. The RGB pixels are arranged in a non-uniform Gaussian distribution, constructing a high-density RGB sampling point in the central region of the image. A Gaussian probability distribution function controls the sampling density, and an exponential decay model ensures that the sampling density decreases non-linearly from the center to the periphery. The sampling probability formula is:

[0052] in This represents the Euclidean distance from a pixel to the center of the image. The half length of the image diagonal. Controlling the Gaussian distribution spread width, Adjusting the gradient rate of change and edge decay characteristics, and Define the maximum sampling rate for the central region and the minimum sampling rate for the edge region. The Gaussian arrangement parameters can be determined by combining sensor size, scene attention distribution, or bandwidth requirements to adapt to different semantic accuracy and power consumption balance needs.

[0053] Figure 4 A schematic diagram of a dynamic semantic compensation algorithm based on event optical flow is shown. The optical flow prediction module first converts the continuous event flow into a voxel mesh representation and divides it into the current event sequence according to a time window. and adjacent sequences First, the temporal continuity of the original events is preserved. Second, a dual-branch feature encoding network is constructed using the feature correlation idea and sequence processing mechanism in traditional image optical flow estimation: 1) A CNN feature encoder with shared weights is used to calculate... and The spatial correlation of the sequence feature maps is analyzed, and an event temporal correlation feature body is constructed. 2) A context encoder with independent weights is used to process only... The sequence is processed to extract semantic context features of the current frame. Finally, an iterative optical flow optimization is performed using a GRU recurrent network, while the predicted optical flow of the previous frame is forward-deformed through bilinear interpolation to initialize the current estimate, thereby extracting the pixel-level dense optical flow field of the event camera. The center RGB dynamic compensation module is based on the optical flow prediction results of the current frame. First, the compensated image from the previous frame... Perform reverse deformation, the deformation formula is: ,in For the optical flow vector, bilinear interpolation is used to ensure deformation accuracy. Secondly, the central sparse sampling region (ROI) uses the precise pixel values ​​of the current frame, while the non-ROI regions use the deformation results from the previous frame to dynamically diffuse the semantic information of the central region, thereby recovering a fused RGB image covering the entire pixel range. Finally, the predicted optical flow field is used to perform forward deformation on the fused image, progressively updating the compensation results for the current frame.

[0054] Figure 5 This paper presents a data processing and algorithm design framework for object detection tasks. In object detection, this invention designs a feature enhancement module based on a bidirectional long short-term memory (LSTM) network and an asymmetric weight allocation fusion module. This module leverages the temporal dependencies of events to enhance the feature representation capabilities of the central RGB region and adaptively adjusts the contribution weights of RGB and event features in the fusion process based on scene motion characteristics and semantic complexity. Firstly, in terms of event data representation, an improved two-dimensional time surface model is adopted, constructing a spatiotemporal feature representation by introducing a bipolar time decay function. Specifically, a time surface is defined... The third dimension contains positive and negative event polarity channels, and each pixel value is decayed by an exponential decay function. calculate, The use of time decay weights effectively preserves the temporal characteristics of the event stream. Secondly, YOLOX and a spiking neural network are employed to process the optically flow-compensated RGB images and event data, respectively obtaining features under the corresponding modalities. Subsequently, a bidirectional LSTM timing coding module was used to extract and refine the temporal attributes of the events, resulting in... Then, a cross-modal attention mechanism and an asymmetric weight allocation fusion strategy are used to perform weighted fusion of the features to obtain the fused features. Finally, the prediction result is obtained through the detection head. This scheme significantly improves the target detection accuracy in complex dynamic scenes while maintaining a 10% central RGB sampling rate.

[0055] The experimental results demonstrate the following significant effects of the present invention: (1) Breaking through the dependence of traditional active light source technology on lighting conditions, through the biomimetic hybrid pixel array design, stable semantic acquisition under high dynamic range lighting environment is achieved without the need for external auxiliary light source, thus solving the performance degradation problem under strong light interference.

[0056] (2) By adopting a dynamic compensation algorithm with central concave RGB arrangement and event guidance, while maintaining the low power consumption characteristics of the event camera, color and detail acquisition and continuous semantic updates are achieved, overcoming the shortcomings of traditional optical optimization methods that cannot acquire key semantics such as color and category.

[0057] (3) Through the collaborative design of non-uniform sampling and dynamic compensation, the amount of RGB data is reduced by more than 85% compared with full-resolution hybrid sensors such as DVS346. Under the same semantic accuracy, the system power consumption and bandwidth requirements are significantly reduced, and the deployment feasibility of edge devices is significantly improved.

[0058] (4) By designing a feature enhancement mechanism based on time-series modeling and a feature fusion strategy based on asymmetric weights, the detection accuracy in the target detection task is comparable to that of a full-resolution hybrid sensor, while significantly reducing system resource consumption.

[0059] In summary, this invention proposes a design and imaging method for a central foveal biomimetic hybrid pixel dynamic visual sensor, which is a novel biomimetic event camera design method based on semantic enhancement. Inspired by the rod-cone cell dual-pathway photosensitive mechanism, this method combines hardware-level hybrid pixel array design with algorithm-level collaborative optimization to achieve high-precision, low-power semantic perception in driving scenarios.

[0060] The key innovative contributions and technical advantages of this invention include: This invention proposes a novel biomimetic event camera design method based on semantic enhancement, comprising a hardware and software co-design framework. By simulating the division of labor mechanism between rod cells and cone cells in the human eye, and fully utilizing the low-power characteristics of event data and the rich semantic information of RGB data, a high-dynamic event pixel and a center-enhanced RGB array are integrated at the hardware level. At the algorithm level, a dual-modal data dynamic fusion compensation algorithm is developed, achieving high-precision and low-power visual perception in downstream tasks such as color and shape detection, feature point detection, and object detection.

[0061] This invention proposes a non-uniform Gaussian distribution of RGB pixels, employing high-density RGB sampling in the central region of the sensor (bionic concave area) and gradually decreasing the sampling rate towards the periphery. This design ensures color accuracy in critical areas (such as target detection ROIs) while reducing the overall RGB data volume and lowering transmission and computational power consumption.

[0062] This invention proposes a dynamic semantic compensation algorithm based on event optical flow. By utilizing the high-frequency motion information of the event flow to estimate the scene optical flow, the semantic features of the central RGB region are adaptively diffused to the peripheral event perception region. While maintaining the low-power characteristics of event data, this algorithm effectively solves the problem of incomplete semantic information caused by sparse RGB sampling, achieving stable and persistent semantic enhancement across the entire scene.

[0063] This invention proposes a standard data processing-algorithm design framework for downstream tasks of biomimetic event cameras. It constructs a complete benchmark algorithm flow for visual tasks such as color and shape detection, feature point detection, and object detection. The flow includes core modules such as raw data preprocessing for this scheme, sparse optical flow estimation, task-oriented feature extraction, and multimodal feature adaptive fusion.

[0064] In summary, the method proposed in this invention addresses the inherent limitations of traditional event cameras in semantic perception through a novel biomimetic event camera design based on semantic enhancement. This method achieves semantic understanding of complex scenes while maintaining the inherent advantages of low power consumption and high dynamic range of event cameras. By simulating the dual-pathway mechanism of rod-cone cells in the human eye, employing a biomimetic foveal-arranged RGB pixel array and an event-guided optical flow compensation algorithm, it overcomes the limitations of existing event cameras in terms of scarce and unstable semantic information. This invention innovatively achieves the organic integration of high-speed event streams and central RGB, providing edge computing devices with a visual solution that combines low power consumption and rich semantic perception capabilities. This is of great significance for promoting the practical application of event cameras in fields such as autonomous driving and robot navigation. This invention can be widely used in dynamic scene perception and edge computing vision systems, and is particularly suitable for real-time visual processing in UAV visual navigation, mobile robot environmental perception, and intelligent monitoring systems.

[0065] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0066] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0067] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0068] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0069] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0070] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0071] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0072] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0074] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0075] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0076] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0077] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A foveated biomimetic hybrid-pixel dynamic vision sensor design and imaging method, characterized by, Comprise the following steps: S1, bionic center mixed pixel array design: adopt the RGB pixel arrangement mode of non-uniform Gaussian distribution, arrange high-density RGB pixels in the center area of the sensor, gradually reduce the sampling rate outward, and integrate high dynamic event pixels to simulate the division mechanism of cone cells and rod cells in the human eye retina, output the center RGB data and full-pixel event stream; S2, dynamic semantic compensation based on event optical flow: estimate the scene optical flow using the high-frequency motion information of the event stream, and through the optical flow prediction and center RGB dynamic compensation modules, adaptively diffuse the semantic information of the center area to the peripheral event sensing area to restore the fused RGB image in the full-pixel range; S3, multi-modal data processing and fusion for downstream tasks: Construct a standard algorithm framework including raw data preprocessing, sparse optical flow estimation, task-oriented feature extraction and multi-modal feature fusion to realize semantic enhancement for color and shape detection, feature point detection and target detection tasks.

2. The foveated, biomimetic, hybrid-pixel, dynamic-vision-sensor design and imaging method of claim 1, wherein, The RGB pixel arrangement mode of non-uniform Gaussian distribution in step S1 specifically comprises: Control the sampling density through the Gaussian probability distribution function to make the center area have the highest sampling rate, and the sampling rate decays exponentially outward, so as to reduce the overall RGB data amount while ensuring the color accuracy of the key area.

3. The foveated, biomimetic, hybrid-pixel, dynamic-vision-sensor design and imaging method of claim 1 or 2, wherein, The optical flow prediction module in step S2 specifically comprises: Convert the continuous event stream into a voxel grid representation, and divide it into a current event sequence and an adjacent event sequence according to a time window; Use a double-branch feature coding network for processing, wherein the first branch uses a CNN feature encoder with shared weights to calculate the feature map spatial correlation of the adjacent event sequence and construct an event time sequence correlation feature body, and the second branch uses an independent weight context encoder to extract the current frame semantic context feature; Use a GRU recurrent network for iterative optical flow optimization, and combine the previous frame optical flow prediction result to initialize through bilinear interpolation to generate a pixel-level dense optical flow field.

4. The foveated, biomimetic, hybrid-pixel, dynamic-vision-sensor design and imaging method of claim 3, wherein, The center RGB dynamic compensation module in step S2 specifically comprises: Based on the current optical flow prediction result, deform the previous frame compensation image in reverse, and the deformation offsets the pixel position through the optical flow vector and uses bilinear interpolation to ensure accuracy; Use the accurate pixel value of the current frame in the center sparse sampling area, and use the deformation result of the previous frame image in the non-center area to dynamically diffuse the semantic information of the center area; Use the predicted optical flow field to deform the fused image forward, progressively update the current frame compensation result, and thus restore the fused RGB image in the full-pixel range.

5. The foveated, biomimetic, hybrid-pixel, dynamic-vision-sensor design and imaging method of claim 1, wherein, The multi-modal data processing and fusion framework in step S3 specifically comprises for the target detection task: Use an improved two-dimensional time surface model to construct the spatio-temporal feature representation of event data, and retain the time sequence characteristics of the event stream by introducing a bipolar time decay function; Use a target detection network and a pulse neural network to process the RGB image and the event data after optical flow compensation, respectively, to extract the respective modal features; Use a bidirectional LSTM time sequence coding module to extract and refine the time attributes of events; The RGB feature and the event feature are weighted and fused by using a cross-modal attention mechanism and an asymmetric weight distribution strategy, and the fused feature is input into a detection head to obtain a prediction result.

6. The foveated, biomimetic, hybrid-pixel, dynamic-vision-sensor design and imaging method of claim 5, wherein, The two-dimensional time surface model is constructed in the following manner: A time surface tensor containing positive and negative event polarity channels is established, and each pixel value is calculated by an exponential decay function to preserve the timing characteristics of the event stream.

7. The foveated, biomimetic, hybrid-pixel, dynamic-vision-sensor design and imaging method of claim 5 or 6, wherein, The asymmetric weight distribution strategy is specifically as follows: The contribution weights of the RGB feature and the event feature in the fusion process are adaptively adjusted according to the scene motion characteristics and the semantic complexity, so that the RGB feature and the event feature are weighted and fused.

8. The foveated, biomimetic, hybrid-pixel, dynamic-vision-sensor design and imaging method of claim 1, wherein, The method further comprises: Through the cooperative design of the bionic center mixed pixel array and the event optical flow compensation algorithm, the semantic integrity of the whole scene is enhanced while maintaining the low power consumption characteristics, so as to be used for automatic driving, unmanned aerial vehicle navigation or mobile robot visual perception tasks.

9. The foveated, biomimetic, hybrid-pixel, dynamic-vision-sensor design and imaging method of claim 1, wherein, The method reduces the amount of RGB data and utilizes the event stream to guide semantic diffusion, so as to reduce the system power consumption and bandwidth demand under the same semantic accuracy, and improve the deployment feasibility of edge devices.

10. A foveated biomimetic hybrid-pixel dynamic vision sensor, characterized in that, It comprises: A bionic center mixed pixel array is used for sensing light signals and outputting data, the array adopts a non-uniform Gaussian distribution of RGB pixel arrangement, high-density RGB pixels are arranged in the center area of the sensor, the sampling rate is gradually reduced outward, and high dynamic event pixels are integrated to simulate the division mechanism of cone cells and rod cells in the human eye retina, so as to output center RGB data and full-pixel event stream; A processing circuit is configured to perform: Dynamic semantic compensation based on event optical flow: the high-frequency motion information of the event stream is used to estimate the scene optical flow, and the semantic information of the center area is adaptively diffused to the peripheral event sensing area through optical flow prediction and center RGB dynamic compensation, so as to restore the fused RGB image in the full-pixel range; Multi-modal data processing and fusion for downstream tasks: a processing flow containing original data preprocessing, sparse optical flow estimation, task-oriented feature extraction and multi-modal feature fusion is constructed, so as to realize semantic enhancement for color shape detection, feature point detection and target detection tasks.

Images