A method, system, terminal and medium for regional segmentation and noise reduction based on fusion imaging collaboration

Abstract

The present application relates to the field of image processing technology, and discloses a method, system, terminal, and medium for regional segmentation and noise reduction in fusion imaging collaboration. The method for regional segmentation and noise reduction in fusion imaging collaboration includes: acquiring an event stream and a visible light image in the same light field; obtaining an event stream confidence map based on the event stream, and obtaining a visual frame confidence map based on the visible light image; obtaining a motion confidence map based on the event stream confidence map and the visual frame confidence map, and dividing the motion confidence map into static areas, dynamic areas, and transition areas; performing noise reduction processing on the static areas, dynamic areas, and transition areas according to different strategies, to obtain a target visible light image and a target event stream. The present application can use different noise reduction strategies for processing the divided dynamic areas, static areas, and transition areas, thereby improving the background noise suppression effect of the event camera, improving the noise reduction effect, and improving the imaging effect in mixed static and dynamic scenes.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, system, terminal and computer-readable storage medium for regional segmentation and noise reduction in fusion imaging collaboration. Background Art

[0002] In areas such as intelligent driving and industrial precision inspection, which require extremely high temporal and spatial resolution, visual perception systems must simultaneously meet the dual requirements of microsecond temporal resolution and sub-pixel spatial resolution. However, event cameras and traditional visual cameras (including RGB and grayscale cameras) exhibit significant complementarity and contradiction in performance: while event cameras have microsecond temporal resolution and can capture the polarity of brightness changes in high-speed motion in real time, they are extremely sensitive to changes in ambient light. Sudden changes in lighting or slight fluctuations in static backgrounds can easily trigger a large number of false triggering events, resulting in noise events accounting for over 30%, seriously interfering with target information extraction. While traditional visual cameras can output high-resolution brightness / color images and accurately restore static scene details, they are limited by frame rate, suffer from severe motion blur in high-speed motion scenes, and significantly increase noise in low light.

[0003] Existing technologies primarily focus on providing one-way assistance to traditional vision cameras through event cameras, such as compensating for motion blur in traditional vision cameras through event streams. However, research on technologies that feed back noise reduction from traditional vision cameras to event cameras is still immature. There is a lack of effective methods to suppress background noise in event cameras. Existing fixed-threshold filtering methods achieve a noise removal rate of less than 50% and are prone to mistakenly deleting valid events. Furthermore, the lack of a bidirectional collaborative mechanism often leads to the dual problems of "residual event noise in static areas" and "texture blur in dynamic areas" in mixed static and dynamic scenes. In scenarios such as intelligent driving (e.g., prominent obstacle detection) and industrial precision inspection (e.g., submicron defect identification), there is an urgent need to capture high-speed motion time series and restore static backgrounds with high resolution. However, existing technologies lack a systematic solution for cross-modal collaborative noise reduction, making it difficult to address the combined noise and blur issues under complex lighting and high-speed motion.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide a method, system, terminal and medium for regional segmentation and noise reduction of fused imaging collaboration, aiming to solve the problem in the prior art that when an event camera provides one-way assistance to a traditional visual camera, the background noise suppression effect of the event camera is poor, resulting in the frequent occurrence of "static area event noise residue" and "dynamic area texture blur" in mixed static and dynamic scenes.

[0006] A first aspect of an embodiment of the present application provides a collaborative regional segmentation and denoising method for fused imaging, which includes the following steps: acquiring an event stream and a visible light image under the same light field; obtaining an event stream confidence map based on the event stream, and obtaining a visual frame confidence map based on the visible light image; obtaining a motion confidence map based on the event stream confidence map and the visual frame confidence map, and dividing the motion confidence map into a static area, a dynamic area, and a transition area; performing denoising processing on the static area, the dynamic area, and the transition area according to different strategies to obtain a target visible light image and a target event stream.

[0007] Optionally, in one embodiment of the present application, obtaining an event flow confidence map based on the event flow specifically includes: determining the number of events and the global maximum number of events of the pixel point in the time window according to the event flow; obtaining the event number ratio according to the event number and the global maximum number of events; fusing the event number ratio with the direction consistency to obtain an event flow confidence map; the calculation formula of the event flow confidence map is:

[0008] ;

[0009] in, is the event stream confidence, For the time window Inner pixel The number of events, For the time window The global maximum number of events within , is the temporal polarity of adjacent pixels.

[0010] Optionally, in one embodiment of the present application, obtaining a visual frame confidence map based on the visible light image specifically includes: determining the gradient amplitude and the global maximum gradient amplitude of the image based on the visible light image; obtaining a gradient amplitude ratio based on the gradient amplitude and the global maximum gradient amplitude; fusing the gradient amplitude ratio with motion reliability to obtain a visual frame confidence map; the calculation formula of the visual frame confidence map is:

[0011] ;

[0012] in, is the visual frame confidence, Pixel The gradient amplitude, is the global maximum gradient amplitude, The inverse error for optical flow estimation.

[0013] Optionally, in one embodiment of the present application, the motion confidence map is obtained based on the event flow confidence map and the visual frame confidence map, specifically including: obtaining the dynamic feature weights of the event flow confidence map and the static feature weights of the visual frame confidence map; and performing weighted fusion of the event flow confidence map and the visual frame confidence map based on the dynamic feature weights and the static feature weights to obtain a motion confidence map.

[0014] Optionally, in one embodiment of the present application, the fusion formula of the motion confidence map is:

[0015] ;

[0016] ;

[0017] in, is the motion confidence, is the event stream confidence, is the visual frame confidence, is the dynamic feature weight, is the static feature weight.

[0018] Optionally, in one embodiment of the present application, the segmenting of the motion confidence map into static areas, dynamic areas and transition areas specifically includes: obtaining a static determination threshold and a dynamic determination threshold; treating an area in the motion confidence map where the motion confidence is less than the static determination threshold as a static area, treating an area where the motion confidence is greater than the dynamic determination threshold as a dynamic area, and treating an area where the motion confidence is greater than or equal to the static determination threshold and less than or equal to the dynamic determination threshold as a transition area.

[0019] Optionally, in one embodiment of the present application, the static area includes a spatially corresponding static image layer and a static event flow layer, the dynamic area includes a spatially corresponding dynamic image layer and a dynamic event flow layer, and the transition area includes a spatially corresponding transition image layer and a transition event flow layer; the denoising process is performed on the static area, the dynamic area and the transition area according to different strategies to obtain a target visible light image and a target event flow, specifically including: for the static area, retaining the high resolution of the static image layer to obtain an updated static image layer, and performing low-weight denoising process on the static event flow layer to obtain an updated static event stream layer; for the dynamic area, performing motion compensation optimization on the dynamic image layer to obtain an updated dynamic image layer, and performing high-weight denoising on the dynamic event stream layer to obtain an updated dynamic event stream; for the transition area, performing detail gradient descent on the transition image layer to obtain an updated transition image layer, and performing medium-weight denoising on the transition event stream layer to obtain an updated transition event stream; based on the updated static image layer, the updated static event stream layer, the updated dynamic image layer, the updated dynamic event stream, the updated transition image layer and the updated transition event stream, the spatially corresponding target visible light image and target event stream are obtained.

[0020] A second aspect of the embodiments of the present application further provides a fusion imaging collaborative regional segmentation and noise reduction system, wherein the fusion imaging collaborative regional segmentation and noise reduction system includes:

[0021] A data acquisition module, used to acquire event streams and visible light images under the same light field;

[0022] a confidence calculation module, configured to obtain an event stream confidence map based on the event stream, and obtain a visual frame confidence map based on the visible light image;

[0023] A region segmentation module is used to obtain a motion confidence map based on the event flow confidence map and the visual frame confidence map, and to segment the motion confidence map into a static region, a dynamic region, and a transition region;

[0024] The partition noise reduction module is used to perform noise reduction processing on the static area, the dynamic area and the transition area according to different strategies to obtain a target visible light image and a target event stream.

[0025] The third aspect of an embodiment of the present application also provides a terminal, wherein the terminal includes: a memory, a processor, and a fusion imaging collaborative regional segmentation and denoising program stored on the memory and runnable on the processor, wherein when the fusion imaging collaborative regional segmentation and denoising program is executed by the processor, the steps of the fusion imaging collaborative regional segmentation and denoising method described above are implemented.

[0026] The fourth aspect of an embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a regional segmentation and denoising program for fusion imaging collaboration, and when the regional segmentation and denoising program for fusion imaging collaboration is executed by a processor, the steps of the regional segmentation and denoising method for fusion imaging collaboration as described above are implemented.

[0027] Beneficial effects: The present application provides a method, system, terminal and medium for regional segmentation and noise reduction in fusion imaging collaboration. The present application divides the field of view into dynamic areas, static areas and transition areas based on motion confidence through spatially corresponding event streams and visible light images, and adopts different noise reduction strategies for processing different areas. The event stream compensates for the motion blur of the visible light image, and the visible light multi-frame fusion suppresses the event stream noise. The threshold weight is adjusted by two-way feedback, thereby solving the problems of excessive smoothing of static areas and residual noise in dynamic areas in traditional noise reduction, thereby improving the background noise suppression effect of the event camera, improving the noise reduction effect, and thus improving the imaging effect in mixed dynamic and static scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a schematic diagram of the structural principle of the fusion imaging device of this application;

[0030] Figure 2 This is a flow chart of a preferred embodiment of the regional segmentation and denoising method for fusion imaging collaboration of the present application;

[0031] Figure 3 This is a flowchart of specific implementation steps of the entire execution process in a preferred embodiment of the method for regional segmentation and noise reduction in fusion imaging collaboration of the present application;

[0032] Figure 4 This is a structural diagram of a preferred embodiment of the regional segmentation and noise reduction system for fusion imaging collaboration of the present application;

[0033] Figure 5 This is a structural diagram of a preferred embodiment of the terminal of this application.

[0034] Description of reference numerals:

[0035] 100, data acquisition module; 200, confidence calculation module; 300, region segmentation module; 400, partition noise reduction module. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and effects of this application clearer and more specific, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. The described embodiments are only possible technical implementations of this application and are not all possible implementations. Based on the embodiments in this application, those skilled in the art can fully combine the embodiments of this application to obtain other embodiments without creative work, and these embodiments are also within the scope of protection of this application.

[0037] First, the nouns involved in the embodiments of this application are introduced:

[0038] RGB camera: Red-Green-Blue camera, which captures color images through a three-channel color sensor;

[0039] CMOS / CCD: Complementary Metal Oxide Semiconductor / Charge Coupled Device, image sensor type (CMOS has low power consumption, CCD has high image quality);

[0040] ADC: Analog-to-Digital Converter, which converts analog electrical signals into digital image data;

[0041] Event Camera: Based on Dynamic Vision Sensor (DVS) technology, it asynchronously records pixel brightness changes.

[0042] The following is an introduction to traditional visual cameras and event cameras in related technologies:

[0043] Traditional vision cameras include but are not limited to RGB cameras and grayscale cameras. Traditional vision cameras use optical lenses to focus light in the scene onto an image sensor (such as CMOS or CCD). The sensor converts the light signal into an electrical signal, and then converts the electrical signal into digital image data through an analog-to-digital converter (ADC). Traditional vision cameras usually capture image sequences at a fixed frame rate (such as 30 frames per second or 60 frames per second) to generate a continuous video stream. Technical advantages: High resolution, capable of capturing high-resolution still images or videos, suitable for scenes requiring fine details; Mature algorithm support: There are a large number of mature image processing and computer vision algorithms (such as target detection, recognition, tracking, etc.); Strong versatility: Suitable for a variety of application scenarios, including monitoring, industrial inspection, autonomous driving, etc.; Low cost: The technology is mature, the price is relatively low, and it is easy to deploy on a large scale.

[0044] However, traditional vision cameras have a limited dynamic range. In environments with strong or weak light, traditional cameras are prone to overexposure or underexposure problems. Traditional vision cameras have low temporal resolution. Due to the fixed frame rate limitation, traditional cameras have a certain time delay and cannot capture rapidly changing scenes in real time. When the object exceeds the camera frame rate, imaging quality problems such as image blur will occur.

[0045] Event cameras utilize an asynchronous, pixel-level light intensity change detection mechanism, recording only pixels with varying brightness and their timestamps. They offer microsecond temporal resolution, but at the expense of spatial resolution. Technical advantages include a high dynamic range, enabling the capture of scenes with extreme brightness fluctuations without over- or underexposure; low power consumption, as only pixels with varying brightness are recorded, consuming far less than traditional visual cameras; and high temporal resolution, enabling rapid response to rapidly changing scenes and suitable for capturing high-speed dynamic events.

[0046] However, event cameras are non-visual detection devices that can only output pixel change information and cannot directly detect specific information about the target object; the resolution and resolution of event cameras are limited, and they are limited to high temporal resolution detection. The resolution of event cameras is quite limited, and due to their sensitivity to illumination vectors, the noise suppression effect is poor and the signal quality is poor.

[0047] In addition, high-speed cameras are expensive and cannot meet the requirements of high temporal and spatial resolution, making them difficult to popularize.

[0048] In related technologies, event cameras are usually used to assist RGB cameras (event cameras enable visible light), and RGB cameras are rarely used to assist event cameras (visible light cameras cannot be used to feed back to event cameras). In other words, the advantages of the two types of sensors are not fused through dynamic area segmentation and two-way collaborative technology, and the technical bottlenecks of spatiotemporal resolution coordination and noise suppression still exist.

[0049] When the event camera provides one-way assistance to the traditional visual camera, the background noise suppression effect of the event camera is poor, resulting in the problems of "residual event noise in static areas" and "texture blur in dynamic areas" in mixed static and dynamic scenes. This application divides the field of view into dynamic areas, static areas and transition areas based on motion confidence through spatially corresponding event streams and visible light images, and adopts different noise reduction strategies for processing different areas. The event stream compensates for the motion blur of the visible light image (dynamic area), and the visible light multi-frame fusion suppresses the event stream noise (static area). The two-way feedback adjusts the threshold weight (transition area), thereby solving the problems of excessive smoothing in static areas and residual noise in dynamic areas in traditional noise reduction, thereby improving the background noise suppression effect of the event camera, improving the noise reduction effect, and thus improving the imaging effect in mixed static and dynamic scenes.

[0050] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0051] like Figure 1 As shown, an embodiment of the present application provides a fusion imaging device for a method for regional segmentation and denoising in collaborative fusion imaging. The fusion imaging device includes a visible light camera, a spectroscopic optical component (beam splitter prism), an event camera, an external trigger box, and a computer. The visible light camera and the event camera are coaxially arranged, and the visible light camera and the event camera are respectively arranged opposite the beam splitter prism. The computer is connected to the visible light camera and the event camera, respectively, and the external trigger box is connected to the visible light camera and the event camera, respectively. The optical path coupling between the visible light camera and the event camera is achieved by using a beam splitter prism. The beam splitter prism evenly distributes the incident light to the two cameras, ensuring the consistency of their observation fields. That is, the event stream captured by the time camera and the visible light image captured by the visible light camera are in the same light field. This application uses hardware synchronization and light field sharing, employing a beam splitter prism and a synchronous trigger module to enable the event camera and the visible light camera to synchronously capture the spatiotemporal alignment signal of the same light field, eliminating cross-modal data registration errors at the hardware level and providing a spatiotemporally consistent input foundation for subsequent collaborative denoising.

[0052] In response to the problems of redundant event stream triggering, serious interference of stray ambient light on event streams in existing technologies, failure of event cameras and traditional visual cameras to effectively combine high temporal resolution and high spatial resolution characteristics, over-smoothing of static areas by traditional noise reduction methods and residual noise in dynamic areas, the present application proposes a regional segmentation and noise reduction method based on fusion imaging collaboration, namely a dynamic region segmentation and noise reduction strategy based on bidirectional collaboration. This method generates a pixel-level motion confidence map by fusing the spatiotemporal derivatives of the event stream with the frame gradient features of the traditional visual camera, and divides the scene into static, dynamic, and transition regions. In the static region, noise is suppressed by multi-frame fusion of the traditional visual camera, and the event stream is fed back to filter out false triggering events caused by ambient light. In the dynamic region, a short-exposure image is reconstructed through the event stream and motion-compensated fusion is performed with the traditional visual camera frame to suppress motion blur. In the transition region, the wavelet threshold is dynamically adjusted based on the motion confidence to achieve a smooth transition of the noise reduction intensity. By constructing a bidirectional collaborative information interaction closed loop, adaptive noise reduction is achieved in complex scenes, effectively improving the accuracy and robustness of the visual perception system.

[0053] Compared with related technologies, the advantages of this application are as follows:

[0054] Cross-modal bidirectional collaboration mechanism: By constructing a beam-splitting prism optical path, the light field synchronization between the event camera and traditional vision camera is achieved, establishing a closed loop of bidirectional information exchange, breaking through the traditional one-way auxiliary model. The event camera's microsecond temporal resolution is used to reconstruct short-exposure images, compensating for motion blur in traditional cameras. A high signal-to-noise ratio background model is generated through multi-frame fusion of visible light, filtering out ambient light noise in the event stream. Fusion weights are adjusted in real time based on scene complexity, achieving an adaptive balance between temporal and spatial resolution.

[0055] Three-level region segmentation and adaptive noise reduction: Based on event flow density and motion confidence (integrating spatiotemporal gradients and optical flow errors), the field of view is divided into dynamic, static, and transition regions. In dynamic regions, the event flow is used to drive motion compensation or amplification, eliminating traditional camera blur or increasing sensitivity to small vibrations. In static regions, a dual sampling verification mechanism (event trigger frequency + background model matching) preserves texture details. In transition regions, the wavelet denoising threshold is dynamically adjusted based on motion confidence. After wavelet transform, the threshold function and threshold are set to denoise the information. An inverse transform is then performed to obtain a dynamically adjusted fused image, achieving balanced noise reduction and edge preservation. This addresses the issues of over-smoothing in static regions and residual noise in dynamic regions encountered in traditional noise reduction.

[0056] The preferred embodiment of the present application describes a method for regional segmentation and noise reduction in fusion imaging collaboration, such as Figure 2 As shown, the fusion imaging collaborative regional segmentation and noise reduction method includes the following steps:

[0057] In step S101 , an event stream and a visible light image in the same light field are acquired.

[0058] It should be noted that this application performs dynamic region segmentation based on motion confidence maps through hardware synchronization and spatiotemporal calibration of event cameras and visible light cameras, designs bidirectional collaborative noise reduction strategies for different regions, and realizes real-time optimization of noise reduction parameters through a feedback adjustment module.

[0059] Specifically, during the data acquisition and synchronization process, a beam splitter precisely distributes the same light field to the event camera and visible light camera, ensuring that the scenes captured by the two are completely synchronized in space and time. The event camera records pixel-level brightness changes with microsecond temporal resolution. When the brightness change of a pixel exceeds a preset threshold, an event is triggered, recording the pixel coordinates (identifying the pixel location that triggered the event), timestamp (recording the exact time of the event), and polarity (indicating the direction of the brightness change, whether it increases or decreases). Visible light image frames (such as RGB or grayscale images) synchronized with the event stream are acquired. If the input is an RGB image, grayscale conversion is performed to convert the color image into a grayscale image.

[0060] In step S102, an event stream confidence map is obtained according to the event stream, and a visual frame confidence map is obtained according to the visible light image.

[0061] In one possible implementation, based on the event stream, the number of events at the pixel point in the time window and the global maximum number of events are determined; based on the number of events and the global maximum number of events, the proportion of the number of events is obtained; and the proportion of the number of events is fused with the directional consistency to obtain an event stream confidence map.

[0062] Specifically, set a time window , used to count event activities in a short period of time; within each time window, the number of events triggered by each pixel P is counted ; In the same time window, count the number of events of all pixels and find the global maximum ; Through the pixel high frequency motion intensity formula , normalize the number of pixel-level events to the range of [0, 1]. It can be understood that the normalized value represents the relative activity intensity of the pixel in the local area. The closer the value is to 1, the more events the pixel has triggered in a short period of time, which may be the edge of a dynamic object or a high-speed motion area. The closer the value is to 0, the weaker the pixel activity is, which may be a static background or a low-speed area. Then, the motion direction consistency analysis is performed to analyze the consistency of the temporal polarity (brightness change direction) of adjacent pixels. If adjacent pixels have the same direction of brightness change in a short period of time (such as brightening or dimming at the same time), they are considered to belong to the same motion trajectory. To calculate the temporal polarity of adjacent pixels, The value range is [0, 1]. The closer the value is to 1, the higher the consistency of the motion direction of adjacent pixels is. The closer the value is to 0, the stronger the directional randomness is. Event stream confidence fusion is performed to fuse the high-frequency motion intensity with the directional consistency to generate the event stream confidence.

[0063] The calculation formula of the event flow confidence graph is:

[0064] ;

[0065] in, is the event stream confidence, For the time window Inner pixel The number of events, For the time window The global maximum number of events (i.e. the total number of events) within , It is the temporal polarity of adjacent pixels, indicating the proportion of the same direction, i.e. the consistency of motion direction, and its value range is [0, 1].

[0066] It should be noted that the 5ms time window is a compromise between temporal resolution and computational efficiency. A window that is too short may lead to statistical noise (too few events), while a window that is too long may reduce temporal resolution (unable to capture rapid changes). In practical applications, the window length can be dynamically adjusted according to the scenario. Used to normalize the number of events for different pixels to ensure comparability of confidence between different pixels, such as in scenes with high global activity (such as fast-moving objects). The normalized high-frequency motion intensity can suppress the difference in absolute event number and highlight the relative activity intensity. It directly reflects the frequency of pixel-level brightness changes. In dynamic areas (such as object edges), brightness changes frequently and the number of events is high. Through normalization processing, the confidence of the event stream is The relative intensity of pixel-level dynamic activity is quantified, providing key input for subsequent regional segmentation and feedback regulation.

[0067] In one possible implementation, the gradient amplitude and the global maximum gradient amplitude of the image are determined based on the visible light image; the gradient amplitude ratio is obtained based on the gradient amplitude and the global maximum gradient amplitude; and the gradient amplitude ratio is fused with the motion reliability to obtain a visual frame confidence map.

[0068] Specifically, the gradient magnitude is calculated by the Sobel (Sobel Operator) operator (edge ​​detection algorithm in image processing). The gradient amplitude reflects the severity of the local brightness change in the image. The gradient amplitude indicates the intensity of the brightness change at the pixel point. The larger the value, the pixel is located at the edge or texture-rich area (such as the boundary of an object). The smaller the value, the pixel is located in a smooth area (such as the background).

[0069] is the gradient component in the horizontal direction: ;

[0070] is the gradient component in the vertical direction: ;

[0071] in, is the input image.

[0072] Traverse all pixels in the entire image and find the maximum gradient amplitude (the maximum value of the gradient amplitude of all pixels in the image); perform normalization , eliminate the influence of brightness differences between different images, convert the gradient amplitude into relative intensity, so that the confidence in different scenes is comparable. Through normalization, it ensures that the confidence only reflects local relative changes.

[0073] Then further combine the optical flow to estimate the reverse error , (Lucas-Kanade algorithm), after normalization, the value is [0,1], which is used to measure the accuracy of optical flow estimation. The smaller the error, the higher the confidence, which quantifies the reliability of motion estimation; the final confidence Both structural information (gradient, pixel gradient amplitude) and motion information (optical flow, motion reliability) are considered at the same time, providing a reliable basis for dynamic area segmentation.

[0074] The calculation formula of the visual frame confidence map is:

[0075] ;

[0076] in, is the visual frame confidence, Pixel The gradient amplitude, is the global maximum gradient amplitude, The inverse error for optical flow estimation.

[0077] In step S103, a motion confidence map is obtained according to the event flow confidence map and the visual frame confidence map, and the motion confidence map is divided into a static area, a dynamic area, and a transition area.

[0078] In one possible implementation, the dynamic feature weights of the event stream confidence map and the static feature weights of the visual frame confidence map are obtained; based on the dynamic feature weights and the static feature weights, the event stream confidence map and the visual frame confidence map are weightedly fused to obtain a motion confidence map.

[0079] This application generates a pixel motion confidence map by fusing event stream features with traditional visual frame gradient information, thereby achieving self-adaptive block segmentation.

[0080] Pixels Movement confidence Motion confidence for event streams and traditional visual frame motion confidence The weighted fusion of the motion confidence map is:

[0081] ;

[0082] ;

[0083] in, is the motion confidence, is the event stream confidence, is the visual frame confidence, is the dynamic feature weight, is the static feature weight. The feature weights for fusion are determined through offline training to maximize the accuracy of scene segmentation, and the parameters can be adjusted according to the specific scene.

[0084] It should be noted that the event stream dominates the dynamic characteristics, and the event camera has a microsecond time resolution and is sensitive to fast motion, so it is given a higher weight ( > ) to capture dynamic areas. The visual frame dominates the static characteristics. The visible light camera has high spatial resolution and is good at capturing details of static scenes, so it is given a lower weight ( ), to suppress event stream noise. Weight and It can be adjusted dynamically according to the specific scene, such as in high-speed sports scenes, it can be increased (like =0.7), improve the contribution of event flow confidence; in low-light scenes, it can reduce (like = 0.5), reducing the impact of event stream noise.

[0085] In one possible implementation, a static determination threshold and a dynamic determination threshold are obtained; an area in the motion confidence map where the motion confidence is less than the static determination threshold is taken as a static area, an area where the motion confidence is greater than the dynamic determination threshold is taken as a dynamic area, and an area where the motion confidence is greater than or equal to the static determination threshold and less than or equal to the dynamic determination threshold is taken as a transition area.

[0086] Specifically, such as Figure 3 As shown in the three-layer area division rule, if Set to a static area with minimal or no motion, preserving the original details of the visible light image; if Set as dynamic area, high-frequency motion area, give priority to retaining event flow features; if Set as transition area, blur the motion state, and fuse spatiotemporal information; among them, the static judgment threshold , dynamic decision threshold , determined by the maximum inter-class variance method, and the adaptive threshold can be adjusted by parameters according to specific scenarios.

[0087] In step S104 , noise reduction processing is performed on the static area, the dynamic area, and the transition area respectively according to different strategies to obtain a target visible light image and a target event stream.

[0088] It should be noted that in this application, the same light field is assigned to the event camera and visible light camera through hardware synchronization. The visible light image and event stream are simultaneously divided into dynamic, static, and transition regions, with a one-to-one spatial correspondence. From the perspective of the visible light image, the dynamic region is affected by the integration of high-temporal-resolution dynamic object features of the event stream, resulting in a decrease in image detail. However, unlike the unprocessed blurred imaging of dynamic objects, the image compensates for motion information and retains a certain motion trajectory, allowing the image features of dynamic objects to appear. The pixels in the static region do not change significantly from the original visible light image. The pixels in the transition region show an exponentially decreasing trend, connecting the high spatial resolution features of the static region and the high temporal resolution features of the dynamic region, presenting an intermediate state. From the perspective of the event stream, the dynamic region event stream is given a high weight, meaning the denoising level is the lowest and the event stream feature points are most completely preserved. The transition region is second, and the static region is given a low weight and the denoising level is the highest, thereby filtering out stray light and non-attentional events.

[0089] This application performs dynamic region segmentation through cross-modal collaborative dynamic partitioning and noise reduction of video stream and event stream, integrates the spatiotemporal density of event stream and visible light gradient features, and combines a static background model (multi-frame non-local mean generation). On this basis, the event stream compensates for the motion blur of the visible light camera (dynamic area), the visible light multi-frame fusion suppresses the event stream noise (static area), and the two-way feedback adjusts the threshold weight (transition area).

[0090] In a possible implementation, the static region includes a spatially corresponding static image layer and a static event stream layer, the dynamic region includes a spatially corresponding dynamic image layer and a dynamic event stream layer, and the transition region includes a spatially corresponding transition image layer and a transition event stream layer. For the static region, the static image layer is subjected to high-resolution preservation to obtain an updated static image layer, and the static event stream layer is subjected to low-weight denoising to obtain an updated static event stream layer; for the dynamic region, the dynamic image layer is subjected to motion compensation optimization to obtain an updated dynamic image layer, and the dynamic event stream layer is subjected to high-weight denoising to obtain an updated dynamic event stream; for the transition region, the transition image layer is subjected to detail gradient descent to obtain an updated transition image layer, and the transition event stream layer is subjected to medium-weight denoising to obtain an updated transition event stream; based on the updated static image layer, the updated static event stream layer, the updated dynamic image layer, the updated dynamic event stream, the updated transition image layer, and the updated transition event stream, a spatially corresponding target visible light image and target event stream are obtained.

[0091] In bidirectional feedback regulation, event streams drive feedback, leveraging the high temporal resolution of event cameras. When the temporal density of event streams in a certain area increases dramatically within a short period of time, this indicates the presence of rapidly moving objects, referred to as dynamic regions. By combining frame and event stream fusion, techniques such as motion amplification or motion compensation are employed to optimize and reduce noise in fast-moving visible light images, where the image quality is significantly reduced. This includes, but is not limited to, frame insertion, contrast enhancement, and edge sharpening to enhance detail in rapidly moving objects. Visual quality drives feedback, leveraging the high spatial resolution of visible light cameras to construct a static background model, compare pixel differences between adjacent frames, and distinguish between static and slowly changing areas. For event streams in this area that are likely to be noise events, the event streams within the static region are weighted by synchronizing the event stream to the event camera using the same light field. This includes, but is not limited to, adjusting internal event trigger thresholds to reduce false triggers and achieve noise reduction in the event stream information.

[0092] Specifically, in the dynamic area noise reduction strategy: from the perspective of event streams, in dynamic areas, event streams are given high weights, which means the degree of denoising is the lowest, so as to retain the feature points of the event stream to the greatest extent possible; by retaining the high-frequency motion information of the event stream, the motion trajectory and details of dynamic objects can be accurately captured. From the perspective of visible light images, motion blur that may appear in dynamic areas in visible light images is optimized using technologies such as motion amplification, interpolation, contrast enhancement, and edge sharpening; by compensating for motion information, the features of dynamic objects in visible light images are made more apparent. Although the resolution may decrease slightly, the motion trajectory is preserved. It is understandable that dynamic areas are the most active parts of the scene, so the core of the noise reduction strategy is to retain motion information. By retaining the event stream features with high weights and combining them with the motion compensation technology of visible light images, it is possible to sacrifice resolution to a certain extent in exchange for clearer dynamic object features.

[0093] In the static area noise reduction strategy: from the perspective of event streams, in static areas, the event stream is assigned a low weight, which means the highest degree of denoising is achieved to filter out stray light and non-attention events; noise suppression reduces falsely triggered event stream information in static areas by adjusting parameters such as the event trigger threshold. From the perspective of visible light images, static areas maintain their original high spatial resolution in visible light images without significant changes; leveraging the high spatial resolution of visible light cameras, a static background model is constructed, and static areas and slowly changing areas are distinguished by comparing pixel differences between adjacent frames. It is understandable that static areas are the parts of the scene with the least or no motion, so the core of the noise reduction strategy is to suppress noise. By denoising event stream information with low weights and combining the high resolution of visible light images with background modeling technology, it is possible to accurately distinguish between static and dynamic areas while maintaining high-quality images in static areas.

[0094] In the transition region denoising strategy, from the perspective of event flow, the event flow in the transition region is given a medium weight to balance temporal resolution and noise. This strategy not only retains certain event flow characteristics to capture possible motion information, but also reduces noise interference through denoising. From the perspective of visible light images, the details of the transition region in the visible light image decrease exponentially, connecting the high spatial resolution characteristics of the static region with the high temporal resolution characteristics of the dynamic region. By fusing spatiotemporal information, the transition region visually appears to be an intermediate state, avoiding the abrupt sense of modal switching. It can be understood that the transition region is the part of the scene where the motion state is blurred, including the transition state from static to dynamic or from dynamic to static. Therefore, the denoising strategy needs to balance temporal resolution and noise suppression. By processing the event flow information with a medium weight and combining it with the detail gradient descent technique of visible light images, it can reduce noise interference while maintaining a certain amount of motion information, achieving a smooth transition of spatiotemporal information.

[0095] In another embodiment of the present application, the same light field is synchronously assigned to the event camera and the visible light camera through hardware, and image processing is performed through cross-modal collaborative dynamic partitioning and noise reduction of video stream and event stream to achieve bidirectional noise reduction of event stream and video stream.

[0096] In another embodiment of the present application, an attention mechanism is added to adjust the event stream weight to adjust the event triggering threshold.

[0097] Next, the fusion imaging collaborative region segmentation and denoising system proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0098] Figure 4 This is a structural diagram of the fusion imaging collaborative regional segmentation and denoising system in an embodiment of the present application.

[0099] like Figure 4 As shown, the fusion imaging collaborative region segmentation and noise reduction system includes: a data acquisition module 100, a confidence calculation module 200, a region segmentation module 300 and a partition noise reduction module 400.

[0100] Specifically, the data acquisition module 100 is used to acquire the event stream and visible light image under the same light field;

[0101] A confidence calculation module 200 is configured to obtain an event stream confidence map based on the event stream and a visual frame confidence map based on the visible light image;

[0102] A region segmentation module 300 is configured to obtain a motion confidence map based on the event flow confidence map and the visual frame confidence map, and to segment the motion confidence map into a static region, a dynamic region, and a transition region;

[0103] The partition noise reduction module 400 is configured to perform noise reduction processing on the static area, the dynamic area, and the transition area according to different strategies to obtain a target visible light image and a target event stream.

[0104] Figure 5 This is a diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0105] Memory 501 , processor 502 , and computer programs stored in the memory 501 and executable on the processor 502 .

[0106] When the processor 502 executes the program, the region segmentation and denoising method for fusion imaging collaboration provided in the above embodiment is implemented.

[0107] Furthermore, the terminal further includes:

[0108] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0109] The memory 501 is used to store computer programs that can be run on the processor 502 .

[0110] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0111] If the memory 501, processor 502, and communication interface 503 are implemented independently, the communication interface 503, memory 501, and processor 502 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EIS) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0112] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0113] The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0114] This embodiment further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the above-mentioned method for regional segmentation and denoising in collaborative fusion imaging is implemented.

[0115] One embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the Figure 2 The corresponding embodiment provides a method for regional segmentation and denoising of fused imaging collaboration provided by any embodiment.

[0116] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0117] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0118] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0119] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0120] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0121] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0122] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0123] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

[0124] It should be understood that the application of this application is not limited to the above examples. For ordinary technicians in this field, they can make improvements or changes based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to this application.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A fusion imaging collaborative region segmentation denoising method, characterized in that: The fusion imaging collaborative regional segmentation and noise reduction method includes: Acquire event streams and visible light images under the same light field; Obtaining an event stream confidence map according to the event stream, and obtaining a visual frame confidence map according to the visible light image; Obtaining a motion confidence map according to the event flow confidence map and the visual frame confidence map, and dividing the motion confidence map into a static area, a dynamic area, and a transition area; performing noise reduction processing on the static area, the dynamic area, and the transition area according to different strategies to obtain a target visible light image and a target event stream; The static area includes a spatially corresponding static image layer and a static event flow layer, the dynamic area includes a spatially corresponding dynamic image layer and a dynamic event flow layer, and the transition area includes a spatially corresponding transition image layer and a transition event flow layer; The denoising process is performed on the static area, the dynamic area, and the transition area according to different strategies to obtain a target visible light image and a target event stream, specifically including: For the static area, performing high-resolution retention on the static image layer to obtain an updated static image layer, and performing low-weight denoising processing on the static event flow layer to obtain an updated static event flow layer; For the dynamic area, performing motion compensation optimization on the dynamic image layer to obtain an updated dynamic image layer, and performing high-weight denoising processing on the dynamic event stream layer to obtain an updated dynamic event stream; For the transition region, performing detail gradient descent on the transition image layer to obtain an updated transition image layer, and performing weighted denoising on the transition event flow layer to obtain an updated transition event flow; According to the updated static image layer, the updated static event stream layer, the updated dynamic image layer, the updated dynamic event stream, the updated transition image layer and the updated transition event stream, a spatially corresponding target visible light image and target event stream are obtained.

2. The method for regional segmentation and denoising based on fusion imaging collaboration according to claim 1, characterized in that: Obtaining an event stream confidence graph according to the event stream specifically includes: Determine the number of events and the global maximum number of events for the pixel in the time window according to the event stream; Obtaining a proportion of the number of events according to the number of events and the global maximum number of events; The event count ratio and direction consistency are combined to obtain an event flow confidence map; The calculation formula of the event flow confidence graph is: ； in, is the event stream confidence, For the time window Inner pixel The number of events, For the time window The global maximum number of events within , is the temporal polarity of adjacent pixels.

3. The method for regional segmentation and denoising based on fusion imaging collaboration according to claim 1, characterized in that: The obtaining of a visual frame confidence map according to the visible light image specifically includes: Determining the image gradient amplitude and the global maximum gradient amplitude according to the visible light image; Obtaining a gradient amplitude ratio according to the gradient amplitude and the global maximum gradient amplitude; The gradient amplitude ratio is integrated with the motion reliability to obtain a visual frame confidence map; The calculation formula of the visual frame confidence map is: ； in, is the visual frame confidence, Pixel The gradient amplitude, is the global maximum gradient amplitude, The inverse error for optical flow estimation.

4. The method for regional segmentation and denoising based on fusion imaging collaboration according to claim 1, characterized in that: Obtaining a motion confidence map according to the event flow confidence map and the visual frame confidence map specifically includes: Obtaining dynamic feature weights of the event stream confidence map and static feature weights of the visual frame confidence map; The event stream confidence map and the visual frame confidence map are weightedly fused according to the dynamic feature weights and the static feature weights to obtain a motion confidence map.

5. The method for regional segmentation and denoising based on fusion imaging collaboration according to claim 4, characterized in that: The fusion formula of the motion confidence map is: ； ； in, is the motion confidence, is the event stream confidence, is the visual frame confidence, is the dynamic feature weight, is the static feature weight.

6. The method for regional segmentation and denoising based on fusion imaging collaboration according to claim 4, characterized in that: The step of dividing the motion confidence map into a static area, a dynamic area, and a transition area specifically includes: Obtaining static judgment thresholds and dynamic judgment thresholds; The area in the motion confidence map where the motion confidence is less than the static determination threshold is taken as a static area, the area where the motion confidence is greater than the dynamic determination threshold is taken as a dynamic area, and the area where the motion confidence is greater than or equal to the static determination threshold and less than or equal to the dynamic determination threshold is taken as a transition area.

7. A fusion imaging collaborative regional segmentation and noise reduction system, characterized by: The fusion imaging collaborative regional segmentation and denoising system is applied to the fusion imaging collaborative regional segmentation and denoising method according to any one of claims 1 to 6; The fusion imaging collaborative regional segmentation and noise reduction system includes: A data acquisition module, used to acquire event streams and visible light images under the same light field; a confidence calculation module, configured to obtain an event stream confidence map based on the event stream, and obtain a visual frame confidence map based on the visible light image; A region segmentation module is used to obtain a motion confidence map based on the event flow confidence map and the visual frame confidence map, and to segment the motion confidence map into a static region, a dynamic region, and a transition region; The partition noise reduction module is used to perform noise reduction processing on the static area, the dynamic area and the transition area according to different strategies to obtain a target visible light image and a target event stream.

8. A terminal, characterized in that: The terminal includes: a memory, a processor, and a fusion imaging collaborative regional segmentation and denoising program stored in the memory and runnable on the processor. When the fusion imaging collaborative regional segmentation and denoising program is executed by the processor, the steps of the fusion imaging collaborative regional segmentation and denoising method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a fusion imaging collaborative regional segmentation and denoising program, which, when executed by a processor, implements the steps of the fusion imaging collaborative regional segmentation and denoising method according to any one of claims 1 to 6.

Images