Camera image processing method, device, and storage medium

Abstract

The application discloses a camera image processing method and device and a storage medium, relates to the technical field of image processing, and comprises the following steps: acquiring an event stream output by an event camera within a preset time length, and performing spatial correlation analysis on each event in the event stream to classify each event into a structured event and an unstructured event; calculating a proportion of the structured events in the event stream within the preset time length; adjusting exposure parameters of an ordinary camera according to the proportion of the structured events, and controlling the ordinary camera to acquire a first image according to the adjusted exposure parameters; performing signal-to-noise ratio compensation on the first image according to the structured events to obtain a second image; performing image reconstruction based on the event stream to obtain a third image; and fusing the second image and the third image based on the proportion of the structured events to obtain a fused image. The application can realize continuous and stable output of a perception image in a high dynamic range scene.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a camera image processing method, apparatus, and storage medium. Background Technology

[0002] During the operation of industrial robots, service robots, and autonomous vehicles, visual perception systems need to work stably in lighting environments with extremely wide dynamic ranges of brightness. Examples include situations like direct sunlight when moving from indoors to outdoors, oncoming headlights at night, electric arcs from welding and laser cutting, and large-area specular reflections on metal surfaces. The brightness dynamic range in these scenarios can exceed 120 decibels, while the dynamic range of ordinary cameras is typically only 60 to 70 decibels. This results in large areas of overexposure or underexposure in images captured by ordinary cameras, causing the perception system to lose its effective observation capability of the environment and the target being operated on.

[0003] In existing technologies, multi-exposure fusion is commonly used to deal with high dynamic range scenes, which involves capturing multiple frames with different exposure parameters and then combining them into a single image. However, multi-exposure fusion has significant drawbacks: this method requires capturing multiple frames sequentially, resulting in a total delay of tens of milliseconds, making it unable to respond promptly to microsecond-level brightness changes; motion artifacts in the synthesized image also lead to blurred target edges and positioning errors.

[0004] Furthermore, existing technologies combine event cameras with ordinary cameras. Event cameras asynchronously output brightness change events with microsecond-level time resolution and a dynamic range exceeding 120 decibels. However, existing methods typically use event cameras only as auxiliary sensors, utilizing event streams to deblur or interpolate images from ordinary cameras, without fully utilizing the scene brightness information carried by the event streams themselves. Moreover, in existing perception schemes, highlight detection, exposure adjustment, image denoising, and modal fusion are usually set independently, relying on manually set thresholds and parameters. These steps cannot coordinate and adjust according to scene conditions, and parameter conflicts between steps can lead to fluctuations in the quality of perception results, thereby affecting the stability of downstream decision-making and control.

[0005] Therefore, how to obtain usable perceived images in a timely, continuous and stable manner when the dynamic range of scene brightness exceeds the capabilities of ordinary cameras is an urgent technical problem to be solved.

[0006] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0007] The main objective of this application is to provide a camera image processing method, device, and storage medium, which aims to solve the technical problem of how to obtain usable perceived images in a timely, continuous, and stable manner when the dynamic range of scene brightness exceeds the capabilities of ordinary cameras.

[0008] This application provides a camera image processing method, the method comprising: The event stream output by the event camera within a preset time period is acquired, and spatial correlation analysis is performed on each event in the event stream to classify each event into structured events and unstructured events. The proportion of the number of structured events to the sum of the number of structured events and the number of unstructured events is calculated as the proportion of structured events in the event stream within the preset duration. The exposure parameters of the ordinary camera are adjusted according to the proportion of the structured events, and the ordinary camera is controlled to acquire the first image according to the adjusted exposure parameters; The first image is subjected to signal-to-noise ratio compensation based on the structured events to obtain the second image; Based on the event stream, image reconstruction is performed to obtain a third image; The second image and the third image are fused based on the proportion of structured events to obtain a fused image.

[0009] In one embodiment, the step of performing signal-to-noise ratio compensation on the first image based on the structured events to obtain a second image includes: A spatial confidence map is generated based on the spatial distribution of the structured events. The confidence level at each location in the spatial confidence map represents the reliability of signal-to-noise ratio compensation at the corresponding location in the first image. At the location where the confidence level is greater than a preset confidence threshold, texture enhancement is performed on the corresponding location in the first image based on the timestamp of the structured event and the polarity of the brightness change. At the position where the confidence level is less than the preset confidence threshold, based on the motion information of the structured event, the first image and the historical images taken by the ordinary camera before the first image are motion aligned, and the motion-aligned first image and the motion-aligned historical images are superimposed to obtain the second image.

[0010] In one embodiment, the step of performing signal-to-noise ratio compensation on the first image based on the structured events to obtain a second image includes: If the proportion of structured events is less than a preset threshold, it is determined that the event camera is in a high dynamic range scene; When the event camera is in the high dynamic range scene, the signal-to-noise ratio compensation is performed to obtain the second image.

[0011] In one embodiment, fusing the second image and the third image based on the proportion of structured events to obtain a fused image includes: The fusion weights of the second image and the third image are determined based on the proportion of structured events. The second image and the third image are fused according to the fusion weights to obtain the fused image; The larger the proportion of structured events, the smaller the fusion weight of the third image, and the larger the fusion weight of the second image.

[0012] In one embodiment, determining the fusion weight between the second image and the third image based on the proportion of structured events includes: The transition speed of the fusion weight is determined based on the rate of change of the proportion of structured events over time. According to the aforementioned transition speed, the fusion weight is adjusted to a value corresponding to the proportion of the structured events; The greater the rate of change, the greater the transition speed.

[0013] In one embodiment, performing spatial correlation analysis on each event in the event stream to classify the events into structured events and unstructured events includes: For any of the events, determine the number of neighborhood events within a spatial neighborhood window centered on the pixel coordinates of the event; The event density is obtained by the ratio of the number of neighborhood events to the area of ​​the spatial neighborhood window. If the event density is greater than a preset density threshold and the distribution direction consistency of the neighborhood events is greater than a preset consistency threshold, the event is classified as the structured event. If the event density is less than or equal to the preset density threshold, or if the distribution direction consistency is less than or equal to the preset consistency threshold, the event is classified as an unstructured event.

[0014] In one embodiment, adjusting the exposure parameters of the ordinary camera according to the proportion of structured events includes: Obtain the calibration reference ratio of the ordinary camera and the reference exposure parameters corresponding to the calibration reference ratio; The exposure parameters are determined based on the ratio of the structured event proportion to the calibration benchmark proportion, and the benchmark exposure parameters. The smaller the proportion of structured events, the shorter the exposure time corresponding to the exposure parameters.

[0015] In one embodiment, fusing the second image and the third image based on the proportion of structured events to obtain a fused image includes: The second image and the third image are normalized. The normalized second image and the normalized third image are registered together; Based on the proportion of structured events, the registered second image and the registered third image are fused to obtain the fused image.

[0016] Furthermore, to achieve the above objectives, this application also proposes a camera image processing apparatus, which includes: The event classification module is used to acquire the event stream output by the event camera within a preset time period, and to perform spatial correlation analysis on each event in the event stream to classify each event into structured events and unstructured events. The event percentage calculation module is used to calculate the proportion of the number of structured events in the sum of the number of structured events and the number of unstructured events, as the proportion of structured events in the event stream within the preset duration. The first image acquisition module is used to adjust the exposure parameters of the ordinary camera according to the proportion of the structured events, and control the ordinary camera to acquire the first image according to the adjusted exposure parameters; The second image acquisition module is used to perform signal-to-noise ratio compensation on the first image based on the structured event to obtain the second image; The third image acquisition module is used to reconstruct the image based on the event stream to obtain a third image; The image fusion acquisition module is used to fuse the second image and the third image based on the proportion of structured events to obtain a fused image.

[0017] In addition, to achieve the above objectives, this application also proposes a camera image processing apparatus, the apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the camera image processing method as described above.

[0018] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the camera image processing method as described above.

[0019] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the camera image processing method as described above.

[0020] Therefore, the technical solution provided in this application performs spatial correlation analysis on each event in the event stream, dividing the events into structured events triggered by scene edges and motion, and unstructured events triggered by photon shot noise. The proportion of structured events is used as a unified indicator reflecting scene brightness, enabling the system to directly obtain scene brightness information from within the event stream without the need for additional photometers. Based on the proportion of structured events, the exposure parameters of a regular camera are continuously adjusted, allowing the camera to acquire effective images even in extreme lighting conditions. The spatial distribution of structured events is used to compensate the signal-to-noise ratio of the first image, restoring image quality under low exposure conditions. Furthermore, the proportion of structured events drives the fusion of the second and third images; for example, increasing the fusion weight of the third image when scene brightness increases. This achieves that exposure adjustment, signal-to-noise ratio compensation, and dual-modal fusion are driven by the same physical indicator, with parameters coordinated between each stage, avoiding fluctuations in perception results caused by parameter conflicts between stages.

[0021] Therefore, the technical solution provided in this application can adjust exposure parameters in a timely and continuous manner and obtain fused images when the dynamic range of scene brightness exceeds the capabilities of ordinary cameras. It avoids the perception failure caused by overexposure or underexposure of ordinary camera images during sudden brightness changes, and maintains the stability of perception output when the lighting is stable. This achieves continuous and stable output of perception images in high dynamic range scenes, improves the adaptability of the perception system to extreme lighting scenes, and reduces the impact of sudden lighting changes on downstream decision-making and control. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating an embodiment of the camera image processing method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the camera image processing method of this application; Figure 3This is a flowchart illustrating Embodiment 3 of the camera image processing method of this application; Figure 4 This is a schematic diagram of the module structure of the camera image processing device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the camera image processing method in this application embodiment.

[0025] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0027] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with specific implementation methods.

[0028] It should be noted that the English abbreviations used in this application have the following meanings: DVS (Dynamic Vision Sensor); E2VID (a network model for reconstructing images from event streams based on recurrent convolutional neural networks); FireNet (a network model for reconstructing images from event streams based on recurrent neural networks); RGB (Red Green Blue).

[0029] It should be noted that the executing entity of this application can be a processor or controller with data processing and program execution functions in the perception processing device of the embodied intelligent agent, wherein the embodied intelligent agent includes devices such as industrial robots, service robots, autonomous vehicles, and drones. The perception processing device is connected to event cameras and ordinary cameras, processes the data collected by the cameras, and outputs fused images.

[0030] Example 1 like Figure 1 As shown, Figure 1 This is a flowchart illustrating an embodiment of the camera image processing method of this application. This embodiment provides a camera image processing method, which includes the following steps: Step S10: Obtain the event stream output by the event camera within a preset time period, and perform spatial correlation analysis on each event in the event stream to classify each event into structured events and unstructured events.

[0031] It's important to note that a Distance Vision Camera (DVS) is a biomimetic vision sensor where each pixel independently detects changes in the logarithm of incident light. An event is asynchronously output only when the brightness change of a pixel exceeds a response threshold. This event includes pixel coordinates, a microsecond-level timestamp, and the polarity of the brightness change (brighter or darker). The DVS features microsecond-level temporal resolution, a dynamic range exceeding 120 decibels, and low data redundancy by outputting only the changed portion. An event stream refers to an ordered set of events output by the DVS, arranged chronologically. The preset duration refers to the length of a sliding time window used for event statistics. Events output by the DVS within the preset duration constitute an event packet. Events within an event packet are used to calculate the proportion of structured events. The event packet is continuously updated as the sliding time window progresses, allowing the proportion of structured events to reflect the current scene state in real time. For example, the preset duration can be set to several milliseconds; the specific value can be set based on the frame rate of a typical camera and the output event rate of the DVS.

[0032] It should be noted that spatial correlation analysis refers to the process of determining whether there is a correlation between the spatial distribution of each event and its neighboring events. For example, a spatial neighborhood window (e.g., a 5×5 or 7×7 square area centered on the pixel coordinates of the event to be judged) is defined. The number of neighboring events appearing within the spatial neighborhood window is counted, and the ratio of the number of neighboring events to the area of ​​the spatial neighborhood window is used as the event density. Event density reflects the degree of event clustering around the current event. Simultaneously, directional consistency analysis is performed on the spatial distribution of neighboring events within the spatial neighborhood window. For example, the covariance matrix of the pixel coordinates of the neighboring events is calculated, and the ratio of the largest eigenvalue to the second largest eigenvalue represents the directional consistency of the distribution. The larger the ratio, the more concentrated the neighboring events are in a certain direction. When the event density is greater than a preset density threshold and the directional consistency is greater than a preset consistency threshold, the event is classified as a structured event; other events that do not meet either condition are classified as unstructured events. Structured events refer to spatially continuous and correlated pixel events, triggered by the edge contours and motion textures of real objects in the scene, forming a continuous linear or regional distribution in space. Unstructured events refer to spatially random and isolated pixel events, originating from photon shot noise. Photon shot noise refers to the random fluctuations in the number of photons reaching sensor pixels due to the quantum properties of photons; the standard deviation of the fluctuation is proportional to the square root of the number of photons. Under low brightness conditions, the total number of photons is small, and the number of unstructured events triggered by shot noise is small; under high brightness conditions, the total number of photons is large, and the absolute amplitude of shot noise increases, which can independently and randomly trigger a large number of unstructured events on each pixel of the event camera.

[0033] In one feasible implementation, spatial correlation analysis can also be implemented using clustering. For example, all events within an event packet can be spatially clustered based on their pixel coordinates according to density, with events within clusters being considered structured events and isolated events not belonging to any cluster being considered unstructured events. Alternatively, it can be implemented using an event graph approach. For example, an event graph can be constructed with each event as a node and the spatial distance between events being less than a set distance as a connection condition. Events with a connectivity scale greater than a set threshold in the event graph can be considered structured events, and events with a connectivity scale less than the set threshold can be considered unstructured events. It is understood that regardless of the method used to implement spatial correlation analysis, as long as spatially continuously correlated events can be distinguished from spatially randomly isolated events, they fall within the protection scope of this application. The calculation details of event density and directional consistency, as well as the threshold values, are further explained in Embodiment 2.

[0034] Step S20: Calculate the proportion of the number of structured events to the sum of the number of structured events and the number of unstructured events, and use this as the proportion of structured events in the event stream within a preset duration.

[0035] It should be noted that the structured event percentage refers to the ratio of the number of structured events to the sum of the number of unstructured events in the event stream within a preset duration. In other words, it represents the proportion of structured events in an event packet within the preset duration, relative to the total number of both types of events. The structured event percentage consistently falls between zero and one, naturally possessing cross-device and cross-scene comparability without requiring additional offline or online normalization operations. Because higher scene brightness leads to stronger photon shot noise, the proportion of unstructured events in both types of events increases, while the structured event percentage continuously decreases as scene brightness increases. Therefore, the structured event percentage directly reflects the absolute brightness level of the scene and is calculated entirely from within the event stream, requiring no additional light metering sensors or external calibration equipment.

[0036] In one feasible implementation, the preset duration can be adaptively adjusted based on the event rate output by the event camera: when the event rate is low, the preset duration is appropriately extended to ensure that the number of events within an event packet is sufficient to support stable proportion statistics; when the event rate is high, the preset duration is appropriately shortened to ensure that the structured event proportion can promptly track rapid changes in brightness. In another feasible implementation, the calculation of the structured event proportion can also introduce time weighting, for example, assigning greater statistical weight to events within an event packet that are closer to the current time, allowing the structured event proportion to reflect the latest changes in event composition more quickly. It is understood that the structured event proportion corresponds one-to-one with the ratio of the number of the two types of events.

[0037] Step S30: Adjust the exposure parameters of the ordinary camera according to the proportion of structured events, and control the ordinary camera to acquire the first image according to the adjusted exposure parameters.

[0038] It should be noted that a standard camera refers to a camera that outputs a complete image at a fixed frame rate, such as an RGB camera. The dynamic range of a standard camera is typically only 60 to 70 decibels. Exposure parameters refer to the exposure-related parameters used by a standard camera when shooting, such as exposure time. The lower the proportion of structured events, the brighter the scene, and the shorter the required exposure time. The adjustment range of the exposure parameters corresponds continuously to the proportion of structured events, ensuring a smooth transition of the perceived output between scenes with different brightness levels through continuous exposure adjustment and no sudden on / off changes. For example, the perception processing device generates an exposure adjustment amount based on the proportion of structured events and sends it to the standard camera. The standard camera then shoots according to the adjusted exposure parameters to obtain the first image. The determination of exposure parameters can be based on establishing a mapping relationship between the proportion and the exposure parameters through calibration: in the initial stage of system deployment, the calibration baseline proportion and the corresponding baseline exposure parameters are measured. During operation, the ratio of the structured event proportion to the calibration baseline proportion is used as a scaling factor, and the product of the scaling factor and the baseline exposure parameters is determined as the exposure parameters. The reduction in exposure time is proportional to the intensity of the high brightness.

[0039] In one feasible implementation, the exposure parameters, in addition to exposure time, may include at least one of analog gain, digital gain, and aperture size. Adjusting the exposure parameters can involve adjusting one or more of these parameters in combination, as long as the exposure amount can be continuously adjusted according to the proportion of structured events. In one feasible implementation, the mapping from the proportion of structured events to the exposure parameters can be achieved using a lookup table method, i.e., based on the proportion-exposure parameter lookup table established during offline calibration, the corresponding exposure parameters are queried according to the real-time proportion during runtime; alternatively, a curve fitting method can be used, i.e., fitting the calibration data as a monotonically continuous function of the proportion and exposure parameters, and inputting the real-time proportion to calculate the exposure parameters during runtime; or a piecewise linear interpolation method can be used, i.e., interpolating between adjacent calibration points according to a linear relationship. The specific calibration process and typical values ​​are further explained in Example 2.

[0040] It is understandable that a cross-frame closed loop is formed between exposure parameter adjustment and first image acquisition: the proportion of structured events for the current preset duration drives the exposure parameter adjustment, the ordinary camera takes a picture of the first image according to the adjusted exposure parameters, the first image is output after subsequent compensation and fusion, the proportion of structured events for the next preset duration drives the exposure parameter adjustment again, and so on, so that the exposure parameters continuously track the changes in scene brightness.

[0041] Step S40: Perform signal-to-noise ratio compensation on the first image based on the structured events to obtain the second image.

[0042] It should be noted that in extremely bright scenes, the signal-to-noise ratio (SNR) of the first image captured by a regular camera with a shortened exposure time significantly decreases due to reduced photon collection. Structured events naturally depict the edge contours and motion textures of objects in the scene. The event camera and the regular camera operate independently. The event camera pixels continuously monitor logarithmic brightness changes and trigger an event only when the brightness change exceeds a response threshold (typically 10% to 15% of the logarithmic brightness change). Event triggering depends solely on scene brightness changes and is independent of the regular camera's exposure time. Therefore, the decrease in image SNR caused by shortened exposure in the regular camera does not affect the quality of the event signal. Structured events can still provide clear spatial structure with microsecond-level temporal accuracy even under low exposure conditions. SNR compensation refers to the process of recovering the first image using the spatial structure and motion information carried by the structured event.

[0043] For example, one implementation of signal-to-noise ratio (SNR) compensation is partitioned compensation: a spatial confidence map is generated based on the spatial distribution of structured events. This map represents the reliability of compensation at each location in the first image, with higher confidence at locations covered by structured event edges and lower confidence in flat areas uncovered by structured events. At locations with confidence levels greater than a preset confidence threshold, the sub-pixel displacement and gradient direction of the edges are estimated using the timestamps and brightness change polarities of the structured events. High temporal resolution gradient information from the event stream is then superimposed onto the first image for texture enhancement. At locations with confidence levels less than the preset confidence threshold, motion alignment is performed between the first image and historical images taken by a regular camera before the first image, using the motion information of the structured events. The two motion-aligned images are then superimposed, and the information redundancy in the temporal dimension is used to improve the equivalent SNR of flat areas, resulting in the second image. In a feasible implementation, SNR compensation can also be achieved using a global compensation approach. For example, the overall distribution information of structured events is used to perform global structure restoration and noise suppression on the first image, combined with a partitioning strategy for local refinement. It is understood that regardless of whether partition compensation or global compensation is used, as long as the signal-to-noise ratio compensation process utilizes the spatial structure and motion information provided by structured events, it falls within the protection scope of this application. Details of spatial confidence map generation and partition compensation are further explained in Embodiment 3.

[0044] Step S50: Reconstruct the image based on the event stream to obtain the third image.

[0045] It should be noted that image reconstruction refers to the process of converting an event stream accumulated within a time window into a pseudo-image, i.e., event-to-image reconstruction. For example, image reconstruction can be achieved using neural network-based reconstruction models, such as E2VID and FireNet, which learn a mapping function from the event stream to a grayscale image to generate a pseudo-image containing the spatial structure of the scene. In a feasible implementation, image reconstruction can also be achieved using traditional methods, such as accumulating events within the time window by pixel position and polarity into an event count map, or encoding the timestamps of the events into a time surface map to obtain a third image. The third image is updated at time intervals much smaller than the frame period of a normal camera, preserving the high temporal resolution motion information of the event camera; the third image does not strive to match the brightness of the real scene, but focuses on capturing spatial structure and high-frequency motion details. It is understood that the specific implementation of image reconstruction is not limited, as long as the reconstructed third image can express the spatial structure and motion information carried by the event stream, it falls within the scope of protection of this application.

[0046] Step S60: The second and third images are fused based on the proportion of structured events to obtain a fused image.

[0047] It should be noted that dual-modal fusion refers to the process of combining a second image corresponding to a regular camera with a third image corresponding to event reconstruction. The second image provides accurate absolute brightness reference and texture information, while the third image provides motion edge information with microsecond-level temporal resolution. For example, the fusion weights of the second and third images are determined based on the proportion of structured events; the larger the proportion of structured events, the larger the fusion weight of the second image and the smaller the fusion weight of the third image. The second and third images are then combined pixel-by-pixel according to the fusion weights to obtain the fused image. In one feasible implementation, the fusion weights can be determined using a continuous monotonic function mapping method, that is, inputting the proportion of structured events into a preset continuous monotonic function and outputting the fusion weights. For example, when the proportion changes between zero and one, the fusion weight of the third image monotonically decreases from near one to near zero. Alternatively, a lookup table method can be used, i.e., looking up a proportion-weight lookup table. The fusion weights are driven by the proportion of structured events, and the continuity of the proportion of structured events ensures a continuous transition of the fusion weights, avoiding abrupt changes in perception results caused by mode switching. The fused image is output at a fixed frequency for use by the embodied agent's perception-decision-control pipeline. Downstream tasks (object detection, pose estimation, motion planning, operation execution, etc.) receive the fused image using the same interface as in normal operation, without needing to perceive changes in lighting conditions, and behavioral strategies do not need to be adapted to changes in lighting. Details of the determination of fusion weights and weighted combination are further explained in Example 3.

[0048] It is understood that in this embodiment, the adjustment of exposure parameters and the determination of fusion weight are both driven by the same structured event proportion. When the scene brightness increases, the exposure parameters are shortened synchronously and the fusion weight of the third image is increased synchronously. The two are updated in conjunction with the same index, which avoids the parameter conflict between links caused by the decrease in exposure and the failure to update the fusion weight synchronously from the root, and ensures the quality stability of the perception result during the brightness change process.

[0049] In this embodiment, spatial correlation analysis of the event stream is used to classify events into structured and unstructured events. The proportion of structured events is used as a unified physical index to simultaneously drive the exposure parameter adjustment of the ordinary camera and dual-modal fusion. This achieves that exposure adjustment, signal-to-noise ratio compensation, and modal fusion are driven by the same index, and the parameters of each link are coordinated without conflict. The fused image has both the absolute brightness reference of the ordinary camera and the high dynamic range and high temporal resolution information of the event camera. It can continuously and stably output the perceived image in scenarios where the brightness dynamic range exceeds the capability of the ordinary camera, and the response speed reaches the microsecond level of the event camera.

[0050] Example 2 Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description, and will not be repeated hereafter.

[0051] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the second embodiment of the camera image processing method of this application. This embodiment provides supplementary explanations on the parameter settings and typical values ​​for event classification and determination, the judgment method for high dynamic range scenes, and the calibration process for exposure parameters.

[0052] Step S210: For any event among the events, determine the number of neighborhood events within the spatial neighborhood window centered on the pixel coordinates of the event.

[0053] It should be noted that the spatial neighborhood window refers to a defined spatial range centered on the pixel coordinates of the event to be determined. For example, it could be a circular area with a radius of five pixels centered on the pixel containing the event, or a 5×5 or 7×7 square area centered on the pixel containing the event. The number of neighborhood events refers to the number of other events besides the event to be determined that appear within the spatial neighborhood window, including events of the same polarity and opposite polarity. In one feasible implementation, the size of the spatial neighborhood window can be adaptively set according to the event camera resolution and the event rate: a smaller window is used when the event rate is high to avoid interference between events at adjacent edges, and a larger window is used when the event rate is low to ensure that there are enough events within the window for determination.

[0054] Step S220: Obtain the event density of the event based on the ratio of the number of neighborhood events to the area of ​​the spatial neighborhood window.

[0055] It's important to note that event density refers to the ratio of the number of neighborhood events within a spatial neighborhood window to the area of ​​the spatial neighborhood window (the number of pixels contained within the window), essentially indicating the density of the neighborhood event distribution. Higher event density indicates a stronger clustering of events around the event, making it more likely to be a structured event. For example, a typical structured event region triggered continuously along an object's edge typically has eight to fifteen neighborhood events clustered within a 5×5 window; while an unstructured event triggered by shot noise usually has only zero to two randomly occurring events in its neighborhood.

[0056] Step S230: When the event density is greater than a preset density threshold and the distribution direction consistency of neighboring events is greater than a preset consistency threshold, the event is classified as a structured event.

[0057] It should be noted that distribution direction consistency refers to the degree to which neighborhood events are concentrated in a certain direction in spatial distribution. For example, by calculating the covariance matrix of the pixel coordinates of neighborhood events and extracting the largest and second-largest eigenvalues, distribution direction consistency can be expressed as r = λ1 / λ2, where r represents distribution direction consistency, λ1 represents the largest eigenvalue of the covariance matrix, and λ2 represents the second-largest eigenvalue of the covariance matrix. When r is significantly greater than one, for example, r exceeds three, neighborhood events exhibit obvious spatial direction consistency, showing a linear distribution along the edge direction. The preset density threshold and preset consistency threshold can be set according to the actual scenario. For example, when the window area is 5×5, i.e., twenty-five pixels, the preset density threshold can be set to 0.2, i.e., at least five neighborhood events; the preset consistency threshold can be set to 3.0. In a feasible implementation, distribution direction consistency can also be calculated in other ways, such as performing principal component analysis on the neighborhood event coordinates and using the variance proportion of the first principal component to represent direction consistency; or performing linear fitting on the neighborhood events and using the fitting residuals to represent direction consistency. Typical examples of structured events include: continuous event bands triggered by the relative motion between the camera and the scene at the edges of objects in the scene, and clusters of related events formed in space by the gradual change of lighting on textured surfaces.

[0058] Step S240: If the event density is less than or equal to a preset density threshold, or the distribution direction consistency is less than or equal to a preset consistency threshold, the event is classified as an unstructured event.

[0059] It should be noted that events that do not meet either the density condition or the orientation consistency condition are classified as unstructured events, falling under complete binary classification. A typical example of unstructured events is photon shot noise triggered independently and randomly on each pixel under high-brightness conditions. These events have no spatial correlation and appear as isolated, random scattered points in the event stream. In one feasible implementation, if the event stream also contains other events that are neither structured nor unstructured, such as abnormal events triggered by hot pixels, these other events are not included in the calculation of the structured event proportion. The structured event proportion is still calculated as the ratio of the number of structured events to the sum of the number of structured and unstructured events, maintaining a one-to-one correspondence between the structured event proportion and the ratio of the two types of events. In another feasible implementation, the event stream can be preprocessed before classification, for example, by removing events exceeding the effective field of view of the event camera or events with abnormal timestamps, to improve classification accuracy.

[0060] In this embodiment, events are classified using the dual conditions of event density and distribution direction consistency. This accurately distinguishes between spatially continuous and related edge events and spatially random and isolated shot noise events, ensuring that the proportion of structured events can truly reflect the scene brightness and avoiding misclassification caused by a single condition.

[0061] Step S250: If the proportion of structured events is less than a preset threshold, determine that the event camera is in a high dynamic range scene.

[0062] It should be noted that the preset threshold refers to the percentage threshold used to determine whether a scene has entered a high dynamic range (HDR) state. For example, the preset threshold can be set to 0.3, corresponding to an illuminance of approximately 10,000 to 50,000 lux. A HDR scene refers to a scene where the scene brightness dynamic range exceeds the dynamic range capability of a normal camera. When the percentage of structured events is less than the preset threshold, it indicates that the scene brightness has increased to a level where a normal camera cannot capture images properly, and the sensing processing device determines that the event camera is in a HDR scene. In one feasible implementation, the preset threshold can be adaptively set based on historical operating data. For example, the long-term distribution of the percentage of structured events under normal lighting conditions can be statistically analyzed, and the low quantile of the distribution can be used as the preset threshold to avoid mismatch between fixed thresholds in different deployment environments. In another feasible implementation, to avoid frequent changes in the judgment result due to small fluctuations in the percentage of structured events around the preset threshold, the percentage of structured events can be smoothed first, and then compared with the preset threshold. For example, a moving average of the percentages over multiple consecutive preset durations can be used for the judgment.

[0063] Step S260: When the event camera is in a high dynamic range scene, perform signal-to-noise ratio compensation to obtain a second image.

[0064] It should be noted that when the event camera is determined to be in a high dynamic range scene, the sensing processing device performs signal-to-noise ratio (SNR) compensation on the first image to obtain the second image. The specific process of SNR compensation is the same as in Embodiment 1. It is understood that the adjustment of exposure parameters is performed continuously according to the proportion of structured events and does not depend on the scene judgment result; SNR compensation is only performed when the event camera is in a high dynamic range scene, thereby avoiding unnecessary compensation processing and reducing computational overhead in normal lighting scenes.

[0065] In this embodiment, the comparison between the proportion of structured events and a preset threshold determines whether the event camera is in a high dynamic range scene. If the event camera is in a high dynamic range scene, signal-to-noise ratio compensation is automatically triggered, realizing the on-demand triggering of the automatic detection and compensation process for high dynamic range scenes.

[0066] Step S270: Obtain the calibration reference ratio and the reference exposure parameters of the ordinary camera corresponding to the calibration reference ratio.

[0067] It should be noted that the calibration baseline proportion refers to the proportion of steady-state structured events measured under calibrated illuminance during the initial stage of system deployment, while the baseline exposure parameters refer to the exposure parameters required for a normal-brightness image to be obtained by a regular camera under calibrated illuminance. For example, during the initial stage of system deployment, using a standard uniform light source at multiple levels of known illuminance (e.g., 100 lux, 500 lux, 10000 lux, 5000 lux, 10,000 lux, 100,000 lux), the proportion of steady-state structured events at each illuminance level is recorded progressively, establishing a proportion-illuminance calibration comparison table. This table can be fitted with a monotonically decreasing proportion-illuminance curve; for example, the proportion and the logarithm of illuminance have an approximately linear inverse relationship. Simultaneously, the calibration baseline proportion and the corresponding baseline exposure parameters are measured under calibrated illuminance. In one feasible implementation, calibration can be performed offline at the factory, writing the calibration data into the memory of the sensing and processing device; it can also be performed online at the deployment site, using a standard light source with known illuminance to perform calibration after equipment installation; or it can be periodically recalibrated during operation to compensate for drift caused by sensor aging or environmental changes. During the operation phase, the equivalent illuminance estimate of the current scene can be obtained by looking up the table or substituting the fitted curve based on the proportion of real-time structured events, without relying on any additional photometer sensors.

[0068] Step S280: Determine the exposure parameters based on the ratio of the proportion of structured events to the proportion of the calibration baseline, and the baseline exposure parameters.

[0069] It should be noted that the ratio of the proportion of structured events to the proportion of the calibration baseline is used as the scaling factor, and the product of the scaling factor and the baseline exposure parameter is determined as the exposure parameter. The reduction in the exposure parameter is proportional to the brightness intensity. For example, taking exposure time as an example, when the current proportion of structured events drops to half of the calibration baseline proportion, the exposure time is reduced to half of the baseline exposure time. This continuous exposure adjustment, without sudden on / off changes, ensures a smooth transition in perceived output between different brightness scenes. In one feasible implementation, when the scene brightness changes drastically, the scaling factor can be smoothed using an exponential moving average to avoid instantaneous jumps in the exposure parameter. In another feasible implementation, when the exposure parameter includes exposure time and gain, the exposure time can be adjusted first. After the exposure time is adjusted to its lower limit, further exposure control is achieved by adjusting the gain.

[0070] In this embodiment, a mapping relationship between the proportion of structured events and exposure parameters is established through calibration, so that the exposure parameters are continuously adjusted according to the proportion of structured events, and the perception output transitions smoothly between different brightness scenes, avoiding the abrupt change in perception results caused by the switching abrupt change of exposure parameters.

[0071] Example 3 Based on the first and / or second embodiments of this application, in the third embodiment of this application, the contents that are the same as or similar to those in the first and second embodiments described above can be referred to the above description and will not be repeated hereafter.

[0072] like Figure 3 As shown, Figure 3 This is a flowchart of the camera image processing method according to Embodiment 3 of this application. This embodiment provides supplementary explanations on the specific implementation details of signal-to-noise ratio compensation and dual-modal fusion.

[0073] Step S310: Generate a spatial confidence map based on the spatial distribution of structured events. The confidence level of each location in the spatial confidence map represents the reliability of signal-to-noise ratio compensation at the corresponding location in the first image.

[0074] It should be noted that the spatial confidence map is a two-dimensional matrix with the same resolution as the image, representing the reliability of signal-to-noise ratio compensation at each pixel location in the low-exposure first image. Pixels falling on the edge contours of structured events are assigned high confidence because the gradient information at the corresponding location can be accurately recovered from the event stream; pixels falling on flat areas covered by unstructured events are assigned low confidence because they lack the additional structural constraints provided by the events. The spatial confidence map is generated in real time from the spatial distribution of the event stream and dynamically updated with the scene, without a fixed pattern. In one feasible implementation, the confidence can be represented by a continuous value between zero and one, for example, using the density of structured events in the pixel's neighborhood as the confidence; it can also be represented binaryly, with a confidence of one at locations covered by structured events and zero at other locations. In one feasible implementation, the update frequency of the spatial confidence map is consistent with the update frequency of the event packets, ensuring that the confidence map always reflects the latest spatial distribution of structured events.

[0075] Step S320: At positions where the confidence level is greater than a preset confidence threshold, texture enhancement is performed on the corresponding positions in the first image based on the timestamp of the structured event and the polarity of the brightness change.

[0076] It should be noted that texture enhancement refers to using the timestamps and brightness change polarity information of structured events to estimate the sub-pixel displacement and gradient direction of each edge in the first image, and superimposing the high temporal resolution gradient information in the event stream onto the corresponding positions in the first image. Essentially, it replaces the motion blur caused by noise in the first image with the high temporal precision motion information of the event stream, rather than simply filtering and denoising the first image. For example, a preset confidence threshold can be set according to the actual scene; the position where the confidence level is greater than the preset confidence threshold is the position covered by the event edge. In one feasible implementation, the estimation of the edge gradient direction can be achieved using a structure tensor, i.e., calculating the local structure tensor from the spatial distribution of structured events, and using the direction of the principal feature vector of the structure tensor as the gradient direction; or it can be achieved using event polarity, i.e., events on both sides of the same edge have opposite polarities, and the gradient direction is determined based on the polarity distribution. In one feasible implementation, the estimation of sub-pixel displacement can be achieved using the timestamp difference between adjacent events and the edge normal velocity, i.e., determining the edge motion velocity by the time interval between adjacent events on the edge and the distance between the event intervals, and then combining the temporal alignment relationship between ordinary camera frames and events to obtain the sub-pixel displacement.

[0077] Step S330: At positions where the confidence level is less than a preset confidence threshold, based on the motion information of the structured event, the first image and historical images captured by a regular camera before the first image are motion aligned, and the motion-aligned first image and the motion-aligned historical images are superimposed to obtain the second image.

[0078] It should be noted that motion alignment refers to estimating pixel motion vectors in flat regions using motion information from structured events, and aligning pixels of the same scene point in the first image with those in historical images. Overlay refers to weighted superposition of the motion-aligned first image and the motion-aligned historical images at the same pixel location, i.e., multi-frame temporal integration in the time dimension. The weights can be determined based on the time interval between the historical and first images or the motion alignment quality, or they can be added with equal weights. For example, the motion vectors are directly provided by the event stream, eliminating the need for additional optical flow calculations and avoiding the motion blur problem caused by uncompensated motion in traditional multi-frame integration. Utilizing the information redundancy in the time dimension for multi-frame integration can improve the equivalent signal-to-noise ratio in flat regions. In one feasible implementation, the weights for weighted superposition can be set using a time decay method, i.e., historical images closer to the first image in time have a larger weight; the number of historical image frames participating in the superposition can be adaptively determined based on the intensity of scene motion—the more intense the scene motion, the fewer frames participate in the superposition—to avoid the accumulation of motion alignment errors.

[0079] In this embodiment, a spatial confidence map is generated based on the spatial distribution of structured events. At locations where the confidence level is greater than a preset confidence threshold, the texture is enhanced with event gradient information. At locations where the confidence level is less than the preset confidence threshold, multiple frames are aligned with event motion information and superimposed for integration. This achieves spatial adaptation and temporal dynamics of signal-to-noise ratio compensation, transforming signal-to-noise ratio compensation from a globally unified fixed operation to a differentiated operation, thereby restoring image quality under low exposure conditions.

[0080] Step S340: Determine the fusion weights of the second and third images based on the proportion of structured events.

[0081] It should be noted that the fusion weight refers to the respective weight of the second and third images during fusion. When the proportion of structured events is higher, the fusion weight of the second image is greater, and the second image dominates the fusion process, while the third image is only used to enhance the sharpness of motion edges. When the proportion of structured events decreases, the fusion weight of the second image smoothly decreases, while the fusion weight of the third image smoothly increases. When the proportion of structured events drops to an extremely low value, the fusion is dominated by the third image, and the second image only provides a reference for the absolute brightness of dark areas after signal-to-noise ratio compensation. The transition of the fusion weight is continuous and directly determined by the proportion of structured events, eliminating the need for manually setting weights or training a weight prediction network, thus avoiding abrupt changes in perceptual results caused by mode switching. In one feasible implementation, the fusion weight can be determined using a continuous monotonic function mapping. For example, the proportion of structured events can be input into a preset continuous monotonic function, and the fusion weight of the third image can be output; the closer the proportion is to zero, the closer the fusion weight of the third image is to one. Alternatively, it can be determined using a lookup table, i.e., by querying an offline proportion-weight lookup table. In one feasible implementation, the sum of the fusion weights of the second and third images is one, and the fusion weights correspond one-to-one with the proportion of structured events, ensuring that there are no jumps in the fusion output.

[0082] Step S350: The second image and the third image are fused according to the fusion weights to obtain a fused image.

[0083] It should be noted that the perception processing device weights and combines the second and third images pixel-by-pixel according to fusion weights to obtain the fused image. The fused image simultaneously contains the high dynamic range and high temporal resolution information of the event camera, the absolute brightness reference and texture information of the ordinary camera, and the complementary redundancy of the two in the weight transition region. The fused image is output at a fixed frequency for use by the embodied agent's perception-decision-control pipeline. Downstream tasks receive the fused image using the same interface as in normal operation, and the internal perception mode switching driven by the structured event proportion is completely transparent to all downstream tasks.

[0084] Step S360: Determine the transition speed of the fusion weights based on the rate of change of the proportion of structured events over time.

[0085] It should be noted that the time-series trajectory of the proportion of structured events reflects changes in the scene's lighting conditions: a stable high proportion indicates normal lighting conditions, a rapid decrease indicates the transition to a high-brightness scene, a low proportion indicates the high-brightness scene is ongoing, and a rebound indicates the end of the high-brightness scene. Transition speed refers to how quickly the fusion weights transition from one value to another, controlled by the rate of change of the proportion over time: the more drastic the change in proportion, such as suddenly exiting a tunnel and facing direct sunlight, the faster the transition speed for rapid adaptation; the more gradual the change in proportion, such as the gradual change at sunset, the slower the transition speed to reduce unnecessary parameter fluctuations. In one feasible implementation, the transition speed can be expressed as the adjustment step size of the fusion weights in each update cycle; the greater the rate of change, the larger the adjustment step size. Alternatively, it can be expressed as the time constant for adjusting the fusion weights; the greater the rate of change, the smaller the time constant.

[0086] Step S370: Adjust the fusion weight to a value corresponding to the proportion of structured events according to the transition speed.

[0087] It should be noted that the sensing and processing device gradually adjusts the fusion weights to values ​​corresponding to the current proportion of structured events according to the transition speed. The smoothness of the transition is naturally guaranteed by the continuity of the proportion itself, meaning the entire adjustment process is synchronized with the acquisition of the structured event proportion. The entire process is driven entirely by the physical signals themselves, without hyperparameter adjustment, manual thresholding, or state machine logic, ensuring the continuous and stable perception-decision-control pipeline of the embodied intelligent agent during sudden changes in illumination. In one feasible implementation, as the scene gradually recovers from high brightness to normal illumination, the proportion of structured events rebounds, and the fusion weights gradually tilt towards the second image as the proportion recovers. The fused image smoothly recovers to being dominated by the ordinary camera image. The entire process is synchronized with the change in proportion, and there are no sudden changes caused by mode switching boundaries.

[0088] In this embodiment, the fusion weight changes continuously with the proportion of structured events, and the transition speed matches the rate of change of the proportion over time. It can quickly adapt to sudden changes in illumination and avoid parameter fluctuations when illumination changes gradually, thus realizing a fully continuous sensing mode switching based on physical indicators and ensuring the continuity and stability of the fused image during illumination changes.

[0089] Step S380: Normalize the second and third images.

[0090] It should be noted that normalization processing refers to adjusting the brightness distribution of the second and third images to a consistent numerical range, eliminating the difference in brightness dimensions between the two modalities, thus facilitating subsequent registration and fusion. For example, normalization processing can employ mean-variance normalization, adjusting the pixel values ​​of the two images to a distribution with a mean of zero and a variance of one; it can also employ maximum-minimum normalization, scaling the pixel values ​​of the two images to the range of zero to one; or it can employ histogram matching, matching the brightness histogram of the third image to the brightness histogram of the second image. It is understood that the specific method of normalization processing is not limited, as long as it eliminates the difference in brightness distribution between the two modalities, it falls within the scope of protection of this application.

[0091] Step S390: Register the normalized second image with the normalized third image.

[0092] It should be noted that registration refers to spatially aligning a normalized second image with a normalized third image, ensuring that points of the same scene in both images are located at the same pixel position. For example, registration can be achieved based on edge features in the two images: extracting edge feature points and matching them, calculating the transformation relationship between the two images based on the matching results, and aligning the two images according to the transformation relationship; it can also be achieved based on calibration extrinsic parameters between an event camera and a regular camera: using a pre-calibrated rotation and translation relationship between the two cameras to project the third image onto the pixel coordinate system of the second image; or it can be achieved based on mutual information: maximizing the mutual information between the two images to determine the optimal alignment transformation. After registration, the second image provides accurate absolute brightness reference and texture information, while the third image provides motion edge information with microsecond-level temporal resolution.

[0093] Step S400: Based on the proportion of structured events, the registered second image and the registered third image are fused to obtain a fused image.

[0094] It should be noted that the sensing processing device performs fusion weight determination and weighted combination on the registered second image and the registered third image based on the proportion of structured events to obtain the fused image. The specific method of fusion weight determination and weighted combination is the same as in Embodiment 1, and will not be repeated here.

[0095] In this embodiment, by normalizing and registering the second and third images, the differences in brightness distribution and spatial position between the two modes are eliminated, so that the fusion is performed on the basis of accurate pixel-by-pixel correspondence, thereby improving the fusion accuracy.

[0096] To illustrate the linkage process of each stage of this application, let's take the example of an embodied intelligent agent exiting a tunnel: Before exiting the tunnel, the proportion of structured events remains high, the exposure parameters are close to the baseline exposure parameters, and the fusion is dominated by the second image, resulting in a fused image that closely resembles an ordinary camera image. At the moment of exiting the tunnel, the scene brightness suddenly increases, photon shot noise triggers a large number of unstructured events, and the proportion of structured events rapidly decreases. The sensing and processing device simultaneously shortens the exposure time of the ordinary camera to avoid overexposure and determines that the event camera is in a high dynamic range scene, triggering signal-to-noise ratio compensation. Simultaneously, the fusion weight of the third image increases rapidly, and the fused image smoothly switches to being dominated by the third image, ensuring uninterrupted sensing output during moments of intense light. During the period of sustained high brightness outside the tunnel, the proportion of structured events remains low, and the fusion is stable with the third image as the main component, while the second image provides an absolute brightness reference for dark areas. When re-entering the tunnel, the proportion of structured events rebounds, the exposure parameters and fusion weight continuously recover with the proportion, and the fused image smoothly returns to being dominated by the second image. Throughout the process, exposure adjustment and fusion weights are driven by the same physical index, parameters are updated synchronously, the sensing output remains unchanged, and the downstream decision-making and control pipelines do not need to sense changes in lighting conditions.

[0097] It should be noted that the technical features in the above embodiments can be combined arbitrarily according to actual needs. As long as the combined technical solutions do not contradict each other, they all fall within the protection scope of this application. This application does not make specific limitations on the combination methods between the various embodiments.

[0098] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the camera image processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0099] This application also provides a camera image processing apparatus, please refer to... Figure 4 The camera image processing device includes: The event classification module 10 is used to acquire the event stream output by the event camera within a preset time period, and to perform spatial correlation analysis on each event in the event stream to classify each event into structured events and unstructured events. The event proportion calculation module 20 is used to calculate the proportion of the number of structured events in the sum of the number of structured events and the number of unstructured events, as the proportion of structured events in the event flow within a preset duration. The first image acquisition module 30 is used to adjust the exposure parameters of the ordinary camera according to the proportion of structured events, and control the ordinary camera to acquire the first image according to the adjusted exposure parameters; The second image acquisition module 40 is used to perform signal-to-noise ratio compensation on the first image based on structured events to obtain the second image; The third image acquisition module 50 is used to perform image reconstruction based on the event stream to obtain a third image; The image fusion acquisition module 60 is used to fuse the second image and the third image based on the proportion of structured events to obtain a fused image.

[0100] The camera image processing apparatus provided in this application, employing the camera image processing method in the above embodiments, can solve the technical problem of how to obtain usable perceived images in a timely, continuous, and stable manner when the dynamic range of scene brightness exceeds the capabilities of ordinary cameras. Compared with the prior art, the beneficial effects of the camera image processing apparatus provided in this application are the same as those of the camera image processing method provided in the above embodiments, and other technical features in the camera image processing apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0101] This application provides a camera image processing apparatus, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the camera image processing method in the first embodiment described above.

[0102] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of a camera image processing device suitable for implementing embodiments of this application. The camera image processing device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The camera image processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0103] like Figure 5As shown, the camera image processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the camera image processing device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the camera image processing device to communicate wirelessly or wiredly with other devices to exchange data. Although various systems are shown in the figure, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0104] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0105] The camera image processing device provided in this application, employing the camera image processing method in the above embodiments, can solve the technical problem of how to obtain usable perceived images in a timely, continuous, and stable manner when the dynamic range of scene brightness exceeds the capabilities of ordinary cameras. Compared with the prior art, the beneficial effects of the camera image processing device provided in this application are the same as those of the camera image processing method provided in the above embodiments, and other technical features in this camera image processing device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0106] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0107] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0108] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to perform the camera image processing method in the above embodiments.

[0109] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0110] The aforementioned computer-readable storage medium may be included in the camera image processing device; or it may exist independently and not assembled into the camera image processing device.

[0111] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the camera image processing device, cause the camera image processing device to: The event stream output by the event camera within a preset duration is acquired, and spatial correlation analysis is performed on each event in the event stream to classify each event into structured events and unstructured events. The proportion of structured events in the total number of structured events and unstructured events is calculated and used as the proportion of structured events in the event stream within a preset duration. The exposure parameters of the ordinary camera are adjusted according to the proportion of structured events, and the ordinary camera is controlled to acquire the first image according to the adjusted exposure parameters. The signal-to-noise ratio of the first image is compensated based on the structured events to obtain the second image; Image reconstruction is performed based on the event stream to obtain a third image; The second and third images are fused based on the proportion of structured events to obtain a fused image.

[0112] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0114] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0115] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described camera image processing method. This solves the technical problem of how to obtain usable perceived images in a timely, continuous, and stable manner when the scene brightness dynamic range exceeds the capabilities of ordinary cameras. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the camera image processing method provided in the above embodiments, and will not be repeated here.

[0116] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the camera image processing method described above.

[0117] The computer program product provided in this application can solve the technical problem of how to obtain usable perceived images in a timely, continuous, and stable manner when the dynamic range of scene brightness exceeds the capabilities of ordinary cameras. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the camera image processing method provided in the above embodiments, and will not be repeated here.

[0118] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A camera image processing method, characterized in that, The method includes: The event stream output by the event camera within a preset time period is acquired, and spatial correlation analysis is performed on each event in the event stream to classify each event into structured events and unstructured events. The proportion of the number of structured events to the sum of the number of structured events and the number of unstructured events is calculated as the proportion of structured events in the event stream within the preset duration. The exposure parameters of the ordinary camera are adjusted according to the proportion of the structured events, and the ordinary camera is controlled to acquire the first image according to the adjusted exposure parameters; The first image is subjected to signal-to-noise ratio compensation based on the structured events to obtain the second image; Based on the event stream, image reconstruction is performed to obtain a third image; The second image and the third image are fused based on the proportion of structured events to obtain a fused image.

2. The method as described in claim 1, characterized in that, The step of performing signal-to-noise ratio compensation on the first image based on the structured events to obtain the second image includes: A spatial confidence map is generated based on the spatial distribution of the structured events. The confidence level at each location in the spatial confidence map represents the reliability of signal-to-noise ratio compensation at the corresponding location in the first image. At the location where the confidence level is greater than a preset confidence threshold, texture enhancement is performed on the corresponding location in the first image based on the timestamp of the structured event and the polarity of the brightness change. At the position where the confidence level is less than the preset confidence threshold, based on the motion information of the structured event, the first image and the historical images taken by the ordinary camera before the first image are motion aligned, and the motion-aligned first image and the motion-aligned historical images are superimposed to obtain the second image.

3. The method as described in claim 1, characterized in that, The step of performing signal-to-noise ratio compensation on the first image based on the structured events to obtain the second image includes: If the proportion of structured events is less than a preset threshold, it is determined that the event camera is in a high dynamic range scene; When the event camera is in the high dynamic range scene, the signal-to-noise ratio compensation is performed to obtain the second image.

4. The method as described in claim 1, characterized in that, The process of fusing the second image and the third image based on the proportion of structured events to obtain a fused image includes: The fusion weights of the second image and the third image are determined based on the proportion of structured events. The second image and the third image are fused according to the fusion weights to obtain the fused image; The larger the proportion of structured events, the smaller the fusion weight of the third image, and the larger the fusion weight of the second image.

5. The method as described in claim 4, characterized in that, The step of determining the fusion weight between the second image and the third image based on the proportion of structured events includes: The transition speed of the fusion weight is determined based on the rate of change of the proportion of structured events over time. According to the aforementioned transition speed, the fusion weight is adjusted to a value corresponding to the proportion of the structured events; The greater the rate of change, the greater the transition speed.

6. The method as described in claim 1, characterized in that, The step of performing spatial correlation analysis on each event in the event stream to classify the events into structured events and unstructured events includes: For any of the events, determine the number of neighborhood events within a spatial neighborhood window centered on the pixel coordinates of the event; The event density is obtained by the ratio of the number of neighborhood events to the area of ​​the spatial neighborhood window. If the event density is greater than a preset density threshold and the distribution direction consistency of the neighborhood events is greater than a preset consistency threshold, the event is classified as the structured event. If the event density is less than or equal to the preset density threshold, or if the distribution direction consistency is less than or equal to the preset consistency threshold, the event is classified as an unstructured event.

7. The method as described in claim 1, characterized in that, The step of adjusting the exposure parameters of the ordinary camera based on the proportion of structured events includes: Obtain the calibration reference ratio of the ordinary camera and the reference exposure parameters corresponding to the calibration reference ratio; The exposure parameters are determined based on the ratio of the structured event proportion to the calibration benchmark proportion, and the benchmark exposure parameters. The smaller the proportion of structured events, the shorter the exposure time corresponding to the exposure parameters.

8. The method as described in claim 1, characterized in that, The process of fusing the second image and the third image based on the proportion of structured events to obtain a fused image includes: The second image and the third image are normalized. The normalized second image and the normalized third image are registered together; Based on the proportion of structured events, the registered second image and the registered third image are fused to obtain the fused image.

9. A camera image processing device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the camera image processing method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the camera image processing method as described in any one of claims 1 to 8.

Images