Image processing method and device based on event camera, equipment and medium

Abstract

The invention belongs to the technical field of image processing, and discloses an event camera-based image processing method and device, equipment and a medium. The method comprises the following steps: acquiring event data of a blurred image collected by an event camera; constructing an initial PINN model, and embedding an event generation equation in the initial PINN model; inputting the event data into the PINN model to obtain a predicted brightness value, performing time derivative self-supervision based on the predicted brightness value and an actual brightness value, introducing Tikhonov regularization constraint conditions to optimize the PINN model, and determining an optimized PINN model; inputting event data of an image to be processed into the optimized PINN model to obtain logarithmic brightness values of three color channels of the image to be processed; the logarithmic luminance values are converted into reconstructed image frames using tone mapping. Noise and artifacts of a reconstructed image can be reduced, and the quality of a model reconstructed image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, equipment and medium based on an event camera. Background Art

[0002] With the continuous advancement of computer vision technology, traditional frame cameras have encountered some challenges in processing dynamic scenes and high dynamic range (HDR) environments. To overcome these challenges, event cameras have emerged as a new type of sensor. With its low power consumption, high dynamic range and high temporal resolution, it has gradually attracted attention and become a new hotspot in computer vision research. Unlike traditional cameras, event cameras can capture instantaneous changes in scenes with a temporal resolution of microseconds and generate sparse and asynchronous event stream data. This data format is significantly different from traditional frame images, bringing new challenges to information processing and algorithm design.

[0003] Existing event stream processing methods mainly rely on supervised learning, which requires a large amount of labeled data to train the model. However, the synthetic training data often differs from the real scene, resulting in unsatisfactory performance of the algorithm in practical applications. In addition, existing methods often lack robustness to noise when dealing with complex lighting conditions and high dynamic range scenes, affecting the quality of the reconstruction results. Summary of the invention

[0004] To this end, the embodiments of the present application provide an image processing method, apparatus, device and medium based on an event camera, which reduces the dependence on labeled data and can reconstruct blurred images with high quality.

[0005] In a first aspect, the present application provides an image processing method based on an event camera.

[0006] This application is achieved through the following technical solutions:

[0007] An image processing method based on an event camera, the image processing method comprising:

[0008] Obtaining event data of the blurred image captured by the event camera;

[0009] Constructing an initial PINN model, embedding an event generation equation in the initial PINN model, wherein an input item of the initial PINN model is event data, and an output item is a predicted brightness value that satisfies the event generation equation;

[0010] The event data of the blurred image is input into the initial PINN model to obtain the predicted brightness value of the event data. The time derivative is self-supervised based on the predicted brightness value and the actual brightness value, and the Tikhonov regularization constraint is introduced to optimize the initial PINN model. When the prediction performance of the PINN model reaches the preset standard, the optimized PINN model is determined.

[0011] Input the event data of the image to be processed into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed;

[0012] Tone mapping is used to transform the logarithmic brightness values ​​of the three color channels into the reconstructed image frame.

[0013] In a preferred example of the present application, it can be further configured that tone mapping is used to convert the logarithmic brightness values ​​of the three color channels into a reconstructed image frame, including:

[0014] The logarithmic brightness values ​​of the three color channels are converted into HDR range brightness values ​​through exponential functions;

[0015] The brightness and contrast of the image are adjusted by the Reinhard function, and the HDR range brightness value is converted to the LDR range brightness value;

[0016] A reconstructed image frame is generated based on the LDR range of brightness values.

[0017] In a preferred example of the present application, it can be further configured that the event data of the image to be processed is input into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed, including:

[0018] The time coordinates of the event data are input into the optimized PINN model to obtain the logarithmic brightness value of the red channel, the logarithmic brightness value of the green channel, and the logarithmic brightness value of the blue channel.

[0019] In a preferred example of the present application, it can be further configured that the hidden layer of the PINN model is a multi-layer fully connected neural network, and the initial PINN model uses randomized parameter values.

[0020] In a preferred example of the present application, it can be further configured to perform self-supervision of the time derivative based on the predicted brightness value and the actual brightness value, including:

[0021] Calculate the brightness change difference between the predicted brightness value output by the PINN model and the actual brightness value calculated by the event generation equation:

[0022] The mean square error loss function is determined according to the brightness change difference, and the time derivative of the PINN model is self-supervised based on the mean square error loss function.

[0023] In a preferred example of the present application, it can be further set to introduce Tikhonov regularization constraints to optimize the initial PINN model, including:

[0024] The spatial gradient of logarithmic brightness is constrained by Tikhonov regularization, which is expressed as:

[0025]

[0026] in, represents the regularization constraint, represents the derivative in the x dimension, Represents the derivative in the y dimension.

[0027] In a preferred example of the present application, it can be further set as follows:

[0028] The event generation equation is formulated as:

[0029]

[0030] Where L represents the brightness change, P i represents the direction of brightness change, θ represents the brightness change threshold that triggers the event, and δ(tt i ) represents the Dirac function, t1 and t2 represent the time coordinates.

[0031] In a second aspect, the present application provides an image processing device based on an event camera.

[0032] This application is achieved through the following technical solutions:

[0033] An image processing device based on an event camera, used to execute the image processing method based on an event camera according to the first aspect above, the image processing device comprising:

[0034] A data acquisition module, used to acquire event data of the blurred image captured by the event camera;

[0035] A model building module, used to build an initial PINN model, in which an event generation equation is embedded, wherein the input item of the initial PINN model is event data, and the output item is a predicted brightness value that satisfies the event generation equation;

[0036] A model optimization module is used to input the event data of the blurred image into the initial PINN model to obtain the predicted brightness value of the event data, perform self-supervision of the time derivative based on the predicted brightness value and the actual brightness value, and introduce Tikhonov regularization constraints to optimize the initial PINN model. When the prediction performance of the PINN model reaches the preset standard, the optimized PINN model is determined;

[0037] A brightness value prediction module is used to input the event data of the image to be processed into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed;

[0038] The tone mapping module is used to convert the logarithmic brightness values ​​of three color channels into a reconstructed image frame by using tone mapping.

[0039] On the third aspect, the present application is realized through the following technical solutions:

[0040] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned image processing methods based on event cameras when executing the computer program.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium.

[0042] This application is achieved through the following technical solutions:

[0043] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned image processing methods based on event cameras are implemented.

[0044] In summary, compared with the prior art, the technical solution provided by the embodiment of the present application has at least the following beneficial effects: the present application obtains the event data of the blurred image collected by the event camera; constructs an initial PINN model, and embeds the event generation equation in the initial PINN model; inputs the event data of the blurred image into the initial PINN model to obtain the predicted brightness value of the event data, performs self-supervision of the time derivative based on the predicted brightness value and the actual brightness value, and introduces the Tikhonov regularization constraint condition to optimize the initial PINN model, and when the prediction performance of the PINN model reaches the preset standard, determines the optimized PINN model; inputs the event data of the image to be processed into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed; and uses tone mapping to convert the logarithmic brightness values ​​of the three color channels into reconstructed image frames. This application uses the PINN model to directly solve the physical equations in the event generation model through self-supervised learning, reducing the dependence on a large amount of labeled data, thereby significantly reducing the cost and time investment of data annotation, and improving the applicability of the model in real scenarios; using the dual constraints of time derivative self-supervision and spatial regularization, the prediction error of the model is improved, the noise and artifacts of the reconstructed image are reduced, and the quality of the model-reconstructed image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of an image processing method based on an event camera provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of the structure of an image processing device based on an event camera provided in one embodiment of the present application;

[0047] Description of reference numerals:

[0048] Data acquisition module-101, model construction module-201, model optimization module-301, brightness value prediction module-401, tone mapping module-501. DETAILED DESCRIPTION

[0049] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed, but such modifications are protected by the patent law as long as they are within the scope of the claims of the present application.

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0052] In this application, the terms "first", "second", etc. are used to distinguish identical or similar items with basically the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is there any limitation on quantity and execution order.

[0053] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings of the specification. The image processing method based on the event camera of the present application can be applied to multiple application scenarios such as low-light enhancement of autonomous driving, image de-shaking, and efficient compression of video data.

[0054] like Figure 1 As shown, the image processing method based on event camera proposed in this application includes:

[0055] S10: Obtain event data of the blurred image captured by the event camera.

[0056] An event camera is a new type of visual sensor. Different from traditional cameras that capture images continuously, each pixel in an event camera is an independent sensor that can respond to changes in brightness signals. When the brightness change of a pixel exceeds a preset threshold, an event is triggered, and a series of sparse and asynchronous event sequences (also called event streams) are generated. Event cameras output data asynchronously in units of events, without waiting for the acquisition and transmission of the entire frame, so they have extremely low latency. In the actual implementation process, any event camera from DAVIS346, DVXplorer, PROPHESEE Metavision EVK4 or PROPHESEE Metavision EVK5 can be used. Each event in the event sequence is represented as: (x i ,y i ,t i ,P i ), where x i Indicates the horizontal coordinate of the pixel point of the current event i, y i Indicates the vertical coordinate of the pixel point of the current event i, t i represents the time coordinate corresponding to the current event i, P i Indicates the direction (also called polarity) of the brightness change of the pixel point of the current event i, P i Indicates brightness increase or brightness decrease. i The value of can be negative one or positive one, P i When the value of is negative one, it means that the brightness of the pixel decreases; i When the value of is positive one, it means that the brightness of the pixel increases.

[0057] S20: construct an initial PINN model, embed an event generation equation in the initial PINN model, the input item of the initial PINN model is the event data, and the output item is the predicted brightness value that satisfies the event generation equation.

[0058] First, this application constructs an initial PINN model (Physics-Informed Neural Networks, physical information neural network model). The PINN model takes event data as input, specifically the timestamp (i.e., time coordinate) of the current event of the input event data; the output is the predicted brightness value of the three color channels corresponding to the pixel point of the current event, specifically the predicted brightness value of the red channel, the predicted brightness value of the green channel, and the predicted brightness value of the blue channel. The structure of the PINN model includes an input layer, a hidden layer, and an output layer, wherein the hidden layer is a multi-layer fully connected neural network consisting of multiple layers, each layer containing multiple neurons. When constructing the initial PINN model, randomized parameter values ​​are used. Later in the training process, the parameter values ​​of the PINN model are optimized and adjusted.

[0059] In some embodiments, in order to simplify the analysis, the present application assumes that the brightness of a single pixel changes under fixed spatial coordinates, that is, ignores the change of the pixel position in the image, and only focuses on the brightness value of each pixel. The brightness of each pixel can change, and the difference in pixel brightness represents different colors. The Dirac function δ can be used to describe the event e on the pixel in the event data i :

[0060] e i =P i θδ(tt i ), θ represents the brightness change threshold of the trigger event;

[0061] The brightness increment can be expressed as the accumulation of events on the pixel within a preset time interval [t1, t2]:

[0062] ΔL=L(t2)-L(t1),

[0063]

[0064] Assuming that the time interval [t1, t2] is short and the noise is negligible, the nonlinear brightness change is converted into a linear form using Taylor expansion, that is, the first-order time derivative of the brightness increment, which represents the rate of change of logarithmic brightness over time:

[0065]

[0066] Then the event generation equation can be expressed as:

[0067]

[0068] Where L represents the brightness change, P i represents the direction of brightness change, θ represents the brightness change threshold that triggers the event, and δ(tt i) represents the Dirac function, t1 and t2 represent the time coordinates.

[0069] The event generation equation is embedded into the PINN model and used as a physical constraint when the PINN model processes event data, so that the output result of the model satisfies the constraint conditions of the physical equation.

[0070] When constructing the initial PINN model, the PINN model uses randomized parameter values. The PINN model represents complex information through a parameterized fully connected multilayer perceptron (MLP). By incorporating the above event generation equation into the training process of the neural network, it guides the learning of the network and enables effective training and prediction even when data is scarce or noisy.

[0071] S30: Input the event data of the blurred image into the initial PINN model to obtain the predicted brightness value of the event data, perform self-supervision of the time derivative based on the predicted brightness value and the actual brightness value, and introduce the Tikhonov regularization constraint condition to optimize the initial PINN model. When the prediction performance of the PINN model reaches the preset standard, determine the optimized PINN model.

[0072] In some embodiments, the predicted brightness value output by the PINN model is directly minimized The brightness change difference between the brightness value ΔL calculated by the event generation equation:

[0073]

[0074] The mean square error loss function is determined according to the brightness change difference, and the PINN model is self-supervised on the time derivative based on the mean square error loss function:

[0075]

[0076] Although the temporal derivative self-supervision can estimate the brightness function, considering that the PINN model has no prior knowledge of the initial brightness value of each pixel, in order to eliminate the unnatural artifacts that may be contained in the estimation result, this application further introduces the Tikhonov regularization constraint in the loss function to keep the model output result in the natural image space. The specific steps are:

[0077] The spatial gradient of logarithmic brightness is constrained by Tikhonov regularization, which is expressed as:

[0078]

[0079] in, represents the regularization constraint, represents the derivative in the x dimension, Represents the derivative in the y dimension.

[0080] Tikhonov regularization limits the complexity of the model by penalizing large parameter changes, which helps the model perform on unknown data and reduces the risk of overfitting. At the same time, compared with some complex regularization methods, such as CNN denoisers, Tikhonov regularization is computationally simpler and more efficient, does not require large-scale network models, and thus significantly reduces training time.

[0081] The overall objective function of the PINN model during training is: The PINN model is trained based on the objective function by gradient descent. When the value of the objective function drops to a preset threshold, or when the value of the objective function no longer drops significantly after a certain number of epochs of training, the model can be considered to have converged and the training can be terminated.

[0082] S40: Inputting the event data of the image to be processed into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed.

[0083] The event data of the image to be processed is input into the optimized PINN model, and the PINN model predicts the brightness values ​​of the three color channels (red channel, green channel, and blue channel) based on the time coordinate t of the event data. The PINN model constructed in this application is not limited to a single brightness prediction, but can be extended to brightness prediction of three color channels (RGB).

[0084] S50: Using tone mapping to convert the logarithmic brightness values ​​of the three color channels into a reconstructed image frame.

[0085] Specifically, the logarithmic brightness values ​​of the three color channels are converted into HDR (high dynamic range) range brightness values ​​through an exponential function:

[0086] I(x,y,t)=exp(F Θ (x,y,t));

[0087] The brightness and contrast of the image are adjusted by the Reinhard function, and the logarithmic brightness value of the HDR range is converted to the logarithmic brightness value of the LDR range. The specific expression is:

[0088]

[0089] I represents the logarithmic brightness value in the HDR range, Γ(I) represents the logarithmic brightness value in the LDR range, and γ represents a hyperparameter for controlling the contrast;

[0090] Generate a reconstructed image frame based on the LDR range brightness values.

[0091] In some feasible embodiments, an event camera is mounted on an autonomous driving vehicle, and the position of the event camera is adjusted to the front of the autonomous driving vehicle, so as to more realistically obtain the images obtained by the event camera during the driving process of the autonomous driving vehicle. During the night driving process, the event camera captures event data in a low-light environment, and the event data contains the brightness change information of roads, pedestrians, and vehicles during the night driving process. First, the event data is used as input data and input into a physical neural network model (PINN model). The PINN model extracts features from the event data, and optimizes the PINN model through time derivative self-supervision and spatial regularization constraints to minimize the mean square error between the predicted brightness change and the actual event data, while reducing noise and artifacts in the reconstructed image. The optimized PINN model is used for low-light enhancement processing to generate high-quality image frames, and tone mapping is used to convert high dynamic range (HDR) brightness values ​​into low dynamic range image frames. The image clarity and brightness during night driving are significantly enhanced, and roads, pedestrians and other vehicles can be accurately identified, improving the perception ability and safety of the autonomous driving system in low-light environments.

[0092] In some embodiments, the processing method of the present application is applicable to image processing when shooting dynamic scenes, and can remove image blur caused by camera shake and generate clear image frames. Through the data acquisition software or interface of the event camera, the event data recording jitter and blur are obtained, and the event data contains the time and space information of the pixel brightness change. The information of the event data is used as the input information of the PINN model, and the PINN model is optimized through time derivative self-supervision and spatial regularization. The mean square error between the predicted brightness change and the actual event data is minimized by time derivative self-supervision, and the noise and artifacts in the reconstructed image are reduced by spatial regularization. The optimized PINN model is used for de-jittering processing to generate clear image frames. Tone mapping is used to convert high dynamic range (HDR) brightness values ​​into low dynamic range image frames. The jitter and blur in the image are removed, and the generated image frame is clearer, which is suitable for high-quality image processing in dynamic scenes, and improves the detail retention and visual effects of the image.

[0093] In some embodiments, the processing method of the present application is applicable to the scene of efficient compression of video data, which can reduce the bandwidth required for storage and transmission while maintaining high-quality video reconstruction. The video event data captured by the event camera is obtained through the data acquisition software or interface of the event camera, and the brightness change information in the video frame is recorded in the video event data. The information of the event data is used as input and input into the PINN model, and the PINN model is optimized by time derivative self-supervision and spatial regularization. The mean square error between the predicted brightness change and the actual event data is minimized by time derivative self-supervision, and the noise and artifacts in the reconstructed image are reduced by Tikhonov regularization. The video data is compressed and represented by the optimized PINN model, and the brightness information in the video frame is converted into the parameters of the PINN model to achieve efficient compression. By decompression, the compressed representation is restored to a high-quality video frame. Tone mapping is used to convert high dynamic range (HDR) brightness values ​​into low dynamic range image frames. Video compression with high compression rate is achieved while maintaining high-quality image reconstruction, significantly reducing the bandwidth and storage space required for video storage and transmission, and improving the economy and practicality of video applications.

[0094] Another embodiment of the present application further provides an image processing device for an event camera, the processing device being used to execute the above processing method, such as Figure 2 As shown, the processing device includes:

[0095] The data acquisition module 101 is used to acquire event data of the blurred image captured by the event camera;

[0096] A model building module 201 is used to build an initial PINN model, embed an event generation equation in the initial PINN model, the input item of the initial PINN model is event data, and the output item is a predicted brightness value that satisfies the event generation equation;

[0097] The model optimization module 301 is used to input the event data of the blurred image into the initial PINN model to obtain the predicted brightness value of the event data, perform self-supervision of the time derivative based on the predicted brightness value and the actual brightness value, and introduce the Tikhonov regularization constraint condition to optimize the initial PINN model, and determine the optimized PINN model when the prediction performance of the PINN model reaches the preset standard;

[0098] The brightness value prediction module 401 is used to input the event data of the image to be processed into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed;

[0099] The tone mapping module 501 is used to convert the logarithmic brightness values ​​of the three color channels into a reconstructed image frame by using tone mapping.

[0100] In some embodiments, the tone mapping module 501 is used to convert the logarithmic brightness values ​​of the three color channels into HDR range brightness values ​​through an exponential function: I(x, y, t) = exp(F Θ (x,y,t));

[0101] The brightness and contrast of the image are adjusted by the Reinhard function, and the logarithmic brightness value of the HDR range is converted to the logarithmic brightness value of the LDR range. The specific expression is:

[0102]

[0103] I represents the logarithmic brightness value in the HDR range, Γ(I) represents the logarithmic brightness value in the LDR range, and γ represents a hyperparameter for controlling the contrast.

[0104] The specific definition of the image processing device of the event camera provided in this embodiment can be found in the embodiment of the image processing method based on the event camera above, and will not be repeated here. Each module in the image processing device of the above-mentioned event camera can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0105] An embodiment of the present application provides a computer device, which may include a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the processor executes the steps of the image processing method based on the event camera as in any of the above embodiments.

[0106] The working process, working details and technical effects of the computer device provided in this embodiment can be found in the embodiment of the image processing method based on the event camera above, and will not be described in detail here.

[0107] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method based on an event camera in any of the above embodiments are implemented. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0108] The working process, working details and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above embodiment of the image processing method based on the event camera, and will not be described in detail here.

[0109] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0110] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system described in the present application is divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. An image processing method based on an event camera, characterized in that: include: Obtaining event data of the blurred image captured by the event camera; Constructing an initial PINN model, embedding an event generation equation in the initial PINN model, wherein an input item of the initial PINN model is event data, and an output item is a predicted brightness value that satisfies the event generation equation; The event data of the blurred image is input into the initial PINN model to obtain the predicted brightness value of the event data. The time derivative is self-supervised based on the predicted brightness value and the actual brightness value, and the Tikhonov regularization constraint is introduced to optimize the initial PINN model. When the prediction performance of the PINN model reaches the preset standard, the optimized PINN model is determined. Input the event data of the image to be processed into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed; Tone mapping is used to transform the logarithmic brightness values ​​of the three color channels into the reconstructed image frame.

2. The image processing method based on event camera according to claim 1, characterized in that: Tone mapping is used to transform the logarithmic brightness values ​​of the three color channels into a reconstructed image frame, including: The logarithmic brightness values ​​of the three color channels are converted into HDR range brightness values ​​through exponential functions; The brightness and contrast of the image are adjusted by the Reinhard function, and the HDR range brightness value is converted to the LDR range brightness value; A reconstructed image frame is generated based on the LDR range of brightness values.

3. The image processing method based on event camera according to claim 1, characterized in that: The event data of the image to be processed is input into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed, including: The time coordinates of the event data are input into the optimized PINN model to obtain the logarithmic brightness value of the red channel, the logarithmic brightness value of the green channel, and the logarithmic brightness value of the blue channel.

4. The image processing method based on event camera according to claim 1, characterized in that: The hidden layer of the PINN model is a multi-layer fully connected neural network, and the initial PINN model uses randomized parameter values.

5. The image processing method based on event camera according to claim 1, characterized in that: Self-supervision of temporal derivatives based on predicted brightness values ​​and actual brightness values, including: Calculate the brightness change difference between the predicted brightness value output by the PINN model and the actual brightness value calculated by the event generation equation: The mean square error loss function is determined according to the brightness change difference, and the time derivative of the PINN model is self-supervised based on the mean square error loss function.

6. The image processing method based on event camera according to claim 4, characterized in that: The Tikhonov regularization constraint is introduced to optimize the initial PINN model, including: The spatial gradient of logarithmic brightness is constrained by Tikhonov regularization, which is expressed as: in, represents the regularization constraint, represents the derivative in the x dimension, Represents the derivative in the y dimension.

7. The image processing method based on event camera according to any one of claims 1 to 6, characterized in that: The event generation equation is formulated as: Where L represents the brightness change, P i represents the direction of brightness change, θ represents the brightness change threshold that triggers the event, and δ(tt i ) represents the Dirac function, t1 and t2 represent the time coordinates.

8. An image processing device based on an event camera, characterized in that: Used to perform the image processing method according to any one of claims 1 to 7, comprising: A data acquisition module, used to acquire event data of the blurred image captured by the event camera; A model building module, used to build an initial PINN model, in which an event generation equation is embedded, wherein the input item of the initial PINN model is event data, and the output item is a predicted brightness value that satisfies the event generation equation; A model optimization module is used to input the event data of the blurred image into the initial PINN model to obtain the predicted brightness value of the event data, perform self-supervision of the time derivative based on the predicted brightness value and the actual brightness value, and introduce Tikhonov regularization constraints to optimize the initial PINN model. When the prediction performance of the PINN model reaches the preset standard, the optimized PINN model is determined; A brightness value prediction module is used to input the event data of the image to be processed into the optimized PINN model to obtain the logarithmic brightness values ​​of the three color channels of the image to be processed; The tone mapping module is used to convert the logarithmic brightness values ​​of three color channels into a reconstructed image frame by using tone mapping.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images