Image processing method and related apparatus therefor
The target model uses event stream to process blurred LDR images, and realizes coupling processing of high dynamic range imaging and motion debuming, solving the problem of insufficient image quality in the prior art, improving the quality of the generated HDR image and saving model training costs.
Patent Information
- Application Number
- PCT/CN2024/095676
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-05-28
- Publication Date
- 2024-12-05
AI Technical Summary
In the prior art, high dynamic range imaging and motion defuzzing processes are independent of each other, resulting in poor quality of the final clear HDR image, and problems of color flaws, abnormal image brightness and detail blur.
Through the target model, the blurred LDR image and the event stream within its exposure time are input, and the event stream is used to fusion dynamic range enhancement, intensity component debuffering and color component fusion is realized to achieve coupling processing of high dynamic range imaging and motion debuffering, and generate High quality HDR images.
The coupled processing of high dynamic range imaging and motion debuffing is realized, and the quality of the generated clear HDR image is improved, which avoids color flaws and blurred details, and saves the training cost of neural network models.
Smart Images

Figure CN2024095676_05122024_PF_FP_ABST
Abstract
Description
Image processing method and related device
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 1, 2023, with application number 202310647524.9 and invention name “An image processing method and related devices”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to an image processing method and related devices. Background Art
[0003] High dynamic range (HDR) imaging and motion deblurring, as key issues in computer vision applications, are gaining increasing attention. With the rapid development of AI technology, more and more device manufacturers are integrating AI-based neural network models into their devices to achieve clear HDR images.
[0004] In related technologies, two neural network models are usually used to achieve high dynamic range imaging and motion deblurring respectively. Generally, the blurred low dynamic range (LDR) image is processed by the first neural network model to obtain a blurred HDR image, and then the blurred HDR image is processed by the second neural network model to obtain a clear HDR image. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0005] In the above process, since high dynamic range imaging and motion deblurring are implemented independently of each other, this often results in poor quality of the final clear HDR image. For example, the final clear HDR image may have various quality issues such as color defects, abnormal image brightness, and varying degrees of detail blurring.
[0006] Summary of the Invention
[0007] The embodiments of the present application provide an image processing method and related devices, which can use event streams to restore high-quality and clear HDR images from blurred LDR images.
[0008] A first aspect of an embodiment of the present application provides an image processing method, which can be implemented by a target model. The method includes:
[0009] When high dynamic range imaging and motion deblurring are required, the blurred LDR image and the event stream triggered during the blurred LDR image's exposure time can be input into the target model. It should be noted that the event stream is triggered by changes in the brightness of the captured scene during the blurred LDR image's exposure time. It is understood that the blurred LDR image is used to represent the aforementioned captured scene.
[0010] After obtaining the blurred LDR image and event stream, the target model can use the event stream to perform dynamic range enhancement on the LDR image, thereby obtaining a blurred HDR image. After obtaining the blurred HDR image, the target model can split the blurred HDR image into the intensity component of the blurred HDR image and the color component of the blurred HDR image, and use the event stream to deblur the intensity component of the blurred HDR image, thereby obtaining a clear and colorless HDR image. After obtaining the clear and colorless HDR image, the target model can also fuse the color component of the blurred HDR image with the clear and colorless HDR image, thereby obtaining a clear and color-rich HDR image. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0011] From the above method, it can be seen that the target model can use the event stream to recover the blurred HDR image from the blurred LDR image, and then continue to use the event stream to recover the clear and colorful HDR image from the blurred HDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together and there is a connection between the two. Therefore, the clear HDR image obtained in this way often has higher quality. In other words, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0012] In one possible implementation, dynamic range enhancement processing is performed on an LDR image based on an event stream to obtain a blurred HDR image, including: extracting features from the LDR image and the event stream separately to obtain features of the LDR image and features of the event stream; and fusing the features of the LDR image and the features of the event stream to obtain a blurred HDR image. In the aforementioned implementation, the target model may first perform feature extraction on the blurred LDR image to obtain initial features of the LDR image, then perform further feature extraction on the initial features of the LDR image to obtain intermediate features of the LDR image, and then perform further feature extraction on the intermediate features of the LDR image to obtain final features of the LDR image. Similarly, the target model may first perform feature extraction on the event stream to obtain initial features of the event stream, then perform further feature extraction on the initial features of the event stream to obtain intermediate features of the event stream, and then perform further feature extraction on the intermediate features of the event stream to obtain final features of the event stream. The target model can first perform feature fusion on the final features of the LDR image and the final features of the event stream to obtain a first fused feature, then perform feature fusion on the first fused feature, the intermediate features of the LDR image, and the intermediate features of the event stream to obtain a second fused feature, and then perform feature fusion on the second fused feature, the initial features of the LDR image, and the initial features of the event stream to obtain a third fused feature, which is a blurred HDR image. It can be seen that the target model can project the blurred LDR image and the event stream into the same feature space, and then fuse the obtained features to accurately estimate the blurred HDR image.
[0013] In one possible implementation, deblurring the intensity component of a blurred HDR image based on an event stream to obtain a clear and colorless HDR image includes: dividing the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; extracting features from the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimposing the features of the first event stream and the features of the second event stream to obtain superimposed features; and performing operations on the intensity component of the blurred HDR image and the superimposed features to obtain a clear and colorless HDR image. In the aforementioned implementation, after obtaining the blurred HDR image, the target model can first divide the exposure time of the blurred LDR image into a first exposure time and a second exposure time. Based on this, the target model can divide the event stream into a first event stream triggered within the first exposure time and a second event stream triggered within the second exposure time. After obtaining the first event stream and the second event stream, the target model can extract features from the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream respectively. Then, the target model can superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features. After obtaining the superimposed features, the target model can extract the intensity component of the blurred HDR image from the blurred HDR image, and perform operations on the intensity component of the blurred HDR image and the superimposed features to obtain a clear HDR image without color. It can be seen that the target model can use the high temporal resolution of the event stream to perform motion prior on the intensity component of the blurred HDR image, thereby obtaining a fuzzy factor of nonlinear motion (the aforementioned superimposed features), so as to accurately obtain a clear HDR image without color based on this factor.
[0014] In one possible implementation, the color components of a blurred HDR image and a clear HDR image without color are fused to obtain a clear HDR image with color, including: extracting features from the color components of the blurred HDR image to obtain features of the color components; and fusing the features of the color components with the clear HDR image without color to obtain a clear HDR image with color. In the aforementioned implementation, after obtaining the clear HDR image without color, the target model can extract the color components of the blurred HDR image from the blurred HDR image, and extract features from the color components of the blurred HDR image to obtain features of the color components. After obtaining the features of the color components, the target model can fuse the features of the color components with the clear HDR image without color to obtain a clear HDR image with color. It can be seen that the target model can colorize the clear HDR image without color, thereby ultimately obtaining a clear HDR image with color.
[0015] In a possible implementation, obtaining the blurred LDR image includes: obtaining the blurred LDR image presented in RGB format; and converting the LDR image presented in RGB format into an LDR image presented in YUV format.
[0016] In one possible implementation, obtaining the event stream triggered within the exposure time of the LDR image includes: obtaining the event stream triggered within the exposure time of the LDR image presented as discrete data; and converting the event stream presented as discrete data into an event stream presented as sequence data.
[0017] A second aspect of an embodiment of the present application provides an image processing method, which can be implemented through a target model. The method includes: acquiring a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; deblurring the blurred LDR image based on the event stream to obtain a clear LDR image; dynamically enhancing the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; and fusing the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0018] In one possible implementation, deblurring is performed on a blurred LDR image based on an event stream to obtain a clear LDR image, including: extracting features from the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and fusing features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0019] In one possible implementation, based on the event stream, the intensity component of the clear LDR image is dynamically enhanced to obtain a clear HDR image without color, including: dividing the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, the exposure time including the first exposure time and the second exposure time; performing feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimposing the features of the first event stream and the features of the second event stream to obtain superimposed features; performing calculations on the intensity component of the clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0020] In one possible implementation, the color components of the clear LDR image and the clear HDR image without color are fused to obtain a clear HDR image with color, including: extracting features of the color components of the clear LDR image to obtain features of the color components; and fusing features of the color components with the clear HDR image without color to obtain a clear HDR image with color.
[0021] In a possible implementation, obtaining the blurred LDR image includes: obtaining the blurred LDR image presented in RGB format; and converting the LDR image presented in RGB format into an LDR image presented in YUV format.
[0022] In one possible implementation, obtaining the event stream triggered within the exposure time of the LDR image includes: obtaining the event stream triggered within the exposure time of the LDR image presented as discrete data; and converting the event stream presented as discrete data into an event stream presented as sequence data.
[0023] A third aspect of an embodiment of the present application provides a model training method, which includes: processing a blurred LDR image and an event stream triggered within the exposure time of the LDR image through a to-be-trained model to obtain a clear HDR image with color, wherein the to-be-trained model is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color; and train the to-be-trained model based on the clear HDR image with color to obtain a target model.
[0024] In one possible implementation, the model to be trained is used to: perform feature extraction on the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image.
[0025] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform operations on the intensity components of the blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0026] In one possible implementation, the model to be trained is used to: extract features of the color components of a blurred HDR image to obtain features of the color components; and fuse features of the color components with a clear HDR image without color to obtain a clear HDR image with color.
[0027] In one possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0028] In one possible implementation, the model to be trained is used to: obtain an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0029] A fourth aspect of an embodiment of the present application provides a model training method, which includes: processing a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image through a to-be-trained model to obtain a clear HDR image with color, wherein the to-be-trained model is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblur the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhance the intensity component of the clear LDR image to obtain a clear HDR image without color; fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color; and train the to-be-trained model based on the clear HDR image with color to obtain a target model.
[0030] In one possible implementation, the model to be trained is used to: perform feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0031] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0032] In one possible implementation, the model to be trained is used to: extract features of the color components of a clear LDR image to obtain features of the color components; and fuse features of the color components with a clear HDR image without color to obtain a clear HDR image with color.
[0033] In one possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0034] In one possible implementation, the model to be trained is used to: obtain an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0035] A fifth aspect of an embodiment of the present application provides an image processing device, which includes a target model and comprises: an acquisition module for acquiring a blurred LDR image and an event stream triggered within the exposure time of the LDR image; a first processing module for performing dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image; a second processing module for performing deblurring processing on the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color; and a third processing module for performing fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0036] In one possible implementation, the first processing module 1402 is used to: perform feature extraction on the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image.
[0037] In one possible implementation, the second processing module 1403 is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform operations on the intensity components of the blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0038] In one possible implementation, the third processing module 1404 is configured to: extract features of the color components of the blurred HDR image to obtain features of the color components; and perform feature fusion on the features of the color components and the clear HDR image without color to obtain a clear HDR image with color.
[0039] In a possible implementation, the acquisition module 1401 is configured to: acquire a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0040] In one possible implementation, the acquisition module 1401 is configured to: acquire an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0041] A sixth aspect of an embodiment of the present application provides an image processing device, which includes a target model and comprises: an acquisition module for acquiring a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; a first processing module for deblurring the blurred LDR image based on the event stream to obtain a clear LDR image; a second processing module for dynamically enhancing the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; and a third processing module for fusing the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0042] In one possible implementation, the first processing module 1502 is used to: perform feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0043] In one possible implementation, the second processing module 1503 is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of the clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0044] In one possible implementation, the third processing module 1503 is configured to: extract features of the color components of the clear LDR image to obtain features of the color components; and perform feature fusion on the features of the color components and the clear HDR image without color to obtain a clear HDR image with color.
[0045] In a possible implementation, the acquisition module 1501 is configured to: acquire a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0046] In one possible implementation, the acquisition module 1501 is configured to: acquire an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0047] In a seventh aspect of an embodiment of the present application, a model training device is provided, which includes: a processing module for processing a blurred LDR image and an event stream triggered within the exposure time of the LDR image through a model to be trained to obtain a clear HDR image with color, and the model to be trained is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color; a training module for training the model to be trained based on the clear HDR image with color to obtain a target model.
[0048] In one possible implementation, the model to be trained is used to: perform feature extraction on the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image.
[0049] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform operations on the intensity components of the blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0050] In one possible implementation, the model to be trained is used to: extract features of the color components of a blurred HDR image to obtain features of the color components; and fuse features of the color components with a clear HDR image without color to obtain a clear HDR image with color.
[0051] In one possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0052] In one possible implementation, the model to be trained is used to: obtain an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0053] An eighth aspect of an embodiment of the present application provides a model training device, which includes: a processing module, used to process a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image through a to-be-trained model to obtain a clear HDR image with color, wherein the to-be-trained model is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblur the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhance the intensity component of the clear LDR image to obtain a clear HDR image without color; fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color; and a training module, used to train the to-be-trained model based on the clear HDR image with color to obtain a target model.
[0054] In one possible implementation, the model to be trained is used to: perform feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0055] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0056] In one possible implementation, the model to be trained is used to: extract features of the color components of a clear LDR image to obtain features of the color components; and fuse features of the color components with a clear HDR image without color to obtain a clear HDR image with color.
[0057] In one possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0058] In one possible implementation, the model to be trained is used to: obtain an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0059] A ninth aspect of an embodiment of the present application provides a fault prediction device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the fault prediction device executes the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.
[0060] The tenth aspect of an embodiment of the present application provides a model training device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes the method described in the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0061] The eleventh aspect of an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0062] The twelfth aspect of an embodiment of the present application provides a chip system, which includes a processor for calling a computer program or computer instruction stored in a memory so that the processor executes a method as described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0063] In a possible implementation, the processor is coupled to the memory through an interface.
[0064] In one possible implementation, the chip system further includes a memory in which computer programs or computer instructions are stored.
[0065] A thirteenth aspect of an embodiment of the present application provides a computer storage medium, which stores a computer program. When the program is executed by a computer, the computer implements the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0066] A fourteenth aspect of the embodiments of the present application provides a computer program product, which stores instructions that, when executed by a computer, enable the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0067] In an embodiment of the present application, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image. Then, the target model can deblur the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can fuse the color component of the blurred HDR image with the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to recover the blurred HDR image from the blurred LDR image, and then continue to use the event stream to recover the clear and colorful HDR image from the blurred HDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together. There is a connection between the two, so the clear HDR image obtained in this way often has higher quality. That is to say, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;
[0069] FIG2a is a schematic structural diagram of an image processing system provided in an embodiment of the present application;
[0070] FIG2 b is another schematic diagram of the structure of the image processing system provided in an embodiment of the present application;
[0071] FIG2c is a schematic diagram of an image processing device according to an embodiment of the present application;
[0072] FIG3 is a schematic diagram of the architecture of the system 100 provided in an embodiment of the present application;
[0073] FIG4 is a schematic diagram of a structure of a target model provided in an embodiment of the present application;
[0074] FIG5 is a flow chart of an image processing method provided in an embodiment of the present application;
[0075] FIG6 is a schematic structural diagram of a dynamic enhancement module provided in an embodiment of the present application;
[0076] FIG7 is a schematic structural diagram of a motion deblurring module provided in an embodiment of the present application;
[0077] FIG8 is a schematic diagram of a structure of a color enhancement network provided in an embodiment of the present application;
[0078] FIG9 is another schematic diagram of the structure of the target model provided in an embodiment of the present application;
[0079] FIG10 is another schematic flow chart of the image processing method provided in an embodiment of the present application;
[0080] FIG11 is a schematic diagram of a comparative structure provided in an embodiment of the present application;
[0081] FIG12 is a flow chart of a model training method provided in an embodiment of the present application;
[0082] FIG13 is another flow chart of the model training method provided in an embodiment of the present application;
[0083] FIG14 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;
[0084] FIG15 is another schematic diagram of the structure of the image processing device provided in an embodiment of the present application;
[0085] FIG16 is a schematic structural diagram of a model training device provided in an embodiment of the present application;
[0086] FIG17 is another structural diagram of the model training device provided in an embodiment of the present application;
[0087] FIG18 is a schematic diagram of the structure of an execution device provided in an embodiment of the present application;
[0088] FIG19 is a schematic structural diagram of a training device provided in an embodiment of the present application;
[0089] FIG20 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION
[0090] The embodiments of the present application provide an image processing method and related devices, which can use event streams to restore high-quality and clear HDR images from blurred LDR images.
[0091] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0092] High dynamic range imaging and motion deblurring, as key issues in computer vision applications, are receiving increasing attention. With the rapid development of AI technology, more and more device manufacturers are integrating AI-based neural network models into their devices to achieve clear HDR images.
[0093] In related technologies, two neural network models are usually used to achieve high dynamic range imaging and motion deblurring respectively. Generally, the blurred LDR image is processed by the first neural network model to obtain a blurred HDR image, and then the blurred HDR image is processed by the second neural network model to obtain a clear HDR image. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0094] In the above process, since high dynamic range imaging and motion deblurring are implemented independently of each other, this often results in poor quality of the final clear HDR image. For example, the final clear HDR image may have various quality issues such as color defects, abnormal image brightness, and varying degrees of detail blurring.
[0095] Furthermore, the two processes of high dynamic range imaging and motion deblurring require two neural network models to implement and cannot be integrated together, resulting in excessively high training costs for the neural network models.
[0096] In order to solve the above problems, an embodiment of the present application provides an image processing method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by perceiving the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for data processing is a common application of artificial intelligence.
[0097] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the AI framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.
[0098] (1) Infrastructure
[0099] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.
[0100] (2) Data
[0101] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.
[0102] (3) Data processing
[0103] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.
[0104] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0105] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0106] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.
[0107] (4) General ability
[0108] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0109] (5) Smart products and industry applications
[0110] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.
[0111] Next, several application scenarios of this application are introduced.
[0112] Figure 2a is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application. The image processing system includes a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, a personal computer, or an information processing center. The user device is the initiator of image processing and is the initiator of image processing requests, typically initiated by a user through the user device.
[0113] The aforementioned data processing device can be a device or server with data processing capabilities, such as a cloud server, network server, application server, or management server. The data processing device receives image processing requests from smart terminals via an interactive interface and then performs image processing using methods such as machine learning, deep learning, search, reasoning, and decision-making through its memory and data processing processor. The memory in the data processing device is a general term that includes local storage and a database that stores historical data. The database can be located on the data processing device or on other network servers.
[0114] In the image processing system shown in FIG2a, a user device may receive a user instruction. For example, the user device may obtain an LDR image input / selected by the user, and then initiate a request to a data processing device, causing the data processing device to execute an image processing application on the LDR image obtained by the user device, thereby obtaining a clear HDR image. For example, the user device may obtain a blurred LDR image input by the user, and then initiate an image processing request to the data processing device, causing the data processing device to perform a series of processing on the blurred LDR image based on the image processing request, thereby obtaining a processed result of the blurred LDR image, that is, a clear HDR image.
[0115] In FIG. 2 a , the data processing device may execute the image processing method according to the embodiment of the present application.
[0116] Figure 2b is another structural schematic diagram of the image processing system provided in an embodiment of the present application. In Figure 2b, the user device directly serves as a data processing device. The user device can directly obtain input from the user and directly process it by the hardware of the user device itself. The specific process is similar to that of Figure 2a. Please refer to the above description and will not be repeated here.
[0117] In the image processing system shown in FIG2b , the user device may receive instructions from the user. For example, the user device may obtain a blurred LDR image input by the user, and then perform a series of processing on the blurred LDR image to obtain a processing result of the blurred LDR image, that is, a clear HDR image.
[0118] In FIG2 b , the user equipment itself can execute the image processing method of the embodiment of the present application.
[0119] FIG2c is a schematic diagram of an image processing related device provided in an embodiment of the present application.
[0120] The user device in the above Figures 2a and 2b can specifically be the local device 301 or the local device 302 in Figure 2c, and the data processing device in Figure 2a can specifically be the execution device 210 in Figure 2c, wherein the data storage system 250 can store the data to be processed of the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or it can be set on the cloud or other network servers.
[0121] The processors in Figures 2a and 2b can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to perform image processing applications on the image, thereby obtaining corresponding processing results.
[0122] Figure 3 is a schematic diagram of the system 100 architecture provided in an embodiment of the present application. In Figure 3, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data can include: various tasks to be scheduled, callable resources and other parameters in the embodiment of the present application.
[0123] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing (such as implementing the functions of the neural network in this application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.
[0124] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.
[0125] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from training samples collected by the data collection device 160.
[0126] In the scenario shown in FIG3 , the user can manually input data, which can be performed through the interface provided by I / O interface 112. Alternatively, client device 140 can automatically send input data to I / O interface 112. If user authorization is required for client device 140 to automatically send input data, the user can set the corresponding permissions in client device 140. The user can view the output of execution device 110 on client device 140, which can be presented in a display, sound, action, or other specific form. Client device 140 can also serve as a data acquisition terminal, collecting input data and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, collection can also be performed without client device 140, with I / O interface 112 directly storing the input data and output results from I / O interface 112 as new sample data in database 130.
[0127] It is worth noting that FIG3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG3, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110. As shown in FIG3, a neural network can be trained according to the training device 120.
[0128] The present application also provides a chip including a neural network processor (NPU). The chip can be provided in the execution device 110 shown in FIG3 to complete the computational work of the computation module 111. The chip can also be provided in the training device 120 shown in FIG3 to complete the training work of the training device 120 and output the target model / rule.
[0129] The neural network processor (NPU) is a coprocessor mounted on the host central processing unit (CPU), which assigns tasks to it. The core of the NPU is the arithmetic circuit, which is controlled by a controller to extract data from memory (weight memory or input memory) and perform calculations.
[0130] In some implementations, the arithmetic circuit includes multiple processing engines (PEs). In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.
[0131] For example, consider an input matrix A, a weight matrix B, and an output matrix C. The computational circuit retrieves the corresponding data for matrix B from the weight memory and caches it on each PE within the computational circuit. The computational circuit then performs a matrix operation on the matrix A data from the input memory and matrix B, storing the partial or final matrix results in the accumulator.
[0132] The vector computation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit can be used for network calculations in non-convolutional / non-FC layers in neural networks, such as pooling, batch normalization, and local response normalization.
[0133] In some implementations, the vector computation unit can store the processed output vectors in a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the computation circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vectors can be used as activation inputs to the computation circuit, such as for use in subsequent layers in a neural network.
[0134] The unified memory is used to store input data and output data.
[0135] The weight data is directly transferred from the external memory to the input memory and / or unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.
[0136] The bus interface unit (BIU) is used to implement interaction between the main CPU, DMAC and instruction fetch memory through the bus.
[0137] An instruction fetch buffer connected to the controller, used to store instructions used by the controller;
[0138] The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.
[0139] Generally, unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and external memory is memory outside the NPU, which can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
[0140] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.
[0141] (1) Neural Network
[0142] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:
[0143] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.
[0144] The operation of each layer in a neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the operation of each layer in a neural network can be understood as transforming the input space (a set of input vectors) into the output space (i.e., from the row space to the column space of a matrix) through five operations. These operations include: 1. Dimensionality increase / decrease; 2. Scaling / reduction; 3. Rotation; 4. Translation; and 5. "Bending." Operations 1, 2, and 3 are performed by Wx, 4 by +b, and 5 by a(). The word "space" is used here because the objects being classified are not individual things, but rather a class of things, and space refers to the collection of all individuals within that class. W is the weight vector, each value in which represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from input space to output space described above. That is, the weights W of each layer control how the space is transformed. The goal of training a neural network is to ultimately obtain the weight matrix for all layers of the trained neural network (a weight matrix formed by the vectors W of many layers). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, and more specifically, about learning the weight matrix.
[0145] Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value, and this adjustment is continued until the neural network can predict the desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which is an important equation used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.
[0146] (2) Backpropagation algorithm
[0147] Neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during training, reducing the reconstruction error loss of the neural network model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial neural network model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.
[0148] The method provided in this application is described below from the training side of the neural network and the application side of the neural network.
[0149] The model training method provided in the embodiment of the present application involves the processing of data sequences, and can be specifically applied to methods such as data training, machine learning, and deep learning. The training data (for example, the blurred LDR image in the model training method provided in the embodiment of the present application) is subjected to symbolic and formalized intelligent information modeling, extraction, preprocessing, and training, and finally a trained neural network (for example, the target model in the model training method provided in the embodiment of the present application) is obtained; and the image processing method provided in the embodiment of the present application can use the above-mentioned trained neural network to input the input data (for example, the blurred LDR image in the image processing method provided in the embodiment of the present application) into the trained neural network to obtain output data (for example, a clear and colorful HDR image in the image processing method provided in the embodiment of the present application). It should be noted that the model training method and image processing method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.
[0150] The image processing method provided in the embodiment of the present application can be implemented through a target model, and the target model can have a variety of structures. The following is a brief introduction to the first structure of the target model. Figure 4 is a structural diagram of the target model provided in the embodiment of the present application. As shown in Figure 4, the target model includes an event-based dynamic enhancement module, an event-based motion deblurring module, and a color enhancement network, wherein the input end of the dynamic enhancement module serves as the input end of the entire target model, the first output end of the dynamic enhancement module is connected to the input end of the motion deblurring module, the second output end of the dynamic enhancement module is connected to the first input end of the color enhancement network, the output end of the motion deblurring module is connected to the second input end of the color enhancement network, and the output end of the color enhancement network serves as the output end of the entire target model. In order to understand the workflow of the target model shown in Figure 4, the workflow is introduced in conjunction with Figure 5 below. Figure 5 is a flow chart of the image processing method provided in the embodiment of the present application. As shown in Figure 5, the method includes:
[0151] 501. Obtain a blurred LDR image and an event stream triggered within the exposure time of the LDR image.
[0152] In this embodiment, when high dynamic range imaging and motion deblurring are required, a blurred LDR image can be captured first, and an event stream triggered during the exposure time of the blurred LDR image can be collected. It should be noted that the event stream is triggered by a change in the brightness of the captured scene (i.e., the scene corresponding to the blurred LDR image) during the exposure time of the blurred LDR image.
[0153] After obtaining the blurred LDR image and event stream, the blurred LDR image and event stream can be input into the target model, so the target model can successfully obtain the blurred LDR image and event stream.
[0154] Specifically, the target model can obtain the blurred LDR image in the following way:
[0155] Since the blurred LDR image obtained by shooting is presented in RGB format, after the blurred LDR image presented in RGB format is input into the target model, the dynamic enhancement module of the target model can convert the blurred LDR image presented in RGB format into a blurred LDR image presented in YUV format (also known as YCrCb format).
[0156] For example, as shown in Figure 6 (Figure 6 is a structural diagram of a dynamic enhancement module provided in an embodiment of the present application), the dynamic enhancement module includes a first conversion unit, a second conversion unit, and a multimodal fusion unit. After receiving a blurred LDR image L in RGB format, the first conversion unit can convert L into a blurred LDR image L' in YCrCb format and send L' to the multimodal fusion unit.
[0157] More specifically, the target model can obtain the event stream in the following ways:
[0158] Since the captured event stream is triggered during the exposure time of the blurred LDR image and presented as discrete data, after the event stream presented as discrete data is input into the target model, the dynamic enhancement module of the target model can convert the event stream presented as discrete data into an event stream presented as sequence data. It should be noted that the event stream presented as sequence data includes multiple event images sorted in chronological order, and these multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0159] As in the above example, when receiving the event stream ε of discrete data T After that, the second conversion unit can T Perform stacking operations to obtain the event stream ε of sequence data D , and ε D Send to the multimodal fusion unit and motion deblurring module. At this time, ε D It includes multiple event images sorted in chronological order, and these multiple event images correspond one-to-one to multiple moments in the exposure time of the blurred LDR image.
[0160] 502. Based on the event stream, dynamic range enhancement processing is performed on the LDR image to obtain a blurred HDR image.
[0161] After obtaining the blurred LDR image and event stream, the target model can use the event stream to perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image.
[0162] Specifically, the target model can obtain the blurred HDR image in the following way:
[0163] (1) After obtaining the blurred LDR image presented in YUV format and the event stream presented in sequence data, the dynamic enhancement module can extract features from the blurred LDR image and the event stream respectively, thereby obtaining the features of the LDR image and the features of the event stream accordingly.
[0164] It should be noted that the features of the LDR image include the initial features of the LDR image, the intermediate features of the LDR image, and the final features of the LDR image, and the features of the event stream include the initial features of the event stream, the intermediate features of the event stream, and the final features of the event stream. Therefore, the dynamic enhancement module can first perform feature extraction on the blurred LDR image to obtain the initial features of the LDR image, and then perform further feature extraction on the initial features of the LDR image to obtain the intermediate features of the LDR image, and then perform further feature extraction on the intermediate features of the LDR image to obtain the final features of the LDR image. Similarly, the dynamic enhancement module can first perform feature extraction on the event stream to obtain the initial features of the event stream, and then perform further feature extraction on the initial features of the event stream to obtain the intermediate features of the event stream, and then perform further feature extraction on the intermediate features of the event stream to obtain the final features of the event stream.
[0165] Still as in the above example, after obtaining L′ and ε D After that, the multimodal fusion unit can extract features from L′ to obtain the initial feature h1, then extract features from h1 to obtain the intermediate feature h2, and then extract features from h2 to obtain the final feature h3. Similarly, the multimodal fusion unit can extract features from ε D Perform feature extraction to obtain the initial feature h4, then perform feature extraction on h4 to obtain the intermediate feature h5, and then perform feature extraction on h5 to obtain the final feature h6.
[0166] (2) After obtaining the features of the LDR image and the features of the event stream, the dynamic enhancement module can perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image, and send the blurred HDR image to the motion deblurring module and the color enhancement network.
[0167] It should be noted that the dynamic enhancement module can first perform feature fusion on the final features of the LDR image and the final features of the event stream to obtain a first fused feature, then perform feature fusion on the first fused feature, the intermediate features of the LDR image, and the intermediate features of the event stream to obtain a second fused feature, and then perform feature fusion on the second fused feature, the initial features of the LDR image, and the initial features of the event stream to obtain a third fused feature, that is, a blurred HDR image. Acquiring a blurred HDR image in accordance with the combination of (1) and (2) can prevent the loss of introduced spatial information during the image acquisition process.
[0168] Still as in the above example, after obtaining h1 to h6, the multimodal fusion unit can perform feature fusion on h3 and h6 to obtain fused feature h7, then perform feature fusion on h2, h5 and h7 to obtain fused feature h8, and then perform feature fusion on h1, h4 and h8 to obtain a blurred HDR image B, and provide B to the motion deblurring module and the color enhancement network.
[0169] 503. Based on the event stream, deblur the intensity component of the blurred HDR image to obtain a clear HDR image without color.
[0170] After obtaining the blurred HDR image, the target model can split the blurred HDR image into an intensity component of the blurred HDR image and a color component of the blurred HDR image, and use the event stream to deblur the intensity component of the blurred HDR image, thereby obtaining a clear HDR image without color.
[0171] Specifically, the target model can obtain a clear and colorless HDR image by:
[0172] (1) After obtaining the blurred HDR image, the motion deblurring module can first divide the exposure time of the blurred LDR image into a first exposure time (also called a more concerned time period, which includes several moments ranked relatively high) and a second exposure time (also called a less concerned time period, which includes several moments ranked relatively low). Based on this, the motion deblurring module can divide the event stream into a first event stream triggered within the first exposure time (the first event stream includes several event images ranked relatively high) and a second event stream triggered within the second exposure time (the second event stream includes several event images ranked relatively low). This can make the time distinction more obvious.
[0173] For example, as shown in FIG7 (FIG. 7 is a structural diagram of a motion deblurring module provided in an embodiment of the present application, and FIG7 is drawn based on FIG6), the motion deblurring module includes an event separator, a deblurring factor generation unit, and a division operation unit. D After that, the event partitioning unit can D Divide into event streams and event streams in, Contains several event images that are ranked first, and these event images correspond one-to-one to several moments that are ranked first in the exposure time. Contains several event images that are ranked later, and these event images correspond one-to-one to several moments that are ranked later in the exposure time, and as well as Sent to the defuzzification factor generation unit.
[0174] (2) After obtaining the first event stream and the second event stream, the motion deblurring module may perform feature extraction on the first event stream and the second event stream, respectively, thereby obtaining features of the first event stream and features of the second event stream. The motion deblurring module may then superimpose the features of the first event stream and the features of the second event stream, thereby obtaining superimposed features.
[0175] Still as in the above example, after getting as well as After that, the defuzzification factor generation unit can first as well as Perform feature extraction to obtain the corresponding features and features Then, the defuzzification factor generation unit can as well as Superposition is performed to obtain superposition features and will Sent to the division unit.
[0176] (3) After obtaining the superimposed features, the motion deblurring module can extract the intensity component of the blurred HDR image from the blurred HDR image, and perform operations (e.g., division operations, etc.) on the intensity component of the blurred HDR image and the superimposed features, thereby obtaining a clear HDR image without color, and sending the clear HDR image without color to the color enhancement network. It should be noted that the clear HDR image without color includes multiple clear HDR sub-images without color, and these multiple HDR sub-images are sorted in chronological order, and these multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0177] Still as in the above example, after getting and the intensity component B Y After that, the division unit can and B Y Perform division operation to obtain a clear and colorless HDR image y T , and y T Sent to the color enhancement network. The division operation is shown in the following formula:
[0178] In the above formula, y T It includes multiple HDR sub-images sorted in time sequence, and the multiple HDR sub-images correspond one-to-one to multiple moments in the exposure time of the blurred LDR image.
[0179] It should be understood that this embodiment only uses the example of dividing the event stream into two sub-event streams, the first event stream and the second event stream, for schematic introduction, and does not limit the number of sub-event streams into which the event stream is divided. In actual applications, the number of sub-event streams can be more, for example, dividing the event stream into three sub-event streams, the first event stream, the second event stream, and the third event stream, and so on.
[0180] 504. Perform fusion processing on the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0181] After obtaining a clear and colorless HDR image, the target model can also fuse the color components of the blurred HDR image with the clear and colorless HDR image to obtain a clear and color-rich HDR image. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0182] Specifically, the target model can obtain clear and colorful HDR images through the following methods:
[0183] (1) After obtaining a clear and colorless HDR image, the color enhancement network can extract the color components of the blurred HDR image from the blurred HDR image, and perform feature extraction on the color components of the blurred HDR image to obtain the features of the color components.
[0184] For example, as shown in FIG8 (FIG8 is a structural diagram of a color enhancement network provided by an embodiment of the present application, FIG8 is drawn based on FIG7), the color enhancement network includes a mapping network and a coloring network. CrCb After that, the mapping network can be used to map B CrCb Perform feature extraction to obtain color feature b, and send b to the coloring network.
[0185] (2) After obtaining the features of the color components, the color enhancement network can perform feature fusion on the features of the color components and the clear HDR image without color, thereby obtaining a clear HDR image with color. It should be noted that the clear HDR image with color contains multiple clear HDR sub-images without color, and these multiple HDR sub-images are sorted in chronological order, and these multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0186] Still as in the above example, after getting b and y T After that, the coloring network can be used to calculate b and y T Fusion is performed to obtain a clear and colorful HDR image I T , I T It includes multiple HDR sub-images sorted in time sequence, and the multiple HDR sub-images correspond one-to-one to multiple moments in the exposure time of the blurred LDR image.
[0187] In an embodiment of the present application, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image. Then, the target model can deblur the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can fuse the color component of the blurred HDR image with the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to recover the blurred HDR image from the blurred LDR image, and then continue to use the event stream to recover the clear and colorful HDR image from the blurred HDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together. There is a connection between the two, so the clear HDR image obtained in this way often has higher quality. That is to say, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0188] Furthermore, in an embodiment of the present application, in the process of restoring a clear and colorful HDR image from a blurred LDR image, an event stream triggered during the exposure time of the blurred LDR image is used. Since the event stream describes the brightness changes of the shooting scene (i.e., the scene corresponding to the blurred LDR image), the target model is based on the event stream, which can more comprehensively understand and analyze the shooting scene, and more realistically display the brightness information of each part of the shooting scene. The clear HDR image obtained based on this can have high-quality visual effects.
[0189] Furthermore, in the embodiment of the present application, the two processes of high dynamic range imaging and motion deblurring only require the target model, a neural network model, to be implemented, and can be integrated together to achieve this, thereby saving the training cost of the neural network model.
[0190] The above is a detailed description of the workflow of the target model of the first structure, and the target model of the second structure will be introduced below. Figure 9 is another structural schematic diagram of the target model provided in an embodiment of the present application. As shown in Figure 9, the target model includes an event-based motion deblurring module, an event-based dynamic enhancement module, and a color enhancement network, wherein the input end of the motion deblurring module serves as the input end of the entire target model, the first output end of the motion deblurring module is connected to the input end of the dynamic enhancement module, the second output end of the motion deblurring module is connected to the first input end of the color enhancement network, the output end of the dynamic enhancement module is connected to the second input end of the color enhancement network, and the output end of the color enhancement network serves as the output end of the entire target model. In order to understand the workflow of the target model shown in Figure 9, the workflow is introduced below in conjunction with Figure 5. Figure 10 is another flow chart of the image processing method provided in an embodiment of the present application. As shown in Figure 10, the method includes:
[0191] 1001. Obtain a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image.
[0192] In this embodiment, when high dynamic range imaging and motion deblurring are required, a blurred LDR image may be captured first, and an event stream triggered within the exposure time of the blurred LDR image may be collected.
[0193] In a possible implementation, obtaining the blurred LDR image includes: obtaining the blurred LDR image presented in RGB format; and converting the LDR image presented in RGB format into an LDR image presented in YUV format.
[0194] In one possible implementation, obtaining the event stream triggered within the exposure time of the LDR image includes: obtaining the event stream triggered within the exposure time of the LDR image presented as discrete data; and converting the event stream presented as discrete data into an event stream presented as sequence data.
[0195] It should be noted that, for the introduction of step 1001, reference may be made to the relevant description of step 501 in the embodiment shown in FIG5 , which will not be repeated here.
[0196] 1002. Deblur the blurred LDR image based on the event stream to obtain a clear LDR image.
[0197] After obtaining the blurred LDR image and the event stream, the target model can use the event stream to deblur the blurred LDR image, thereby obtaining a clear LDR image.
[0198] Specifically, the target model can obtain a clear LDR image in the following ways:
[0199] (1) After obtaining the blurred LDR image presented in YUV format and the event stream presented in sequence data, the motion deblurring module can extract features from the blurred LDR image and the event stream respectively, thereby obtaining the features of the LDR image and the features of the event stream accordingly.
[0200] It should be noted that the features of the LDR image include the initial features of the LDR image, the intermediate features of the LDR image, and the final features of the LDR image, and the features of the event stream include the initial features of the event stream, the intermediate features of the event stream, and the final features of the event stream. Therefore, the motion deblurring module can first perform feature extraction on the blurred LDR image to obtain the initial features of the LDR image, and then perform further feature extraction on the initial features of the LDR image to obtain the intermediate features of the LDR image, and then perform further feature extraction on the intermediate features of the LDR image to obtain the final features of the LDR image. Similarly, the motion deblurring module can first perform feature extraction on the event stream to obtain the initial features of the event stream, and then perform further feature extraction on the initial features of the event stream to obtain the intermediate features of the event stream, and then perform further feature extraction on the intermediate features of the event stream to obtain the final features of the event stream.
[0201] (2) After obtaining the features of the LDR image and the features of the event stream, the motion deblurring module can perform feature fusion on the features of the LDR image and the features of the event stream to obtain a clear LDR image, and then send the clear LDR image to the dynamic enhancement module and the color enhancement network.
[0202] It should be noted that the motion deblurring module can first perform feature fusion on the final features of the LDR image and the final features of the event stream to obtain a first fused feature, then perform feature fusion on the first fused feature, the intermediate features of the LDR image, and the intermediate features of the event stream to obtain a second fused feature, and then perform feature fusion on the second fused feature, the initial features of the LDR image, and the initial features of the event stream to obtain a third fused feature, that is, a clear LDR image. Obtaining a clear LDR image by combining (1) and (2) can prevent the loss of introduced spatial information during the image acquisition process.
[0203] 1003. Based on the event stream, the intensity component of the clear LDR image is dynamically enhanced to obtain a clear HDR image without color.
[0204] After obtaining a clear LDR image, the target model can split the clear LDR image into an intensity component of the clear LDR image and a color component of the clear LDR image, and dynamically enhance the intensity component of the clear LDR image using the event stream, thereby obtaining a clear HDR image without color.
[0205] Specifically, the target model can obtain a clear and colorless HDR image by:
[0206] (1) After obtaining a clear LDR image, the dynamic enhancement module can first divide the exposure time of the blurred LDR image into a first exposure time (also called a time period of more concern, which includes several moments ranked relatively high) and a second exposure time (also called a time period of less concern, which includes several moments ranked relatively low). Based on this, the dynamic enhancement module can divide the event stream into a first event stream triggered within the first exposure time (the first event stream includes several event images ranked relatively high) and a second event stream triggered within the second exposure time (the second event stream includes several event images ranked relatively low). This can make the time distinction more obvious.
[0207] (2) After obtaining the first event stream and the second event stream, the dynamic enhancement module may extract features from the first event stream and the second event stream, respectively, thereby obtaining features of the first event stream and features of the second event stream. The dynamic enhancement module may then superimpose the features of the first event stream and the features of the second event stream, thereby obtaining superimposed features.
[0208] (3) After obtaining the superimposed features, the dynamic enhancement module can extract the intensity component of the clear LDR image from the clear LDR image, and perform operations (e.g., division operations, etc.) on the intensity component of the clear LDR image and the superimposed features, thereby obtaining a clear HDR image without color, and sending the clear HDR image without color to the color enhancement network. It should be noted that the clear HDR image without color includes multiple clear HDR sub-images without color, and these multiple HDR sub-images are sorted in chronological order, and these multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0209] It should be understood that this embodiment only uses the example of dividing the event stream into two sub-event streams, the first event stream and the second event stream, for schematic introduction, and does not limit the number of sub-event streams into which the event stream is divided. In actual applications, the number of sub-event streams can be more, for example, dividing the event stream into three sub-event streams, the first event stream, the second event stream, and the third event stream, and so on.
[0210] 1004. Perform fusion processing on the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0211] After obtaining a clear and colorless HDR image, the target model can also fuse the color components of the clear LDR image and the clear and colorless HDR image to obtain a clear and color-rich HDR image. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0212] Specifically, the target model can obtain clear and colorful HDR images through the following methods:
[0213] (1) After obtaining a clear and colorless HDR image, the color enhancement network can extract the color components of the clear LDR image from the clear LDR image, and perform feature extraction on the color components of the clear LDR image to obtain the features of the color components.
[0214] (2) After obtaining the features of the color components, the color enhancement network can perform feature fusion on the features of the color components and the clear HDR image without color, thereby obtaining a clear HDR image with color. It should be noted that the clear HDR image with color contains multiple clear HDR sub-images without color, and these multiple HDR sub-images are sorted in chronological order, and these multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0215] In an embodiment of the present application, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into a target model. Then, the target model can first deblur the blurred LDR image based on the event stream to obtain a clear LDR image. Then, the target model can perform dynamic range enhancement processing on the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can fuse the color component of the clear LDR image with the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to restore a clear LDR image from the blurred LDR image, and then continue to use the event stream to restore a clear and colorful HDR image from the clear LDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together. There is a connection between the two, so the clear HDR image obtained in this way often has higher quality. That is to say, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0216] Furthermore, in an embodiment of the present application, in the process of restoring a clear and colorful HDR image from a blurred LDR image, an event stream triggered during the exposure time of the blurred LDR image is used. Since the event stream describes the brightness changes of the shooting scene (i.e., the scene corresponding to the blurred LDR image), the target model is based on the event stream, which can more comprehensively understand and analyze the shooting scene, and more realistically display the brightness information of each part of the shooting scene. The clear HDR image obtained based on this can have high-quality visual effects.
[0217] Furthermore, in the embodiment of the present application, the two processes of high dynamic range imaging and motion deblurring only require the target model, a neural network model, to be implemented, and can be integrated together to achieve this, thereby saving the training cost of the neural network model.
[0218] In addition, the target model provided in the embodiment of the present application (for example, Ours in Figure 11) can be compared with the models provided by some related technologies (for example, GT in Figure 11). The comparison results are shown in Figure 11 (Figure 11 is a schematic diagram of the comparison structure provided in the embodiment of the present application). The target model provided in the embodiment of the present application can obtain clear HDR images with better visual quality.
[0219] Furthermore, the target model provided in the embodiment of the present application (for example, Ours in Table 1) can be compared with the model provided by another part of the related technology (for example, HDRCNN EDI, HDRCNN eSL-Net, etc. in Table 1). The comparison results are shown in Table 1:
[0220] Table 1
[0221] Based on Table 1, it can be seen that when tested on the same data set, the target model provided by the embodiment of the present application achieved significantly the best results in the three indicators of PSNR, SSIM and HDR-VDP-2, and the target model provided by the embodiment of the present application achieved a great performance improvement (>4dB).
[0222] The above is a detailed description of the image processing method provided by the embodiment of the present application. The following is an introduction to the model training method provided by the embodiment of the present application. Figure 12 is a flow chart of the model training method provided by the embodiment of the present application. As shown in Figure 12, the method includes:
[0223] 1201. A blurred LDR image and an event stream triggered within the exposure time of the LDR image are processed by a to-be-trained model to obtain a clear HDR image with color. The to-be-trained model is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; and perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0224] In this embodiment, when the to-be-trained model needs to be trained, a batch of training data can be first obtained, where the batch of training data includes a blurred LDR image and an event stream triggered during the exposure time of the LDR image. It should be noted that for the blurred LDR image, a clear and color-rich true HDR image is known. Therefore, the clear and color-rich true HDR image can be processed to obtain a clear and colorless true HDR image and a blurred true HDR image.
[0225] After obtaining the blurred LDR image, the blurred LDR image and the event stream can be input into the model to be trained. Therefore, after obtaining the blurred LDR image and the event stream, the model to be trained can first perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred (predicted) HDR image. Then, the model to be trained can deblur the intensity component of the blurred HDR image based on the event stream to obtain a clear and colorless (predicted) HDR image. Finally, the model to be trained can fuse the color component of the blurred HDR image with the clear and colorless HDR image to obtain a clear and color-rich (predicted) HDR image.
[0226] In one possible implementation, the model to be trained is used to: perform feature extraction on the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image.
[0227] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform operations on the intensity components of the blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0228] In one possible implementation, the model to be trained is used to: extract features of the color components of a blurred HDR image to obtain features of the color components; and fuse features of the color components with a clear HDR image without color to obtain a clear HDR image with color.
[0229] In one possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0230] In one possible implementation, the model to be trained is used to: obtain an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0231] It should be noted that, for the introduction of step 1201, reference may be made to the relevant description of steps 501 to 504 in the embodiment shown in FIG5, which will not be repeated here.
[0232] 1202. Based on the clear and colorful HDR image, the to-be-trained model is trained to obtain a target model.
[0233] After obtaining a clear and colorful HDR image, the to-be-trained model can be trained based on the clear and colorful HDR image, thereby obtaining the target model in the embodiment shown in FIG5 .
[0234] Specifically, the target model can be trained in the following ways:
[0235] (1) After obtaining a clear and colorful HDR image, since a clear and colorful true HDR image is known, a first loss and a second loss can be obtained based on the clear and colorful HDR image and the clear and colorful true HDR image. The first loss is used to indicate the difference between the clear and colorful HDR image and the clear and colorful true HDR image, and the second loss is used to indicate the difference between the first discrimination value and the second discrimination value (the first discrimination value is calculated based on the clear and colorful HDR image, and the second discrimination value is calculated based on the clear and colorful true HDR image). The second loss can be obtained by the following formula:
[0236] In the above formula, L gan is the second loss, D(I T ) is the first discriminant value, is the second discriminant value, I T For clear and colorful HDR images, A true HDR image with clarity and color.
[0237] (2) Since the real HDR image that is clear and has color, the real HDR image that is clear and has no color, and the blurred real HDR image are known, a third loss can be obtained based on the blurred real HDR image and the blurred HDR image, and a fourth loss can be obtained based on the real HDR image that is clear and has no color and the real HDR image that is clear and has no color. The third loss is used to indicate the difference between the blurred real HDR image and the blurred HDR image, and the fourth loss is used to indicate the difference between the real HDR image that is clear and has no color and the real HDR image that is clear and has no color. The third loss can be obtained by the following formula:
[0238] In the above formula, L bc is the third loss, B is the blurred HDR image, A blurred real HDR image.
[0239] The fourth loss can be obtained by the following formula:
[0240] In the above formula, L lc is the fourth loss, y T For a clear HDR image without color, A true HDR image that is clear and colorless.
[0241] (3) The first loss, the second loss, the third loss, and the fourth loss are superimposed to obtain the target loss. Then, the parameters of the model to be trained can be updated based on the target loss to obtain the model to be trained after the updated parameters are obtained. The model to be trained after the updated parameters is further trained using the next batch of training data until the model training conditions are met (for example, the target loss meets convergence, etc.), thereby obtaining the target model.
[0242] The target model obtained by training in the embodiment of the present application has high dynamic range imaging and motion deblurring functions. Specifically, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image. Then, the target model can deblur the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can fuse the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to recover the blurred HDR image from the blurred LDR image, and then continue to use the event stream to recover the clear and colorful HDR image from the blurred HDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together. There is a connection between the two, so the clear HDR image obtained in this way often has higher quality. That is to say, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0243] FIG13 is another flow chart of the model training method provided in an embodiment of the present application. As shown in FIG13 , the method includes:
[0244] 1301. A blurred LDR image and an event stream triggered during the exposure time of the blurred LDR image are processed by a to-be-trained model to obtain a clear HDR image with color. The to-be-trained model is used to: obtain a blurred LDR image and an event stream triggered during the exposure time of the blurred LDR image; deblur the blurred LDR image based on the event stream to obtain a clear LDR image; dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; and fuse the color component of the clear LDR image with the clear HDR image without color to obtain a clear HDR image with color.
[0245] In this embodiment, when the training model needs to be trained, a batch of training data can be first obtained, where the batch of training data includes a blurred LDR image and an event stream triggered during the exposure time of the LDR image. It should be noted that for the blurred LDR image, a clear and color-rich true HDR image is known. Therefore, the clear and color-rich true HDR image can be processed to obtain a clear and colorless true HDR image and a clear true LDR image.
[0246] After obtaining a blurred LDR image, the blurred LDR image and the event stream can be input into the model to be trained. Therefore, after obtaining the blurred LDR image and the event stream, the model to be trained can first deblur the blurred LDR image based on the event stream to obtain a clear LDR image. Then, the model to be trained can dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color. Finally, the model to be trained can fuse the color component of the clear LDR image with the clear HDR image without color to obtain a clear HDR image with color.
[0247] In one possible implementation, the model to be trained is used to: perform feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0248] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0249] In one possible implementation, the model to be trained is used to: extract features of the color components of a clear LDR image to obtain features of the color components; and fuse features of the color components with a clear HDR image without color to obtain a clear HDR image with color.
[0250] In one possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0251] In one possible implementation, the model to be trained is used to: obtain an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0252] It should be noted that, for the introduction of step 1301, reference may be made to the relevant description of steps 1001 to 1004 in the embodiment shown in FIG10, which will not be repeated here.
[0253] 1302. Based on the clear and colorful HDR image, the to-be-trained model is trained to obtain a target model.
[0254] After obtaining a clear and colorful HDR image, the to-be-trained model can be trained based on the clear and colorful HDR image, thereby obtaining the target model in the embodiment shown in FIG5 .
[0255] Specifically, the target model can be trained in the following ways:
[0256] (1) After obtaining a clear and colorful HDR image, since a clear and colorful true HDR image is known, a first loss and a second loss can be obtained based on the clear and colorful HDR image and the clear and colorful true HDR image. The first loss is used to indicate the difference between the clear and colorful HDR image and the clear and colorful true HDR image, and the second loss is used to indicate the difference between a first discrimination value and a second discrimination value (the first discrimination value is calculated based on the clear and colorful HDR image, and the second discrimination value is calculated based on the clear and colorful true HDR image).
[0257] (2) Since the clear real HDR image with color, the clear real HDR image without color, and the blurred real HDR image are known, a third loss can be obtained based on the clear real LDR image and the clear LDR image, and a fourth loss can be obtained based on the clear real HDR image without color and the clear HDR image without color. The third loss is used to indicate the difference between the clear real LDR image and the clear LDR image, and the fourth loss is used to indicate the difference between the clear real HDR image without color and the clear HDR image without color.
[0258] (3) The first loss, the second loss, the third loss, and the fourth loss are superimposed to obtain the target loss. Then, the parameters of the model to be trained can be updated based on the target loss to obtain the model to be trained after the updated parameters are obtained. The model to be trained after the updated parameters is further trained using the next batch of training data until the model training conditions are met (for example, the target loss meets convergence, etc.), thereby obtaining the target model.
[0259] The target model obtained by training in the embodiment of the present application has high dynamic range imaging and motion deblurring functions. Specifically, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first deblur the blurred LDR image based on the event stream to obtain a clear LDR image. Then, the target model can perform dynamic range enhancement processing on the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to restore a clear LDR image from the blurred LDR image, and then continue to use the event stream to restore a clear and colorful HDR image from the clear LDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together. There is a connection between the two, so the clear HDR image obtained in this way often has higher quality. That is to say, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0260] The above is a detailed description of the model training method provided in the embodiment of the present application. The image processing device and model training device provided in the embodiment of the present application will be introduced below. Figure 14 is a structural schematic diagram of the image processing device provided in the embodiment of the present application. As shown in Figure 14, the device includes a target model, and the device includes:
[0261] An acquisition module 1401 is configured to acquire a blurred LDR image and an event stream triggered during the exposure time of the LDR image;
[0262] A first processing module 1402 is configured to perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image;
[0263] The second processing module 1403 is configured to deblur the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color;
[0264] The third processing module 1404 is configured to fuse the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0265] In one possible implementation, the first processing module 1402 is used to: perform feature extraction on the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image.
[0266] In one possible implementation, the second processing module 1403 is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform operations on the intensity components of the blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0267] In one possible implementation, the third processing module 1404 is configured to: extract features of the color components of the blurred HDR image to obtain features of the color components; and perform feature fusion on the features of the color components and the clear HDR image without color to obtain a clear HDR image with color.
[0268] In a possible implementation, the acquisition module 1401 is configured to: acquire a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0269] In one possible implementation, the acquisition module 1401 is configured to: acquire an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0270] FIG15 is another schematic diagram of the structure of an image processing device provided in an embodiment of the present application. As shown in FIG15 , the device includes a target model, and the device includes:
[0271] An acquisition module 1501 is configured to acquire a blurred LDR image and an event stream triggered during an exposure time of the blurred LDR image;
[0272] A first processing module 1502 is configured to deblur the blurred LDR image based on the event stream to obtain a clear LDR image;
[0273] The second processing module 1503 is configured to dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color;
[0274] The third processing module 1504 is configured to fuse the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0275] In one possible implementation, the first processing module 1502 is used to: perform feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0276] In one possible implementation, the second processing module 1503 is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of the clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0277] In one possible implementation, the third processing module 1503 is used to: extract features of the color components of the clear LDR image to obtain features of the color components; and perform feature fusion on the features of the color components and the clear HDR image without color to obtain a clear HDR image with color.
[0278] In a possible implementation, the acquisition module 1501 is configured to: acquire a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0279] In one possible implementation, the acquisition module 1501 is configured to: acquire an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0280] FIG16 is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. As shown in FIG16 , the device includes:
[0281] The processing module 1601 is configured to process the blurred LDR image and the event stream triggered during the exposure time of the LDR image using a to-be-trained model to obtain a clear HDR image with color. The to-be-trained model is configured to: obtain the blurred LDR image and the event stream triggered during the exposure time of the LDR image; perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image; perform deblurring processing on the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color; and perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0282] The training module 1602 is used to train the to-be-trained model based on the clear and colorful HDR image to obtain a target model.
[0283] In one possible implementation, the model to be trained is used to: perform feature extraction on the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image.
[0284] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform operations on the intensity components of the blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0285] In one possible implementation, the model to be trained is used to: extract features of the color components of a blurred HDR image to obtain features of the color components; and fuse features of the color components with a clear HDR image without color to obtain a clear HDR image with color.
[0286] In one possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0287] In one possible implementation, the model to be trained is used to: obtain an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0288] FIG17 is another structural diagram of the model training device provided in an embodiment of the present application. As shown in FIG17 , the device includes:
[0289] The processing module 1701 is configured to process the blurred LDR image and the event stream triggered during the exposure time of the blurred LDR image using a to-be-trained model to obtain a clear HDR image with color. The to-be-trained model is configured to: obtain the blurred LDR image and the event stream triggered during the exposure time of the blurred LDR image; deblur the blurred LDR image based on the event stream to obtain a clear LDR image; dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; and fuse the color component of the clear LDR image with the clear HDR image without color to obtain a clear HDR image with color.
[0290] The training module 1702 is used to train the to-be-trained model based on a clear and colorful HDR image to obtain a target model.
[0291] In one possible implementation, the model to be trained is used to: perform feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0292] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0293] In one possible implementation, the model to be trained is used to: extract features of the color components of a clear LDR image to obtain features of the color components; and fuse features of the color components with a clear HDR image without color to obtain a clear HDR image with color.
[0294] In one possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB format; and convert the LDR image presented in RGB format into an LDR image presented in YUV format.
[0295] In one possible implementation, the model to be trained is used to: obtain an event stream triggered during the exposure time of an LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0296] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.
[0297] The embodiment of the present application also relates to an execution device, and Figure 18 is a structural diagram of the execution device provided by the embodiment of the present application. As shown in Figure 18, the execution device 1800 can be specifically manifested as a mobile phone, a tablet, a laptop computer, a smart wearable device, a server, etc., which are not limited here. Among them, the image processing device described in the embodiment corresponding to Figure 14 or Figure 15 can be deployed on the execution device 1800 to implement the image processing function in the embodiment corresponding to Figure 5 or Figure 10. Specifically, the execution device 1800 includes: a receiver 1801, a transmitter 1802, a processor 1803 and a memory 1804 (wherein the number of processors 1803 in the execution device 1800 can be one or more, and Figure 18 takes one processor as an example), wherein the processor 1803 may include an application processor 18031 and a communication processor 18032. In some embodiments of the present application, the receiver 1801, the transmitter 1802, the processor 1803 and the memory 1804 can be connected via a bus or other means.
[0298] Memory 1804 may include read-only memory and random access memory, and provides instructions and data to processor 1803. A portion of memory 1804 may also include non-volatile random access memory (NVRAM). Memory 1804 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.
[0299] Processor 1803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.
[0300] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1803. Processor 1803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1803. The above processor 1803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1803 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1804, and processor 1803 reads information from memory 1804 and, in conjunction with its hardware, completes the steps of the above method.
[0301] Receiver 1801 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1802 can be used to output digital or character information through the first interface. Transmitter 1802 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1802 can also include a display device such as a display screen.
[0302] In one embodiment of the present application, the processor 1803 is configured to obtain a clear and colorful HDR image through the target model in the embodiment corresponding to FIG. 5 or FIG. 10 .
[0303] The present application also relates to a training device. FIG19 is a schematic diagram of the structure of the training device provided by the present application. As shown in FIG19 , the training device 1900 is implemented by one or more servers. The training device 1900 may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1919 (e.g., one or more processors) and memory 1932, and one or more storage media 1930 (e.g., one or more mass storage devices) storing application programs 1942 or data 1944. The memory 1932 and storage medium 1930 may be temporary storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown), each module may include a series of instruction operations in the training device. Furthermore, the CPU 1919 may be configured to communicate with the storage medium 1930 to execute the series of instruction operations in the storage medium 1930 on the training device 1900.
[0304] The training device 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958; or, one or more operating systems 1941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0305] Specifically, the training device can execute the model training method in the embodiment corresponding to Figure 12 or Figure 13 to obtain the target model.
[0306] An embodiment of the present application also relates to a computer storage medium, which stores a program for signal processing. When the computer storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.
[0307] An embodiment of the present application also relates to a computer program product, which stores instructions that, when executed by a computer, enable the computer to execute the steps executed by the aforementioned execution device, or enable the computer to execute the steps executed by the aforementioned training device.
[0308] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
[0309] Specifically, see Figure 20, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 2000. NPU 2000 is mounted on the host CPU (host CPU) as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 2003, which is controlled by controller 2004 to extract matrix data from memory and perform multiplication operations.
[0310] In some implementations, the arithmetic circuit 2003 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 2003 is a two-dimensional systolic array. The arithmetic circuit 2003 can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2003 is a general-purpose matrix processor.
[0311] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 2002 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 2001 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 2008.
[0312] Unified memory 2006 is used to store input and output data. Weight data is directly transferred to weight memory 2002 through the Direct Memory Access Controller (DMAC) 2005. Input data is also transferred to unified memory 2006 through the DMAC.
[0313] BIU stands for Bus Interface Unit, i.e., bus interface unit 2013 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 2009 .
[0314] The bus interface unit 2013 (BIU) is used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
[0315] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2006 or transfer weight data to the weight memory 2002 or transfer input data to the input memory 2001.
[0316] The vector calculation unit 2007 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 2003, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of the predicted label plane.
[0317] In some implementations, the vector calculation unit 2007 can store the processed output vector in the unified memory 2006. For example, the vector calculation unit 2007 can apply a linear function or a nonlinear function to the output of the operation circuit 2003, such as linear interpolation of the predicted label plane extracted by the convolution layer, or another example is a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 2007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2003, for example, for use in subsequent layers in a neural network.
[0318] An instruction fetch buffer 2009 connected to the controller 2004 is used to store instructions used by the controller 2004;
[0319] Unified memory 2006, input memory 2001, weight memory 2002, and instruction fetch memory 2009 are all on-chip memories. External memories are private to the NPU hardware architecture.
[0320] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.
[0321] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0322] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.
[0323] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0324] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).
Claims
1. An image processing method, characterized in that: The method is implemented by a target model, and the method comprises: Acquire a blurred low dynamic range (LDR) image and a stream of events triggered during an exposure time of the LDR image; Based on the event stream, dynamic range enhancement processing is performed on the LDR image to obtain a blurred high dynamic range HDR image; Based on the event stream, deblurring the intensity component of the blurred HDR image to obtain a clear HDR image without color; The color components of the blurred HDR image and the clear HDR image without color are fused to obtain a clear HDR image with color.
2. The method according to claim 1, characterized in that The step of performing dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image comprises: Extracting features from the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; The features of the LDR image and the features of the event stream are fused to obtain a blurred HDR image.
3. The method according to claim 1 or 2, characterized in that: The deblurring the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color comprises: Dividing the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, wherein the exposure time includes the first exposure time and the second exposure time; Extracting features from the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; Superimposing the features of the first event stream and the features of the second event stream to obtain superimposed features; The intensity component of the blurred HDR image and the superimposed features are operated to obtain a clear HDR image without color.
4. The method according to any one of claims 1 to 3, characterized in that: The step of fusing the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color includes: Performing feature extraction on color components of the blurred HDR image to obtain features of the color components; The features of the color components and the clear HDR image without color are fused to obtain a clear HDR image with color.
5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining a blurred LDR image comprises: Get the blurred LDR image in RGB form; Convert an LDR image presented in RGB form to an LDR image presented in YUV form.
6. The method according to any one of claims 1 to 5, characterized in that: The event flow triggered within the exposure time of acquiring the LDR image includes: Acquire an event stream triggered within the exposure time of the LDR image presented as discrete data; The event stream presented as discrete data is converted into an event stream presented as sequence data.
7. An image processing method, characterized in that: The method is implemented by a target model, and the method comprises: Acquire a blurred LDR image and an event stream triggered during an exposure time of the blurred LDR image; Based on the event stream, deblurring the blurred LDR image to obtain a clear LDR image; Based on the event stream, dynamically enhancing the intensity component of the clear LDR image to obtain a clear HDR image without color; The color components of the clear LDR image and the clear HDR image without color are fused to obtain a clear HDR image with color.
8. The method according to claim 7, characterized in that The deblurring the blurred LDR image based on the event stream to obtain a clear LDR image comprises: Performing feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; The features of the blurred LDR image and the features of the event stream are fused to obtain a clear LDR image.
9. The method according to claim 7 or 8, characterized in that: The step of dynamically enhancing the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color comprises: Dividing the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, wherein the exposure time includes the first exposure time and the second exposure time; Extracting features from the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; Superimposing the features of the first event stream and the features of the second event stream to obtain superimposed features; The intensity component of the clear LDR image and the superimposed features are operated to obtain a clear HDR image without color.
10. The method according to any one of claims 7 to 9, characterized in that: The step of fusing the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color comprises: Performing feature extraction on the color components of the clear LDR image to obtain features of the color components; The features of the color components and the clear HDR image without color are fused to obtain a clear HDR image with color.
11. The method according to any one of claims 7 to 10, characterized in that: The step of obtaining a blurred LDR image comprises: Get the blurred LDR image in RGB form; Convert an LDR image presented in RGB form to an LDR image presented in YUV form.
12. The method according to any one of claims 7 to 11, characterized in that: The event flow triggered within the exposure time of acquiring the LDR image includes: Acquire an event stream triggered within the exposure time of the LDR image presented as discrete data; The event stream presented as discrete data is converted into an event stream presented as sequence data.
13. A model training method, characterized in that: The method comprises: The blurred LDR image and the event stream triggered within the exposure time of the LDR image are processed by the model to be trained to obtain a clear HDR image with color. The model to be trained is used to: obtain the blurred LDR image and the event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color; Based on the clear and colorful HDR image, the model to be trained is trained to obtain a target model.
14. A model training method, characterized in that: The method comprises: The blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image are processed by the model to be trained to obtain a clear HDR image with color. The model to be trained is used to: obtain the blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblur the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhance the intensity component of the clear LDR image to obtain a clear HDR image without color; fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color; Based on the clear and colorful HDR image, the model to be trained is trained to obtain a target model.
15. An image processing device, characterized in that: The device comprises a target model, and the device comprises: An acquisition module, used for acquiring a blurred LDR image and an event stream triggered during the exposure time of the LDR image; A first processing module, configured to perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image; A second processing module is used for deblurring the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color; The third processing module is used to fuse the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
16. An image processing device, characterized in that: The device comprises a target model, and the device comprises: An acquisition module, used for acquiring a blurred LDR image and an event stream triggered during an exposure time of the blurred LDR image; A first processing module, configured to perform a deblurring process on the blurred LDR image based on the event stream to obtain a clear LDR image; A second processing module is used to dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; The third processing module is used to fuse the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
17. A model training device, characterized in that: The device comprises: A processing module, used for processing a blurred LDR image and an event stream triggered within the exposure time of the LDR image through a to-be-trained model to obtain a clear HDR image with color, wherein the to-be-trained model is used for: acquiring a blurred LDR image and an event stream triggered within the exposure time of the LDR image; performing dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image; performing deblurring processing on an intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color; performing fusion processing on a color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color; The training module is used to train the to-be-trained model based on the clear and colorful HDR image to obtain a target model.
18. A model training device, characterized in that: The device comprises: A processing module, used for processing a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image through a model to be trained to obtain a clear HDR image with color, wherein the model to be trained is used for: acquiring a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; deblurring the blurred LDR image based on the event stream to obtain a clear LDR image; dynamically enhancing the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; fusing the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color; The training module is used to train the to-be-trained model based on the clear and colorful HDR image to obtain a target model.
19. An image processing device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the image processing device performs the method according to any one of claims 1 to 14.
20. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 14.
21. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 14.
Citation Information
Patent Citations
Reverse tone mapping algorithm based on frequency domain
CN104463820A
Image enhancement method and image enhancement device
CN113284054A
Self-adaptive visual tracking method, system and device and storage medium
CN115984327A
Fire detection system and method employing digital images processing
US20150213621A1
Event detection method and device, movable platform, and computer-readable storage medium
WO2022027444A1
Cited By
Image deblurring method based on improved U-Net
CN120451003A
