Image processing method and related device thereof
Through the target model, dynamic range enhancement, debuffering and fusion processing of blurred LDR images and event streams is solved, and the image quality problems caused by independent independent high-dynamic range imaging and motion debuffing in the prior art are generated and high-quality HDR images are reduced. model training cost.
Patent Information
- Application Number
- CN202310647524.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-01
- Publication Date
- 2024-12-03
AI Technical Summary
When the prior art realizes high dynamic range imaging and motion deblurring, the two processes are carried out independently, resulting in poor quality of the final clear HDR image, and problems such as color flaws, abnormal image brightness and blurred details.
Through the target model, the blurred LDR images and event stream input within their exposure time are carried out, dynamic range enhancement, intensity component debuffering and color component fusion are realized, and the coupling processing of high dynamic range imaging and motion debuffering is achieved to generate high-quality HDR images.
The generated clear HDR image quality is improved, there are no color flaws, abnormal image brightness and blurred details, and the training cost of neural network models is reduced.
Smart Images

Figure CN119067879A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to an image processing method and related devices. Background Art
[0002] As key issues in computer vision applications, high dynamic range (HDR) imaging and motion deblurring have received increasing attention. With the rapid development of AI technology, more and more device manufacturers are embedding neural network models in AI technology into their devices to obtain clear HDR images through the models.
[0003] In the related art, two neural network models are usually used to realize high dynamic range imaging and motion deblurring respectively. Generally, the blurred low dynamic range (LDR) image is processed by the first neural network model to obtain a blurred HDR image, and then the blurred HDR image is processed by the second neural network model to obtain a clear HDR image. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0004] In the above process, since the high dynamic range imaging and motion deblurring processes are implemented independently of each other, this often results in poor quality of the final clear HDR image. For example, the final clear HDR image has various quality problems such as color defects, abnormal image brightness, and varying degrees of detail blur. Summary of the invention
[0005] The embodiments of the present application provide an image processing method and a related device thereof, which can utilize an event stream to restore a high-quality and clear HDR image from a blurred LDR image.
[0006] A first aspect of an embodiment of the present application provides an image processing method, which can be implemented by a target model. The method includes:
[0007] When high dynamic range imaging and motion deblurring are required, the blurred LDR image and the event stream triggered during the exposure time of the blurred LDR image can be input into the target model. It should be noted that the event stream is triggered by the change in brightness of the shooting scene during the exposure time of the blurred LDR image. It can be understood that the blurred LDR image is used to present the aforementioned shooting scene.
[0008] After obtaining the blurred LDR image and event stream, the target model can use the event stream to perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image. After obtaining the blurred HDR image, the target model can split the blurred HDR image into the intensity component of the blurred HDR image and the color component of the blurred HDR image, and use the event stream to deblur the intensity component of the blurred HDR image to obtain a clear HDR image without color. After obtaining the clear HDR image without color, the target model can also fuse the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0009] It can be seen from the above method that the target model can use the event stream to recover the blurred HDR image from the blurred LDR image, and then continue to use the event stream to recover the clear and colorful HDR image from the blurred HDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together and there is a connection between the two. Therefore, the clear HDR image obtained in this way often has a higher quality. In other words, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0010] In one possible implementation, based on the event stream, the dynamic range enhancement processing is performed on the LDR image to obtain a blurred HDR image, which includes: extracting features from the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and fusing the features of the LDR image and the features of the event stream to obtain a blurred HDR image. In the aforementioned implementation, the target model may first perform feature extraction on the blurred LDR image to obtain initial features of the LDR image, then perform further feature extraction on the initial features of the LDR image to obtain intermediate features of the LDR image, and then perform further feature extraction on the intermediate features of the LDR image to obtain final features of the LDR image. Similarly, the target model may first perform feature extraction on the event stream to obtain initial features of the event stream, then perform further feature extraction on the initial features of the event stream to obtain intermediate features of the event stream, and then perform further feature extraction on the intermediate features of the event stream to obtain final features of the event stream. The target model can first perform feature fusion on the final features of the LDR image and the final features of the event stream to obtain the first fused feature, and then perform feature fusion on the first fused feature, the intermediate features of the LDR image, and the intermediate features of the event stream to obtain the second fused feature, and then perform feature fusion on the second fused feature, the initial features of the LDR image, and the initial features of the event stream to obtain the third fused feature, which is the blurred HDR image. It can be seen that the target model can project the blurred LDR image and the event stream into the same feature space, and then fuse the obtained features to accurately estimate the blurred HDR image.
[0011] In a possible implementation, based on the event stream, deblurring the intensity component of the blurred HDR image to obtain a clear HDR image without color includes: dividing the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, the exposure time including the first exposure time and the second exposure time; extracting features from the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimposing the features of the first event stream and the features of the second event stream to obtain superimposed features; and calculating the intensity component of the blurred HDR image and the superimposed features to obtain a clear HDR image without color. In the aforementioned implementation, after obtaining the blurred HDR image, the target model can first divide the exposure time of the blurred LDR image into the first exposure time and the second exposure time, based on which, the target model can divide the event stream into the first event stream triggered within the first exposure time and the second event stream triggered within the second exposure time. After obtaining the first event stream and the second event stream, the target model can extract features from the first event stream and the second event stream respectively, thereby obtaining features of the first event stream and features of the second event stream accordingly. Then, the target model can superimpose the features of the first event stream and the features of the second event stream to obtain the superimposed features. After obtaining the superimposed features, the target model can extract the intensity component of the blurred HDR image from the blurred HDR image, and perform operations on the intensity component of the blurred HDR image and the superimposed features to obtain a clear HDR image without color. It can be seen that the target model can use the high temporal resolution of the event stream to perform motion prior on the intensity component of the blurred HDR image, thereby obtaining a fuzzy factor of nonlinear motion (the aforementioned superimposed features), so as to accurately obtain a clear HDR image without color based on the factor.
[0012] In one possible implementation, the color components of a blurred HDR image and a clear HDR image without color are fused to obtain a clear HDR image with color, including: extracting features from the color components of the blurred HDR image to obtain features of the color components; and fusing features of the color components with the clear HDR image without color to obtain a clear HDR image with color. In the aforementioned implementation, after obtaining a clear HDR image without color, the target model can extract the color components of the blurred HDR image from the blurred HDR image, and extract features from the color components of the blurred HDR image to obtain features of the color components. After obtaining the features of the color components, the target model can fuse the features of the color components with the clear HDR image without color to obtain a clear HDR image with color. It can be seen that the target model can colorize the clear HDR image without color, thereby finally obtaining a clear HDR image with color.
[0013] In a possible implementation manner, obtaining a blurred LDR image includes: obtaining a blurred LDR image presented in an RGB form; and converting the LDR image presented in an RGB form into an LDR image presented in a YUV form.
[0014] In one possible implementation, acquiring the event stream triggered within the exposure time of the LDR image includes: acquiring the event stream triggered within the exposure time of the LDR image presented as discrete data; and converting the event stream presented as discrete data into an event stream presented as sequence data.
[0015] A second aspect of an embodiment of the present application provides an image processing method, which can be implemented through a target model. The method includes: acquiring a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblurring the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhancing the intensity component of the clear LDR image to obtain a clear HDR image without color; and fusing the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0016] In one possible implementation, based on the event stream, the blurred LDR image is deblurred to obtain a clear LDR image, including: extracting features from the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and fusing features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0017] In one possible implementation, based on an event stream, the intensity component of a clear LDR image is dynamically enhanced to obtain a clear HDR image without color, including: dividing the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, the exposure time including the first exposure time and the second exposure time; performing feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimposing the features of the first event stream and the features of the second event stream to obtain superimposed features; and performing calculations on the intensity component of the clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0018] In one possible implementation, the color components of a clear LDR image and a clear HDR image without color are fused to obtain a clear HDR image with color, including: extracting features of the color components of the clear LDR image to obtain features of the color components; and fusing features of the color components and the clear HDR image without color to obtain a clear HDR image with color.
[0019] In a possible implementation manner, obtaining a blurred LDR image includes: obtaining a blurred LDR image presented in an RGB form; and converting the LDR image presented in an RGB form into an LDR image presented in a YUV form.
[0020] In one possible implementation, acquiring the event stream triggered within the exposure time of the LDR image includes: acquiring the event stream triggered within the exposure time of the LDR image presented as discrete data; and converting the event stream presented as discrete data into an event stream presented as sequence data.
[0021] A third aspect of an embodiment of the present application provides a model training method, which includes: processing a blurred LDR image and an event stream triggered within the exposure time of the LDR image through a to-be-trained model to obtain a clear HDR image with color, wherein the to-be-trained model is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color; and based on the clear HDR image with color, train the to-be-trained model to obtain a target model.
[0022] In one possible implementation, the model to be trained is used to: extract features from the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and features of the event stream to obtain a blurred HDR image.
[0023] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0024] In one possible implementation, the model to be trained is used to: extract features of color components of a blurred HDR image to obtain features of the color components; and fuse features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0025] In a possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0026] In one possible implementation, the model to be trained is used to: obtain the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0027] A fourth aspect of an embodiment of the present application provides a model training method, which includes: processing a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image through a model to be trained to obtain a clear HDR image with color, wherein the model to be trained is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblur the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhance the intensity component of the clear LDR image to obtain a clear HDR image without color; fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color; and train the model to be trained based on the clear HDR image with color to obtain a target model.
[0028] In one possible implementation, the model to be trained is used to: extract features from the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0029] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0030] In one possible implementation, the model to be trained is used to: extract features of color components of a clear LDR image to obtain features of the color components; and fuse features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0031] In a possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0032] In one possible implementation, the model to be trained is used to: obtain the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0033] A fifth aspect of an embodiment of the present application provides an image processing device, which includes a target model, and the device includes: an acquisition module, used to acquire a blurred LDR image and an event stream triggered within the exposure time of the LDR image; a first processing module, used to perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image; a second processing module, used to deblur the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color; a third processing module, used to perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0034] In one possible implementation, the first processing module 1402 is used to: perform feature extraction on the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image.
[0035] In one possible implementation, the second processing module 1403 is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, the exposure time including the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; perform calculations on the intensity components of the blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0036] In one possible implementation, the third processing module 1404 is used to: extract features of the color components of the blurred HDR image to obtain features of the color components; and perform feature fusion on the features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0037] In a possible implementation, the acquisition module 1401 is used to: acquire a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0038] In a possible implementation, the acquisition module 1401 is used to: acquire the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0039] A sixth aspect of an embodiment of the present application provides an image processing device, which includes a target model, and the device includes: an acquisition module, used to acquire a blurred LDR image, and an event stream triggered within the exposure time of the blurred LDR image; a first processing module, used to deblur the blurred LDR image based on the event stream to obtain a clear LDR image; a second processing module, used to dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; a third processing module, used to fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0040] In one possible implementation, the first processing module 1502 is used to: perform feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0041] In one possible implementation, the second processing module 1503 is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, the exposure time including the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of the clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0042] In one possible implementation, the third processing module 1503 is used to: extract features of the color components of the clear LDR image to obtain features of the color components; and perform feature fusion on the features of the color components and the clear HDR image without color to obtain a clear HDR image with color.
[0043] In a possible implementation, the acquisition module 1501 is used to: acquire a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0044] In a possible implementation, the acquisition module 1501 is used to: acquire the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0045] A seventh aspect of an embodiment of the present application provides a model training device, which includes: a processing module, which is used to process a blurred LDR image and an event stream triggered within the exposure time of the LDR image through a model to be trained to obtain a clear HDR image with color, and the model to be trained is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color; a training module, which is used to train the model to be trained based on the clear HDR image with color to obtain a target model.
[0046] In one possible implementation, the model to be trained is used to: extract features from the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and features of the event stream to obtain a blurred HDR image.
[0047] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0048] In one possible implementation, the model to be trained is used to: extract features of color components of a blurred HDR image to obtain features of the color components; and fuse features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0049] In a possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0050] In one possible implementation, the model to be trained is used to: obtain the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0051] An eighth aspect of an embodiment of the present application provides a model training device, which includes: a processing module, used to process a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image through a model to be trained to obtain a clear HDR image with color, and the model to be trained is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblur the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhance the intensity component of the clear LDR image to obtain a clear HDR image without color; fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color; a training module, used to train the model to be trained based on the clear HDR image with color to obtain a target model.
[0052] In one possible implementation, the model to be trained is used to: extract features from the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0053] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0054] In one possible implementation, the model to be trained is used to: extract features of color components of a clear LDR image to obtain features of the color components; and fuse features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0055] In a possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0056] In one possible implementation, the model to be trained is used to: obtain the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0057] A ninth aspect of an embodiment of the present application provides a fault prediction device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the fault prediction device executes the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.
[0058] The tenth aspect of an embodiment of the present application provides a model training device, which includes a memory and a processor; the memory stores code, and the processor is configured to execute the code. When the code is executed, the model training device executes the method described in the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0059] The eleventh aspect of an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0060] The twelfth aspect of an embodiment of the present application provides a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes a method as described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0061] In a possible implementation manner, the processor is coupled to the memory through an interface.
[0062] In a possible implementation, the chip system also includes a memory, in which a computer program or computer instructions are stored.
[0063] A thirteenth aspect of an embodiment of the present application provides a computer storage medium, which stores a computer program. When the program is executed by a computer, the computer implements the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0064] A fourteenth aspect of an embodiment of the present application provides a computer program product, which stores instructions, which, when executed by a computer, enable the computer to implement the method described in the first aspect, any possible implementation of the first aspect, the second aspect, any possible implementation of the second aspect, the third aspect, any possible implementation of the third aspect, the fourth aspect, or any possible implementation of the fourth aspect.
[0065] In an embodiment of the present application, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and an event stream triggered during the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image. Then, the target model can perform deblurring processing on the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to recover the blurred HDR image from the blurred LDR image, and then continue to use the event stream to recover the clear and colorful HDR image from the blurred HDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together and there is a connection between the two. Therefore, the clear HDR image obtained in this way often has higher quality. In other words, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 A structural diagram of the main framework of artificial intelligence;
[0067] Figure 2a A schematic diagram of the structure of an image processing system provided in an embodiment of the present application;
[0068] Figure 2b Another structural schematic diagram of the image processing system provided in an embodiment of the present application;
[0069] Figure 2c A schematic diagram of an image processing device provided in an embodiment of the present application;
[0070] Figure 3 A schematic diagram of the system 100 architecture provided in an embodiment of the present application;
[0071] Figure 4 A schematic diagram of the structure of the target model provided in the embodiment of the present application;
[0072] Figure 5 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application;
[0073] Figure 6 A schematic diagram of the structure of a dynamic enhancement module provided in an embodiment of the present application;
[0074] Figure 7 A schematic diagram of the structure of a motion deblurring module provided in an embodiment of the present application;
[0075] Figure 8 A schematic diagram of the structure of a color enhancement network provided in an embodiment of the present application;
[0076] Fig. 9 Another schematic diagram of the structure of the target model provided in the embodiment of the present application;
[0077] Fig.10 Another schematic diagram of the image processing method provided in the embodiment of the present application;
[0078] Fig.11 A schematic diagram of a comparative structure provided for an embodiment of the present application;
[0079] Fig.12 A schematic diagram of a flow chart of a model training method provided in an embodiment of the present application;
[0080] Fig.13 Another schematic diagram of the flow chart of the model training method provided in the embodiment of the present application;
[0081] Fig.14 A structural schematic diagram of an image processing device is provided for an embodiment of the present application;
[0082] Fig.15 Another structural schematic diagram of the image processing device provided in an embodiment of the present application;
[0083] Fig.16 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;
[0084] Fig.17 Another structural schematic diagram of the model training device provided in an embodiment of the present application;
[0085] Fig.18 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;
[0086] Fig.19 A schematic diagram of the structure of the training device provided in the embodiment of the present application;
[0087] Fig. 20 A schematic diagram of the structure of the chip provided in the embodiment of the present application. DETAILED DESCRIPTION
[0088] The embodiments of the present application provide an image processing method and a related device thereof, which can utilize an event stream to restore a high-quality and clear HDR image from a blurred LDR image.
[0089] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0090] As key issues in computer vision applications, high dynamic range imaging and motion deblurring have received increasing attention. With the rapid development of AI technology, more and more device manufacturers are embedding neural network models in AI technology into their devices to obtain clear HDR images through the models.
[0091] In the related art, two neural network models are usually used to realize high dynamic range imaging and motion deblurring respectively. Generally, the blurred LDR image is processed by the first neural network model to obtain a blurred HDR image, and then the blurred HDR image is processed by the second neural network model to obtain a clear HDR image. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0092] In the above process, since the high dynamic range imaging and motion deblurring processes are implemented independently of each other, this often results in poor quality of the final clear HDR image. For example, the final clear HDR image has various quality problems such as color defects, abnormal image brightness, and varying degrees of detail blur.
[0093] Furthermore, the two processes of high dynamic range imaging and motion deblurring require two neural network models to implement and cannot be integrated together, resulting in excessively high training costs for the neural network models.
[0094] In order to solve the above problems, the embodiment of the present application provides an image processing method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technical discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by sensing the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Using artificial intelligence for data processing is a common application of artificial intelligence.
[0095] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 It is a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.
[0096] (1) Infrastructure
[0097] The infrastructure provides computing power support for the AI system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.
[0098] (2) Data
[0099] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.
[0100] (3) Data processing
[0101] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.
[0102] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0103] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0104] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.
[0105] (4) General ability
[0106] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0107] (5) Smart products and industry applications
[0108] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.
[0109] Next, several application scenarios of this application are introduced.
[0110] Figure 2a A schematic diagram of the structure of an image processing system provided in an embodiment of the present application, the image processing system includes a user device and a data processing device. The user device includes an intelligent terminal such as a mobile phone, a personal computer or an information processing center. The user device is the initiator of image processing, and as the initiator of the image processing request, the request is usually initiated by the user through the user device.
[0111] The above-mentioned data processing device can be a device or server with data processing function such as a cloud server, a network server, an application server and a management server. The data processing device receives the image processing request from the intelligent terminal through the interactive interface, and then performs image processing in the form of machine learning, deep learning, search, reasoning, decision-making, etc. through the memory for storing data and the processor link for data processing. The memory in the data processing device can be a general term, including local storage and databases for storing historical data. The database can be on the data processing device or on other network servers.
[0112] exist Figure 2a In the image processing system shown, the user device can receive the user's instruction, for example, the user device can obtain the LDR image input / selected by the user, and then initiate a request to the data processing device, so that the data processing device executes the image processing application for the LDR image obtained by the user device, thereby obtaining a clear HDR image. Exemplarily, the user device can obtain the blurred LDR image input by the user, and then initiate an image processing request to the data processing device, so that the data processing device performs a series of processing on the blurred LDR image based on the image processing request, thereby obtaining the processing result of the blurred LDR image, that is, a clear HDR image.
[0113] exist Figure 2a In the embodiment of the present application, the data processing device can execute the image processing method of the embodiment of the present application.
[0114] Figure 2b Another structural diagram of the image processing system provided in the embodiment of the present application is shown in FIG. Figure 2b In the process, the user device directly acts as a data processing device. The user device can directly obtain input from the user and directly process it by the hardware of the user device itself. The specific process is similar to Figure 2a Similarly, please refer to the above description and will not repeat them here.
[0115] exist Figure 2bIn the image processing system shown, the user device can receive instructions from the user. For example, the user device can obtain a blurred LDR image input by the user, and then perform a series of processing on the blurred LDR image to obtain a processing result of the blurred LDR image, that is, a clear HDR image.
[0116] exist Figure 2b In the embodiment of the present application, the user equipment itself can execute the image processing method.
[0117] Figure 2c A schematic diagram of image processing related equipment provided in an embodiment of the present application.
[0118] Above Figure 2a and Figure 2b The user equipment in the example may specifically be Figure 2c The local device 301 or the local device 302 in Figure 2a The data processing device in the embodiment may be Figure 2c The execution device 210 in the embodiment, wherein the data storage system 250 can store the data to be processed by the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or can be set on the cloud or other network servers.
[0119] Figure 2a and Figure 2b The processor in the image processing apparatus can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to execute image processing applications on the image, thereby obtaining corresponding processing results.
[0120] Figure 3 A schematic diagram of the system 100 architecture provided in an embodiment of the present application, in Figure 3 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data may include: various tasks to be scheduled, callable resources and other parameters in the embodiment of the application.
[0121] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.
[0122] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.
[0123] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or tasks, and the corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from the training samples collected by the data collection device 160.
[0124] exist Figure 3 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.
[0125] It is worth noting that Figure 3 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110. Figure 3 As shown, a neural network can be obtained by training according to the training device 120.
[0126] The present application also provides a chip including a neural network processor NPU. The chip can be arranged in Figure 3 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule.
[0127] Neural network processor NPU, NPU is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of NPU is the operation circuit, and the controller controls the operation circuit to extract data from the memory (weight memory or input memory) and perform operations.
[0128] In some implementations, the arithmetic circuit includes multiple processing units (process engines, PEs) inside. In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.
[0129] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory and performs matrix operations with matrix B. The partial results or final results of the matrix are stored in the accumulator.
[0130] The vector computing unit can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computing unit can be used for network calculations of non-convolutional / non-FC layers in neural networks, such as pooling, batch normalization, local response normalization, etc.
[0131] In some implementations, the vector computation unit can store the processed output vector to a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the computation circuit, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector computation unit generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit, such as for use in a subsequent layer in a neural network.
[0132] The unified memory is used to store input data and output data.
[0133] The weight data is directly transferred from the external memory to the input memory and / or the unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.
[0134] The bus interface unit (BIU) is used to implement interaction between the main CPU, DMAC and instruction fetch memory through the bus.
[0135] An instruction fetch buffer connected to the controller, used to store instructions used by the controller;
[0136] The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.
[0137] Generally, the unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDRSDRAM), a high bandwidth memory (HBM) or other readable and writable memory.
[0138] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.
[0139] (1) Neural Network
[0140] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:
[0141]
[0142] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.
[0143] The work of each layer in the neural network can be described by the mathematical expression y=a(Wx+b): From a physical level, the work of each layer in the neural network can be understood as completing the transformation from the input space to the output space (i.e., the row space to the column space of the matrix) through five operations on the input space (the set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Zoom in / out; 3. Rotation; 4. Translation; 5. "Bending". Among them, operations 1, 2, and 3 are completed by Wx, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network is to finally obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially about learning how to control spatial transformations, or more specifically, learning the weight matrix.
[0144] Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.
[0145] (2) Back propagation algorithm
[0146] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.
[0147] The method provided in the present application is described below from the training side of the neural network and the application side of the neural network.
[0148] The model training method provided in the embodiment of the present application involves the processing of data sequences, and can be specifically applied to methods such as data training, machine learning, and deep learning, and symbolizes and formalizes intelligent information modeling, extraction, preprocessing, and training of training data (for example, the blurred LDR image in the model training method provided in the embodiment of the present application), and finally obtains a trained neural network (for example, the target model in the model training method provided in the embodiment of the present application); and the image processing method provided in the embodiment of the present application can use the above-mentioned trained neural network to input input data (for example, the blurred LDR image in the image processing method provided in the embodiment of the present application) into the trained neural network to obtain output data (for example, a clear and colorful HDR image in the image processing method provided in the embodiment of the present application). It should be noted that the model training method and image processing method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.
[0149] The image processing method provided in the embodiment of the present application can be implemented by a target model, and the target model can have a variety of structures. The first structure of the target model is briefly introduced below. Figure 4 A schematic diagram of the structure of the target model provided in the embodiment of the present application, such as Figure 4 As shown, the target model includes an event-based dynamic enhancement module, an event-based motion deblurring module and a color enhancement network, wherein the input end of the dynamic enhancement module serves as the input end of the entire target model, the first output end of the dynamic enhancement module is connected to the input end of the motion deblurring module, the second output end of the dynamic enhancement module is connected to the first input end of the color enhancement network, the output end of the motion deblurring module is connected to the second input end of the color enhancement network, and the output end of the color enhancement network serves as the output end of the entire target model. In order to understand Figure 4 The workflow of the target model shown below is combined with Figure 5This workflow is introduced. Figure 5 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method includes:
[0150] 501. Obtain a blurred LDR image and an event stream triggered within the exposure time of the LDR image.
[0151] In this embodiment, when high dynamic range imaging and motion deblurring are required, a blurred LDR image may be captured first, and an event stream triggered during the exposure time of the blurred LDR image may be collected. It should be noted that the event stream is triggered by a change in brightness of the captured scene (i.e., the scene corresponding to the blurred LDR image) during the exposure time of the blurred LDR image.
[0152] After obtaining the blurred LDR image and the event stream, the blurred LDR image and the event stream can be input into the target model, so the target model can successfully obtain the blurred LDR image and the event stream.
[0153] Specifically, the target model can obtain the blurred LDR image in the following way:
[0154] Since the blurred LDR image obtained by shooting is presented in RGB format, after the blurred LDR image presented in RGB format is input into the target model, the dynamic enhancement module of the target model can convert the blurred LDR image presented in RGB format into a blurred LDR image presented in YUV format (also called YCrCb format).
[0155] For example, Figure 6 As shown ( Figure 6 A structural schematic diagram of a dynamic enhancement module provided in an embodiment of the present application) The dynamic enhancement module includes a first conversion unit, a second conversion unit, and a multimodal fusion unit. After receiving a blurred LDR image L in RGB format, the first conversion unit may convert L into a blurred LDR image L′ in YCrCb format, and send L′ to the multimodal fusion unit.
[0156] More specifically, the target model can obtain the event stream in the following ways:
[0157] Since the event stream triggered during the exposure time of the blurred LDR image is captured and presented as discrete data, after the event stream presented as discrete data is input into the target model, the dynamic enhancement module of the target model can convert the event stream presented as discrete data into an event stream presented as sequence data. It should be noted that the event stream presented as sequence data includes multiple event images sorted in chronological order, and these multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0158] As in the above example, when receiving the event stream ε of discrete data T After that, the second conversion unit can T Perform stacking operations to obtain the event stream ε of sequence data D , and ε D is sent to the multimodal fusion unit and the motion deblurring module. At this time, ε D It includes a plurality of event images which are arranged in chronological order, and the plurality of event images correspond one-to-one to a plurality of moments in the exposure time of the blurred LDR image.
[0159] 502. Based on the event stream, dynamic range enhancement processing is performed on the LDR image to obtain a blurred HDR image.
[0160] After obtaining the blurred LDR image and the event stream, the target model can use the event stream to perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image.
[0161] Specifically, the target model can obtain the blurred HDR image in the following way:
[0162] (1) After obtaining the blurred LDR image presented in YUV format and the event stream presented in sequence data, the dynamic enhancement module can extract features of the blurred LDR image and the event stream respectively, thereby obtaining features of the LDR image and features of the event stream accordingly.
[0163] It should be noted that the features of the LDR image include the initial features of the LDR image, the intermediate features of the LDR image, and the final features of the LDR image, and the features of the event stream include the initial features of the event stream, the intermediate features of the event stream, and the final features of the event stream. Then, the dynamic enhancement module can first perform feature extraction on the blurred LDR image to obtain the initial features of the LDR image, and then perform further feature extraction on the initial features of the LDR image to obtain the intermediate features of the LDR image, and then perform further feature extraction on the intermediate features of the LDR image to obtain the final features of the LDR image. Similarly, the dynamic enhancement module can first perform feature extraction on the event stream to obtain the initial features of the event stream, and then perform further feature extraction on the initial features of the event stream to obtain the intermediate features of the event stream, and then perform further feature extraction on the intermediate features of the event stream to obtain the final features of the event stream.
[0164] As in the above example, after obtaining L′ and ε D After that, the multimodal fusion unit can extract features from L′ to obtain the initial feature h1, then extract features from h1 to obtain the intermediate feature h2, and then extract features from h2 to obtain the final feature h3. Similarly, the multimodal fusion unit can extract features from ε D Perform feature extraction to obtain the initial feature h4, then perform feature extraction on h4 to obtain the intermediate feature h5, and then perform feature extraction on h5 to obtain the final feature h6.
[0165] (2) After obtaining the features of the LDR image and the features of the event stream, the dynamic enhancement module can perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image, and send the blurred HDR image to the motion deblurring module and the color enhancement network.
[0166] It should be noted that the dynamic enhancement module can first perform feature fusion on the final features of the LDR image and the final features of the event stream to obtain a first fused feature, then perform feature fusion on the first fused feature, the intermediate features of the LDR image and the intermediate features of the event stream to obtain a second fused feature, and then perform feature fusion on the second fused feature, the initial features of the LDR image and the initial features of the event stream to obtain a third fused feature, that is, a blurred HDR image. Acquiring a blurred HDR image by combining (1) and (2) can prevent the loss of introduced spatial information in the process of acquiring the image.
[0167] Still as in the above example, after obtaining h1 to h6, the multimodal fusion unit can perform feature fusion on h3 and h6 to obtain fused feature h7, and then perform feature fusion on h2, h5 and h7 to obtain fused feature h8, and then perform feature fusion on h1, h4 and h8 to obtain a blurred HDR image B, and provide B to the motion deblurring module and the color enhancement network.
[0168] 503. Based on the event stream, deblur the intensity component of the blurred HDR image to obtain a clear HDR image without color.
[0169] After obtaining the blurred HDR image, the target model can split the blurred HDR image into an intensity component of the blurred HDR image and a color component of the blurred HDR image, and use the event stream to deblur the intensity component of the blurred HDR image, so as to obtain a clear HDR image without color.
[0170] Specifically, the target model can obtain a clear and colorless HDR image in the following ways:
[0171] (1) After obtaining the blurred HDR image, the motion deblurring module can first divide the exposure time of the blurred LDR image into a first exposure time (also called a time period of more concern, which includes several moments ranked relatively high) and a second exposure time (also called a time period of less concern, which includes several moments ranked relatively low). Based on this, the motion deblurring module can divide the event stream into a first event stream triggered within the first exposure time (the first event stream includes several event images ranked relatively high) and a second event stream triggered within the second exposure time (the second event stream includes several event images ranked relatively low). This can make the time distinction more obvious.
[0172] For example, Figure 7 As shown ( Figure 7 A schematic diagram of the structure of a motion deblurring module provided in an embodiment of the present application. Figure 7 is Figure 6 The motion deblurring module includes an event separator, a deblurring factor generation unit, and a division operation unit. D After that, the event partitioning unit can D Divide into event streams And the event stream ,in, It contains several event images with the highest ranking, and these event images correspond one by one to several moments with the highest ranking in the exposure time. Contains several event images that are ranked later, and these event images correspond one by one to several moments that are ranked later in the exposure time, and as well as Sent to the defuzzification factor generation unit.
[0173] (2) After obtaining the first event stream and the second event stream, the motion deblurring module may extract features from the first event stream and the second event stream, respectively, thereby obtaining features of the first event stream and features of the second event stream. Then, the motion deblurring module may superimpose the features of the first event stream and the features of the second event stream, thereby obtaining superimposed features.
[0174] Still as in the above example, after getting as well as After that, the defuzzification factor generation unit can first as well as Perform feature extraction to obtain the corresponding features And features Then, the defuzzification factor generation unit can as well as Superimpose to obtain superimposed features , and Sent to the division unit.
[0175] (3) After obtaining the superimposed features, the motion deblurring module can extract the intensity component of the blurred HDR image from the blurred HDR image, and perform operations (e.g., division operations, etc.) on the intensity component of the blurred HDR image and the superimposed features, thereby obtaining a clear HDR image without color, and sending the clear HDR image without color to the color enhancement network. It should be noted that the clear HDR image without color includes multiple clear HDR sub-images without color, and the multiple HDR sub-images are sorted in chronological order, and the multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0176] Still as in the above example, after getting and the intensity component B Y After that, the division unit can and B Y Perform division operation to obtain a clear HDR image y without color T , and y T Sent to the color enhancement network. The division operation is shown in the following formula:
[0177]
[0178] In the above formula, y T It comprises a plurality of HDR sub-images which are arranged in chronological order, and the plurality of HDR sub-images correspond one by one to a plurality of moments in the exposure time of the blurred LDR image.
[0179] It should be understood that in this embodiment, only the division of the event stream into two sub-event streams, namely the first event stream and the second event stream, is used as an example for schematic introduction, and there is no limitation on the number of sub-event streams into which the event stream is divided. In actual applications, the number of sub-event streams can be more, for example, the event stream can be divided into three sub-event streams, namely the first event stream, the second event stream and the third event stream, and so on.
[0180] 504. Perform fusion processing on the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0181] After obtaining a clear HDR image without color, the target model can also fuse the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0182] Specifically, the target model can obtain clear and colorful HDR images in the following ways:
[0183] (1) After obtaining a clear HDR image without color, the color enhancement network can extract the color components of the blurred HDR image from the blurred HDR image, and perform feature extraction on the color components of the blurred HDR image to obtain the features of the color components.
[0184] For example, Figure 8 As shown ( Figure 8 A schematic diagram of a color enhancement network provided in an embodiment of the present application is shown in FIG. Figure 8 is Figure 7 The color enhancement network includes a mapping network and a coloring network. CrCb After that, the mapping network can be used for B CrCb Feature extraction is performed to obtain color feature b, and b is sent to the coloring network.
[0185] (2) After obtaining the features of the color components, the color enhancement network can perform feature fusion on the features of the color components and the clear HDR image without color, thereby obtaining a clear HDR image with color. It should be noted that the clear HDR image with color includes multiple clear HDR sub-images without color, and the multiple HDR sub-images are sorted in chronological order, and the multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0186] As in the above example, after obtaining b and y T After that, the coloring network can be used to calculate b and y T Fusion is performed to obtain a clear and colorful HDR image I T , I T It comprises a plurality of HDR sub-images which are arranged in chronological order, and the plurality of HDR sub-images correspond one by one to a plurality of moments in the exposure time of the blurred LDR image.
[0187] In an embodiment of the present application, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and an event stream triggered during the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image. Then, the target model can perform deblurring processing on the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to recover the blurred HDR image from the blurred LDR image, and then continue to use the event stream to recover the clear and colorful HDR image from the blurred HDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together and there is a connection between the two. Therefore, the clear HDR image obtained in this way often has higher quality. In other words, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0188] Furthermore, in an embodiment of the present application, in the process of restoring a clear and colorful HDR image from a blurred LDR image, an event stream triggered during the exposure time of the blurred LDR image is used. Since the event stream describes the brightness changes of the shooting scene (i.e., the scene corresponding to the blurred LDR image), the target model is based on the event stream, and can more comprehensively understand and analyze the shooting scene, and more realistically display the brightness information of each part of the shooting scene. The clear HDR image obtained based on this can have high-quality visual effects.
[0189] Furthermore, in the embodiment of the present application, the two processes of high dynamic range imaging and motion deblurring only require the implementation of the neural network model of the target model, and can be integrated together to achieve this, thereby saving the training cost of the neural network model.
[0190] The above is a detailed description of the workflow of the target model of the first structure. The following will introduce the target model of the second structure. Fig. 9 Another structural diagram of the target model provided in the embodiment of the present application is as follows: Fig. 9 As shown, the target model includes an event-based motion deblurring module, an event-based dynamic enhancement module and a color enhancement network, wherein the input end of the motion deblurring module serves as the input end of the entire target model, the first output end of the motion deblurring module is connected to the input end of the dynamic enhancement module, the second output end of the motion deblurring module is connected to the first input end of the color enhancement network, the output end of the dynamic enhancement module is connected to the second input end of the color enhancement network, and the output end of the color enhancement network serves as the output end of the entire target model. In order to understand Fig. 9 The workflow of the target model shown below is combined with Figure 5 This workflow is introduced. Fig.10 Another flowchart of the image processing method provided in the embodiment of the present application is as follows: Fig.10 As shown, the method includes:
[0191] 1001. Obtain a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image.
[0192] In this embodiment, when high dynamic range imaging and motion deblurring are required, a blurred LDR image may be captured first, and an event stream triggered within the exposure time of the blurred LDR image may be collected.
[0193] In a possible implementation manner, obtaining a blurred LDR image includes: obtaining a blurred LDR image presented in an RGB form; and converting the LDR image presented in an RGB form into an LDR image presented in a YUV form.
[0194] In one possible implementation, acquiring the event stream triggered within the exposure time of the LDR image includes: acquiring the event stream triggered within the exposure time of the LDR image presented as discrete data; and converting the event stream presented as discrete data into an event stream presented as sequence data.
[0195] It should be noted that for the introduction of step 1001, please refer to Figure 5 The relevant description of step 501 in the illustrated embodiment will not be repeated here.
[0196] 1002. Based on the event stream, the blurred LDR image is deblurred to obtain a clear LDR image.
[0197] After obtaining the blurred LDR image and the event stream, the target model can use the event stream to deblur the blurred LDR image, thereby obtaining a clear LDR image.
[0198] Specifically, the target model can obtain a clear LDR image in the following ways:
[0199] (1) After obtaining the blurred LDR image presented in YUV format and the event stream presented in sequence data, the motion deblurring module can extract features from the blurred LDR image and the event stream respectively, thereby obtaining features of the LDR image and features of the event stream accordingly.
[0200] It should be noted that the features of the LDR image include the initial features of the LDR image, the intermediate features of the LDR image, and the final features of the LDR image, and the features of the event stream include the initial features of the event stream, the intermediate features of the event stream, and the final features of the event stream. Then, the motion deblurring module may first perform feature extraction on the blurred LDR image to obtain the initial features of the LDR image, and then perform further feature extraction on the initial features of the LDR image to obtain the intermediate features of the LDR image, and then perform further feature extraction on the intermediate features of the LDR image to obtain the final features of the LDR image. Similarly, the motion deblurring module may first perform feature extraction on the event stream to obtain the initial features of the event stream, and then perform further feature extraction on the initial features of the event stream to obtain the intermediate features of the event stream, and then perform further feature extraction on the intermediate features of the event stream to obtain the final features of the event stream.
[0201] (2) After obtaining the features of the LDR image and the features of the event stream, the motion deblurring module can perform feature fusion on the features of the LDR image and the features of the event stream to obtain a clear LDR image, and then send the clear LDR image to the dynamic enhancement module and the color enhancement network.
[0202] It should be noted that the motion deblurring module can first perform feature fusion on the final features of the LDR image and the final features of the event stream to obtain a first fused feature, then perform feature fusion on the first fused feature, the intermediate features of the LDR image and the intermediate features of the event stream to obtain a second fused feature, and then perform feature fusion on the second fused feature, the initial features of the LDR image and the initial features of the event stream to obtain a third fused feature, that is, a clear LDR image. Acquiring a clear LDR image by combining (1) and (2) can prevent the loss of introduced spatial information in the process of acquiring the image.
[0203] 1003. Based on the event stream, the intensity component of the clear LDR image is dynamically enhanced to obtain a clear HDR image without color.
[0204] After obtaining a clear LDR image, the target model can split the clear LDR image into an intensity component of the clear LDR image and a color component of the clear LDR image, and dynamically enhance the intensity component of the clear LDR image by using an event stream, thereby obtaining a clear HDR image without color.
[0205] Specifically, the target model can obtain a clear and colorless HDR image in the following ways:
[0206] (1) After obtaining a clear LDR image, the dynamic enhancement module can first divide the exposure time of the blurred LDR image into a first exposure time (also called a time period of more concern, which includes several moments ranked relatively high) and a second exposure time (also called a time period of less concern, which includes several moments ranked relatively low). Based on this, the dynamic enhancement module can divide the event stream into a first event stream triggered within the first exposure time (the first event stream includes several event images ranked relatively high) and a second event stream triggered within the second exposure time (the second event stream includes several event images ranked relatively low). This can make the time distinction more obvious.
[0207] (2) After obtaining the first event stream and the second event stream, the dynamic enhancement module can extract features from the first event stream and the second event stream respectively, thereby obtaining features of the first event stream and features of the second event stream respectively. Then, the dynamic enhancement module can superimpose the features of the first event stream and the features of the second event stream, thereby obtaining superimposed features.
[0208] (3) After obtaining the superimposed features, the dynamic enhancement module can extract the intensity component of the clear LDR image from the clear LDR image, and perform operations (e.g., division operations, etc.) on the intensity component of the clear LDR image and the superimposed features, thereby obtaining a clear HDR image without color, and sending the clear HDR image without color to the color enhancement network. It should be noted that the clear HDR image without color includes multiple clear HDR sub-images without color, and the multiple HDR sub-images are sorted in chronological order, and the multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0209] It should be understood that in this embodiment, only the division of the event stream into two sub-event streams, namely the first event stream and the second event stream, is used as an example for schematic introduction, and there is no limitation on the number of sub-event streams into which the event stream is divided. In actual applications, the number of sub-event streams can be more, for example, the event stream can be divided into three sub-event streams, namely the first event stream, the second event stream and the third event stream, and so on.
[0210] 1004. Perform fusion processing on the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0211] After obtaining a clear HDR image without color, the target model can also fuse the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color. At this point, high dynamic range imaging and motion deblurring are successfully completed.
[0212] Specifically, the target model can obtain clear and colorful HDR images in the following ways:
[0213] (1) After obtaining a clear HDR image without color, the color enhancement network can extract the color components of the clear LDR image from the clear LDR image, and perform feature extraction on the color components of the clear LDR image to obtain the features of the color components.
[0214] (2) After obtaining the features of the color components, the color enhancement network can perform feature fusion on the features of the color components and the clear HDR image without color, thereby obtaining a clear HDR image with color. It should be noted that the clear HDR image with color includes multiple clear HDR sub-images without color, and the multiple HDR sub-images are sorted in chronological order, and the multiple event images correspond one-to-one to multiple moments in the exposure time of the LDR image.
[0215] In an embodiment of the present application, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and an event stream triggered during the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first deblur the blurred LDR image based on the event stream to obtain a clear LDR image. Then, the target model can perform dynamic range enhancement processing on the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color. So far, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to restore a clear LDR image from the blurred LDR image, and then continue to use the event stream to restore a clear and colorful HDR image from the clear LDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together and there is a connection between the two. Therefore, the clear HDR image obtained in this way often has higher quality. That is to say, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0216] Furthermore, in an embodiment of the present application, in the process of restoring a clear and colorful HDR image from a blurred LDR image, an event stream triggered during the exposure time of the blurred LDR image is used. Since the event stream describes the brightness changes of the shooting scene (i.e., the scene corresponding to the blurred LDR image), the target model is based on the event stream, and can more comprehensively understand and analyze the shooting scene, and more realistically display the brightness information of each part of the shooting scene. The clear HDR image obtained based on this can have high-quality visual effects.
[0217] Furthermore, in the embodiment of the present application, the two processes of high dynamic range imaging and motion deblurring only require the implementation of the neural network model of the target model, and can be integrated together to achieve this, thereby saving the training cost of the neural network model.
[0218] In addition, the target model provided in the embodiment of the present application (for example, Fig.11 Ours in ) and some models provided by related technologies (for example, Fig.11 The comparison result is as follows: Fig.11 As shown ( Fig.11 A schematic diagram of a comparison structure provided in an embodiment of the present application), the target model provided in an embodiment of the present application can obtain a clear HDR image with better visual quality.
[0219] Furthermore, the target model provided in the embodiment of the present application (for example, Ours in Table 1) can be compared with the model provided by another part of the related technology (for example, HDRCNN EDI, HDRCNN eSL-Net, etc. in Table 1), and the comparison results are shown in Table 1:
[0220] Table 1
[0221] Methods PSNR-L↑ PSNR-uf PSNR-Y SSIM-L SSIM-ut SSIM-YT HDR-VDP-21 HDRCNN EDI 17.4 16.67 25.97 0.479 0.617 0.867 43.86 HDRCNN eSL-Net 17.39 17.45 25.97 0.479 0.653 0.866 43.67 HDRCNN EVDI 17.86 19.62 26.35 0.66 0.543 0.873 44.05 HDRUnet EDI 15.84 20.5 25.25 0.54 0.705 0.862 43.87 HDRUnet eSL-Net 15.5 20.82 24.67 0.501 0.704 0.845 43.38 HDRUnet EVDI 15.78 20.92 24.69 0.508 0.733 0.873 44.1 EDI HDRCNN 17.81 16.03 25.73 0.462 0.592 0.844 42.67 EDI HDRUnet 17.4 20.34 24.64 0.542 0.72 0.863 42.9 eSL-Net HDRCNN 16.65 9.05 22.72 0.201 0.345 0.792 35.82 eSL-Net HDRUnet 16.42 16.5 24.39 0.511 0.536 0.78 41.41 EVDI HDRCNN 17.75 16.92 25.61 0.445 0.628 0.863 42.35 EVDI HDRUnet 17.74 21.45 25.64 0.543 0.743 0.883 44.62 Ours 22.75 25.87 30.1 0.791 0.829 0.921 47.13
[0222] Based on Table 1, it can be seen that when tested on the same data set, the target model provided by the embodiment of the present application has achieved the best results in the three indicators of PSNR, SSIM and HDR-VDP-2, and the target model provided by the embodiment of the present application has achieved a great performance improvement (>4dB).
[0223] The above is a detailed description of the image processing method provided in the embodiment of the present application. The following is an introduction to the model training method provided in the embodiment of the present application. Fig.12 A flow chart of the model training method provided in the embodiment of the present application is as follows: Fig.12 As shown, the method includes:
[0224] 1201. The blurred LDR image and the event stream triggered within the exposure time of the LDR image are processed by the model to be trained to obtain a clear HDR image with color. The model to be trained is used to: obtain the blurred LDR image and the event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0225] In this embodiment, when the training model needs to be trained, a batch of training data can be obtained first, and the batch of training data includes blurred LDR images and event streams triggered during the exposure time of the LDR images. It should be noted that for the blurred LDR images, the clear and colorful real HDR images are known, so the clear and colorful real HDR images can be processed to obtain clear and colorless real HDR images and blurred real HDR images.
[0226] After obtaining the blurred LDR image, the blurred LDR image and the event stream can be input into the model to be trained. Therefore, after obtaining the blurred LDR image and the event stream, the model to be trained can first perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred (predicted) HDR image. Then, the model to be trained can deblur the intensity component of the blurred HDR image based on the event stream to obtain a clear (predicted) HDR image without color. Finally, the model to be trained can fuse the color component of the blurred HDR image and the clear HDR image without color to obtain a clear (predicted) HDR image with color.
[0227] In one possible implementation, the model to be trained is used to: extract features from the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and features of the event stream to obtain a blurred HDR image.
[0228] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0229] In one possible implementation, the model to be trained is used to: extract features of color components of a blurred HDR image to obtain features of the color components; and fuse features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0230] In a possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0231] In one possible implementation, the model to be trained is used to: obtain the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0232] It should be noted that for the introduction of step 1201, please refer to Figure 5The relevant description parts of steps 501 to 504 in the illustrated embodiment will not be repeated here.
[0233] 1202. Based on the clear and colorful HDR image, the to-be-trained model is trained to obtain a target model.
[0234] After obtaining a clear and colorful HDR image, the training model can be trained based on the clear and colorful HDR image to obtain Figure 5 Target model in the illustrated embodiment.
[0235] Specifically, the target model can be trained in the following ways:
[0236] (1) After obtaining a clear and colorful HDR image, since a clear and colorful real HDR image is known, a first loss and a second loss can be obtained based on the clear and colorful HDR image and the clear and colorful real HDR image. The first loss is used to indicate the difference between the clear and colorful HDR image and the clear and colorful real HDR image, and the second loss is used to indicate the difference between the first discrimination value and the second discrimination value (the first discrimination value is calculated based on the clear and colorful HDR image, and the second discrimination value is calculated based on the clear and colorful real HDR image). The second loss can be obtained by the following formula:
[0237]
[0238] In the above formula, L gan is the second loss, D(I T ) is the first discriminant value, is the second discriminant value, I T For clear and colorful HDR images, A true HDR image with clarity and color.
[0239] (2) Since the real HDR image that is clear and has color, the real HDR image that is clear and has no color, and the blurred real HDR image are known, the third loss can be obtained based on the blurred real HDR image and the blurred HDR image, and the fourth loss can be obtained based on the real HDR image that is clear and has no color and the real HDR image that is clear and has no color. The third loss is used to indicate the difference between the real HDR image that is clear and has no color and the blurred HDR image, and the fourth loss is used to indicate the difference between the real HDR image that is clear and has no color and the real HDR image that is clear and has no color. The third loss can be obtained by the following formula:
[0240]
[0241] In the above formula, L bc is the third loss, B is the blurred HDR image, A real HDR image that is blurred.
[0242] The fourth loss can be obtained by the following formula:
[0243]
[0244] In the above formula, L lc is the fourth loss, y T For a clear HDR image without color, A true HDR image that is clear and colorless.
[0245] (3) The first loss, the second loss, the third loss, and the fourth loss are superimposed to obtain the target loss. Then, the parameters of the model to be trained can be updated based on the target loss to obtain the model to be trained after the updated parameters, and the next batch of training data can be used to continue training the model to be trained after the updated parameters until the model training conditions are met (for example, the target loss meets convergence, etc.), thereby obtaining the target model.
[0246] The target model obtained by training in the embodiment of the present application has the functions of high dynamic range imaging and motion deblurring. Specifically, when high dynamic range imaging and motion deblurring are required, the blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image. Then, the target model can perform deblurring processing on the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color. Finally, the target model can perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color. So far, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to recover the blurred HDR image from the blurred LDR image, and then continue to use the event stream to recover the clear and colorful HDR image from the blurred HDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together and there is a connection between the two. Therefore, the clear HDR image obtained in this way often has higher quality. In other words, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0247] Fig.13Another flowchart of the model training method provided in the embodiment of the present application is as follows: Fig.13 As shown, the method includes:
[0248] 1301. A blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image are processed by a to-be-trained model to obtain a clear HDR image with color. The to-be-trained model is used to: obtain a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblur the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhance the intensity component of the clear LDR image to obtain a clear HDR image without color; and fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0249] In this embodiment, when the training model needs to be trained, a batch of training data can be obtained first, and the batch of training data includes a blurred LDR image and an event stream triggered during the exposure time of the LDR image. It should be noted that for the blurred LDR image, a clear and colorful real HDR image is known, so the clear and colorful real HDR image can be processed to obtain a clear and colorless real HDR image and a clear real LDR image.
[0250] After obtaining the blurred LDR image, the blurred LDR image and the event stream can be input into the model to be trained. Therefore, after obtaining the blurred LDR image and the event stream, the model to be trained can first deblur the blurred LDR image based on the event stream to obtain a clear LDR image. Then, the model to be trained can dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color. Finally, the model to be trained can fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0251] In one possible implementation, the model to be trained is used to: extract features from the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0252] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0253] In one possible implementation, the model to be trained is used to: extract features of color components of a clear LDR image to obtain features of the color components; and fuse features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0254] In a possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0255] In one possible implementation, the model to be trained is used to: obtain the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0256] It should be noted that for the introduction of step 1301, please refer to Fig.10 The relevant description parts of steps 1001 to 1004 in the illustrated embodiment will not be repeated here.
[0257] 1302. Based on the clear and colorful HDR image, the to-be-trained model is trained to obtain a target model.
[0258] After obtaining a clear and colorful HDR image, the training model can be trained based on the clear and colorful HDR image to obtain Figure 5 Target model in the illustrated embodiment.
[0259] Specifically, the target model can be trained in the following ways:
[0260] (1) After obtaining a clear and colorful HDR image, since a clear and colorful real HDR image is known, a first loss and a second loss can be obtained based on the clear and colorful HDR image and the clear and colorful real HDR image. The first loss is used to indicate the difference between the clear and colorful HDR image and the clear and colorful real HDR image, and the second loss is used to indicate the difference between a first discrimination value and a second discrimination value (the first discrimination value is calculated based on the clear and colorful HDR image, and the second discrimination value is calculated based on the clear and colorful real HDR image).
[0261] (2) Since the real HDR image that is clear and has color, the real HDR image that is clear and has no color, and the blurred real HDR image are known, a third loss can be obtained based on the clear real LDR image and the clear LDR image, and a fourth loss can be obtained based on the clear real HDR image without color and the clear HDR image without color. The third loss is used to indicate the difference between the clear real LDR image and the clear LDR image, and the fourth loss is used to indicate the difference between the clear real HDR image without color and the clear HDR image without color.
[0262] (3) The first loss, the second loss, the third loss, and the fourth loss are superimposed to obtain the target loss. Then, the parameters of the model to be trained can be updated based on the target loss to obtain the model to be trained after the updated parameters, and the next batch of training data can be used to continue training the model to be trained after the updated parameters until the model training conditions are met (for example, the target loss meets convergence, etc.), thereby obtaining the target model.
[0263] The target model obtained by training in the embodiment of the present application has high dynamic range imaging and motion deblurring functions. Specifically, when high dynamic range imaging and motion deblurring are required, a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image can be first obtained, and the blurred LDR image and the event stream can be input into the target model. Then, the target model can first deblur the blurred LDR image based on the event stream, thereby obtaining a clear LDR image. Then, the target model can perform dynamic range enhancement processing on the intensity component of the clear LDR image based on the event stream, thereby obtaining a clear HDR image without color. Finally, the target model can fuse the color component of the clear LDR image and the clear HDR image without color, thereby obtaining a clear HDR image with color. So far, high dynamic range imaging and motion deblurring are completed. In the above process, the target model can use the event stream to restore a clear LDR image from the blurred LDR image, and then continue to use the event stream to restore a clear and colorful HDR image from the clear LDR image. It can be seen that the target model can realize high dynamic range imaging and motion deblurring at one time, so that the two processes are coupled together and there is a connection between the two. Therefore, the clear HDR image obtained in this way often has higher quality. That is to say, the final clear HDR image does not have various quality problems such as color defects, abnormal image brightness, and different degrees of detail blur.
[0264] The above is a detailed description of the model training method provided in the embodiment of the present application. The following will introduce the image processing device and model training device provided in the embodiment of the present application. Fig.14 A structural schematic diagram of an image processing device is provided for an embodiment of the present application, such as Fig.14 As shown, the device includes a target model, and the device includes:
[0265] An acquisition module 1401 is used to acquire a blurred LDR image and an event stream triggered during the exposure time of the LDR image;
[0266] A first processing module 1402 is used to perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image;
[0267] The second processing module 1403 is used to deblur the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color;
[0268] The third processing module 1404 is used to perform fusion processing on the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0269] In one possible implementation, the first processing module 1402 is used to: perform feature extraction on the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; perform feature fusion on the features of the LDR image and the features of the event stream to obtain a blurred HDR image.
[0270] In one possible implementation, the second processing module 1403 is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, the exposure time including the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; perform calculations on the intensity components of the blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0271] In one possible implementation, the third processing module 1404 is used to: extract features of the color components of the blurred HDR image to obtain features of the color components; and perform feature fusion on the features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0272] In a possible implementation, the acquisition module 1401 is used to: acquire a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0273] In a possible implementation, the acquisition module 1401 is used to: acquire the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0274] Fig.15 Another structural diagram of the image processing device provided in the embodiment of the present application is as follows Fig.15 As shown, the device includes a target model, and the device includes:
[0275] An acquisition module 1501 is used to acquire a blurred LDR image and an event stream triggered during the exposure time of the blurred LDR image;
[0276] A first processing module 1502 is used to deblur the blurred LDR image based on the event stream to obtain a clear LDR image;
[0277] The second processing module 1503 is used to dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color;
[0278] The third processing module 1504 is used to perform fusion processing on the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
[0279] In one possible implementation, the first processing module 1502 is used to: perform feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0280] In one possible implementation, the second processing module 1503 is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, the exposure time including the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of the clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0281] In one possible implementation, the third processing module 1503 is used to: extract features of the color components of the clear LDR image to obtain features of the color components; and perform feature fusion on the features of the color components and the clear HDR image without color to obtain a clear HDR image with color.
[0282] In a possible implementation, the acquisition module 1501 is used to: acquire a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0283] In a possible implementation, the acquisition module 1501 is used to: acquire the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0284] Fig.16 A structural diagram of a model training device provided in an embodiment of the present application, such as Fig.16 As shown, the device comprises:
[0285] The processing module 1601 is used to process the blurred LDR image and the event stream triggered within the exposure time of the LDR image through the model to be trained to obtain a clear HDR image with color. The model to be trained is used to: obtain the blurred LDR image and the event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color;
[0286] The training module 1602 is used to train the to-be-trained model based on the clear and colorful HDR image to obtain a target model.
[0287] In one possible implementation, the model to be trained is used to: extract features from the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; and perform feature fusion on the features of the LDR image and features of the event stream to obtain a blurred HDR image.
[0288] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a blurred HDR image and the superimposed features to obtain a clear HDR image without color.
[0289] In one possible implementation, the model to be trained is used to: extract features of color components of a blurred HDR image to obtain features of the color components; and fuse features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0290] In a possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0291] In one possible implementation, the model to be trained is used to: obtain the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0292] Fig.17 Another structural diagram of the model training device provided in the embodiment of the present application is as follows Fig.17 As shown, the device comprises:
[0293] The processing module 1701 is used to process the blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image through the model to be trained to obtain a clear HDR image with color. The model to be trained is used to: obtain the blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblur the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhance the intensity component of the clear LDR image to obtain a clear HDR image without color; fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color;
[0294] The training module 1702 is used to train the to-be-trained model based on the clear and colorful HDR image to obtain a target model.
[0295] In one possible implementation, the model to be trained is used to: extract features from the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; and perform feature fusion on the features of the blurred LDR image and features of the event stream to obtain a clear LDR image.
[0296] In one possible implementation, the model to be trained is used to: divide the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, where the exposure time includes the first exposure time and the second exposure time; perform feature extraction on the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; superimpose the features of the first event stream and the features of the second event stream to obtain superimposed features; and perform calculations on the intensity components of a clear LDR image and the superimposed features to obtain a clear HDR image without color.
[0297] In one possible implementation, the model to be trained is used to: extract features of color components of a clear LDR image to obtain features of the color components; and fuse features of the color components and a clear HDR image without color to obtain a clear HDR image with color.
[0298] In a possible implementation, the model to be trained is used to: obtain a blurred LDR image presented in RGB form; and convert the LDR image presented in RGB form into an LDR image presented in YUV form.
[0299] In one possible implementation, the model to be trained is used to: obtain the event stream triggered during the exposure time of the LDR image presented as discrete data; and convert the event stream presented as discrete data into an event stream presented as sequence data.
[0300] It should be noted that the information interaction, execution process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and will not be repeated here.
[0301] The present application also relates to an execution device. Fig.18 A schematic diagram of the structure of the execution device provided in the embodiment of the present application. Fig.18 As shown, the execution device 1800 can be specifically a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which is not limited here. Fig.14 or Fig.15 The image processing device described in the corresponding embodiment is used to implement Figure 5 or Fig.10 The image processing function in the corresponding embodiment. Specifically, the execution device 1800 includes: a receiver 1801, a transmitter 1802, a processor 1803 and a memory 1804 (wherein the number of the processor 1803 in the execution device 1800 can be one or more, Fig.18 In the example of FIG. 1803 , the processor 1803 may include an application processor 18031 and a communication processor 18032. In some embodiments of the present application, the receiver 1801, the transmitter 1802, the processor 1803 and the memory 1804 may be connected via a bus or other means.
[0302] The memory 1804 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1803. A portion of the memory 1804 may also include a non-volatile random access memory (NVRAM). The memory 1804 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.
[0303] The processor 1803 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.
[0304] The method disclosed in the above embodiment of the present application can be applied to the processor 1803, or implemented by the processor 1803. The processor 1803 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1803. The above processor 1803 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1803 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1804, and the processor 1803 reads the information in the memory 1804 and completes the steps of the above method in combination with its hardware.
[0305] The receiver 1801 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1802 can be used to output digital or character information through the first interface; the transmitter 1802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1802 can also include a display device such as a display screen.
[0306] In one embodiment of the present application, the processor 1803 is configured to: Figure 5 or Fig.10 Corresponding to the target model in the embodiment, a clear and colorful HDR image is obtained.
[0307] The present application also relates to a training device. Fig.19 A schematic diagram of the structure of the training device provided in the embodiment of the present application. Fig.19 As shown, the training device 1900 is implemented by one or more servers. The training device 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1919 (for example, one or more processors) and memory 1932, one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1919 can be configured to communicate with the storage medium 1930 to execute a series of instruction operations in the storage medium 1930 on the training device 1900.
[0308] The training device 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958; or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0309] Specifically, the training device can perform Fig.12 or Fig.13 The model training method in the corresponding embodiment is used to obtain the target model.
[0310] An embodiment of the present application also relates to a computer storage medium, in which a program for signal processing is stored. When the program is run on a computer, the computer executes the steps executed by the aforementioned execution device, or the computer executes the steps executed by the aforementioned training device.
[0311] An embodiment of the present application also relates to a computer program product, which stores instructions, which, when executed by a computer, enable the computer to execute the steps executed by the aforementioned execution device, or enable the computer to execute the steps executed by the aforementioned training device.
[0312] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
[0313] For details, please refer to Fig. 20 , Fig. 20 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 2000, NPU 2000 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.
[0314] In some implementations, the operation circuit 2003 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2003 is a two-dimensional systolic array. The operation circuit 2003 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2003 is a general-purpose matrix processor.
[0315] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2002 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2001 and performs matrix operation with matrix B. The partial result or final result of the matrix is stored in the accumulator 2008.
[0316] The unified memory 2006 is used to store input data and output data. The weight data is directly transferred to the weight memory 2002 through the direct memory access controller (DMAC) 2005. The input data is also transferred to the unified memory 2006 through the DMAC.
[0317] BIU stands for Bus Interface Unit 2013 , which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 2009 .
[0318] The bus interface unit 2013 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
[0319] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2006 or to transfer weight data to the weight memory 2002 or to transfer input data to the input memory 2001.
[0320] The vector calculation unit 2007 includes multiple operation processing units, and further processes the output of the operation circuit 2003 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of predicted label planes, etc.
[0321] In some implementations, the vector calculation unit 2007 can store the processed output vector to the unified memory 2006. For example, the vector calculation unit 2007 can apply a linear function; or a nonlinear function to the output of the operation circuit 2003, such as linear interpolation of the predicted label plane extracted by the convolution layer, and then, for example, a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 2007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2003, for example, for use in a subsequent layer in a neural network.
[0322] An instruction fetch buffer 2009 connected to the controller 2004, for storing instructions used by the controller 2004;
[0323] The unified memory 2006, the input memory 2001, the weight memory 2002 and the instruction fetch memory 2009 are all on-chip memories. The external memory is private to the NPU hardware architecture.
[0324] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.
[0325] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0326] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0327] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0328] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. An image processing method, characterized in that: The method is implemented by a target model, and the method comprises: Acquire a blurred low dynamic range (LDR) image and a stream of events triggered during an exposure time of the LDR image; Based on the event stream, dynamic range enhancement processing is performed on the LDR image to obtain a blurred high dynamic range HDR image; Based on the event stream, deblurring the intensity component of the blurred HDR image to obtain a clear HDR image without color; The color components of the blurred HDR image and the clear HDR image without color are fused to obtain a clear HDR image with color.
2. The method according to claim 1, characterized in that: The step of performing dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image comprises: Extracting features from the LDR image and the event stream respectively to obtain features of the LDR image and features of the event stream; The features of the LDR image and the features of the event stream are fused to obtain a blurred HDR image.
3. The method according to claim 1 or 2, characterized in that: The deblurring the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color comprises: Dividing the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, wherein the exposure time includes the first exposure time and the second exposure time; Extracting features from the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; Superimposing the features of the first event stream and the features of the second event stream to obtain superimposed features; The intensity component of the blurred HDR image and the superimposed features are operated to obtain a clear HDR image without color.
4. The method according to any one of claims 1 to 3, characterized in that: The step of fusing the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color includes: Performing feature extraction on color components of the blurred HDR image to obtain features of the color components; The features of the color components and the clear HDR image without color are fused to obtain a clear HDR image with color.
5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining a blurred LDR image comprises: Get the blurred LDR image in RGB form; Convert an LDR image presented in RGB form to an LDR image presented in YUV form.
6. The method according to any one of claims 1 to 5, characterized in that: The event flow triggered within the exposure time of acquiring the LDR image includes: Acquire an event stream triggered within the exposure time of the LDR image presented as discrete data; The event stream presented as discrete data is converted into an event stream presented as sequence data.
7. An image processing method, characterized in that: The method is implemented by a target model, and the method comprises: Acquire a blurred LDR image and an event stream triggered during an exposure time of the blurred LDR image; Based on the event stream, deblurring the blurred LDR image to obtain a clear LDR image; Based on the event stream, dynamically enhancing the intensity component of the clear LDR image to obtain a clear HDR image without color; The color components of the clear LDR image and the clear HDR image without color are fused to obtain a clear HDR image with color.
8. The method according to claim 7, characterized in that The deblurring the blurred LDR image based on the event stream to obtain a clear LDR image comprises: Performing feature extraction on the blurred LDR image and the event stream respectively to obtain features of the blurred LDR image and features of the event stream; The features of the blurred LDR image and the features of the event stream are fused to obtain a clear LDR image.
9. The method according to claim 7 or 8, characterized in that: The step of dynamically enhancing the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color comprises: Dividing the event stream into a first event stream triggered within a first exposure time and a second event stream triggered within a second exposure time, wherein the exposure time includes the first exposure time and the second exposure time; Extracting features from the first event stream and the second event stream respectively to obtain features of the first event stream and features of the second event stream; Superimposing the features of the first event stream and the features of the second event stream to obtain superimposed features; The intensity component of the clear LDR image and the superimposed features are operated to obtain a clear HDR image without color.
10. The method according to any one of claims 7 to 9, characterized in that: The step of fusing the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color comprises: Performing feature extraction on the color components of the clear LDR image to obtain features of the color components; The features of the color components and the clear HDR image without color are fused to obtain a clear HDR image with color.
11. The method according to any one of claims 7 to 10, characterized in that: The step of obtaining a blurred LDR image comprises: Get the blurred LDR image in RGB form; Convert an LDR image presented in RGB form to an LDR image presented in YUV form.
12. The method according to any one of claims 7 to 11, characterized in that: The event flow triggered within the exposure time of acquiring the LDR image includes: Acquire an event stream triggered within the exposure time of the LDR image presented as discrete data; The event stream presented as discrete data is converted into an event stream presented as sequence data.
13. A model training method, characterized in that: The method comprises: The blurred LDR image and the event stream triggered within the exposure time of the LDR image are processed by the model to be trained to obtain a clear HDR image with color. The model to be trained is used to: obtain the blurred LDR image and the event stream triggered within the exposure time of the LDR image; based on the event stream, perform dynamic range enhancement processing on the LDR image to obtain a blurred HDR image; based on the event stream, perform deblurring processing on the intensity component of the blurred HDR image to obtain a clear HDR image without color; perform fusion processing on the color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color; Based on the clear and colorful HDR image, the model to be trained is trained to obtain a target model.
14. A model training method, characterized in that: The method comprises: The blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image are processed by the model to be trained to obtain a clear HDR image with color. The model to be trained is used to: obtain the blurred LDR image and the event stream triggered within the exposure time of the blurred LDR image; based on the event stream, deblur the blurred LDR image to obtain a clear LDR image; based on the event stream, dynamically enhance the intensity component of the clear LDR image to obtain a clear HDR image without color; fuse the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color; Based on the clear and colorful HDR image, the model to be trained is trained to obtain a target model.
15. An image processing device, characterized in that: The device comprises a target model, and the device comprises: An acquisition module, used for acquiring a blurred LDR image and an event stream triggered during the exposure time of the LDR image; A first processing module, configured to perform dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image; A second processing module is used for deblurring the intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color; The third processing module is used to fuse the color components of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color.
16. An image processing device, characterized in that: The device comprises a target model, and the device comprises: An acquisition module, used for acquiring a blurred LDR image and an event stream triggered during an exposure time of the blurred LDR image; A first processing module, configured to perform a deblurring process on the blurred LDR image based on the event stream to obtain a clear LDR image; A second processing module is used to dynamically enhance the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; The third processing module is used to fuse the color components of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color.
17. A model training device, characterized in that: The device comprises: A processing module, used for processing a blurred LDR image and an event stream triggered within the exposure time of the LDR image through a to-be-trained model to obtain a clear HDR image with color, wherein the to-be-trained model is used for: acquiring a blurred LDR image and an event stream triggered within the exposure time of the LDR image; performing dynamic range enhancement processing on the LDR image based on the event stream to obtain a blurred HDR image; performing deblurring processing on an intensity component of the blurred HDR image based on the event stream to obtain a clear HDR image without color; performing fusion processing on a color component of the blurred HDR image and the clear HDR image without color to obtain a clear HDR image with color; The training module is used to train the to-be-trained model based on the clear and colorful HDR image to obtain a target model.
18. A model training device, characterized in that: The device comprises: A processing module, used for processing a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image through a model to be trained to obtain a clear HDR image with color, wherein the model to be trained is used for: acquiring a blurred LDR image and an event stream triggered within the exposure time of the blurred LDR image; deblurring the blurred LDR image based on the event stream to obtain a clear LDR image; dynamically enhancing the intensity component of the clear LDR image based on the event stream to obtain a clear HDR image without color; fusing the color component of the clear LDR image and the clear HDR image without color to obtain a clear HDR image with color; The training module is used to train the to-be-trained model based on the clear and colorful HDR image to obtain a target model.
19. An image processing device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to execute the codes. When the codes are executed, the image processing device performs the method according to any one of claims 1 to 14.
20. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 14.
21. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 14.
Citation Information
Cited By
Image reconstruction method and device, equipment and storage medium
CN121685347A
