Image processing method, device and equipment and readable storage medium

Abstract

The invention discloses an image processing method, device and equipment and a readable storage medium, and belongs to the technical field of video processing. The method is executed by a terminal and comprises the following steps: acquiring an original image, a first image and a second image; wherein the original image comprises a long exposure image and a short exposure image, the first image is a fused image obtained by fusing the original image and a corresponding registration image, and the second image is an image output by a first model last time; inputting the original image, the first image and the second image into the first model, and taking an image output by the first model as a target image; wherein the first model is used for outputting the first image of which the artifact region is removed.

Description

Technical Field

[0001] The present application belongs to the field of video processing technology, and specifically relates to an image processing method, device, equipment and readable storage medium. Background Art

[0002] With the development of terminal devices, more users use mobile terminal devices to take photos and prefer to watch photos and preview videos on mobile terminal devices. In real scenes, the color of objects has a very high dynamic range. Most terminal devices can support 24 bits of information to represent colors, but the pixel value of a single channel is 8 bits of data. Therefore, the dynamic range that can be covered by a single photo taken by an ordinary terminal device is extremely limited. Using too short an exposure time will make the low-brightness parts of the scene dark and produce noise, while using too long an exposure time will cause the high-brightness parts of the scene to be overexposed and lose details. In order to make the image present rich color details and brightness levels, High Dynamic Range (HDR) technology that combines alternate exposure of multiple frames of pictures has become a mainstream solution.

[0003] Among them, traditional HDR videos use long and short frames for fusion, which may result in short frame fusion based on long frames or long frame fusion based on short frames. The design of fusion weights and the strategy of optical flow estimation will cause artifacts in the single-frame HDR fusion results in occluded areas and large motion scenes; in the entire HDR video fusion process, some areas are prone to flickering and jittering. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, equipment and readable storage medium to avoid artifacts, flickering, jitter and other phenomena in video fusion.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is executed by a terminal and includes:

[0006] Acquire an original image, a first image, and a second image; wherein the original image includes a long-exposure image and a short-exposure image, the first image is a fused image obtained by fusing the original image and a corresponding registration image, and the second image is an image outputted by the first model last time;

[0007] Input the original image, the first image and the second image into the first model, and use the image output by the first model as the target image;

[0008] The first model is used to output the first image with the artifact area removed.

[0009] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0010] A first processing module is used to obtain an original image, a first image, and a second image; wherein the original image includes a long-exposure image and a short-exposure image, the first image is a fused image obtained by fusing the original image and a corresponding registration image, and the second image is an image outputted by the first model last time;

[0011] A second processing module, configured to input the original image, the first image and the second image into the first model, and use the image output by the first model as a target image;

[0012] The first model is used to output the first image with the artifact area removed.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0017] In an embodiment of the present application, after acquiring the original image, the first image and the second image, they can be input into the first model, and the output of the first model can obtain the desired fused image with the artifact area removed, thereby reducing fusion abnormalities, reducing the differences between the fused images, and improving image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of an image processing method according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the application of the first model;

[0020] Figure 3 It is a schematic diagram of the temporal consistency loss function;

[0021] Figure 4 It is a schematic diagram of texture consistency loss function;

[0022] Figure 5 This is a schematic diagram of the training and application of the first model;

[0023] Figure 6 It is a schematic diagram of the terminal hardware structure;

[0024] Figure 7 This is a schematic diagram of the HDR-MV-LITE module structure;

[0025] Figure 8 It is a schematic diagram of the terminal interface;

[0026] Fig. 9 This is a schematic diagram of the HDR video processing flow;

[0027] Fig.10 This is a schematic diagram of the offline HDR video processing process;

[0028] Fig.11 yes Figure 1 Schematic diagram of the module structure of the corresponding device;

[0029] Fig.12 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0030] Fig.13 It is a schematic diagram of the structure of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0032] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0033] The electronic device provided in the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0034] like Figure 1 As shown, an image processing method in an embodiment of the present application is executed by a terminal, including:

[0035] Step 101, obtaining an original image, a first image, and a second image; wherein the original image includes a long exposure image and a short exposure image, the first image is a fused image obtained by fusing the original image and the corresponding registration image, and the second image is an image output by the first model last time. The first model is used to output the first image with the artifact area removed.

[0036] Among them, the original image can be a multi-frame image continuously captured by the image sensor, or it can be a multi-frame image obtained by processing the multi-frame images continuously captured by the image sensor through at least one fusion strategy (such as 2-in-1 half-frame rate output strategy, 2-in-1 equal frame rate output strategy, 3-in-1 reduced frame rate output strategy, 3-in-1 equal frame rate output strategy, etc.).

[0037] Optionally, the acquired original image includes a plurality of consecutive long exposure images and short exposure images, wherein the long exposure images and the short exposure images appear alternately, for example, the original image includes long exposure image 1 and short exposure image 2; or, the original image includes short exposure image 1 and long exposure image 2.

[0038] The first image is a fused image obtained by synthesizing the original image through long and short exposure.

[0039] The first model is a trained model for outputting the first image with the artifact area removed, for example, an artificial intelligence (AI) neural network model. The first model is a model that takes the original image, the first image, and the second image as input. In this step, the original image, the first image, and the second image are obtained so as to subsequently obtain the first image with the artifact area removed based on the first model.

[0040] Step 102, inputting the original image, the first image and the second image into the first model, and taking the image output by the first model as the target image;

[0041] In this step, after the original image, the first image and the second image obtained in step 101 are input into the first model, the image output by the first model is the first image with the artifact area removed, that is, the target image is the fused image with the artifact area removed.

[0042] In this way, according to the above steps 101-104, the terminal executes the method of the embodiment of the present application. After obtaining the original image, the first image and the second image, it can input them into the first model, and obtain the desired fused image with the artifact area removed from the output of the first model, thereby reducing fusion abnormalities, reducing the differences between the fused images, and improving image quality.

[0043] It should be known that the method of this embodiment can be applied to HDR video generation, and HDR video can be generated based on the fused image with artifact areas removed, thereby avoiding jitter, flicker, stroboscopic and other phenomena in HDR video, enhancing the stability of HDR video output, and improving the user's terminal shooting and viewing experience.

[0044] Optionally, in this embodiment, the original image is a RAW domain image. RAW domain images have a higher dynamic range, a wider range of light intensity, and a higher color depth, and can retain more highlights and shadows. In addition, each pixel on the RAW domain image is linear data, which is proportional to the exposure intensity. Compared with RGB domain images, HDR fusion in the RAW domain can generate higher quality HDR images.

[0045] Optionally, in this embodiment, the long exposure image is also called a long frame (i.e., long exposure frame) image, and the short exposure image is also called a short frame (i.e., short exposure frame) image. As an implementation method, in order to reduce power consumption, the long frame image acquisition size is full size, such as 1920*980; the short frame image acquisition size is half size, such as 960*490. Short frame images have high noise, and using bilinear interpolation to restore the short frame size to full size can reduce noise intensity and reduce power consumption of storing and transmitting short frame images.

[0046] Specifically, the long and short exposure synthesis is based on the original image, the optical flow motion vector map corresponding to the original image, and the registered image corresponding to the original image. The different areas of the baseline reference frame image are determined by the optical flow motion vector map, and then the target strategy corresponding to each area of ​​the baseline reference frame image is determined according to the area types to which the different areas belong. Further, the determined target strategy can be used to fuse the original image and the configured image to obtain a fused image.

[0047] Optional, such as Figure 2 As shown, the first model mainly includes a convolutional network, a convolutional long short-term memory network (ConvLSTM) and a deconvolutional network.

[0048] Among them, the acquired original images (such as long exposure images (Lx) and short exposure images (Sx)), the first image (such as the current HDR fusion image (Processed_Lx_Sx)) and the second image (the image output last time (Output_Sx-Lx)) are input into the convolutional network of the first model. The convolutional network performs feature dimensionality reduction, and the deconvolutional network performs image feature recovery. In the middle, ConvLSTM is used to learn the relationship features between the previous and next frames. The input of ConvLSTM is four images. According to the similarity of movement, color, texture and other characteristics between the original long and short exposure images, the processed image features similar to the last output image are generated, so as to ensure the HDR video stability of the previous and next frame HDR images on the time axis.

[0049] Of course, in the first model, ConvLSTM can be replaced by a feedforward neural network (Transfomrer), and the convolutional network can be replaced by a residual network (ResNet). The main idea of ​​the structure of the first model is to extract features by downsampling the image, restore features by upsampling, and use the association network in the middle layer to learn the continuity between sequences.

[0050] In addition, optionally, in this embodiment, the method further includes:

[0051] constructing the first model;

[0052] The first model is trained according to multiple groups of images in a training set; wherein each group of images includes a third image, a fourth image, a fifth image and a sixth image, the third image is a long exposure image, the fourth image is a short exposure image, the fifth image is the image outputted by the first model last time during the training process, and the sixth image is a fused image obtained by fusing the third image, the fourth image and the corresponding registration image.

[0053] That is, the training of the first model will be combined with the color features, texture details, picture style, etc. of the historical frame (fifth image) and the current frame (third image, fourth image), so that the trained first model can ensure that the target image is free of artifacts. Accordingly, in HDR video generation, the output video frame content can be stable and free of jitter.

[0054] The multiple groups of images in the training set constitute multiple samples for the first model training, that is, each sample includes a group of third images, fourth images, fifth images and sixth images. The number of samples can be preset.

[0055] In which, considering that there is motion between the previous and next frame images, different optical flow calculation strategies in the occluded area lead to different effects, thereby affecting the convergence of the temporal consistency loss function. Optionally, in this embodiment, the training of the first model according to the multiple groups of images in the training set includes:

[0056] Input the third image, the fourth image, the fifth image, and the sixth image into the first model to obtain an eighth image output by the first model;

[0057] Calculating a loss of the first model based on the input and output of the first model;

[0058] Parameters of the first model are adjusted based on the loss of the first model.

[0059] That is to say, in one training, for one sample, the third image, the fourth image, the fifth image and the sixth image are input into the first model to obtain the eighth image as output. In this way, the loss can be calculated based on the input and output of the first model, and the parameters of the first model can be adjusted based on the loss until the training is completed.

[0060] Optionally, in this embodiment, the loss of the first model includes a time consistency loss value;

[0061] The calculating the loss of the first model based on the input and output of the first model includes:

[0062] When the temporal consistency loss value includes a short-time loss value, the short-time loss value is calculated by using the fifth image and the seventh image; wherein the seventh image is a residual image after removing the occluded area from the registration image of the third image and the fourth image;

[0063] When the temporal consistency loss value includes a long-term loss value, the long-term loss value is obtained by the difference between the eighth image and the ninth image, and the ninth image is a de-occluded area registration map corresponding to the first output image of the first model during the training process.

[0064] That is, the temporal consistency loss value is composed of the short-term loss value L st and long-term loss value L lt Among them, the short-term loss value L st It can be understood as the adjacent frame loss, which indicates the difference between two consecutive output images of the first model before and after; the long-term loss value L lt It can be understood as a cross-frame loss, which indicates the difference between the current output image of the first model and the first output image of the training.

[0065] In calculating L through the fifth and seventh images st When, optional, Lst The calculation formula (i.e. short-term loss function) is:

[0066]

[0067] Where i represents the i-th pixel, N represents the total number of pixels, t represents the t-th original image input to the first model during training, T represents the total number of original images in the training set, and O (i) t Represents the pixel value of the i-th pixel in the current output image, Represents the pixel value of the i-th pixel in the registered image after removing the occluded area.

[0068] In calculating L through the eighth and ninth images st When, optional, L lt The calculation formula (i.e. short-term loss function) is:

[0069]

[0070] Where i represents the i-th pixel, N represents the total number of pixels, t represents the t-th original image input to the first model during training, T represents the total number of original images in the training set, and O (i) t Represents the pixel value of the i-th pixel in the current output image, Represents the pixel value of the registration map of the removed occlusion area corresponding to the i-th pixel in the first output image of training.

[0071] For example, Figure 3 As shown, in the four trainings, the first training input is the third image L1, the fourth image L2, the fifth image O1 (L1-S1), and the sixth image P2 (S1-L1), and the eighth image is O2 after being processed by the first model; the second training input is the third image L2, the fourth image S2, the fifth image O2 (L2-S2), and the sixth image P3 (S2-L2)), and the eighth image is O3 after being processed by the first model; the third training input is the third image L3, the fourth image S3, the fifth image O3 (L3-S3), and the sixth image P4 (S3-L3)), and the eighth image is O4 after being processed by the first model; the fourth training input is the third image L4, the fourth image S4, the fifth image O4 (L4-S4), and the sixth image P5 (S4-L4)), and the eighth image is O5 after being processed by the first model.

[0072] In this embodiment, before calculating the loss of the first model, it is necessary to first determine the seventh image for the third image and the fourth image in the current sample, that is, the remaining image after the registration map removes the occluded area, which can also be understood as the registration map after the occluded area is removed. Optionally, the method further includes:

[0073] Acquire an optical flow motion vector map and a registration map of the third image and the fourth image;

[0074] Determine an occluded area in the registration map based on the optical flow motion vector map and the registration map;

[0075] The seventh image is determined based on the occluded area.

[0076] The optical flow motion vector diagram may be obtained by traditional optical flow estimation, or may be based on the third image and the fourth image, and each pixel in the two images is divided into a plurality of regions by the transmission channel thereof, and then further determined by the region type to which each region belongs. For example, the third image is the short-exposure image 1, and the fourth image is the long-exposure image 2, and the obtained optical flow motion vector diagram is the optical flow motion vector diagram from the short-exposure image 1 to the long-exposure image 2.

[0077] The registration (warp) map is obtained by performing an optical flow registration operation after obtaining the optical flow motion vector map. For example, the optical flow registration operation may be to combine the optical flow motion vector map with the short exposure image 1 to generate a map warped to the long exposure image 2.

[0078] After obtaining the optical flow motion vector map and the registration map, the optical flow motion vector map can be used to determine the occluded area in the registration map, reducing the impact of different optical flow strategies on the convergence of the loss function. Among them, the occluded area can be determined by calculating the difference between the registration map and the current frame image. If the difference is greater than the preset threshold, it is judged as an occluded area. The calculation formula is as follows:

[0079]

[0080] Among them, M t-1-＞t is the judgment value of whether the pixel is in the occlusion area, t is the image sequence number, α is the weight coefficient, I t is the pixel value of the pixel in the current frame image, is the pixel value of the registration map from the previous frame image to the current frame image, and Y is the preset threshold. Here, the current frame image is the third image or the fourth image.

[0081] Therefore, after the difference values ​​are obtained through calculation, the occluded area in the registration image can be determined according to the difference values ​​of each pixel point, thereby obtaining the seventh image.

[0082] Optionally, in this embodiment, the loss of the first model includes a texture consistency loss value.

[0083] Optional, texture consistency loss value L p The calculation formula (i.e., texture consistency loss function) is:

[0084]

[0085] Where i represents the i-th pixel, N represents the total number of pixels, t represents the t-th original image input to the first model during training, T represents the total number of original images in the training set, and O (i) t represents the pixel value of the i-th pixel in the current output image, P (i) t represents the pixel value of the fused image at the i-th pixel, φ l Represents the characteristic result of the output of the lth layer of the VGG network. The value of l is preset.

[0086] The VGG network consists of 3x3 convolutional layers and pooling layers, which can extract useful features from images. These features can measure the content differences and style differences between two images. The output results from different layers are the image feature information of the layer. The output results of different layers represent different features, so it is sufficient to extract several important feature layers to construct the loss function. Therefore, optionally, the loss of the first model is calculated, including:

[0087] In combination with the feature results output by each convolutional layer of the first model, the difference between the sixth image and the eighth image is calculated to determine the texture consistency loss value.

[0088] That is, the texture consistency loss value is calculated based on the images output by the previous and next two frames of the model to ensure that the texture features of the two output images are consistent, such as Figure 4 shown.

[0089] Thus, in this embodiment, the training phase can obtain the loss L of the first model by weighted summation based on the weight parameters corresponding to the short-term loss value, the long-term loss value, and the texture consistency loss value.

[0090] Specifically, the calculation formula of loss L is:

[0091] L=λ1L st +λ2L lt +λ3L p ; Among them, λ1 is the weight parameter of the short-term loss value, λ2 is the weight parameter of the long-term loss value, and λ3 is the weight parameter of the texture consistency loss value.

[0092] Further, it can be expanded to:

[0093]

[0094] In summary, in this embodiment, in HDR video generation, the process of training and applying the first model (also called forward reasoning) is as follows: Figure 5 As shown. Among them, optical flow estimation, registration and occlusion area judgment are only used in the training stage. A total of four images are input in the training stage, the original image (i.e., long exposure (Lx) and short exposure (Sx) images), the HDR image output by the last model (Output_Sx-Lx), and the current HDR fusion image (Processed_Lx_Sx). In the calculation of temporal consistency loss, based on the registration results between the previous and next long and short exposure image slices, the continuity of the pixel point change values ​​of the current image output by the model and the HDR image output by the previous frame of the model in time is inferred. In the calculation of texture consistency loss, the HDR images output by the previous and next frames of the model are input at the same time, and the calculated loss value ensures that the texture features of the previous and next images are consistent. The above two losses ensure the stability of the previous and next frames of HDR video.

[0095] During the training process, the original images are the third image and the fourth image in the training set samples; during the application process, the original images are the images acquired in real time.

[0096] In this embodiment, in the input stage of the first model, the current frame HDR image can be output according to the characteristics of the historical frame, or the current frame HDR image can be output by combining the characteristics of the historical frame and the future frame. The HDR image generation strategy includes but is not limited to the strategy described above, and the historical and future multiple frames can also be used to perform similar registration and generation on the current HDR image.

[0097] In addition, in this embodiment, the loss function of the training phase includes but is not limited to a temporal consistency loss function and a texture consistency loss function, and the main idea is to ensure the consistency, stability, and smoothness of the color, texture, and details of the previous and next two frames of images.

[0098] The method of the embodiment of the present application is applicable to HDR video processing and can achieve:

[0099] 1) Add one-dimensional information to the input and output of the first model, increasing the three-channel information of RGB to the four-channel information of RGGB;

[0100] 2) The optical flow motion vector map output by the optical flow estimation is used to determine the registration image where the occluded area is removed, which is used to calculate the temporal consistency loss during the training process.

[0101] 3) The original long and short frame images are introduced as input to alternately generate long-short HDR fusion and short-long HDR fusion video frames.

[0102] 4) Make full use of the performance of the terminal device's SoC CPU, SoC GPU, and SoC NPU. The flicker artifact removal module calls the SoC CPU for frame splitting and merging, and the first model calculation calls the SoC NPU. The chip linkage improves the real-time performance of HDR video.

[0103] It should be noted that in this embodiment, the hardware components of the terminal shown are as follows Figure 6 As shown, it includes (a) audio / video system, such as: camera (image sensor), audio equipment, display equipment; (b) chip processing system, such as central processing unit (CPU), neural network processor (NPU), graphics processing unit (GPU); (c) memory, such as system memory, local cache, etc. (d) communication bus. Of course, other hardware parts such as communication interface can also be included. Software components include kernel layer, system layer, application framework layer, and application layer. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor executes instructions in the memory, combined with its hardware completion.

[0104] The camera converts the optical image into an electronic signal (image Raw data), and the processor (CPU, NPU, GPU) is used for system coordination, scheduling, management, image-related processing flow and AI neural network processing flow, providing systematic computing support for the algorithm model. The system memory is used to temporarily store the calculation data in the CPU and exchange cache data with the CPU. The local cache is a buffer for CPU data exchange, which is used to solve the speed difference problem between the CPU speed and the memory speed. The processor, memory, and communication structure are connected through a communication bus; the memory is used to store execution instructions, the processor executes calculation instructions, and the required previous and next frame image data. The display device is used to present the digital image signal on the display to ensure the normal operation of the LCD display. The chip processing system is responsible for processing the data captured by the camera in the HDR video (High Dynamic Range Video Module-Lite, HDR-VM-Lite) module to output HDR-fused RGB images and HDR videos. The details of the HDR-MV-LITE module are as follows: Figure 7As shown in the figure, it mainly includes four submodules: (a) HDR video frame acquisition submodule; (b) HDR single frame synthesis submodule; (c) storage submodule; (d) HDR video single frame processing submodule. The HDR-MV-LITE module inputs continuous long and short exposure RAW domain images and outputs RAW domain HDR synthesis images. The HDR video acquisition strategy (the strategy used by the image sensor to acquire images) is located in the HDR video frame acquisition module.

[0105] The HDR-VM-Lite module is used for terminal photo taking, video recording, video calling, live broadcasting and other functions.

[0106] Among them, the HDR video frame acquisition submodule automatically adjusts the fusion strategy, acquisition interval, and sensitivity according to the light intensity of the external environment and the movement of objects to ensure the optimal front-end long and short frame input in different scenarios.

[0107] Among them, the HDR single-frame synthesis submodule includes (a) optical flow estimation unit 1, (b) long and short exposure registration unit, and (c) long and short exposure synthesis unit.

[0108] The optical flow estimation unit 1 is used to estimate the optical flow motion vector between the long and short exposure frames.

[0109] The long-short exposure registration unit is used to align the pixels of the long-short exposure images according to the optical flow motion information, and can be calculated using the following formula I(x',y')=I(x+Δx,y+Δy). Where I represents the pixel value, (x',y') represents the pixel position of the target image; (x,y) represents the pixel position of the original image, and (Δx,Δy) represents the optical flow motion vector calculated by the optical flow estimation unit 1.

[0110] The long and short exposure synthesis unit processes the long and short exposure images respectively and outputs a fusion weight map (fusion weight corresponding image). The fusion weight can be calculated based on factors such as exposure ratio, pixel brightness, regional similarity, and time similarity, and can also be output through an AI neural network. In this embodiment, the fusion weight is associated with the brightness weight, saturation weight, and contrast weight.

[0111] Among them, the HDR video single frame processing submodule includes an optical flow estimation unit 2, a preprocessing unit, an HDR multi-frame feature extraction unit, an HDR multi-frame feature association unit, an HDR single-frame feature generation unit, and an artifact elimination unit. The optical flow estimation unit 2 can use a method similar to that of the optical flow estimation unit 1. The optical flow estimation result is used for HDR multi-frame feature fusion. After extracting the HDR video frame features, an alignment operation is performed on the feature layer. Finally, a jitter-free, color-stable HDR video is output.

[0112] Among them, the artifact elimination unit can be understood as the first module, the preprocessing unit is used for preprocessing before the first model inference, and the HDR multi-frame feature extraction unit, the HDR multi-frame feature association unit, and the HDR single-frame feature generation unit are used for training the first model.

[0113] In one embodiment, the HDR-VM-Lite module is input as a RAW domain image, which is directly acquired by the front-end camera sensor. The pixel value is proportional to the exposure time, brightness, etc., and is linear data. Most traditional optical flow estimation is applied to RGG domain nonlinear feature images after demosaicing, and is not applicable to RAW domain linear feature images. In addition, compared with the RGB domain, the pixel arrangement distribution in the Raw domain is RGGB, so the Raw domain pixels have a higher value on the G (green) channel, and the pixel values ​​on the B (blue) and R (red) channels are relatively low. The image processing method of the embodiment of the present application is used to improve the accuracy of inter-frame optical flow estimation and reduce HDR video fusion anomalies on the basis of ensuring the real-time performance of the algorithm, thereby improving the user's terminal shooting and reading experience.

[0114] Specifically, when a user uses a terminal to enable HDR video mode, such as clicking Figure 8 The HDR mode function button shown in the figure, the terminal camera will capture video frame images with a fixed frame rate and alternating long and short exposures. After that, the HDR video processing flow is as follows Fig. 9 As shown:

[0115] The terminal can determine at least one fusion strategy and input a group of images into the HDR single-frame synthesis submodule. The HDR single-frame synthesis submodule selects alternating long and short exposure frames from the DDR (a) memory for corresponding HDR single-frame fusion; at the same time, the optical flow estimation unit 1 in the HDR single-frame synthesis submodule determines the optical flow motion vector map, and the calculation result is stored in the DDR (c) memory; the HDR single-frame synthesis submodule or the long and short exposure synthesis unit applies the image processing method of the embodiment of the present application, and saves one copy of the fusion result (fused image) in the DDR (b) memory for subsequent HDR video single-frame processing submodule use, and directly inputs one copy into the HDR video single-frame processing submodule. The HDR video single-frame processing submodule reads the original long and short exposure data, optical flow data (such as optical flow motion vector map), HDR fusion picture, and finally outputs an HDR picture that is consistent with the texture, color, motion and other features of the historical frame as a frame in the HDR output video.

[0116] In the HDR video processing flow, optical flow calculation runs on the SoC GPU, and other functions of the HDR single-frame synthesis submodule are implemented on the SoC CPU; other functions of the HDR video single-frame processing submodule are implemented on the SoC NPU. The performance of each SoC chip is fully utilized to output HDR video data in real time while ensuring the stability of HDR video.

[0117] In this way, the real-time HDR video output function can be completed on the terminal side, and the front-to-back stability of the HDR video frame can be improved by combining the optical flow information of the HDR video frame and relying on the first model (AI neural network model), so as to achieve real-time online HDR video output. This solution can be used to reduce the jitter phenomenon of HDR video, and HDR video truly reflects the color range of real scenes.

[0118] Or, specifically, after using the terminal for HDR shooting, the user can set the HDR offline processing function. After the setting is completed, the terminal background starts the HDR video synthesis effect when the current process task is not busy. At this time, the original long and short exposure data collected by the front-end camera has been cleared from the DDR, so it is necessary to perform frame insertion processing on the synthesized HDR video, and at the same time call the optical flow model to calculate the optical flow vector information of the front and back frames of the HDR video. The offline processing process is as follows Fig.10 As shown:

[0119] The HDR video is first deframed, and the result is stored in the DDR (a) memory; the deframed image enters the optical flow estimation unit 1 to output the optical flow motion vector information (such as the optical flow motion vector diagram) between the previous and next frames; the HDR video single frame processing submodule is called, and the HDR video single frame processing submodule performs self-training on the HDR video. After the training is completed, the trained model is used to read the HDR deframe data to regenerate each frame in the HDR video and remove the HDR video jitter. In this way, the HDR video offline output function can be completed on the terminal side, and self-training can be performed on the HDR video to make the model more compatible with the HDR video content and generate a more stable HDR video.

[0120] It should be noted that the terminal uses the CPU to send a control strategy to allocate corresponding GPU resources for the optical flow calculation of each block, and parallel calculation accelerates the optical flow calculation results of the picture to predict the optical flow motion vector field between long and short exposures. Taking the use of Lucas-Kanade for optical flow estimation as an example, before calculating the optical flow motion information of long and short exposures (such as the optical flow motion vector map), the two images need to be converted into a unified grayscale image, and then the optical flow information of the two images is calculated according to the Lucas-Kanade method. After obtaining the optical flow motion vector information, the pixel point A of the previous frame can be moved to a new position according to the motion vector, and the pixel point B of the current frame can be moved to the next frame position according to the motion vector.

[0121] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0122] like Fig.11 As shown, an image processing device 1100 according to an embodiment of the present application includes:

[0123] A first processing module 1110 is used to obtain an original image, a first image, and a second image; wherein the original image includes a long-exposure image and a short-exposure image, the first image is a fused image obtained by fusing the original image and a corresponding registration image, and the second image is an image outputted by the first model last time;

[0124] A second processing module 1120, configured to input the original image, the first image and the second image into the first model, and use the image output by the first model as a target image;

[0125] The first model is used to output the first image with the artifact area removed.

[0126] Optionally, the device further comprises:

[0127] A third processing module, used for constructing the first model;

[0128] A fourth processing module is used to train the first model according to multiple groups of images in a training set; wherein each group of images includes a third image, a fourth image, a fifth image and a sixth image, the third image is a long exposure image, the fourth image is a short exposure image, the fifth image is the image output by the first model last time during the training process, and the sixth image is a fused image obtained by fusing the third image, the fourth image and the corresponding registration image.

[0129] Optionally, the fourth processing module is further used for:

[0130] Input the third image, the fourth image, the fifth image, and the sixth image into the first model to obtain an eighth image output by the first model;

[0131] Calculating a loss of the first model based on the input and output of the first model;

[0132] Parameters of the first model are adjusted based on the loss of the first model.

[0133] Optionally, the loss of the first model includes a time consistency loss value;

[0134] The fourth processing module is further used for:

[0135] When the temporal consistency loss value includes a short-time loss value, the short-time loss value is calculated by using the fifth image and the seventh image; wherein the seventh image is a residual image after removing the occluded area from the registration image of the third image and the fourth image;

[0136] When the temporal consistency loss value includes a long-term loss value, the long-term loss value is obtained by the difference between the eighth image and the ninth image, and the ninth image is a registration map of the removed occluded area corresponding to the first output image of the first model during the training process.

[0137] Optionally, the device further comprises:

[0138] A fifth processing module, configured to obtain an optical flow motion vector map and a registration map of the third image and the fourth image;

[0139] A sixth processing module, configured to determine an occlusion area in the registration map based on the optical flow motion vector map and the registration map;

[0140] A seventh processing module is used to determine the seventh image based on the occluded area.

[0141] Optionally, the loss of the first model includes a texture consistency loss value.

[0142] Optionally, the fourth processing module is further used for:

[0143] In combination with the feature results output by each convolutional layer of the first model, the difference between the sixth image and the eighth image is calculated to determine the texture consistency loss value.

[0144] After acquiring the original image, the first image and the second image, the device can input them into the first model, and obtain the desired fused image with the artifact area removed from the output of the first model, thereby reducing fusion abnormalities, reducing the differences between fused images, and improving image quality.

[0145] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0146] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0147] The image processing device provided in the embodiment of the present application can achieve Figure 1-Figure 10 To avoid repetition, the various processes implemented in the embodiment of the image processing method are not described again here.

[0148] Alternatively, if Fig.12 As shown, an embodiment of the present application further provides an electronic device 1200, including a processor 1201 and a memory 1202, wherein the memory 1202 stores programs or instructions that can be executed on the processor 1201, and when the program or instructions are executed by the processor 1201, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0149] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0150] Fig.13 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0151] The electronic device 1300 includes but is not limited to: a radio frequency unit 1301, a network module 1302, an audio output unit 1303, an input unit 1304, a sensor 1305, a display unit 1306, a user input unit 1307, an interface unit 1308, a memory 1309, and a processor 1310 and other components.

[0152] Those skilled in the art will appreciate that the electronic device 1300 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1310 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.13 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0153] The processor 1310 is used to obtain an original image, a first image, and a second image; wherein the original image includes a long-exposure image and a short-exposure image, the first image is a fused image obtained by fusing the original image and the corresponding registration image, and the second image is an image output by the first model last time;

[0154] Input the original image, the first image and the second image into the first model, and use the image output by the first model as the target image;

[0155] The first model is used to output the first image with the artifact area removed.

[0156] Optionally, the processor 1310 is further configured to:

[0157] constructing the first model;

[0158] The first model is trained according to multiple groups of images in a training set; wherein each group of images includes a third image, a fourth image, a fifth image and a sixth image, the third image is a long exposure image, the fourth image is a short exposure image, the fifth image is the image outputted by the first model last time during the training process, and the sixth image is a fused image obtained by fusing the third image, the fourth image and the corresponding registration image.

[0159] Optionally, the processor 1310 is further configured to:

[0160] Input the third image, the fourth image, the fifth image, and the sixth image into the first model to obtain an eighth image output by the first model;

[0161] Calculating a loss of the first model based on the input and output of the first model;

[0162] Parameters of the first model are adjusted based on the loss of the first model.

[0163] Optionally, the loss of the first model includes a time consistency loss value;

[0164] The processor 1310 is further configured to:

[0165] When the temporal consistency loss value includes a short-time loss value, the short-time loss value is calculated by using the fifth image and the seventh image; wherein the seventh image is a residual image after removing the occluded area from the registration image of the third image and the fourth image;

[0166] When the temporal consistency loss value includes a long-term loss value, the long-term loss value is obtained by the difference between the eighth image and the ninth image, and the ninth image is a registration map of the removed occluded area corresponding to the first output image of the first model during the training process.

[0167] Optionally, the processor 1310 is further configured to:

[0168] Acquire an optical flow motion vector map and a registration map of the third image and the fourth image;

[0169] Determine an occluded area in the registration map based on the optical flow motion vector map and the registration map;

[0170] The seventh image is determined based on the occluded area.

[0171] Optionally, the loss of the first model includes a texture consistency loss value.

[0172] Optionally, the processor 1310 is further configured to:

[0173] In combination with the feature results output by each convolutional layer of the first model, the difference between the sixth image and the eighth image is calculated to determine the texture consistency loss value.

[0174] It should be understood that in the embodiment of the present application, the input unit 1304 may include a graphics processor (Graphics Processing Unit, GPU) 13041 and a microphone 13042, and the graphics processor 13041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1306 may include a display panel 13061, and the display panel 13061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1307 includes a touch panel 13071 and at least one of other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include two parts: a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0175] The memory 1309 can be used to store software programs and various data. The memory 1309 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1309 may include a volatile memory or a non-volatile memory, or the memory 1309 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1309 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0176] The processor 1310 may include one or more processing units; optionally, the processor 1310 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1310.

[0177] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0178] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0179] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0180] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0181] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0182] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0184] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Executed by the terminal, including: Acquire an original image, a first image, and a second image; wherein the original image includes a long-exposure image and a short-exposure image, the first image is a fused image obtained by fusing the original image and a corresponding registration image, and the second image is an image outputted by the first model last time; Input the original image, the first image and the second image into the first model, and use the image output by the first model as the target image; The first model is used to output the first image with the artifact area removed.

2. The method according to claim 1, characterized in that Also includes: constructing the first model; The first model is trained according to multiple groups of images in a training set; wherein each group of images includes a third image, a fourth image, a fifth image and a sixth image, the third image is a long exposure image, the fourth image is a short exposure image, the fifth image is the image outputted by the first model last time during the training process, and the sixth image is a fused image obtained by fusing the third image, the fourth image and the corresponding registration image.

3. The method according to claim 2, characterized in that The step of training the first model according to the multiple groups of images in the training set includes: Input the third image, the fourth image, the fifth image, and the sixth image into the first model to obtain an eighth image output by the first model; Calculating a loss of the first model based on the input and output of the first model; Parameters of the first model are adjusted based on the loss of the first model.

4. The method according to claim 3, characterized in that The loss of the first model includes a time consistency loss value; The calculating the loss of the first model based on the input and output of the first model includes: When the temporal consistency loss value includes a short-time loss value, the short-time loss value is calculated by using the fifth image and the seventh image; wherein the seventh image is a residual image after removing the occluded area from the registration image of the third image and the fourth image; When the temporal consistency loss value includes a long-term loss value, the long-term loss value is obtained by the difference between the eighth image and the ninth image, and the ninth image is a registration map of the removed occluded area corresponding to the first output image of the first model during the training process.

5. The method according to claim 4, characterized in that Also includes: Acquire an optical flow motion vector map and a registration map of the third image and the fourth image; Determine an occluded area in the registration map based on the optical flow motion vector map and the registration map; The seventh image is determined based on the occluded area.

6. The method according to claim 3 or 4, characterized in that: The loss of the first model includes a texture consistency loss value.

7. The method according to claim 6, characterized in that Calculating the loss of the first model includes: In combination with the feature results output by each convolutional layer of the first model, the difference between the sixth image and the eighth image is calculated to determine the texture consistency loss value.

8. An image processing device, characterized in that: include: A first processing module is used to obtain an original image, a first image, and a second image; wherein the original image includes a long-exposure image and a short-exposure image, the first image is a fused image obtained by fusing the original image and a corresponding registration image, and the second image is an image outputted by the first model last time; A second processing module, configured to input the original image, the first image and the second image into the first model, and use the image output by the first model as a target image; The first model is used to output the first image with the artifact area removed.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images