Methods, apparatus, storage media and electronic devices for enhancing images

Abstract

This application proposes a method, apparatus, storage medium, and electronic device for generating enhanced images. The method includes: determining a target frame image from multiple consecutive frame images; pairing each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs; sending the multiple image pairs to an optical flow calculation unit and obtaining optical flow data obtained by the optical flow calculation unit based on the multiple image pairs; sending the optical flow data and the multiple consecutive frame images to a video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image based on the optical flow data; and outputting the aligned multiple frame images. This technical solution solves the technical problem of low video quality in night scene videos under low light conditions in related technologies.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and more specifically, to a method, apparatus, storage medium, and electronic device for generating enhanced images. Background Technology

[0002] In recent years, with the increasing number of cameras supporting night scene shooting, users' demands for the quality of night scene videos have also been continuously increasing. Among related technologies, the processing of night scene videos under low light conditions has always been a crucial research direction in the field of video processing. The lack of light sources in night scenes leads to poor lighting conditions, resulting in insufficient sensor sensitivity. Compared to images or videos captured under good lighting conditions, images or videos captured in night scenes often exhibit more noise, color casts, and lower resolution. This not only affects the user's visual experience but also impacts the accuracy of motion target identification in scenarios requiring the extraction of moving objects from the video. Therefore, it is essential to improve the quality of night scene videos under low light conditions.

[0003] Currently, adaptive video night scene algorithms based on neural networks can be used to optimize images or videos captured in night scenes, thereby improving video quality. These algorithms belong to the supervised learning type of neural network algorithm, where training samples include input and output samples. The parameters in the neural network algorithm are updated using gradient descent, ensuring that the output of the training samples closely approximates the output sample, achieving a result that is free of noise while preserving as much video image detail as possible. However, these neural network algorithms typically use raw domain images as input samples and sRGB domain images as output samples. The operation on the input images is simply superimposing consecutive frames into the processing network. This often results in motion blur in the output images or videos.

[0004] Regarding the technical issue of low video quality in night scene videos under low light conditions, no effective solution has yet been proposed.

[0005] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0006] This application provides a method, apparatus, storage medium, and electronic device for generating enhanced images, to at least address the technical problem of low video quality in night scene videos under low light conditions.

[0007] According to one aspect of the embodiments of this application, a method for generating an enhanced image is provided, comprising: determining a target frame image from multiple consecutive frame images; pairing each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs; sending the multiple image pairs to an optical flow calculation unit and obtaining optical flow data obtained by the optical flow calculation unit based on the multiple image pairs, wherein the optical flow data is used to indicate the pixel parameters of the images in the multiple image pairs; sending the optical flow data and the multiple consecutive frame images to a video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image based on the optical flow data, and outputting multiple aligned frame images, wherein the multiple aligned frame images are used to indicate multiple consecutive frame images aligned with the target frame image; determining multiple first tensor data based on the multiple aligned frame images and the target frame image, and sending the multiple first tensor data to a video restoration unit, so that the video restoration unit generates an enhanced image based on the multiple first tensor data.

[0008] In an exemplary embodiment, before determining the target frame image from multiple acquired consecutive frame images and pairing each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs, the method includes: acquiring a video file that meets preset conditions through a camera device, wherein the preset conditions include at least one of the following: the duration of the video file is less than a first threshold; the size of the video file is less than a second threshold; the type of the video file is a preset type; and extracting the video file frame by frame to obtain the extracted multiple consecutive frame images.

[0009] In an exemplary embodiment, sending the plurality of image pairs to an optical flow calculation unit and obtaining optical flow data obtained by the optical flow calculation unit based on the plurality of image pairs includes: obtaining each image pair in the plurality of image pairs, wherein each image pair includes the target frame image and one frame image from the plurality of consecutive frame images; sending each image pair to the optical flow calculation unit respectively; determining pixel position change parameters of the images in each image pair through the optical flow calculation unit; wherein the pixel position change parameters of the images in each image pair are used to represent the position change of pixels between the target frame image and the one frame image in each image pair; and determining the optical flow data output by the optical flow calculation unit to include the pixel position change parameters of the images in each image pair.

[0010] In an exemplary embodiment, determining a plurality of first tensor data based on the plurality of aligned frame images and the target frame image includes: adjusting the plurality of aligned frame images and the target frame image to the same size as a plurality of first sample images; determining a plurality of first tensor data based on the image data of the plurality of first sample images, wherein each first tensor data is used to represent the image data of a corresponding first sample image among the plurality of first sample images in the form of an array, and each first tensor data includes at least one of the following: the height of the corresponding first sample image, the width of the corresponding first sample image, and the number of channels of the corresponding first sample image.

[0011] In an exemplary embodiment, after sending the plurality of first tensor data to the video restoration unit, the method further includes: determining a plurality of second tensor data by the video restoration unit based on the plurality of first tensor data, and determining image data of a plurality of second sample images corresponding to the plurality of second tensor data, wherein each second tensor data is used to represent the image data of the corresponding second sample image in the plurality of second sample images in the form of an array, and the product of the height, width and number of channels of each second sample image in the second tensor data is equal to the product of the height, width and number of channels of each first sample image in the first tensor data; determining the image data of the plurality of second sample images based on the plurality of second tensor data; inputting the image data of the plurality of second sample images into a target neural network model, and generating the enhanced image by the target neural network model, wherein the target neural network model is used to perform enhancement processing on the input image data, obtain a processing result, and generate an image corresponding to the processing result.

[0012] In an exemplary embodiment, after sending the plurality of first tensor data to a video restoration unit so that the video restoration unit generates an enhanced image based on the first tensor data, the method further includes: performing enhancement processing on the target frame image to obtain a reference image; determining the image similarity between the enhanced image and the reference image; and determining that the enhanced image passes verification if the image similarity is less than a preset value.

[0013] In an exemplary embodiment, determining the image similarity between the enhanced image and the reference image includes: determining a first weighting coefficient and a second weighting coefficient, wherein the first weighting coefficient corresponds to the pixel difference between the enhanced image and the reference image, and the second weighting parameter corresponds to the structural similarity value between the enhanced image and the reference image, the structural similarity value indicating the degree of similarity of the image structural information of the enhanced image and the reference image; determining the pixel similarity between the enhanced image and the reference image, and determining the structural similarity between the enhanced image and the reference image, wherein the pixel similarity between the enhanced image and the reference image is the product of the first weighting coefficient and the pixel difference, and the structural similarity between the enhanced image and the reference image is the product of the second weighting parameter and the structural similarity value; obtaining the sum of the pixel similarity and the structural similarity; and determining the image similarity according to a preset proportional relationship and the sum, wherein the preset proportional relationship indicating the negative correlation between the sum and the image similarity.

[0014] According to another aspect of the embodiments of this application, an enhanced image generation apparatus is also provided, comprising: a determining module, configured to determine a target frame image from multiple consecutive frame images acquired, and pair each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs; a first sending module, configured to send the multiple image pairs to an optical flow calculation unit, and acquire optical flow data obtained by the optical flow calculation unit based on the multiple image pairs, wherein the optical flow data is used to indicate the pixel parameters of the images in the multiple image pairs; a second sending module, configured to send the optical flow data and the multiple consecutive frame images to a video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image based on the optical flow data, and outputs multiple aligned frame images, wherein the multiple aligned frame images are used to indicate multiple consecutive frame images aligned with the target frame image; and a third sending module, configured to determine first tensor data based on the multiple aligned frame images and the target frame image, and send the first tensor data to a video restoration unit, so that the video restoration unit generates an enhanced image based on the first tensor data.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described method for generating enhanced images when it is run.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described method for generating enhanced images through the computer program.

[0017] This application solves the technical problem of low video quality in night scene videos under low light conditions by sending the optical flow calculation unit to a video alignment unit based on the optical flow data obtained from the multiple sets of image pairs and the multiple consecutive frame images. The video alignment unit aligns the multiple consecutive frame images with the target frame image based on the optical flow data and outputs the aligned frame images. Multiple first tensor data are determined based on the aligned frame images and the target frame image, and these first tensor data are sent to a video restoration unit. The video restoration unit then generates an enhanced image based on the multiple first tensor data. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and, together with the description thereof, serve to explain this application and do not constitute an undue limitation thereof. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal for the enhanced image generation method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a method for generating enhanced images according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a night scene video enhancement network according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of the optical flow calculation unit according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the tensor reconstruction process according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram illustrating the process of generating an enhanced image according to an embodiment of this application;

[0025] Figure 7 This is a structural block diagram of an enhanced image generation apparatus according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for the enhanced image generation method according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data are also shown. In one exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0029] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the enhanced image generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0031] Figure 2 This is a flowchart of an enhanced image generation method according to an embodiment of this application, such as... Figure 2 As shown, the steps of this encapsulation method include:

[0032] Step S202: Determine the target frame image from the acquired multiple consecutive frame images, and pair each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs.

[0033] It should be noted that the intermediate frame image among multiple consecutive frame images can be determined as the target frame image. The intermediate frame image can be determined according to the order of the frame images being captured, or it can be determined according to the size order of the frame images. This application does not impose any restrictions on this.

[0034] Step S204: Send the multiple sets of image pairs to the optical flow calculation unit and obtain the optical flow data obtained by the optical flow calculation unit based on the multiple sets of image pairs, wherein the optical flow data is used to indicate the pixel parameters of the images in the multiple sets of image pairs.

[0035] It should be noted that the pixel parameters of the above images may include, but are not limited to, the pixel position, the number of pixels, the pixel density, and the pixel spacing of the images.

[0036] Step S206: The optical flow data and the multiple consecutive frame images are sent to the video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image according to the optical flow data, and outputs the aligned multiple frame images, wherein the multiple aligned frame images are used to indicate the multiple consecutive frame images aligned with the target frame image.

[0037] It should be noted that the process of aligning the multiple consecutive frame images with the target frame image based on the optical flow data can be understood as aligning the pixel positions in the multiple consecutive frame images with the specified pixel positions in the target frame image, but it is not limited to this.

[0038] Step S208: Determine multiple first tensor data based on the multiple aligned frame images and the target frame image, and send the multiple first tensor data to the video restoration unit so that the video restoration unit can generate an enhanced image based on the multiple first tensor data.

[0039] This embodiment of the application determines a target frame image from multiple consecutive frame images, pairs each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs; sends the multiple image pairs to an optical flow calculation unit, and obtains optical flow data obtained by the optical flow calculation unit based on the multiple image pairs, wherein the optical flow data is used to indicate the pixel parameters of the images in the multiple image pairs; sends the optical flow data and the multiple consecutive frame images to a video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image based on the optical flow data, and outputs multiple aligned frame images, wherein the multiple aligned frame images are used to indicate multiple consecutive frame images aligned with the target frame image, thereby solving the technical problem of low video quality in night scene videos under low light conditions and improving the video quality of night scene videos.

[0040] In an exemplary embodiment, before determining the target frame image from multiple acquired consecutive frame images and pairing each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs, a technical solution is provided. The specific steps include: acquiring a video file that meets preset conditions using a camera device; the preset conditions include at least one of the following: the duration of the video file is less than a first threshold; the size of the video file is less than a second threshold; the type of the video file is a preset type; and extracting the video file frame by frame to obtain the extracted multiple consecutive frame images.

[0041] It should be noted that the above-mentioned camera device can be understood as a device equipped with a camera component. The camera component is a device used to complete image decomposition and photoelectric signal conversion. Camera devices include CCD (Charge Coupled Device) cameras, CMOS (Complementary Metal-Oxide-Semiconductor) cameras, etc., but are not limited to these.

[0042] The aforementioned first threshold can be one hour, one week, or one day; this application does not impose any restrictions on this. For example, if the first threshold is one hour, and the duration of the video file is two hours, then the video file does not meet the preset conditions.

[0043] The second threshold mentioned above can be 1GB, 1TB, or 100MB, and this application does not limit it. For example, if the second threshold is 100MB, and the size of the video file is 50MB, then the video file does not meet the preset conditions.

[0044] The above preset types can be understood as the formats in which computers store videos, including but not limited to avi, wmv, MPEG, rm, rmvb, mov, and asf formats.

[0045] It should be noted that the image information reflected in the enhanced images is consistent with the image information reflected in each consecutive frame of the images. The image information includes: the scene in the image, the objects in the image, and the annotations in the image, etc. This application does not limit this.

[0046] In an exemplary embodiment, to better understand how the multiple sets of image pairs are sent to the optical flow calculation unit in step S204, and to obtain the optical flow data obtained by the optical flow calculation unit based on the multiple sets of image pairs, each set of image pairs can be obtained, wherein each set of image pairs includes the target frame image and one frame image from the multiple consecutive frame images; each set of image pairs is sent to the optical flow calculation unit; the pixel position change parameters of the images in each set of image pairs are determined by the optical flow calculation unit; wherein the pixel position change parameters of the images in each set of image pairs are used to represent the pixel position change between the target frame image and the one frame image in each set of image pairs; the optical flow data output by the optical flow calculation unit is determined to include the pixel position change parameters of the images in each set of image pairs.

[0047] In an exemplary embodiment, to better understand how to determine multiple first tensor data based on the multiple aligned frame images and the target frame image in step S208, a technical solution is proposed. The specific steps include: adjusting the multiple aligned frame images and the target frame image to the same size as multiple first sample images; determining multiple first tensor data based on the image data of the multiple first sample images, wherein each first tensor data is used to represent the image data of a corresponding first sample image among the multiple first sample images in array form, and each first tensor data includes at least one of the following: the height of the corresponding first sample image, the width of the corresponding first sample image, and the number of channels of the corresponding first sample image.

[0048] It should be noted that the image data of the first sample image can be decoded into image data represented in array form by a decoding function in the computer, namely the first tensor data mentioned above. In one embodiment, each first tensor data can be represented as an array of [number of image channels (i.e., image depth), image width, image height], or as an array of [number of samples, number of image channels (i.e., image depth), image width, image height], to store the image data of each first sample image.

[0049] In an exemplary embodiment, after sending the plurality of first tensor data to the video restoration unit, the video restoration unit can further determine a plurality of second tensor data based on the plurality of first tensor data, and determine the image data of a plurality of second sample images corresponding to the plurality of second tensor data. Each second tensor data is used to represent the image data of the corresponding second sample image among the plurality of second sample images in array form. The product of the height, width, and number of channels of each second sample image in the second tensor data is equal to the product of the height, width, and number of channels of each first sample image in the first tensor data. The image data of the plurality of second sample images is determined based on the plurality of second tensor data. The image data of the plurality of second sample images is input into a target neural network model, and the enhanced image is generated through the target neural network model. The target neural network model is used to perform enhancement processing on the input image data to obtain a processing result and generate an image corresponding to the processing result.

[0050] It should be noted that, in one embodiment, the product of the height, width, and number of channels of each second sample image in the second tensor data is equal to the product of the height, width, and number of channels of each first sample image in the first tensor data. For example, if the first tensor data is represented by [1,1,1], that is, the first tensor data represents image data with a height of 1 pixel, a width of 1 pixel, and a depth of 1 pixel, then the second tensor data can be represented by [1 / 2,1 / 2,4], that is, the second tensor data represents image data with a height of 1 / 2 pixel, a width of 1 / 2 pixel, and a depth of 4 pixels.

[0051] It should be noted that the image data enhancement process of the aforementioned target neural network model includes, but is not limited to, image denoising, image brightness enhancement, image contrast enhancement, and image color restoration.

[0052] In an exemplary embodiment, after sending the plurality of first tensor data to the video restoration unit so that the video restoration unit generates an enhanced image based on the first tensor data, a technical solution is also provided, the specific steps of which include: performing enhancement processing on the target frame image to obtain a reference image; determining the image similarity between the enhanced image and the reference image; and determining that the enhanced image passes the verification if the image similarity is less than a preset value.

[0053] It should be noted that the preset value ranges from [0,1]. When the preset value is 0.1, if the image similarity is 0.05, the enhanced image passes the verification, and the image enhancement method in this embodiment can be applied to real-world work scenarios. If the image similarity is 0.3, the enhanced image fails the verification, and the image similarity can be obtained again by re-determining the consecutive frames or optimizing the target neural network model.

[0054] In an exemplary embodiment, a technical solution is provided for determining the image similarity between an enhanced image and a reference image. The specific steps include: determining a first weighting coefficient and a second weighting coefficient, wherein the first weighting coefficient corresponds to the pixel difference between the enhanced image and the reference image, and the second weighting coefficient corresponds to a structural similarity value between the enhanced image and the reference image, the structural similarity value indicating the degree of similarity in the image structural information of the enhanced image and the reference image; determining the pixel similarity between the enhanced image and the reference image, and determining the structural similarity between the enhanced image and the reference image, wherein the pixel similarity between the enhanced image and the reference image is the product of the first weighting coefficient and the pixel difference, and the structural similarity between the enhanced image and the reference image is the product of the second weighting coefficient and the structural similarity value; obtaining the sum of the pixel similarity and the structural similarity; and determining the image similarity according to a preset proportional relationship and the sum, wherein the preset proportional relationship indicates a negative correlation between the sum and the image similarity.

[0055] It should be noted that, in one embodiment, the process of determining the image similarity between the enhanced image and the reference image can be illustrated by example. In this embodiment, an objective function L is used to represent image similarity, and the expression of the objective function L is as follows:

[0056] L=αL enh +(1-α)L ssim (1)

[0057] Where, L enh To enhance the image Compared with reference image I GT The L1 norm of the absolute values ​​of the pixel differences (representing the sum of the absolute values ​​of all pixel differences, equivalent to the pixel similarity between the enhanced image and the reference image mentioned above), i.e.

[0058]

[0059] L ssim This is a structural similarity function (equivalent to the structural similarity value between the enhanced image and the reference image mentioned above).

[0060]

[0061] Where, μ x μ y σ represents the average value. x , σ y Let σ be the variance. xy For covariance, σ xyThis effectively enhances image details. c1 and c2 are regularization constants. α is the first weight coefficient, and (1-α) is the second weight coefficient. The first and second weight coefficients can be set according to specific circumstances. After calculating L, the larger the value of L, the smaller the image similarity.

[0062] The following examples and appendices are then discussed. Figure 3-6 The method for generating enhanced images will be further explained.

[0063] Example 1

[0064] This embodiment provides a method for generating enhanced images, relating to the field of video processing technology. It can be implemented using a night scene video enhancement algorithm, which is detailed below:

[0065] Step 1: Read consecutive image frames from RAW format video as input data for the night scene video enhancement algorithm;

[0066] It should be noted that the aforementioned RAW format video can be understood as a video file that records the raw information of the camera device's sensor. This RAW format video also records video metadata generated by the camera device during shooting, such as shutter speed, aperture value, white balance parameters, etc., but is not limited to these.

[0067] Step 2: Rearrange the extracted RAW format video data and split it according to the Bayer unit. Construct the (2N+1)xHxWx1 image data into (2N+1)xH / 2xW / 2x4 image data.

[0068] It should be noted that the above process of splitting according to Bayer units can be understood as processing RAW format video data using a Bayer filter. A Bayer filter is a filter composed of a 2-pixel x 2-pixel array of green, blue, red, and green pixels arranged in a repetitive pattern. That is, a Bayer unit can be understood as one of the array units in a Bayer filter with a height of 2 pixels and a width of 2 pixels. The aforementioned (2N+1)xHxWx1 image data can be understood as the input data in step 1, representing (2N+1) images with a height of 1 pixel, a width of 1 pixel, and a depth of 1 pixel. The aforementioned (2N+1)xH / 2xW / 2x image data represents (2N+1) images with a height of 1 / 2 pixel, a width of 1 / 2 pixel, and a depth of 4 pixels.

[0069] Step 3: Input the rearranged video data into the pre-trained neural network model (equivalent to the target neural network model mentioned above) for processing;

[0070] Optionally, in step 3 above, the video restoration network uses the aligned 2N+1 images to generate an image similar to one under bright lighting conditions through enhancement processes such as noise reduction, brightening, contrast enhancement, and color restoration.

[0071] Step 4: Obtain the raw format video corresponding to normal lighting conditions output by the neural network model, that is, restore the night scene video under low light conditions to the raw format video under normal lighting conditions.

[0072] It should be noted that, in one embodiment, the method for generating enhanced images can be further explained by combining a night scene video enhancement network. Figure 3 This is a schematic diagram of a night scene video enhancement network according to an embodiment of this application, such as... Figure 3 As shown, a night scene video enhancement network (SSN_net) was constructed. The video night scene enhancement network includes three sub-networks: an optical flow calculation network (equivalent to the optical flow calculation unit mentioned above, represented by flow_net), a video alignment network (equivalent to the video alignment unit mentioned above, represented by alignment_net), and a video restoration network (equivalent to the video restoration unit mentioned above, represented by video_res_net). The video alignment network also includes a superposition network (represented by wrap_net) to superimpose the aligned 2N frames output by the video alignment network with the Nth frame image. Figure 4 This is a schematic diagram of the optical flow calculation unit according to an embodiment of this application. Figure 4 As shown, the optical flow calculation network is responsible for calculating the optical flow data of consecutive frame images and intermediate frame images (equivalent to the target frame image mentioned above). Then, the video alignment network is responsible for aligning the consecutive frame images with the intermediate frame based on the calculated optical flow data. The video restoration network is responsible for restoring the aligned RAW video image from low-light conditions to near-normal lighting conditions, that is, restoring consecutive RAW images in a night scene to the corresponding images under high-light conditions. The specific implementation process is as follows;

[0073] It should be noted that step 1 above also includes:

[0074] Step 11: Enable the camera module. The camera module writes consecutive RAW format video frame images into the DDR. The video night scene network reads consecutive RAW format video frame images from the DDR and forms a data tensor with dimensions (2N+1)xHxWx1.

[0075] The range of N can be determined based on historical records, and the range of N is [1, 5].

[0076] It should be noted that the aforementioned DDR stands for Double Data Rate Synchronous Dynamic Random Access Memory, which is a type of memory space that can be directly addressed.

[0077] Step 12: Take the Nth image as the target frame image from these 2N+1 images, form 2N image pairs with each of them, and input them into the optical flow computing network. The optical flow computing network outputs the optical flow data corresponding to the 2N image pairs.

[0078] It should be noted that the above optical flow computing network structure consists of multiple convolutional layers, and this application does not limit the size of the convolutional kernel or the number of convolutional layers.

[0079] Step 13: Based on the optical flow data output by the optical flow calculation network, input the optical flow data and the original continuous video frame images to the video alignment network, and align the 1st frame image, the 2nd frame image, ..., the (N-1)th frame image, the N+1th frame image, ..., the 2N+1th frame image with the Nth frame image in sequence, and output the image after 2N frames are aligned with the Nth frame.

[0080] It should be noted that the above video alignment network consists of multiple convolutional layers, and this application does not limit the size of the convolutional kernels or the number of convolutional layers.

[0081] Step 14: Overlay the 2N aligned images output by the video alignment network with the Nth frame image to form a tensor with an original dimension of (2N+1)xHxWx1, and then input it into the video recovery network.

[0082] It should be noted that, in one embodiment, it can be combined with Figure 5 Step 2 above will be explained. Figure 5 This is a schematic diagram illustrating the tensor reconstruction process according to an embodiment of this application. Figure 5 As shown, the video restoration network (equivalent to the deep neural network model in step 2 above) is split according to Bayer units. Through the S2D (Space to depth, which is to transform an image with a depth of 1 in a plane into multiple images with a depth of 4 in a three-dimensional space) technique, the tensor with a dimension of (2N+1)xHxWx1 output by the video alignment network is constructed into a tensor of (2N+1)xH / 2xW / 2x4.

[0083] Figure 6 This is a schematic diagram illustrating the process of generating an enhanced image according to an embodiment of this application, such as... Figure 6As shown, this application provides a target neural network model for processing continuous frame images. Multiple continuous frame images are reconstructed into multiple sample images (equivalent to the second sample images mentioned above) using S2D technology. The multiple sample images are input into video_res_net, and the rec format file output by video_res_net (equivalent to the second tensor data mentioned above) is processed using D2S (Depth to space, the reverse of the S2D processing) technology to generate corresponding enhanced images.

[0084] It should be noted that the video restoration network consists of multiple convolutional layers, and this application does not limit the size of the convolutional kernels or the number of convolutional layers.

[0085] Through the above embodiments, the video enhancement method uses continuous raw format video frames as input, which can effectively avoid the video flickering problem caused by processing single-frame image inputs separately; the introduction of the alignment network can effectively avoid the problem of motion blur caused by moving objects in the video frame after algorithm processing; the video restoration network can remove noise from the image to the greatest extent, improve the brightness and contrast of the image, and restore the color, while preserving the original details of the image as much as possible. Moreover, the entire super night scene video enhancement algorithm is an end-to-end approach, which is simple and convenient and can effectively avoid the error accumulation problem of existing multi-stage processing algorithms.

[0086] Through the above embodiments, by taking continuous RAW format video frames as input and then outputting RAW domain images, an optical flow computing network, a video alignment network, and a video restoration network are introduced, which can effectively avoid the problem of motion blur caused by moving objects in video frames after algorithm processing.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0088] This embodiment also provides an enhanced image generation apparatus, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0089] Figure 7 This is a structural block diagram of an enhanced image generation apparatus according to an embodiment of this application. For example... Figure 7 As shown, the apparatus for generating enhanced images includes:

[0090] The determining module 702 is used to determine the target frame image from the acquired multiple consecutive frame images, and to pair each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs;

[0091] It should be noted that the intermediate frame image among multiple consecutive frame images can be determined as the target frame image. The intermediate frame image can be determined according to the order of the frame images being captured, or it can be determined according to the size order of the frame images. This application does not impose any restrictions on this.

[0092] The first sending module 704 is used to send the multiple sets of image pairs to the optical flow calculation unit and obtain the optical flow data obtained by the optical flow calculation unit based on the multiple sets of image pairs, wherein the optical flow data is used to indicate the pixel parameters of the images in the multiple sets of image pairs;

[0093] It should be noted that the pixel parameters of the above images may include, but are not limited to, the pixel position, the number of pixels, the pixel density, and the pixel spacing of the images.

[0094] The second sending module 706 is used to send the optical flow data and the multiple consecutive frame images to the video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image according to the optical flow data, and outputs the aligned multiple frame images, wherein the multiple aligned frame images are used to indicate the multiple consecutive frame images aligned with the target frame image.

[0095] It should be noted that the process of aligning the multiple consecutive frame images with the target frame image based on the optical flow data can be understood as aligning the pixel positions in the multiple consecutive frame images with the specified pixel positions in the target frame image, but it is not limited to this.

[0096] The third sending module 708 is used to determine first tensor data based on the multiple aligned frame images and the target frame image, and send the first tensor data to the video restoration unit so that the video restoration unit can generate an enhanced image based on the first tensor data.

[0097] Using the aforementioned apparatus, a target frame image is determined from multiple consecutive frame images. Each frame image in the multiple consecutive frame images is paired with the target frame image to obtain multiple image pairs. These multiple image pairs are then sent to an optical flow calculation unit, which obtains optical flow data based on the multiple image pairs. The optical flow data indicates the pixel parameters of the images in the multiple image pairs. The optical flow data and the multiple consecutive frame images are then sent to a video alignment unit, which aligns the multiple consecutive frame images with the target frame image based on the optical flow data and outputs multiple aligned frame images. These aligned frame images indicate the multiple consecutive frame images aligned with the target frame image. This process solves the technical problem of low video quality in night scene videos under low light conditions and improves the video quality of night scene videos.

[0098] In an exemplary embodiment, the above-described enhanced image generation apparatus further includes an acquisition module, configured to acquire a video file that meets preset conditions via a camera device, the preset conditions including at least one of the following: the duration of the video file is less than a first threshold; the size of the video file is less than a second threshold; the type of the video file is a preset type; and to extract the video file frame by frame to obtain the extracted multiple consecutive frame images.

[0099] It should be noted that the above-mentioned camera device can be understood as a device equipped with a camera component. The camera component is a device used to complete image decomposition and photoelectric signal conversion. Camera devices include CCD (Charge Coupled Device) cameras, CMOS (Complementary Metal-Oxide-Semiconductor) cameras, etc., but are not limited to these.

[0100] The aforementioned first threshold can be one hour, one week, or one day; this application does not impose any restrictions on this. For example, if the first threshold is one hour, and the duration of the video file is two hours, then the video file does not meet the preset conditions.

[0101] The second threshold mentioned above can be 1GB, 1TB, or 100MB, and this application does not limit it. For example, if the second threshold is 100MB, and the size of the video file is 50MB, then the video file does not meet the preset conditions.

[0102] The above preset types can be understood as the formats in which computers store videos, including but not limited to avi, wmv, MPEG, rm, rmvb, mov, and asf formats.

[0103] It should be noted that the image information reflected in the enhanced images is consistent with the image information reflected in each consecutive frame of the images. The image information includes: the scene in the image, the objects in the image, and the annotations in the image, etc. This application does not limit this.

[0104] In an exemplary embodiment, the first sending module 704 is further configured to: acquire each of the plurality of image pairs, wherein each image pair includes the target frame image and one frame image from the plurality of consecutive frame images; send each image pair to the optical flow calculation unit; determine the pixel position change parameters of the images in each image pair through the optical flow calculation unit; wherein the pixel position change parameters of the images in each image pair are used to represent the position change of pixels between the target frame image and the one frame image in each image pair; and determine the optical flow data output by the optical flow calculation unit as including the pixel position change parameters of the images in each image pair.

[0105] In an exemplary embodiment, the third sending module 708 is further configured to: adjust the plurality of aligned frame images and the target frame image to the same size as a plurality of first sample images; determine a plurality of first tensor data based on the image data of the plurality of first sample images, wherein each first tensor data is used to represent the image data of the corresponding first sample image in the plurality of first sample images in the form of an array, and each first tensor data includes at least one of the following: the height of the corresponding first sample image, the width of the corresponding first sample image, and the number of channels of the corresponding first sample image.

[0106] It should be noted that the image data of the first sample image can be decoded into image data represented in array form by a decoding function in the computer, namely the first tensor data mentioned above. In one embodiment, each first tensor data can be represented as an array of [number of image channels (i.e., image depth), image width, image height], or as an array of [number of samples, number of image channels (i.e., image depth), image width, image height], to store the image data of each first sample image.

[0107] In an exemplary embodiment, the above-described enhanced image generation apparatus further includes a determining module, configured to determine, through the video restoration unit, a plurality of second tensor data based on a plurality of first tensor data, and determine image data of a plurality of second sample images corresponding to the plurality of second tensor data, wherein each second tensor data is used to represent the image data of the corresponding second sample image in the plurality of second sample images in array form, and the product of the height, width, and number of channels of each second sample image in the second tensor data is equal to the product of the height, width, and number of channels of each first sample image in the first tensor data; determine the image data of the plurality of second sample images based on the plurality of second tensor data; input the image data of the plurality of second sample images into a target neural network model, and generate the enhanced image through the target neural network model, wherein the target neural network model is used to perform enhancement processing on the input image data, obtain a processing result, and generate an image corresponding to the processing result.

[0108] It should be noted that, in one embodiment, the product of the height, width, and number of channels of each second sample image in the second tensor data is equal to the product of the height, width, and number of channels of each first sample image in the first tensor data. For example, if the first tensor data is represented by [1,1,1], that is, the first tensor data represents image data with a height of 1 pixel, a width of 1 pixel, and a depth of 1 pixel, then the second tensor data can be represented by [1 / 2,1 / 2,4], that is, the second tensor data represents image data with a height of 1 / 2 pixel, a width of 1 / 2 pixel, and a depth of 4 pixels.

[0109] It should be noted that the image data enhancement process of the aforementioned target neural network model includes, but is not limited to, image denoising, image brightness enhancement, image contrast enhancement, and image color restoration.

[0110] In an exemplary embodiment, the above-described enhanced image generation apparatus further includes a processing module, configured to perform enhancement processing on the target frame image to obtain a reference image; determine the image similarity between the enhanced image and the reference image; and determine that the enhanced image passes verification if the image similarity is less than a preset value.

[0111] It should be noted that the preset value ranges from [0,1]. When the preset value is 0.1, if the image similarity is 0.05, the enhanced image passes the verification, and the image enhancement method in this embodiment can be applied to real-world work scenarios. If the image similarity is 0.3, the enhanced image fails the verification, and the image similarity can be obtained again by re-determining the consecutive frames or optimizing the target neural network model.

[0112] In an exemplary embodiment, the processing module is further configured to: determine a first weighting coefficient and a second weighting coefficient, wherein the first weighting coefficient corresponds to the pixel difference between the enhanced image and the reference image, and the second weighting parameter corresponds to the structural similarity value between the enhanced image and the reference image, the structural similarity value indicating the degree of similarity of the image structural information of the enhanced image and the reference image; determine the pixel similarity between the enhanced image and the reference image, and determine the structural similarity between the enhanced image and the reference image, wherein the pixel similarity between the enhanced image and the reference image is the product of the first weighting coefficient and the pixel difference, and the structural similarity between the enhanced image and the reference image is the product of the second weighting parameter and the structural similarity value; obtain the sum of the pixel similarity and the structural similarity; and determine the image similarity according to a preset proportional relationship and the sum, wherein the preset proportional relationship indicating the negative correlation between the sum and the image similarity.

[0113] It should be noted that, in one embodiment, the process of determining the image similarity between the enhanced image and the reference image can be illustrated by example. In this embodiment, an objective function L is used to represent image similarity, and the expression of the objective function L is as follows:

[0114] LαL enh +(1α)L ssim (1)

[0115] Where, L enh To enhance the image Compared with reference image I GT The L1 norm of the absolute values ​​of the pixel differences (representing the sum of the absolute values ​​of all pixel differences, equivalent to the pixel similarity between the enhanced image and the reference image mentioned above), i.e.

[0116]

[0117] L ssim This is a structural similarity function (equivalent to the structural similarity value between the enhanced image and the reference image mentioned above).

[0118]

[0119] Where, μ x μ y σ represents the average value. x , σ y Let σ be the variance. xy For covariance, σ xyThis effectively enhances image details. c1 and c2 are regularization constants. α is the first weight coefficient, and (1-α) is the second weight coefficient. The first and second weight coefficients can be set according to specific circumstances. After calculating L, the larger the value of L, the smaller the image similarity.

[0120] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0121] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0122] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0123] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0124] S1, determine the target frame image from the acquired multiple consecutive frame images, and pair each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs.

[0125] S2, the multiple sets of image pairs are sent to the optical flow calculation unit, and the optical flow data obtained by the optical flow calculation unit based on the multiple sets of image pairs is obtained, wherein the optical flow data is used to indicate the pixel parameters of the images in the multiple sets of image pairs.

[0126] S3, the optical flow data and the multiple consecutive frame images are sent to the video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image according to the optical flow data, and outputs the aligned multiple frame images, wherein the multiple aligned frame images are used to indicate the multiple consecutive frame images aligned with the target frame image.

[0127] S4, determine multiple first tensor data based on the multiple aligned frame images and the target frame image, and send the multiple first tensor data to the video restoration unit so that the video restoration unit can generate an enhanced image based on the multiple first tensor data.

[0128] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0129] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0130] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0131] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating enhanced images, characterized in that, include: The target frame image is determined from the acquired multiple consecutive frame images, and each frame image in the multiple consecutive frame images is paired with the target frame image to obtain multiple image pairs; The multiple sets of image pairs are sent to the optical flow calculation unit, and the optical flow data obtained by the optical flow calculation unit based on the multiple sets of image pairs is obtained, wherein the optical flow data is used to indicate the pixel parameters of the images in the multiple sets of image pairs; The optical flow data and the multiple consecutive frame images are sent to the video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image according to the optical flow data, and outputs the aligned multiple frame images, wherein the multiple aligned frame images are used to indicate the multiple consecutive frame images aligned with the target frame image. Multiple first tensor data are determined based on the multiple aligned frame images and the target frame image, and the multiple first tensor data are sent to the video restoration unit so that the video restoration unit can generate an enhanced image based on the multiple first tensor data; Based on the multiple aligned frame images and the target frame image, multiple first tensor data are determined, including: The multiple aligned frame images and the target frame image are adjusted to the same size as multiple first sample images; Multiple first tensor data are determined based on the image data of the multiple first sample images, wherein each first tensor data is used to represent the image data of the corresponding first sample image in the multiple first sample images in the form of an array, and each first tensor data includes: the height of the corresponding first sample image, the width of the corresponding first sample image, and the number of channels of the corresponding first sample image. After sending the plurality of first tensor data to the video restoration unit, the method further includes: The video recovery unit determines multiple second tensor data based on multiple first tensor data, and determines the image data of multiple second sample images corresponding to the multiple second tensor data. Each second tensor data is used to represent the image data of the corresponding second sample image in the multiple second sample images in array form. The product of the height, width, and number of channels of each second sample image in the second tensor data is equal to the product of the height, width, and number of channels of each first sample image in the first tensor data. The image data of the multiple second sample images is determined based on the multiple second tensor data; The image data of the multiple second sample images are input into the target neural network model, and the enhanced image is generated by the target neural network model. The target neural network model is used to enhance the input image data, obtain the processing result, and generate the image corresponding to the processing result.

2. The method for generating enhanced images according to claim 1, characterized in that, Before determining the target frame image from multiple consecutive frame images, and pairing each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs, the process includes: A video file that meets preset conditions is acquired by a camera device. The preset conditions include at least one of the following: the duration of the video file is less than a first threshold; the size of the video file is less than a second threshold; and the type of the video file is a preset type. The video file is extracted frame by frame to obtain the extracted multiple consecutive frame images.

3. The method for generating enhanced images according to claim 1, characterized in that, Sending the multiple sets of image pairs to the optical flow calculation unit and obtaining the optical flow data obtained by the optical flow calculation unit based on the multiple sets of image pairs, including: Obtain each of the multiple sets of image pairs, wherein each set of image pairs includes the target frame image and one frame image from the multiple consecutive frame images; Each pair of images is sent to the optical flow calculation unit. The optical flow calculation unit determines the pixel position change parameters of the images in each image pair; wherein, the pixel position change parameters of the images in each image pair are used to represent the position change of pixels between the target frame image and the frame image in each image pair; The optical flow data output by the optical flow calculation unit is determined to include the pixel position change parameters of the images in each image pair.

4. The method for generating enhanced images according to claim 1, characterized in that, After sending the plurality of first tensor data to the video restoration unit so that the video restoration unit generates an enhanced image based on the first tensor data, the method further includes: The target frame image is enhanced to obtain a reference image; Determine the image similarity between the enhanced image and the reference image; If the image similarity is less than a preset value, the enhanced image is determined to pass the verification.

5. The method for generating enhanced images according to claim 4, characterized in that, Determining the image similarity between the enhanced image and the reference image includes: A first weighting coefficient and a second weighting coefficient are determined, wherein the first weighting coefficient corresponds to the pixel difference between the enhanced image and the reference image, and the second weighting coefficient corresponds to the structural similarity value between the enhanced image and the reference image, the structural similarity value being used to indicate the degree of similarity between the image structural information of the enhanced image and the reference image; The pixel similarity between the enhanced image and the reference image is determined, and the structural similarity between the enhanced image and the reference image is determined, wherein the pixel similarity between the enhanced image and the reference image is the product of the first weight coefficient and the pixel difference, and the structural similarity between the enhanced image and the reference image is the product of the second weight coefficient and the structural similarity value. Obtain the sum of the pixel similarity and the structural similarity; The image similarity is determined based on a preset proportional relationship and the sum value, wherein the preset proportional relationship is used to indicate the negative correlation between the sum value and the image similarity.

6. An apparatus for generating enhanced images, characterized in that, include: The determination module is used to determine the target frame image from the acquired multiple consecutive frame images, and to pair each frame image in the multiple consecutive frame images with the target frame image to obtain multiple image pairs; A first sending module is configured to send the multiple sets of image pairs to an optical flow calculation unit, and to obtain optical flow data obtained by the optical flow calculation unit based on the multiple sets of image pairs, wherein the optical flow data is used to indicate the pixel parameters of the images in the multiple sets of image pairs; The second sending module is used to send the optical flow data and the multiple consecutive frame images to the video alignment unit, so that the video alignment unit aligns the multiple consecutive frame images with the target frame image according to the optical flow data, and outputs the aligned multiple frame images, wherein the multiple aligned frame images are used to indicate the multiple consecutive frame images aligned with the target frame image; The third sending module is used to determine first tensor data based on the multiple aligned frame images and the target frame image, and send the first tensor data to the video restoration unit so that the video restoration unit can generate an enhanced image based on the first tensor data. The device is further configured to adjust the plurality of aligned frame images and the target frame image to the same size as the plurality of first sample images; and to determine a plurality of first tensor data based on the image data of the plurality of first sample images, wherein each first tensor data is used to represent the image data of the corresponding first sample image in the plurality of first sample images in the form of an array, and each first tensor data includes: the height of the corresponding first sample image, the width of the corresponding first sample image, and the number of channels of the corresponding first sample image; The device is further configured to, through the video recovery unit, determine multiple second tensor data based on multiple first tensor data, and determine image data of multiple second sample images corresponding to the multiple second tensor data, wherein each second tensor data is used to represent the image data of the corresponding second sample image in the multiple second sample images in array form, and the product of the height, width, and number of channels of each second sample image in the second tensor data is equal to the product of the height, width, and number of channels of each first sample image in the first tensor data; determine the image data of the multiple second sample images based on the multiple second tensor data; input the image data of the multiple second sample images into a target neural network model, and generate the enhanced image through the target neural network model, wherein the target neural network model is used to perform enhancement processing on the input image data, obtain a processing result, and generate an image corresponding to the processing result.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 5 when it is run.

8. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to perform the method of any one of claims 1 to 5 through the computer program.

Images