Image processing method, device, electronic device and computer readable storage medium
Through multi-frame fusion, single-frame super-score and interpolation processing, combined with the super-score model in the preset zoom magnification set, the problem of limited zoom ratio support in the traditional zoom method is solved, and flexible focal segment processing and image quality improvement is achieved.
Patent Information
- Application Number
- CN202111171386.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-08
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-10-08
AI Technical Summary
The zoom ratio supported by traditional zoom mode is limited, and the processing method is inflexible, making it difficult to meet the zoom needs in different focal lengths.
Through multi-frame fusion, single-frame super-segment method and interpolation processing method, flexibly realize zoom processing in different focal segments, obtain the target zoom magnification, and process it according to the super-segment model in the preset zoom magnification set to ensure image details and picture quality .
It realizes flexible zoom processing for different focal segments, reduces image quality loss, improves image detail and accuracy, and supports more zoom magnification.
Smart Images

Figure CN113935934B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art
[0002] The zoom function of mobile phone photography generally includes three processing methods: optical zoom, hybrid zoom and digital zoom. Different zoom methods can be selected according to different focal lengths. However, the zoom multiples that can be supported by traditional zoom methods are limited and the processing methods are not flexible. Summary of the invention
[0003] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can flexibly implement zoom processing at different focal lengths.
[0004] In a first aspect, the present application provides an image processing method, comprising:
[0005] Acquire a target zoom ratio, and when the target zoom ratio is within a first zoom ratio range, perform zoom processing on each frame of the image to be processed by a multi-frame fusion method to obtain a first target image at the target zoom ratio;
[0006] In the case where the target zoom ratio belongs to a preset zoom ratio set, a first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and zoom processing is performed on each frame of the image to be processed by the first super-resolution model using a single-frame super-resolution method and a multi-frame fusion method to obtain a second target image at the target zoom ratio; the preset zoom ratio is an integer multiple of the upper limit ratio of the first zoom ratio range, and the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception;
[0007] When the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, each frame of the image to be processed is zoomed through the multi-frame fusion method, the single-frame super-resolution method and the interpolation processing method to obtain a third target image at the target zoom ratio.
[0008] In a second aspect, the present application further provides an image processing device, comprising:
[0009] An acquisition module is used to obtain a target zoom ratio;
[0010] A first zoom module, configured to perform zoom processing on each frame of the image to be processed by a multi-frame fusion method when the target zoom ratio is within a preset zoom ratio range, so as to obtain a first target image at the target zoom ratio;
[0011] A second zoom module is used to determine, when the target zoom ratio belongs to a preset zoom ratio set, a first super-resolution model corresponding to the target zoom ratio from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and to perform zoom processing on each frame of the image to be processed by using the first super-resolution model in a single-frame super-resolution manner and in a multi-frame fusion manner to obtain a second target image at the target zoom ratio; the preset zoom ratio is an integer multiple of an upper limit ratio of the first zoom ratio range, and the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception;
[0012] The third zoom module is used to perform zoom processing on each frame of the image to be processed by the multi-frame fusion method, the single-frame super-resolution method and the interpolation processing method when the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, so as to obtain a third target image at the target zoom ratio.
[0013] In a third aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the image processing method provided in the first aspect.
[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method provided in the first aspect.
[0015] The above-mentioned image processing method, device, electronic device, and computer-readable storage medium can quickly generate a first target image at the target zoom ratio by zooming each frame of the image to be processed in a multi-frame fusion manner when the target zoom ratio is within the first zoom ratio range. When the target zoom ratio belongs to a preset zoom ratio set, the first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and the details of the image are increased and the loss of image quality is reduced by zooming each frame of the image to be processed in a single-frame super-resolution manner and a multi-frame fusion manner through the first super-resolution model, thereby obtaining a second target image at the target zoom ratio, and the preset zoom ratio is an integer multiple of the upper limit ratio of the first zoom ratio range. In addition, the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception, so that each super-resolution model can process the image to be processed based on the parameters under the same standard, thereby obtaining a more accurate second target image. When the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, zoom processing is performed on each frame of the image to be processed through a multi-frame fusion method, a single-frame super-resolution method and an interpolation processing method to obtain a third target image at the target zoom ratio. According to the preset zoom ratio to which the target zoom ratio belongs, the multi-frame fusion method, the single-frame super-resolution method, the interpolation processing method and a combination thereof can be flexibly selected to perform zoom processing at different focal lengths, and the loss of image quality at each focal length can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0017] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;
[0018] Figure 2 is a flowchart of an image processing method in one embodiment;
[0019] Figure 3 A schematic diagram of a process of performing quantization perception on a super-resolution model corresponding to a single preset zoom factor in one embodiment;
[0020] Figure 4 A schematic diagram of obtaining target loss values corresponding to all super-resolution models in one embodiment;
[0021] Figure 5is a schematic diagram of obtaining a second target image in one embodiment;
[0022] Figure 6 A schematic diagram of interpolating each second super-resolution model to obtain a third super-resolution model corresponding to a target zoom ratio in an embodiment;
[0023] Figure 7 A schematic diagram of a process of performing zoom processing on each frame of an image to be processed by a multi-frame fusion method in an embodiment;
[0024] Figure 8 A schematic diagram of a process of calculating a homography matrix in one embodiment;
[0025] Fig. 9 is a flowchart of an image processing method in another embodiment;
[0026] Fig.10 is a structural block diagram of an image processing device in one embodiment;
[0027] Fig.11 FIG. 1 is a schematic diagram of the internal structure of an electronic device in an embodiment. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0029] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first target image may be referred to as a second target image, and similarly, a second target image may be referred to as a first target image. Both the first target image and the second target image are target images, but they are not the same target image.
[0030] Figure 1 FIG. 1 is a schematic diagram of an application environment of an image processing method in an embodiment. Figure 1As shown, the application environment includes an electronic device 110 and a server 120. The electronic device 110 obtains a target zoom ratio and each frame of the image to be processed, and sends the target ratio and each frame of the image to be processed to the server 120. When the target zoom ratio is within a first zoom ratio range, the server 120 performs zoom processing on each frame of the image to be processed by a multi-frame fusion method to obtain a first target image at the target zoom ratio. The server 120 returns the first target image to the terminal 110. When the target zoom ratio belongs to a second preset zoom ratio, the server 120 performs zoom processing on each frame of the image to be processed by a multi-frame fusion method and a single-frame super-resolution method to obtain a second target image at the target zoom ratio; the second preset zoom ratio is an integer multiple of the upper limit value of the first zoom ratio range. The server 120 returns the second target image to the terminal 110. When the target zoom ratio belongs to the third preset zoom ratio, the server 120 performs zoom processing on each frame of the to-be-processed image by means of a multi-frame fusion method, a single-frame super-resolution method, and an interpolation processing method to obtain a third target image at the target zoom ratio; the third preset zoom ratio is between two different second preset zoom ratios. The server 120 returns the third target image to the terminal 110.
[0031] The terminal 110 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 120 may be implemented as an independent server or a server cluster consisting of multiple servers.
[0032] Figure 2 FIG. 1 is a flowchart of an image processing method in an embodiment. The image processing method in this embodiment is executed on Figure 1 This article describes the terminal or server in the following example. Figure 2 As shown, the image processing method includes steps 202 to 206.
[0033] Step 202, obtaining a target zoom ratio.
[0034] The target zoom ratio refers to the zoom ratio corresponding to the target image to be obtained. The target zoom ratio can be the zoom ratio set by the user before or during shooting with the camera, or it can be the zoom ratio set after the image is acquired, so as to obtain the target image at the target zoom ratio after zooming the image.
[0035] Step 204 , when the target zoom ratio is within the first zoom ratio range, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method to obtain a first target image at the target zoom ratio.
[0036] The image to be processed may be any one of RGB (Red, Green, Blue) images, RAW images, grayscale images, depth images, images corresponding to the Y component in YUV images, etc. RAW images are raw data that the image sensor converts the captured light source signal into a digital signal. The "Y" in the YUV image represents brightness (Luminance or Luma), which is the grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel.
[0037] The first zoom ratio range is a preset zoom ratio range, for example, the first zoom ratio range is 1x-1.6x, that is, the focal length of the camera is 1x-1.6x. Motion-stacking refers to a processing method of fusing at least two frames of images.
[0038] Specifically, the electronic device may obtain the image to be processed from a local device or other device or network, or the electronic device may obtain the image to be processed by photographing a scene with a camera. The image to be processed may also be a video frame in a video obtained from a local device or other device or network, or a video frame in a video photographed with a camera.
[0039] The electronic device may obtain a target zoom ratio, and compare the target zoom ratio with the first zoom ratio range to determine whether the target zoom ratio is within the first zoom ratio range. When the target zoom ratio is within the first zoom ratio range, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method to obtain a first target image at the target zoom ratio. Further, the electronic device enlarges each frame of the image to be processed to the target zoom ratio, and performs fusion processing on each frame of the image enlarged to the target zoom ratio to obtain the first target image.
[0040] Step 206, when the target zoom ratio belongs to a preset zoom ratio set, a first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and each frame of the image to be processed is zoomed using the first super-resolution model by adopting a single-frame super-resolution method and a multi-frame fusion method to obtain a second target image at the target zoom ratio; the preset zoom ratio is an integer multiple of the upper limit ratio of the first zoom ratio range, and the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception.
[0041] Among them, the preset zoom ratio is a zoom ratio that is an integer multiple of the upper limit ratio of the first zoom ratio range. The preset zoom ratio set is a set composed of various preset zoom ratios. For example, if the first zoom ratio range is 1x-1.6x, and the upper limit ratio is 1.6x, then the second preset zoom ratio set may include 1.6x, 3.2x, 4.8x and 6.4x, etc., and the preset zoom ratio is 1.6x, 3.2x, 4.8x or 6.4x. The single-frame super-resolution method refers to a processing method for super-resolution reconstruction of a single-frame image. The first super-resolution model refers to the super-resolution model corresponding to the target zoom ratio.
[0042] Quantization perception means training the model during the quantization process so that the model parameters can better adapt to the information loss caused by quantization. After joint quantization perception, the super-resolution models corresponding to each preset zoom ratio can obtain each super-resolution model with consistent standard parameters. Each super-resolution model can process the image to be processed with consistent standard parameters, thereby obtaining the second target image more accurately.
[0043] The electronic device can obtain a target zoom ratio, and compare the target zoom ratio with each preset zoom ratio in a preset zoom ratio set to determine whether the target zoom ratio belongs to the preset zoom ratio set. In the case where the target zoom ratio belongs to the preset zoom ratio set, a first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and zoom processing is performed on each frame of the image to be processed by using the first super-resolution model using a single-frame super-resolution method and a multi-frame fusion method. The processing result of the multi-frame fusion method is super-reconstructed by the single-frame super-resolution method to obtain a second target image at the target zoom ratio.
[0044] Step 208, when the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method, a single-frame super-resolution method and an interpolation processing method to obtain a third target image at the target zoom ratio.
[0045] The second zoom ratio range refers to a zoom ratio range between two adjacent preset zoom ratios. For example, if the two adjacent preset zoom ratios are 1.6x and 3.2x, the second zoom ratio range is (1.6x-3.2x).
[0046] The interpolation processing method refers to performing interpolation processing on a single frame image to obtain an image with higher resolution. For example, the interpolation processing method may be Bicubic (bicubic interpolation), Bilinear (bilinear interpolation) or Lanczos, but is not limited thereto.
[0047] Specifically, the electronic device can obtain the target zoom ratio, compare the target zoom ratio with the second zoom ratio range, and determine whether the target zoom ratio is within the second zoom ratio range. When the target zoom ratio is within the second zoom ratio range, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method. A new super-resolution model corresponding to the target zoom ratio is interpolated by an interpolation processing method. The processing result of the multi-frame fusion method is super-reconstructed by a single-frame super-resolution method through the new super-resolution model to obtain a third target image at the target zoom ratio.
[0048] In this embodiment, when the target zoom ratio is within the first zoom ratio range, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method, so that the first target image at the target zoom ratio can be quickly generated. When the target zoom ratio belongs to a preset zoom ratio set, the first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and the details of the image are increased and the loss of image quality is reduced by zooming each frame of the image to be processed by the first super-resolution model using a single-frame super-resolution method and a multi-frame fusion method, so as to obtain a second target image at the target zoom ratio, and the preset zoom ratio is an integer multiple of the upper limit ratio of the first zoom ratio range. In addition, the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception, so that each super-resolution model can process the image to be processed based on the parameters under the same standard, so as to obtain a more accurate second target image. When the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, zoom processing is performed on each frame of the image to be processed through a multi-frame fusion method, a single-frame super-resolution method and an interpolation processing method to obtain a third target image at the target zoom ratio. According to the preset zoom ratio to which the target zoom ratio belongs, the multi-frame fusion method, the single-frame super-resolution method, the interpolation processing method and a combination thereof can be flexibly selected to perform zoom processing at different focal lengths, and the loss of image quality at each focal length can be effectively reduced.
[0049] In one embodiment, a joint quantization perception method for super-resolution models corresponding to each preset zoom ratio includes: performing quantization perception on the super-resolution model corresponding to each preset zoom ratio respectively to obtain a target loss value corresponding to all super-resolution models; training the super-resolution models corresponding to each preset zoom ratio according to the target loss value until a training cutoff condition is met to obtain each trained super-resolution model.
[0050] The target loss value refers to the total loss value obtained by quantizing all super-resolution models. The training cutoff condition can be set as needed. For example, the training cutoff condition can be that the training time reaches the preset time, the number of training times reaches the preset number, or the target loss value is less than the preset value.
[0051] Figure 3 The figure is a flowchart of quantization perception of a super-resolution model corresponding to a single preset zoom ratio in an embodiment. The electronic device obtains input data of data type float, and uses the pseudo-quantization node Quantize to count the maximum and minimum values flowing into the current node, and limits the quantization range according to the maximum and minimum values. It is understandable that if each super-resolution model performs a quantization perception process separately, the quantization range of the same Conv (node) of different super-resolution models will obtain different results according to the training, so there is no way to do parameter interpolation, because the actual physical meaning represented by the quantized value is different. For example, if a conv (node) of Model-2x and Model-3x is used for interpolation, assuming that the input value before quantization is 50, the quantization range in Model-2x and Model-3x is [0,100], [0,200], and the quantization precision is 8 bits, then the input after quantization should be (50-0) / (100-0)*255=128 and (50-0) / (200-0)*255=64, respectively. Therefore, although the actual input before quantization is the same, the different input ranges will cause the input values of the quantization model to differ greatly. Therefore, in the part of quantization perception, it is necessary to ensure that the input ranges of different models at the same node are as consistent as possible. Therefore, it is necessary to perform joint quantization perception of the super-resolution models corresponding to each preset zoom ratio to ensure that the range of parameters of each super-resolution model is consistent. Among them, the input value can be the pixel value at a certain position of the image.
[0052] In this embodiment, the electronic device performs quantized perception on the super-resolution model corresponding to each preset zoom ratio respectively, and obtains the target loss value corresponding to all the super-resolution models. Then, according to the target loss value, the super-resolution models corresponding to the preset zoom ratios can be jointly trained until the training cutoff condition is met. In this way, the training is completed and the super-resolution models with consistent standard parameters can be obtained, thereby improving the accuracy of the training of each super-resolution model.
[0053] In one embodiment, the super-resolution models corresponding to each preset zoom ratio are quantized and perceived respectively to obtain target loss values corresponding to all super-resolution models, including: in the process of quantizing the super-resolution models corresponding to each preset zoom ratio, the maximum value and the minimum value flowing through the pseudo-quantization node are counted through the set pseudo-quantization node; the variance of the maximum value is determined based on each maximum value, and the variance of the minimum value is determined based on each minimum value; based on the variance of the maximum value and the variance of the minimum value, the target loss values corresponding to all super-resolution models are determined.
[0054] The pseudo-quantization node is used to count the maximum and minimum values of the data flowing through the pseudo-quantization node during training, which is convenient for quantization and reducing precision loss when using TOCO to convert tflite format. It participates in the forward reasoning process of model training to enable the model to obtain quantization loss.
[0055] The electronic device obtains the maximum value of each super-resolution model passing through the pseudo quantization node during the quantization perception process and determines the variance of the maximum value. Similarly, the electronic device obtains the minimum value of each super-resolution model passing through the pseudo quantization node during the quantization perception process and determines the variance of the minimum value.
[0056] In one embodiment, the electronic device may add the variance of the maximum value to the variance of the minimum value to obtain the target loss value corresponding to all super-resolution models. In another embodiment, the electronic device may multiply the variance of the maximum value by the variance of the minimum value to obtain the target loss value corresponding to all super-resolution models. In another embodiment, the electronic device may multiply the variance of the maximum value and the variance of the minimum value by a preset weight factor respectively, and then add the two products to obtain the target loss value corresponding to all super-resolution models. The specific calculation method for determining the target loss value can be set according to user needs and is not limited here.
[0057] In one embodiment, the electronic device uses the following formula (1) to calculate the variance of the maximum value:
[0058] L MAX =∑(Max i -μ MAX ) 2 / N (1)
[0059] Among them, L MAX is the variance of the maximum value, μ MAX is the mean of the maximum values, Max i is the i-th maximum value, and N is the number of maximum values.
[0060] The electronic device uses the following formula (2) to calculate the variance of the minimum value:
[0061] L MIN =∑(Min i -μ MIN ) 2 / N (2)
[0062] Among them, L MIN is the variance of the minimum value, μ MIN is the mean of the minimum values, Min i is the i-th minimum value, and N is the number of minimum values.
[0063] The electronic device uses the following formula (3) to calculate the target loss value corresponding to all super-resolution models:
[0064] L=L MIN +L MAX (3)
[0065] Among them, L MIN is the variance of the minimum value, L MAX is the variance of the maximum value. MIN , L MAX It refers to the variance of the minimum and maximum values of the same OP (Operation) position in each super-resolution model.
[0066] In this embodiment, in the process of quantizing the super-resolution model corresponding to each preset zoom ratio, the maximum and minimum values flowing through the pseudo-quantization node are counted through the set pseudo-quantization node; the variance of the maximum value is determined based on each maximum value, and the variance of the minimum value is determined based on each minimum value; based on the variance of the maximum value and the variance of the minimum value, the target loss values corresponding to all super-resolution models can be accurately determined.
[0067] Figure 4 The schematic diagram of obtaining the target loss values corresponding to all super-resolution models in an embodiment. Taking the joint quantization perception of super-resolution model-1x, super-resolution model-2x and super-resolution model-3x as an example, in the process of the electronic device respectively performing quantization perception on super-resolution model-1x, super-resolution model-2x and super-resolution model-3x, the maximum value and minimum value flowing through the pseudo-quantization node are counted through the set pseudo-quantization node; then the loss value corresponding to the minimum value is calculated based on each minimum value, the loss value corresponding to the maximum value is calculated based on each maximum value, and then the target loss value is determined based on the loss value corresponding to the minimum value and the loss value corresponding to the maximum value.
[0068] In one embodiment, each frame of the image to be processed is zoomed by a first super-resolution model using a single-frame super-resolution method and a multi-frame fusion method to obtain a second target image at a target zoom ratio, including: zooming each frame of the image to be processed by a multi-frame fusion method to obtain a first magnification image; the first magnification image is an image corresponding to an upper limit magnification of a first zoom ratio range; zooming the first magnification image by a single-frame super-resolution model to obtain a second target image at the target zoom ratio.
[0069] Specifically, the electronic device determines a reference frame image from each frame of the image to be processed according to the image sharpness corresponding to each frame of the image to be processed, zooms in the reference frame image to the upper limit magnification of the first zoom magnification range, and obtains the feature points of the reference frame image at the upper limit magnification and the feature points of each frame of the image to be processed. According to the feature points of the reference frame image at the upper limit magnification and the feature points of each frame of the image to be processed, the matching point pairs between the reference frame image at the upper limit magnification and each frame of the image to be processed are determined respectively. According to the matching point pairs, the homography matrix between the reference frame image at the upper limit magnification and each frame of the image to be processed is determined respectively. Based on the homography matrix, each frame of the image to be processed is registered, and each registered image at the upper limit magnification is obtained respectively. Each registered image is fused with the reference frame image at the upper limit magnification to obtain a first magnification image at the upper limit magnification.
[0070] The electronic device uses a first super-resolution model to perform zoom processing on the first magnification image in a single-frame super-resolution manner to obtain a second target image at a target zoom magnification. Further, the first super-resolution model may be an AI super-resolution model.
[0071] In this embodiment, when the target zoom ratio belongs to the preset zoom ratio set, each frame of the image to be processed is first zoomed by a multi-frame fusion method, so that the first magnification image can be quickly obtained. The first super-resolution model uses a single-frame super-resolution method to perform zoom processing on the first magnification image, so that when the multi-frame fusion method cannot support the target zoom ratio, the single-frame super-resolution method can be used to perform super-resolution reconstruction on the processing result of the multi-frame fusion method, so as to support more zoom ratios. In addition, each preset zoom ratio in the preset zoom ratio set is an integer multiple of the upper limit ratio of the first zoom ratio range, which can ensure the continuity of zooming and the maintenance of high-magnification zoom image quality.
[0072] In one embodiment, a first-magnification image is zoomed using a single-frame super-resolution method through a first super-resolution model to obtain a second target image at a target zoom ratio, including: enlarging the first-magnification image to the target zoom ratio through the first super-resolution model to obtain a target zoom ratio image; performing feature extraction on the target ratio image to obtain a low-level feature map; performing feature conversion on the low-level feature map to obtain a high-level feature map; and fusing each high-level feature map to obtain the second target image at the target zoom ratio.
[0073] The target magnification image is an image obtained by magnifying the first magnification image to the target zoom magnification. The low-level feature map is the feature extracted from the front position in the first super-resolution model. The low-level feature map may include information such as color, image frequency, and gradient. The high-level feature map is the feature extracted from the back position in the first super-resolution model.
[0074] In one embodiment, the first super-resolution model may include a feature extraction layer, a feature conversion layer, an amplification layer, and a fusion layer. The amplification layer amplifies the first magnification image to a target size, which is the size at the target zoom factor, that is, the first magnification image is amplified to the target zoom factor to obtain a target magnification image. The feature extraction layer extracts low-level features from the target magnification image to obtain a low-level feature map. The feature conversion layer obtains a high-level feature map by nonlinearly combining the low-level feature map. The fusion layer fuses the high-level feature maps into a super-resolution image and outputs it, so as to obtain a second target image at the target zoom factor.
[0075] In this embodiment, the first super-resolution model is used to enlarge the first magnification image to the target zoom magnification to obtain the target magnification image, and feature extraction is performed on the target magnification image to obtain the key features of the target magnification image, that is, the low-level feature map; the low-level feature map is feature transformed to obtain the high-level feature map; and the high-level feature maps are fused to reconstruct the second target image at the target zoom magnification. In addition, the second target image incorporates more detailed features and has better image quality.
[0076] Figure 5 The figure is a schematic diagram of obtaining a second target image in an embodiment. The electronic device inputs the first magnification image into the first super-resolution model, and the first super-resolution model includes a feature extraction layer, a feature conversion layer, an amplification layer, and a fusion layer. The first magnification image is magnified to the target zoom magnification by the amplification layer to obtain the target magnification image; the feature extraction layer performs feature extraction on the target magnification image to obtain a low-level feature map; the low-level feature map is feature converted by M feature conversion layers to obtain a high-level feature map; the high-level feature maps are fused by the fusion layer to obtain the second target image at the target zoom magnification.
[0077] The entire super-resolution model consists of four parts: Feature extraction layer, Feature conversion layer, amplification layer and fusion layer.
[0078] The amplification layer in this embodiment is not a convolutional layer, but a Bicubic amplification, which amplifies the first magnification image to the required ratio. For example, if the zoom ratio of the first magnification image is 1.6x and the target zoom ratio is 2.4x, the first magnification image is further amplified by 1.5x, that is, 1.5×1.6=2.4, and the first magnification image can be amplified to an image with a target magnification of 2.4x. This amplification layer is also called an Upsample Layer. The parameters of the amplification layer can refer to Table 1.
[0079] Table 1 Network structure parameters of the amplification layer
[0080]
[0081] The feature extraction layer uses a larger convolution kernel with a larger receptive field of view to extract low-level features of the image, such as gradient, brightness, size relationship, etc. It usually includes two layers, which can be called Feature Extraction Layer A and B. In the feature extraction layer, a relatively large convolution kernel is used, and the depth and size of the output feature map (Feature Map) remain unchanged, and the number of channels increases. The parameters of the feature extraction layer can be referred to in Table 2.
[0082] Table 2 Network structure parameters of the Feature extraction layer
[0083]
[0084] The feature transformation layer is the process of obtaining high-level features (such as structure, shape, etc.) by nonlinearly combining low-level features extracted by the extraction layer. It is called the Feature Transform Layer. The more layers there are, the higher the nonlinearity of the feature, the more complex the image structure can be expressed, and the more conducive to the authenticity of image reconstruction. In this part, a convolution with a small convolution kernel is usually selected, with the same depth, size (×1), and number of channels, and repeated M times. Here we select M=16. The parameters of the feature transformation layer can be referred to Table 3.
[0085] Table 3. Feature conversion layer network structure parameters
[0086]
[0087] The purpose of the fusion layer is to fuse features with 64 channels into an SR (Super-Resolution) image output, which is called the Fusion Layer. The parameters of the fusion layer can be found in Table 4.
[0088]
[0089] In one embodiment, when a target zoom ratio is within a second zoom ratio range between two adjacent preset zoom ratios, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method, a single-frame super-resolution method and an interpolation processing method to obtain a third target image at the target zoom ratio, including: when the target zoom ratio is within a second zoom ratio range between two adjacent preset zoom ratios, second super-resolution models corresponding to two adjacent preset zoom ratios are obtained; the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception; an interpolation processing method is used to interpolate each second super-resolution model into a third super-resolution model corresponding to the target zoom ratio; zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method and a single-frame super-resolution method is used through the third super-resolution model to obtain a third target image at the target zoom ratio.
[0090] The second super-resolution model refers to the super-resolution model corresponding to two adjacent preset zoom magnifications when the target zoom magnification is within the second zoom magnification range. For example, if the target zoom magnification is 2x, and the two adjacent preset zoom magnifications are 1.6x and 3.2x, then the second super-resolution model is the super-resolution model corresponding to 1.6x and the super-resolution model corresponding to 3.2x. The third super-resolution model is the super-resolution model corresponding to the target zoom magnification when the target zoom magnification is within the second zoom magnification range.
[0091] Specifically, the electronic device obtains the parameters of each second super-resolution model, linearly interpolates new parameters based on the parameters of each second super-resolution model, and online infers the third super-resolution model based on the new parameters. The electronic device can interpolate the parameters of the third super-resolution model using the following formula (4):
[0092] θ interp =[(4.8-z)θ 3.2x +(z-3.2)θ 4.8x ] / (4.8-3.2)
[0093] Among them, θ interp is the parameter of the third super-resolution model, z is the target zoom ratio, θ 3.2x is the parameter of the second super-resolution model 3.2x, θ 4.8x are the parameters of the second super-resolution model 4.8x. The parameters of the super-resolution model are the values of the convolution kernels. For example, the 3x3 convolution kernels in a super-resolution model are [1,1,1; 1,8,1;,1,1,1]. These numbers are the parameters of the super-resolution model.
[0094] In this embodiment, when the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, second super-resolution models corresponding to the two adjacent preset zoom ratios are obtained, and the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception. Then, the second super-resolution models corresponding to the two adjacent preset zoom ratios of the target zoom ratio can be interpolated into a third super-resolution model with a consistent parameter range, thereby achieving accurate continuous zooming and ensuring a smooth transition between different focal lengths.
[0095] Figure 6 Schematic diagram of interpolating the second super-resolution models to obtain the third super-resolution model corresponding to the target zoom ratio in one embodiment. The target zoom ratio obtained by the electronic device is 4x, and the second super-resolution models obtained are 3.2x super-resolution models and 4.8x super-resolution models, respectively. The parameter of the 3.2x super-resolution model is θ 3.2x , get the parameters of the 4.8x super-resolution model as θ 4.8x , the parameters θ of the 3.2x super-resolution model 3.2x Multiply by (4.8-4) / (4.8-3.2) to get the parameter θ of the 4.8x super-resolution model. 4.8x Multiply by (4-3.2) / (4.8-3.2), and then add the two products to get the parameters of the third super-resolution model, and a 4x super-resolution model can be generated.
[0096] In one embodiment, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method and a single-frame super-resolution method is adopted through a third super-resolution model to obtain a third target image at a target zoom ratio, including: zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method to obtain a first-magnification image; the first-magnification image is an image corresponding to an upper limit magnification of a first zoom ratio range; zoom processing is performed on the first-magnification image by a single-frame super-resolution method through the third super-resolution model to obtain a third target image at the target zoom ratio.
[0097] Specifically, the electronic device determines a reference frame image from each frame of the image to be processed according to the image sharpness corresponding to each frame of the image to be processed, zooms in the reference frame image to the upper limit magnification of the first zoom magnification range, and obtains the feature points of the reference frame image at the upper limit magnification and the feature points of each frame of the image to be processed. According to the feature points of the reference frame image at the upper limit magnification and the feature points of each frame of the image to be processed, the matching point pairs between the reference frame image at the upper limit magnification and each frame of the image to be processed are determined respectively. According to the matching point pairs, the homography matrix between the reference frame image at the upper limit magnification and each frame of the image to be processed is determined respectively. Based on the homography matrix, each frame of the image to be processed is registered, and each registered image at the upper limit magnification is obtained respectively. Each registered image is fused with the reference frame image at the upper limit magnification to obtain a first magnification image at the upper limit magnification.
[0098] The electronic device uses a third super-resolution model to perform zoom processing on the first magnification image in a single-frame super-resolution manner to obtain a third target image at a target zoom magnification. Further, the third super-resolution model may be an AI super-resolution model.
[0099] In this embodiment, when the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, each frame of the image to be processed is first zoomed by a multi-frame fusion method, so that the first magnification image can be quickly obtained. The first magnification image is zoomed by a single-frame super-resolution method through the third super-resolution model, so that when the multi-frame fusion method cannot support the target zoom ratio, the single-frame super-resolution method can be used to perform super-resolution reconstruction on the processing result of the multi-frame fusion method, so that more zoom ratios can be supported.
[0100] In one embodiment, when a target zoom ratio is within a first zoom ratio range, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method to obtain a first target image at the target zoom ratio, including: when the target zoom ratio is within the first zoom ratio range, a reference frame image is determined from each frame of the image to be processed according to the image sharpness corresponding to each frame of the image to be processed; the reference frame image is enlarged to the target zoom ratio, and each frame of the image to be processed is registered based on the reference frame image at the target zoom ratio to obtain each registered image; each registered image is fused with the reference frame image at the target zoom ratio to obtain the first target image at the target zoom ratio.
[0101] Among them, image sharpness, also known as clarity, is an indicator that reflects the clarity of the image plane and the sharpness of the image edges.
[0102] Specifically, the electronic device obtains a target zoom ratio, compares the target zoom ratio with the first zoom ratio range, and when the target zoom ratio is within the first zoom ratio range, determines the image sharpness corresponding to each frame of the image to be processed, and determines a reference frame image from each frame of the image to be processed according to the image sharpness.
[0103] In one embodiment, before determining the reference frame image from each frame of the image to be processed according to the image sharpness corresponding to each frame of the image to be processed, the method further includes: determining the local brightness of each frame of the image to be processed, and determining the image sharpness of the image to be processed according to the local brightness. Furthermore, the electronic device may divide the image to be processed into multiple windows, determine the maximum brightness value and the minimum brightness value in each window, and determine the local brightness of the image to be processed based on the maximum brightness value and the minimum brightness value in each window.
[0104] For example, the image sharpness of the image to be processed can be calculated by the following formula:
[0105]
[0106] The entire image is divided into k1*k2 windows, where I max,k,l and I min,k,l They represent the maximum and minimum brightness values in the k*lth window respectively.
[0107] After determining the image sharpness of each frame of the image to be processed, the image to be processed with the largest image sharpness is selected as the reference frame image. The electronic device magnifies the reference frame image to the target zoom factor to obtain the reference frame image at the target zoom factor. For example, a 1.2x reference frame image is magnified to 1.6x to obtain a 1.6x reference frame image.
[0108] For each frame of the image to be processed, the image to be processed is registered based on the reference frame image at the target zoom factor to obtain the registered image corresponding to each frame of the image to be processed. Each registered image is fused with the reference frame image at the target zoom factor to obtain the first target image at the target zoom factor.
[0109] Furthermore, the electronic device determines the feature points that match between each registered image and the reference frame image at the target zoom ratio, calculates the pixel mean of the matched feature points, and uses the pixel mean as the pixel value of the corresponding feature point in the first target image at the target zoom ratio, until the pixel values of each feature point in the first target image at the target zoom ratio are obtained, thereby obtaining the first target image.
[0110] In this embodiment, each frame of the image to be processed is aligned with the reference frame image at the target zoom factor, so that each frame of the image to be processed and the reference frame image at the target zoom factor are in the same image space, to ensure that all parameters other than the resolution of each frame of the image to be processed and the reference frame image at the target zoom factor are consistent, so that after the fusion of the aligned image and the reference frame image, sub-pixel high-resolution information can be obtained, and the generated first target image has clearer image quality.
[0111] In one embodiment, the registration processing of the image to be processed may include the correspondence of image content, alignment of color space, etc. The alignment of image content may be performed by detecting and matching image feature points and calculating homography matrix, or by matching image blocks. In order to solve the problem of brightness and color space mismatch between the image to be processed and the reference frame image at the target zoom factor, some photos of standard color cards may be introduced during the shooting process, and the difference between the image to be processed and the reference frame image at the target zoom factor may be determined by the standard color cards. The alignment model of color space and brightness may be modeled and analyzed, and operations such as histogram matching may be performed to achieve brightness and color space alignment.
[0112] like Figure 7 FIG. 1 is a flow chart of zooming each frame of an image to be processed by a multi-frame fusion method in one embodiment.
[0113] Acquire multiple frames of YUV images, for example, 8-10 frames of images. Determine a reference frame image by estimating the sharpness of each frame of YUV image. The reference frame is enlarged to the target zoom factor, and other frames of YUV image are registered with the enlarged reference frame image to obtain each registered image. Perform multi-frame fusion and average of each registered image and the enlarged reference frame image to obtain a multi-frame fusion result, that is, obtain the first target image at the target zoom factor.
[0114] In one embodiment, each frame of the image to be processed is registered based on the reference frame image at the target zoom ratio to obtain each registered image, including: acquiring feature points of the reference frame image at the target zoom ratio and feature points of each frame of the image to be processed; determining matching point pairs between the reference frame image at the target zoom ratio and each frame of the image to be processed according to the feature points of the reference frame image at the target zoom ratio and the feature points of each frame of the image to be processed; determining the homography matrix between the reference frame image at the target zoom ratio and each frame of the image to be processed according to the matching point pairs; and performing registration processing on each frame of the image to be processed based on the homography matrix to obtain each registered image of the target zoom ratio.
[0115] The feature point refers to a point where the grayscale value of the image changes dramatically or a point with a large curvature on the edge of the image (i.e., the intersection of two edges). Feature points include, but are not limited to, eyes, nose tips, corners of the mouth, moles, and the center of an object.
[0116] The electronic device detects the grayscale value of each pixel in the reference frame image at the target zoom ratio, and when the difference between the grayscale values of adjacent pixels is greater than a threshold, the area where the adjacent pixels are located can be used as a feature point. The grayscale value of each pixel in each frame of the image to be processed is detected, and when the difference between the grayscale values of adjacent pixels is greater than a threshold, the area where the adjacent pixels are located can be used as a feature point.
[0117] In one embodiment, the electronic device can extract corresponding feature points from each frame of the image to be processed based on the feature points in the reference frame image at the target zoom ratio. In another embodiment, the grayscale value of each pixel in the image to be processed can also be detected. When the difference in the grayscale values of adjacent pixels is greater than a threshold, the area where the adjacent pixels are located can be used as a feature point, and the corresponding feature points can be extracted from the remaining images to be processed and the reference frame image at the target zoom ratio.
[0118] The electronic device forms matching point pairs with feature points extracted from the reference frame image at the target zoom ratio and corresponding feature points of the image to be processed.
[0119] In one embodiment, the reference frame image at the target zoom factor can be divided into multiple regions, for example, a nine-square grid is added to the reference frame image at the target zoom factor to divide it into nine regions, and each point of the nine-square grid is taken as a feature point. The feature points of the image to be processed are determined according to the feature points in the reference frame image at the target zoom factor to form matching point pairs.
[0120] Homography is a concept in projective geometry, also known as projective transformation. It maps points (three-dimensional homogeneous vectors) on a projective plane to another projective plane, and maps straight lines to straight lines, with the property of line preservation. The homography matrix is a mapping between points. The homography matrix can be used to find the exact position of the corresponding point of the image point on another image. The homography matrix can be used to characterize the mapping relationship between the reference frame image and the image to be processed at the target zoom factor.
[0121] The electronic device can calculate a homography matrix between the reference frame image at the target zoom ratio and the image to be processed based on the matching point pairs determined between the reference frame image at the target zoom ratio and the image to be processed. The image to be processed can be mapped to the same image space as the reference frame image at the target zoom ratio through the homography matrix.
[0122] The electronic device will perform registration processing on the image to be processed according to the homography matrix between the reference frame image at the target zoom factor and the image to be processed, thereby obtaining a corresponding registered image. Further, the image to be processed is mapped to the same image space as the reference frame image at the target zoom factor according to the homography matrix, and the offset between the image to be processed and the reference frame image at the target zoom factor is determined according to the matching point pairs. Then, the electronic device moves each feature point in the image to be processed by the offset to obtain a moved image, which is the registered image.
[0123] According to the same processing method, the homography matrix corresponding to each frame of the image to be processed and the reference frame image at the target zoom factor can be calculated, and the corresponding image to be processed is registered according to the homography matrix to obtain the corresponding registered image.
[0124] In this embodiment, the reference frame image is enlarged to the target zoom ratio before image registration, and other frames of to-be-processed images are registered to the enlarged reference frame image, so that each frame of to-be-processed image can provide sub-pixel-level high-resolution information to the reference frame image at the target zoom ratio. The reference frame image at the target zoom ratio and each registered image are fused, so that pixels of the same content in different frames can be fused together, and the target image at the target zoom ratio can be accurately obtained, and ghosting can be avoided, thereby improving the image quality of the target image.
[0125] In one embodiment, the purpose of image registration is to fuse together pixels of the same content in different frames as much as possible during image fusion. Before image registration, the reference frame needs to be interpolated and enlarged to the target size (e.g., 1.6x), and other frame images need to be registered to the enlarged reference frame, so that other frame images can provide sub-pixel high-resolution information to the reference frame image.
[0126] Image registration is performed by using homography matrix estimation based on SIFT (Scale-invariant feature transform) feature point detection to estimate the Kth image Y to be processed. k The homography matrix H to the reference frame image Y0 k , as shown in the following formula:
[0127]
[0128] Where (x, y) is H k The coordinates of any feature point in the 3x3 matrix are the homography matrix H k , the converted feature point (x`, y`) is the coordinate of (x, y) after registration to the reference frame image Y0, w`=w=1.
[0129] Calculate the homography matrix H k After that, we can use the homography matrix H k Calculate Y k The offset of each feature point relative to Y0 [MV xk , MV yk ], constituted by Y0 and H k The offset vector graph of two channels of the same size has the following main processes: Figure 8 shown.
[0130] The reference frame image Y0 with a size of h*w and the image to be processed Y with a size of h*w are k Perform feature point detection respectively to obtain the feature point description corresponding to the reference frame image Y0 and the image to be processed Y k Corresponding feature point description. According to the feature point description, the image Y to be processed k The feature points of the image Y0 are matched with the feature points of the reference frame image Y0 to obtain matching point pairs. Based on the matching point pairs, the image Y0 to be processed is calculated. k And the homography matrix H of the reference frame image Y0 k .
[0131] SIFT is the original algorithm for key point detection. Its essence is to find key points (feature points) in different scale spaces, calculate the size, direction, and scale information of the key points, and use this information to form key points to describe the feature points. The key points found by SIFT are very prominent and stable feature points that will not change due to factors such as lighting, affine transformation, and noise. After obtaining the feature points, the gradient histogram of the points around the feature points is used to form the feature vector This feature vector is the description of the current feature point, respectively for Y0 and Y k Find the feature points of the two pictures and get two sets of feature points and
[0132] The process of feature point matching is to match the two sets of feature points found by Euclidean distance, as shown in the following formula:
[0133]
[0134] Find Y0 and Y k The four or more feature point pairs closest to each other in the two images can be used to solve the homography matrix H through DLT (Direct Linear Transformation) k The coefficient of Y k The offset of each point relative to Y0 [MV x , MV y ]. For example, the Y k The coordinates of the feature points on are (x1, y1), (x2, y2),…(x t ,y t ), the corresponding feature point coordinates on Y0 are Applying the homography matrix to the corresponding feature point pairs, we can get the following equation:
[0135] or AH K =0
[0136] Where A is a matrix with twice the number of rows as the number of matching point pairs. The coefficients of the matching point pair equations are stacked into a matrix. The SVD (Singular Value Decomposition) algorithm can be used to find the least squares solution of H, thereby calculating each frame Y k The offset relative to Y0 [MV xk , MV yk ].
[0137] In one embodiment, image registration can also use SURF, corner points or other features to detect and describe feature points. It is also possible to solve the optical flow vector of each pixel from each frame of the image to be processed to the reference frame based on the brightness information around each point in the adjacent frames, and then calculate the motion vector of the pixel based on the optical flow vector to perform image registration.
[0138] Fig. 9 Schematic diagram of an image processing method in an embodiment. Taking the first zoom ratio range of 1x-1.6x, the preset zoom ratio set of (1.6x, 3.2x, 4.8x), and the second zoom ratio range of 1.6x-3.2x and 3.2x-4.8x as an example for explanation, when the target zoom ratio is within 1x-1.6x, the electronic device selects 8-10 frames of images to be processed as input, fuses the selected 8-10 frames of images to be processed by multi-frame fusion, and physically increases the details of the high-resolution image by performing sub-pixel interpolation to obtain the first target image.
[0139] When the target zoom ratio belongs to (1.6x, 3.2x, 4.8x), the first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution models corresponding to each preset zoom ratio in (1.6x, 3.2x, 4.8x), that is, the first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution model 1 corresponding to 1.6x, the super-resolution model 2 corresponding to 3.2x and the super-resolution model 3 corresponding to 4.8x, and the first super-resolution model is used to super-resolution the 1.6x first magnification image processed by the multi-frame fusion method using a single-frame super-resolution method to obtain a second target image. Among them, the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception, and each super-resolution model can be an AI (Artificial Intelligence) super-resolution model. AI super-resolution can ensure that the image quality will not be greatly attenuated under different zoom ratios.
[0140] When the target zoom ratio is within 1.6x-3.2x or 3.2x-4.8x, a new super-resolution model corresponding to the target zoom ratio is interpolated based on the super-resolution models corresponding to two adjacent preset zoom ratios. If the target zoom ratio is within 1.6x-3.2x, a new super-resolution model corresponding to the target zoom ratio is interpolated based on super-resolution model 1 and super-resolution model 2. If the target zoom ratio is within 3.2x-4.8x, a new super-resolution model corresponding to the target zoom ratio is interpolated based on super-resolution model 2 and super-resolution model 3. The 1.6x image processed by the multi-frame fusion method is super-resolved by the new super-resolution model to obtain a third target image.
[0141] In another embodiment, an image processing method is provided, which is applied to an electronic device, and the image processing method includes the following steps:
[0142] Step a1, obtaining the target zoom ratio.
[0143] When the target zoom magnification is within the first zoom magnification range, the electronic device executes step a21 to step a23.
[0144] Step a21, determining a reference frame image from each frame of the image to be processed according to the image sharpness corresponding to each frame of the image to be processed.
[0145] Step a22, enlarge the reference frame image to the target zoom ratio, obtain the feature points of the reference frame image at the target zoom ratio and the feature points of each frame of the image to be processed; determine the matching point pairs between the reference frame image at the target zoom ratio and each frame of the image to be processed according to the feature points of the reference frame image at the target zoom ratio and the feature points of each frame of the image to be processed; determine the homography matrix between the reference frame image at the target zoom ratio and each frame of the image to be processed according to the matching point pairs; perform registration processing on each frame of the image to be processed based on the homography matrix, and obtain each registered image of the target zoom ratio.
[0146] Step a23, fusing each registered image with the reference frame image at the target zoom factor to obtain a first target image at the target zoom factor.
[0147] When the target zoom magnification belongs to a preset zoom magnification set, the electronic device executes step a31 to step a33.
[0148] Step a31, in the process of quantizing the super-resolution model corresponding to each preset zoom ratio, the maximum value and the minimum value flowing through the pseudo-quantization node are counted through the set pseudo-quantization node; the variance of the maximum value is determined based on each maximum value, and the variance of the minimum value is determined based on each minimum value; based on the variance of the maximum value and the variance of the minimum value, the target loss value corresponding to all super-resolution models is determined; according to the target loss value, the super-resolution model corresponding to each preset zoom ratio is trained until the training cutoff condition is met, and each super-resolution model that has been trained is obtained.
[0149] Step a32, determining the first super-resolution model corresponding to the target zoom ratio from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set; the preset zoom ratio is an integer multiple of the upper limit ratio of the first zoom ratio range, and the super-resolution model corresponding to each preset zoom ratio is obtained through joint quantization perception.
[0150] Step a33, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method to obtain a first magnification image; the first magnification image is an image corresponding to the upper limit magnification of the first zoom magnification range; the first magnification image is enlarged to the target zoom magnification by a first super-resolution model to obtain a target magnification image; feature extraction is performed on the target magnification image to obtain a low-level feature map; feature conversion is performed on the low-level feature map to obtain a high-level feature map; each high-level feature map is fused to obtain a second target image at the target zoom magnification.
[0151] In a case where the target zoom magnification is within a second zoom magnification range between two adjacent preset zoom magnifications, the electronic device performs steps a41 to a43.
[0152] Step a41, obtaining the second super-resolution models corresponding to two adjacent preset zoom ratios; the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception; and the third super-resolution model corresponding to the target zoom ratio is interpolated from each second super-resolution model using an interpolation processing method.
[0153] Step a42, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method to obtain a first magnification image; the first magnification image is an image corresponding to the upper limit magnification of the first zoom magnification range.
[0154] Step a43, zooming the first-magnification image by a third super-resolution model using a single-frame super-resolution method to obtain a third target image at a target zoom ratio.
[0155] In this embodiment, three processing methods at different zoom ratios are provided, and the different zoom ratios include a first zoom ratio range, a preset zoom ratio set, and a second zoom ratio range. Each preset zoom ratio in the preset zoom ratio set is an integer multiple of the upper limit ratio of the first zoom ratio range, and the ratio between two adjacent preset zoom ratios in the second zoom ratio range constitutes the second zoom ratio range, so that the first zoom ratio range, the preset zoom ratio set, and the second zoom ratio range can cover all zoom ratios, so that zoom processing can be performed using the corresponding method at each zoom ratio.
[0156] When the target zoom ratio is within the first zoom ratio range, each frame of the image to be processed is registered with the reference frame image at the target zoom ratio, so that each frame of the image to be processed and the reference frame image at the target zoom ratio are in the same image space, so as to ensure that all parameters other than the resolution of each frame of the image to be processed and the reference frame image at the target zoom ratio are consistent, so that after the registration image and the reference frame image are fused, sub-pixel high-resolution information can be obtained, thereby improving the image quality of the target image.
[0157] When the target zoom ratio belongs to the preset zoom ratio set, each frame of the image to be processed is first zoomed by multi-frame fusion, so that the first magnification image can be quickly obtained, and then the first super-resolution model obtained by joint quantization perception is used to zoom the first magnification image by single-frame super-resolution, so that when the multi-frame fusion method cannot support the target zoom ratio, the processing result of the multi-frame fusion method can be reconstructed by super-resolution using a single-frame super-resolution method to obtain each second magnification image under an integer multiple of the upper limit magnification of the first zoom ratio range, which can ensure the continuity of zoom and the maintenance of high-magnification zoom image quality. And the super-resolution model corresponding to each preset zoom ratio in the preset zoom ratio set is obtained through joint quantization perception, so each super-resolution model can process the image to be processed based on the parameters under the same standard, so as to obtain a more accurate second target image.
[0158] When the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, the second super-resolution models corresponding to the two adjacent preset zoom ratios are interpolated to obtain a third super-resolution model corresponding to the target zoom ratio, and then the first-magnification image obtained by multi-frame fusion processing is zoomed by the third super-resolution model to obtain a third target image at the target zoom ratio, so that the image can obtain more detailed features, and the obtained third target image at the target zoom ratio has higher clarity, thereby reducing the loss of image quality.
[0159] This embodiment can flexibly select a multi-frame fusion method, a single-frame super-resolution method, an interpolation processing method and a combination thereof to perform zoom processing at different focal lengths according to the preset zoom ratio to which the target zoom ratio belongs, which can effectively reduce the loss of image quality at each focal length, thereby increasing the image resolution while reducing the loss of image quality.
[0160] It should be understood that although Figures 2 to 9 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 2 to 9 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0161] Fig.10 FIG. 1 is a structural block diagram of an image processing device according to an embodiment. Fig.10 As shown, an image processing device is provided, including: an acquisition module 1002, a first zoom module 1004, a second zoom module 1006 and a third zoom module 1008, wherein:
[0162] The acquisition module 1002 is used to acquire a target zoom ratio.
[0163] The first zoom module 1004 is used to perform zoom processing on each frame of the to-be-processed image by a multi-frame fusion method when the target zoom ratio is within a preset zoom ratio range, so as to obtain a first target image under the target zoom ratio.
[0164] The second zoom module 1006 is used to determine a first super-resolution model corresponding to the target zoom ratio from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set when the target zoom ratio belongs to a preset zoom ratio set, and to perform zoom processing on each frame of the image to be processed by the first super-resolution model using a single-frame super-resolution method and a multi-frame fusion method to obtain a second target image at the target zoom ratio; the preset zoom ratio is an integer multiple of the upper limit ratio of the first zoom ratio range, and the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception.
[0165] The third zoom module 1008 is used to perform zoom processing on each frame of the image to be processed by a multi-frame fusion method, a single-frame super-resolution method and an interpolation processing method when the target zoom ratio is within a second zoom ratio range between two adjacent preset zoom ratios, so as to obtain a third target image at the target zoom ratio.
[0166] The above-mentioned image processing device, when the target zoom ratio is within the first zoom ratio range, zoom processing is performed on each frame of the image to be processed by a multi-frame fusion method, and the first target image under the target zoom ratio can be quickly generated. When the target zoom ratio belongs to a preset zoom ratio set, the first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and the details of the image are increased and the loss of image quality is reduced by zooming each frame of the image to be processed by the first super-resolution model using a single-frame super-resolution method and a multi-frame fusion method, thereby obtaining a second target image under the target zoom ratio, and the preset zoom ratio is an integer multiple of the upper limit ratio of the first zoom ratio range. In addition, the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception, so each super-resolution model can process the image to be processed based on the parameters under the same standard, thereby obtaining a more accurate second target image. When the target zoom ratio is within the second zoom ratio range between two adjacent preset zoom ratios, zoom processing is performed on each frame of the image to be processed through a multi-frame fusion method, a single-frame super-resolution method and an interpolation processing method to obtain a third target image at the target zoom ratio. According to the preset zoom ratio to which the target zoom ratio belongs, the multi-frame fusion method, the single-frame super-resolution method, the interpolation processing method and a combination thereof can be flexibly selected to perform zoom processing at different focal lengths, and the loss of image quality at each focal length can be effectively reduced.
[0167] In one embodiment, the above method also includes a training module, which is used to perform quantitative perception on the super-resolution model corresponding to each preset zoom ratio, and obtain the target loss value corresponding to all super-resolution models; the super-resolution model corresponding to each preset zoom ratio is trained according to the target loss value until the training cutoff condition is met, and each super-resolution model that has been trained is obtained.
[0168] In one embodiment, the above-mentioned training module is also used to, in the process of quantizing the super-resolution model corresponding to each preset zoom ratio, count the maximum and minimum values flowing through the pseudo-quantization node through the set pseudo-quantization node; determine the variance of the maximum value based on each maximum value, and determine the variance of the minimum value based on each minimum value; determine the target loss value corresponding to all super-resolution models based on the variance of the maximum value and the variance of the minimum value.
[0169] In one embodiment, the second zoom module 1006 is also used to perform zoom processing on each frame of the image to be processed by a multi-frame fusion method to obtain a first magnification image; the first magnification image is an image corresponding to the upper limit magnification of the first zoom magnification range; the first magnification image is zoomed by a single-frame super-resolution method using the first super-resolution model to obtain a second target image at the target zoom magnification.
[0170] In one embodiment, the second zoom module 1006 is also used to enlarge the first-magnification image to the target zoom ratio through the first super-resolution model to obtain the target zoom ratio image; perform feature extraction on the target zoom ratio image to obtain a low-level feature map; perform feature conversion on the low-level feature map to obtain a high-level feature map; and fuse the high-level feature maps to obtain a second target image at the target zoom ratio.
[0171] In one embodiment, the third zoom module 1008 is also used to obtain second super-resolution models corresponding to two adjacent preset zoom ratios when the target zoom ratio is within a second zoom ratio range between two adjacent preset zoom ratios; the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception; an interpolation processing method is used to interpolate each second super-resolution model into a third super-resolution model corresponding to the target zoom ratio; zoom processing is performed on each frame of the image to be processed through a multi-frame fusion method and a single-frame super-resolution method through the third super-resolution model to obtain a third target image at the target zoom ratio.
[0172] In one embodiment, the third zoom module 1008 is also used to perform zoom processing on each frame of the image to be processed by a multi-frame fusion method to obtain a first magnification image; the first magnification image is an image corresponding to the upper limit magnification of the first zoom magnification range; the first magnification image is zoomed by a single-frame super-resolution method using a third super-resolution model to obtain a third target image at the target zoom magnification.
[0173] In one embodiment, the first zoom module 1004 is also used to determine a reference frame image from each frame of the image to be processed according to the image sharpness corresponding to each frame of the image to be processed when the target zoom ratio is within the first zoom ratio range; enlarge the reference frame image to the target zoom ratio, and perform registration processing on each frame of the image to be processed based on the reference frame image at the target zoom ratio to obtain each registered image; and fuse each registered image with the reference frame image at the target zoom ratio to obtain the first target image at the target zoom ratio.
[0174] In one embodiment, the first zoom module 1004 is also used to obtain feature points of the reference frame image at the target zoom ratio and feature points of each frame of the image to be processed; according to the feature points of the reference frame image at the target zoom ratio and the feature points of each frame of the image to be processed, respectively determine the matching point pairs between the reference frame image at the target zoom ratio and each frame of the image to be processed; according to the matching point pairs, determine the homography matrix between the reference frame image at the target zoom ratio and each frame of the image to be processed; based on the homography matrix, perform registration processing on each frame of the image to be processed, and obtain each registered image of the target zoom ratio.
[0175] The division of the modules in the above-mentioned image processing device is only for illustration. In other embodiments, the image processing device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned image processing device.
[0176] For the specific definition of the image processing device, please refer to the definition of the image processing method above, which will not be repeated here. Each module in the above image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0177] Fig.11 Schematic diagram of the internal structure of an electronic device in an embodiment. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a wearable device, etc. The electronic device includes a processor and a memory connected via a system bus. Among them, the processor may include one or more processing units. The processor may be a CPU (Central Processing Unit) or a DSP (Digital Signal Processing), etc. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image processing method provided in each of the following embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium.
[0178] The implementation of each module in the image processing device provided in the embodiment of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. The program module constituted by the computer program may be stored in a memory of an electronic device. When the computer program is executed by a processor, the steps of the method described in the embodiment of the present application are implemented.
[0179] The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.
[0180] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.
[0181] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include ROM (Read-Only Memory), PROM (Programmable Read-only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-only Memory) or flash memory. Volatile memory may include RAM (Random Access Memory), which is used as an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDRSDRAM (Double Data Rate Synchronous Dynamic Random Access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory), SLDRAM (Sync Link Dynamic Random Access Memory), RDRAM (Rambus Dynamic Random Access Memory), and DRDRAM (Direct Rambus Dynamic Random Access Memory).
[0182] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. An image processing method, characterized in that: include: Acquire a target zoom ratio, and when the target zoom ratio is within a first zoom ratio range, perform zoom processing on each frame of the image to be processed by a multi-frame fusion method to obtain a first target image at the target zoom ratio; In the case where the target zoom ratio belongs to a preset zoom ratio set, a first super-resolution model corresponding to the target zoom ratio is determined from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and zoom processing is performed on each frame of the image to be processed by the first super-resolution model using a single-frame super-resolution method and a multi-frame fusion method to obtain a second target image at the target zoom ratio; the preset zoom ratio is an integer multiple of the upper limit ratio of the first zoom ratio range, and the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception; When the target zoom ratio is within a second zoom ratio range between two adjacent preset zoom ratios, second super-resolution models corresponding to the two adjacent preset zoom ratios are obtained; the super-resolution models corresponding to the respective preset zoom ratios are obtained through joint quantization perception; an interpolation processing method is used to interpolate each of the second super-resolution models into a third super-resolution model corresponding to the target zoom ratio; zoom processing is performed on each frame of the image to be processed through the multi-frame fusion method and the single-frame super-resolution method through the third super-resolution model to obtain a third target image at the target zoom ratio.
2. The method according to claim 1, characterized in that The joint quantization perception method of the super-resolution model corresponding to each of the preset zoom magnifications includes: Quantizing and perceiving the super-resolution models corresponding to each of the preset zoom magnifications to obtain target loss values corresponding to all super-resolution models; The super-resolution models corresponding to the preset zoom ratios are trained according to the target loss value until a training cutoff condition is met, thereby obtaining each trained super-resolution model.
3. The method according to claim 2, characterized in that The quantization perception is performed on each super-resolution model corresponding to the preset zoom ratio to obtain the target loss value corresponding to all super-resolution models, including: In the process of performing quantization perception on the super-resolution model corresponding to each of the preset zoom magnifications, by setting a pseudo-quantization node, counting the maximum value and the minimum value flowing through the pseudo-quantization node; Determine the variance of the maximum value based on each maximum value, and determine the variance of the minimum value based on each minimum value; Based on the variance of the maximum value and the variance of the minimum value, target loss values corresponding to all super-resolution models are determined.
4. The method according to claim 1, characterized in that The method of performing zoom processing on each frame of the image to be processed by using the first super-resolution model in a single-frame super-resolution manner and in a multi-frame fusion manner to obtain a second target image at the target zoom magnification includes: Performing zoom processing on each frame of the image to be processed by the multi-frame fusion method to obtain a first magnification image; the first magnification image is an image corresponding to an upper limit magnification of the first zoom magnification range; The first super-resolution model uses a single-frame super-resolution method to perform zoom processing on the first-magnification image to obtain a second target image at the target zoom ratio.
5. The method according to claim 4, characterized in that The step of performing zoom processing on the first magnification image by using the first super-resolution model in a single-frame super-resolution manner to obtain a second target image at the target zoom magnification includes: The first magnification image is magnified to the target zoom magnification by using the first super-resolution model to obtain a target magnification image; Performing feature extraction on the target magnification image to obtain a low-level feature map; Performing feature conversion on the low-level feature map to obtain a high-level feature map; The high-level feature maps are fused to obtain a second target image at the target zoom factor.
6. The method according to claim 1, characterized in that The method of performing zoom processing on each frame of the image to be processed by the multi-frame fusion method and the third super-resolution model using a single-frame super-resolution method to obtain a third target image at the target zoom factor includes: Performing zoom processing on each frame of the image to be processed by the multi-frame fusion method to obtain a first magnification image; the first magnification image is an image corresponding to an upper limit magnification of the first zoom magnification range; The first magnification image is zoomed by the third super-resolution model in a single-frame super-resolution manner to obtain a third target image at the target zoom magnification.
7. The method according to claim 1, characterized in that When the target zoom ratio is within the first zoom ratio range, zoom processing is performed on each frame of the to-be-processed image by a multi-frame fusion method to obtain a first target image at the target zoom ratio, including: When the target zoom ratio is within a first zoom ratio range, determining a reference frame image from each frame of the image to be processed according to the image sharpness corresponding to each frame of the image to be processed; Enlarging the reference frame image to the target zoom factor, and performing registration processing on each frame of the image to be processed based on the reference frame image at the target zoom factor to obtain each registered image; Each of the registered images is fused with the reference frame image at the target zoom factor to obtain a first target image at the target zoom factor.
8. The method according to claim 7, characterized in that The registering process of each frame of the image to be processed based on the reference frame image at the target zoom factor to obtain each registered image includes: Acquire feature points of the reference frame image and feature points of each frame of the image to be processed at the target zoom factor; According to the feature points of the reference frame image at the target zoom factor and the feature points of the image to be processed in each frame, respectively determining the matching point pairs between the reference frame image at the target zoom factor and the image to be processed in each frame; Determine, according to the matching point pairs, the homography matrices between the reference frame images at the target zoom factor and the images to be processed in each frame; Based on the homography matrix, registration processing is performed on each frame of the image to be processed to obtain each registered image of the target zoom ratio.
9. An image processing device, characterized in that: include: An acquisition module is used to obtain a target zoom ratio; A first zoom module, configured to perform zoom processing on each frame of the image to be processed by a multi-frame fusion method when the target zoom ratio is within a first zoom ratio range, so as to obtain a first target image at the target zoom ratio; A second zoom module is used to determine, when the target zoom ratio belongs to a preset zoom ratio set, a first super-resolution model corresponding to the target zoom ratio from the super-resolution models corresponding to each preset zoom ratio in the preset zoom ratio set, and to perform zoom processing on each frame of the image to be processed by using the first super-resolution model in a single-frame super-resolution manner and in a multi-frame fusion manner to obtain a second target image at the target zoom ratio; the preset zoom ratio is an integer multiple of an upper limit ratio of the first zoom ratio range, and the super-resolution models corresponding to each preset zoom ratio are obtained through joint quantization perception; The third zoom module is used to obtain the second super-resolution models corresponding to the two adjacent preset zoom ratios when the target zoom ratio is within the second zoom ratio range between the two adjacent preset zoom ratios; the super-resolution models corresponding to the respective preset zoom ratios are obtained through joint quantization perception; the interpolation processing method is used to interpolate the second super-resolution models into a third super-resolution model corresponding to the target zoom ratio; the zoom processing is performed on each frame of the image to be processed through the multi-frame fusion method and the single-frame super-resolution method through the third super-resolution model to obtain a third target image at the target zoom ratio.
10. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image processing method according to any one of claims 1 to 8.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.
12. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the computer is caused to execute the steps of the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Photographing method for terminal, and terminal
CN109906599A
Zoom processing method and device for double cameras and terminal equipment
CN111147755A
Digital zooming method and system, electronic equipment, medium and digital imaging equipment
CN111447359A
Image processing method and device
CN112529775A
Photographing method and device, terminal and storage medium
CN112634160A
