Image generation method and electronic equipment

Abstract

The invention discloses an image generation method and electronic equipment. Belongs to the technical field of image processing. The method comprises the following steps: acquiring an image feature of a first frame image; the super-resolution image corresponding to the second frame image, the first frame image and the image features are input into a weight prediction model, the first weight of each pixel point in the first frame image and the second weight of each pixel point in the super-resolution image are obtained, and the second frame image is an image frame before the first frame image; performing up-sampling on the first frame image based on the first weight to obtain an up-sampled image; and based on the second weight, fusing the up-sampling image and the super-resolution image to generate a super-resolution image corresponding to the first frame image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and specifically to an image generating method and an electronic device. Background Art

[0002] Image super-resolution (Image Super Resolution), also known as image super-resolution, is a technical means to improve the resolution of an image. It is used to restore a high-resolution image from a low-resolution image or image sequence. The image obtained by image super-resolution can be called a super-resolution image.

[0003] In the prior art, super-resolution images are usually generated through traditional difference algorithms, or end-to-end super-resolution image generation is performed through neural networks. These methods rely on limited information, resulting in poor quality of the generated super-resolution images. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image generation method and an electronic device, which can improve the quality of super-resolution images.

[0005] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising: obtaining image features of a first frame image; inputting a super-resolution image corresponding to a second frame image, the first frame image and the image features into a weight prediction model to obtain a first weight for each pixel in the first frame image and a second weight for each pixel in the super-resolution image, wherein the second frame image is an image frame before the first frame image; based on the first weight, upsampling the first frame image to obtain an upsampled image; based on the second weight, fusing the upsampled image with the super-resolution image to generate a super-resolution image corresponding to the first frame image.

[0006] In a second aspect, an embodiment of the present application provides an image generating device, which includes: an acquisition unit, used to acquire image features of a first frame image; a processing unit, used to input a super-resolution image corresponding to a second frame image, the first frame image and the image features into a weight prediction model to obtain a first weight of each pixel in the first frame image and a second weight of each pixel in the super-resolution image, wherein the second frame image is an image frame before the first frame image; an upsampling unit, used to upsample the first frame image based on the first weight to obtain an upsampled image; and a generation unit, used to fuse the upsampled image with the super-resolution image based on the second weight to generate a super-resolution image corresponding to the first frame image.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, the image features of the first frame image are first obtained, and then the super-resolution image corresponding to the second frame image, the first frame image and its image features are input into the weight prediction model to obtain the first weight of each pixel in the first frame image and the second weight of each pixel in the super-resolution image; then, based on the first weight, the first frame image is upsampled to obtain the upsampled image; finally, based on the second weight, the upsampled image and the super-resolution image are fused to generate the super-resolution image corresponding to the first frame image. On the one hand, in the process of generating the super-resolution image corresponding to the first frame image, the weight prediction model is used in combination with the image features of the first frame image to predict the first weight of each pixel in the first frame image, so the determination process of the first weight takes into account the image features of the first frame image, so that when upsampling based on the first weight is performed, the detail information of the first frame image can be effectively retained, thereby improving the quality of the upsampled image, thereby improving the quality of the generated super-resolution image. On the other hand, in the process of generating the super-resolution image corresponding to the first frame image, the weight prediction model is used to combine the image features of the first frame image to predict the second weights of each pixel in the super-resolution image corresponding to the second frame image. Therefore, when the super-resolution image corresponding to the second frame image is fused with the upsampled image of the first frame image, it is possible to more accurately and fully reuse the valid information in the super-resolution image corresponding to the second frame image, and filter out or weaken the invalid information in the super-resolution image corresponding to the second frame image, thereby further improving the quality of the generated super-resolution image corresponding to the first frame image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1is a flow chart of an image generation method provided in an embodiment of the present application;

[0013] Figure 2A It is a visual image of the filtering effect of the upsampling process in the prior art;

[0014] Figure 2B It is a filtering visual effect diagram of the upsampling process in the image generation method provided in the embodiment of the present application;

[0015] Figure 3 is a schematic diagram of an application scenario of the image generation method provided in an embodiment of the present application;

[0016] Figure 4 is a structural schematic diagram of an image generating device provided in an embodiment of the present application;

[0017] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0018] Figure 6 It is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0020] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0021] The image generation method and device provided in the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0022] Please refer to Figure 1, which shows one of the flow charts of the image generation method provided in the embodiment of the present application. The image generation method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above electronic devices can be electronic devices such as smart phones, tablet computers, laptop portable computers, etc.

[0023] The process of the image generation method provided in the embodiment of the present application includes the following steps:

[0024] Step 101, obtaining image features of a first frame of image.

[0025] In this embodiment, the first frame of image may refer to an image frame in any image sequence or video to be subjected to image super-resolution processing. The first frame of image may include image information of three channels of RGB (red Green Blue). Among them, image super-resolution (Image Super Resolution), referred to as image super-resolution, is a technical means to improve the resolution of an image, and is used to restore a high-resolution image from a low-resolution image or image sequence. The image features of the first frame of image can be extracted from the image information of the first frame of image. The above-mentioned image information may include but is not limited to at least one of the following: depth information, motion vector information, normal information, diffuse color information, etc.

[0026] In some optional implementations of this embodiment, before obtaining the image features of the first frame image, the first frame image may be rendered, and the delayed rendering technology may be used to render the first frame image. Specifically, the geometric bodies in the scene may be rendered to a geometric buffer (G-buffer); then, based on the light source in the scene, the geometric bodies in the geometric buffer may be colored to obtain the above-mentioned first frame image. On this basis, when obtaining the image features of the first frame image, the image features of the first frame image may be obtained from the above-mentioned geometric buffer.

[0027] In practice, the geometry buffer is a buffer used in computer graphics, which can usually be applied to the deferred rendering process. The main function of the geometry buffer is to store the geometric information of the geometry, including but not limited to position, normal, color value, mirror value, etc. The basic idea of ​​deferred rendering is to perform depth testing first and then perform shading calculations. By calculating the information of the geometry in advance and storing it in the geometry buffer, the information in the geometry buffer is then used to perform lighting calculations for each fragment during the lighting processing stage. Deferred rendering helps to improve rendering efficiency and image quality.

[0028] By performing delayed rendering on the first frame image, the rendering efficiency and image quality of the first frame image can be improved. By obtaining the image features of the first frame image from the geometry buffer, the information in the geometry buffer can be fully utilized to assist in the subsequent weight determination, so that the detail information of the first frame image can be effectively retained when the first frame image is subsequently upsampled, which helps to improve the quality of the upsampled image, and further helps to improve the quality of the super-resolution image finally generated.

[0029] Step 102, input the super-resolution image corresponding to the second frame image, the first frame image and the image features into a weight prediction model to obtain a first weight of each pixel in the first frame image and a second weight of each pixel in the super-resolution image.

[0030] In this embodiment, the second frame image is an image frame before the first frame image, that is, the second frame image is a frame that has been rendered before the first frame image is rendered. The super-resolution image is a super-resolution image (Super-Resolution Image) of the second frame image. The super-resolution image is a high-resolution image that is restored from a low-resolution image or image sequence through image super-resolution processing.

[0031] In the present embodiment, the weight prediction model can be trained in advance by a machine learning method. The weight prediction model can be used to perform weight prediction. The model structure used by the weight prediction model may include but is not limited to convolutional neural networks (CNN). In practice, a convolutional neural network is a feedforward neural network whose artificial neurons can respond to surrounding units within a portion of the coverage range and have excellent performance for image processing. Therefore, a convolutional neural network can be used to process super-resolution images, first frame images, and image feature inputs. In the present embodiment, the convolutional neural network established may include but a plurality of convolutional layers. Convolutional layers can be used to extract image features.

[0032] In this embodiment, the super-resolution image, the first frame image and the image features corresponding to the second frame image can be input into the weight prediction model to obtain the first weight of each pixel in the first frame image and the second weight of each pixel in the super-resolution image.

[0033] The first weight can be applied to the upsampling stage of the first frame image. Since upsampling is a spatial domain image super-resolution method and can be implemented by a filter, the first weight can be understood as a weight used for filtering in the spatial domain, which can be called a spatial domain filtering weight. The first weights may be different for different pixels in the first frame image. For example, the first frame image includes a moving object and a static background. Since the moving object is usually the focus of the user, the first weight of the area corresponding to the object can be higher, while the first weight of the area corresponding to the static background can be lower. Since the first weights of the pixels in the first frame image are not the same value, the setting of the first weight takes into account the image features of the first frame image. Therefore, when upsampling is performed based on the first weight, the detail information of the first frame image can be effectively retained, and the quality of the upsampled image can be improved.

[0034] The second weight can be applied to the stage of fusing the upsampled image of the first frame image with the super-resolution image. Since image fusion is an image processing process in the time domain, the second weight can be understood as the weight used when performing image fusion in the time domain, which can be called the time domain fusion weight. For different pixels in the super-resolution image, the second weights can be different. For example, the super-resolution image includes a moving object and a static background. Since the position of the moving object in the super-resolution image and the upsampled image obtained by upsampling the first frame image is usually different, the artifact of the object will be generated when the image is fused, so the second weight of the area corresponding to the object can be lower. On the contrary, the static background usually does not produce artifacts when the images overlap, so the second weight of the area corresponding to the background can be lower. Since the second weights of different pixels in the super-resolution image are not the same value, and the setting of the second weight takes into account the image features of the first frame image, when the super-resolution image and the upsampled image of the first frame image are subsequently fused, the effective information in the super-resolution image is reused more accurately and fully, and the invalid information such as artifacts in the super-resolution image is filtered out or weakened, thereby improving the quality of the generated super-resolution image.

[0035] In some optional implementations of this embodiment, the first frame image is any video frame other than the first frame in the target image sequence, and the second frame image is the frame image before the first frame image in the target image sequence. The above-mentioned target image sequence can be an image sequence to be subjected to image super-resolution, for example, it can be a sequence composed of real-time game screens. Since the previous frame is usually correlated with the first frame image, when the image super-resolution processing is performed on the first frame image, the image super-resolution result of the previous frame is integrated to obtain more effective information, thereby improving the quality of the generated super-resolution image.

[0036] In some optional implementations of this embodiment, the weight control model can be trained through the following steps:

[0037] The first step is to obtain a sample set. The sample set may include a large number of samples. Each sample may include a low-resolution image, a high-resolution image, and image features of the low-resolution sample image. The low-resolution image may be a video frame in a video, and the high-resolution image may be obtained by performing image super-resolution processing on the previous frame of the video frame. Each sample includes a label, and the label may include the true weight of each pixel in the low-resolution image in the sample and the true weight of each pixel in the high-resolution image. The label may be set manually.

[0038] The second step is to train the convolutional neural network based on the sample set. Specifically, samples can be first extracted from the sample set, and the extracted samples can be input into the convolutional neural network to obtain the weight prediction results output by the convolutional neural network. The weight prediction results may include the first prediction weight of each pixel in the low-resolution image in the sample and the second prediction weight of each pixel in the high-resolution image. Then, the loss value can be determined based on the weight prediction result and the annotation of the input sample. The above loss value is the value of the loss function, which is a non-negative real-valued function that can be used to characterize the difference between the detection result and the true result. In general, the smaller the loss value, the better the robustness of the model. The loss function can be set according to actual needs. Afterwards, the loss value can be used to update the parameters of the convolutional neural network. Thus, each time a sample is input, the parameters of the convolutional neural network can be updated based on the loss value corresponding to the sample until the training is completed.

[0039] In practice, whether training is completed can be determined in a variety of ways. For example, when the accuracy of the detection results output by the convolutional neural network reaches a preset value (for example, 99%), the training can be determined to be completed. For another example, if the number of training times of the convolutional neural network is equal to the preset number of times, the training can be determined to be completed. For another example, when the loss value of the convolutional neural network converges, the training can be determined to be completed. Here, if the convolutional neural network training is completed, the trained convolutional neural network can be determined as a weight prediction model.

[0040] Since convolutional neural networks have outstanding performance in the field of image processing, the weight prediction model obtained by training the convolutional neural network can make the weight prediction model have higher accuracy.

[0041] Step 103: upsample the first frame image based on the first weight to obtain an upsampled image.

[0042] In this embodiment, upsampling can be performed using various interpolation algorithms, including but not limited to bilinear interpolation, bicubic interpolation, Lanxos interpolation, etc. When performing interpolation, the first weight can be used to replace the original weight.

[0043] In some optional implementations of this embodiment, upsampling the first frame image based on the first weight to obtain the upsampled image includes:

[0044] The first step is to determine the projection point of each pixel in the upsampled image to be generated in the first frame image. Here, the size of the upsampled image can be preset so that the coordinates of each pixel in the upsampled image can be obtained based on the size. Then, based on the size of the upsampled image and the size of the first frame image, the coordinates of each pixel in the upsampled image are transformed to obtain the coordinates of the projection point of each pixel in the upsampled image in the first frame image.

[0045] In the second step, based on the first weight, the pixel values ​​of the pixels in the neighborhood of the projection point are weighted and summed to obtain the pixel value of each pixel in the upsampled image. Here, the neighborhood of the projection point can be preset as needed, for example, it can be set to a rectangular area of ​​a set size centered on the projection point. The weight used in the weighted summation is the above-mentioned first weight.

[0046] The third step is to generate an upsampled image based on the pixel value of each pixel in the upsampled image.

[0047] In the related art, the weighted method only considers the distance from the projection point to the pixel points in the neighborhood. Figure 2A , the point at the position of Dst_p0 in the upsampled image 201 is projected at Src_p0 in the first frame image 202. The neighborhood of the projected point Src_p0 is a rectangular area 203 centered at Src_p0, which contains 16 pixels. When weighting, the pixel values ​​of the 16 pixels in the neighborhood of Src_p0 are taken for weighted calculation, and the weight corresponding to each pixel is determined based on the distance from the pixel to Src_p0, such as Figure 2A R0 and R1 in . It can be seen that in the conventional interpolation strategy, the interpolation weight is closely related to the distance. However, the actual image has a complex texture structure. For high-frequency textures, only the distance relationship is referenced. After interpolation, the image will become very blurred and there will be problems such as loss of details.

[0048] In the embodiment of the present application, the weight prediction model is used in combination with the image features of the first frame image to predict the first weight of each pixel in the first frame image, so the process of determining the first weight takes into account the image features of the first frame image. For example, when the normal of the projection point differs too much from the normal of a pixel in the neighboring domain, the first weight of the pixel will be smaller. Figure 2BSimilarly, the point at the position of Dst_p0 in the upsampled image 201 is projected at Src_p0 in the first frame image 202. The neighborhood of the projected point Src_p0 is a rectangular area 203 centered at Src_p0, which contains 16 pixels. When weighting, the pixel values ​​of the 16 pixels in the neighborhood of Src_p0 are taken for weighted calculation, and the weight corresponding to each pixel is the first weight. The interpolation algorithm can be implemented by a filter. From the actual effect, if only the filter visualization of the distance is considered, it is equivalent to changing Figure 2A The standard hollow circle in Figure 2B Since the image features of the first frame image are taken into consideration in the process of determining the first weight, the detail information of the first frame image can be effectively retained when upsampling is performed based on the first weight, thereby improving the quality of the upsampled image, thereby improving the quality of the generated super-resolution image.

[0049] Step 104: Based on the second weight, the upsampled image and the super-resolution image are fused to generate a super-resolution image corresponding to the first frame image.

[0050] In this embodiment, the pixel values ​​of the pixels at the same position in the upsampled image and the super-resolution image corresponding to the second frame image can be weighted and summed based on the second weight to obtain the super-resolution image corresponding to the first frame image. It should be noted that the pixel value of each pixel in the upsampled image can be set to a value unrelated to the second weight, or can be set based on the second weight. The pixel value of each pixel in the upsampled image can be set to the same value, or can be set to different values ​​for different positions, which is not limited in this embodiment of the present application.

[0051] In some optional implementations of this embodiment, the following steps may be performed:

[0052] In the first step, based on the second weight, a third weight of each pixel in the upsampled image is determined. For example, for each pixel in the super-resolution image, 1 can be subtracted from the pixel value of the pixel to obtain the third weight of the pixel at the same position in the upsampled image.

[0053] In the second step, based on the third weight and the second weight, weighted summation is performed on the pixel values ​​of the same position pixels in the upsampled image and the super-resolution image corresponding to the second frame image to obtain the super-resolution image corresponding to the first frame image.

[0054] Therefore, the setting of the weights of each pixel in the upsampled image and the super-resolution image takes into account the image features of the first frame image. Therefore, when the super-resolution image is fused with the upsampled image of the first frame image, it is possible to more accurately and fully reuse the valid information in the super-resolution image, and filter out or weaken the invalid information in the super-resolution image, thereby further improving the quality of the generated super-resolution image.

[0055] The method provided by the above-mentioned embodiment of the present application first obtains the image features of the first frame image, and then inputs the super-resolution image corresponding to the second frame image, the first frame image and its image features into the weight prediction model to obtain the first weight of each pixel in the first frame image and the second weight of each pixel in the super-resolution image; then, based on the first weight, the first frame image is upsampled to obtain the upsampled image; finally, based on the second weight, the upsampled image and the super-resolution image are fused to generate the super-resolution image corresponding to the first frame image. On the one hand, in the process of generating the super-resolution image corresponding to the first frame image, the weight prediction model is used to combine the image features of the first frame image to predict the first weight of each pixel in the first frame image, so the determination process of the first weight takes into account the image features of the first frame image, so that when upsampling is performed based on the first weight, the detail information of the first frame image can be effectively retained, thereby improving the quality of the upsampled image, thereby improving the quality of the generated super-resolution image. On the other hand, in the process of generating the super-resolution image corresponding to the first frame image, the weight prediction model is used to combine the image features of the first frame image to predict the second weights of each pixel in the super-resolution image corresponding to the second frame image. Therefore, when the super-resolution image corresponding to the second frame image is fused with the upsampled image of the first frame image, it is possible to more accurately and fully reuse the valid information in the super-resolution image corresponding to the second frame image, and filter out or weaken the invalid information in the super-resolution image corresponding to the second frame image, thereby further improving the quality of the generated super-resolution image corresponding to the first frame image.

[0056] See also Figure 3 The first frame image is shown as 301, and its resolution is 540P. The ideal high-resolution image corresponding to the first frame image is shown as 302, and its resolution is 1080P. The super-resolution image obtained by processing the first frame image by traditional image super-resolution means is shown as 303, and the image is relatively blurred. The super-resolution image obtained by processing the first frame image by the embodiment of the present application is shown as 304, and it can be seen that the image super-resolution effect in detail recovery and edge jaggedness is much better than the traditional method.

[0057] It should be noted that the image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device executing the image generation method is taken as an example to illustrate the image generation device provided in the embodiment of the present application.

[0058] like Figure 4 As shown, the image generating device 400 of this embodiment includes: an acquiring unit 401, used to acquire image features of a first frame image; a processing unit 402, used to input a super-resolution image corresponding to a second frame image, the first frame image and the image features into a weight prediction model to obtain a first weight of each pixel in the first frame image and a second weight of each pixel in the super-resolution image, wherein the second frame image is an image frame before the first frame image; an upsampling unit 403, used to upsample the first frame image based on the first weight to obtain an upsampled image; and a generating unit 404, used to fuse the upsampled image with the super-resolution image based on the second weight to generate a super-resolution image corresponding to the first frame image.

[0059] In some optional implementations of this embodiment, the acquisition unit is further used to acquire the image features of the first frame image from the geometric buffer. By acquiring the image features of the first frame image from the geometric buffer, the information in the geometric buffer can be fully utilized to assist in the subsequent weight determination, so that the detail information of the first frame image can be effectively retained when the first frame image is subsequently upsampled, which helps to improve the quality of the upsampled image, and further helps to improve the quality of the super-resolution image finally generated.

[0060] In some optional implementations of the present embodiment, the upsampling unit 403 is further used to: determine the projection point of each pixel in the upsampled image to be generated in the first frame image; based on the first weight, weighted sum the pixel values ​​of the pixel points in the neighborhood of the projection point to obtain the pixel value of each pixel in the upsampled image; based on the pixel value of each pixel in the upsampled image, generate the upsampled image. Since the previous frame and the first frame image are usually correlated, when the image super-resolution result of the previous frame is fused when the first frame image is subjected to image super-resolution processing, more effective information can be obtained, thereby improving the quality of the generated super-resolution image. Since the determination process of the first weight takes into account the image features of the first frame image, the detail information of the first frame image can be effectively retained when upsampling is performed based on the first weight, thereby improving the quality of the upsampled image, thereby improving the quality of the generated super-resolution image.

[0061] In some optional implementations of the present embodiment, the generation unit 404 is further used to: determine the third weight of each pixel in the upsampled image based on the second weight; and perform weighted summation on the pixel values ​​of the same position of the upsampled image and the super-resolution image corresponding to the second frame image based on the third weight and the second weight, to obtain the super-resolution image corresponding to the first frame image. Thus, the setting of the weights of each pixel in the upsampled image and the super-resolution image takes into account the image features of the first frame image, so that when the super-resolution image and the upsampled image of the first frame image are fused, the effective information in the super-resolution image can be reused more accurately and fully, and the invalid information in the super-resolution image can be filtered out or weakened, thereby further improving the quality of the generated super-resolution image.

[0062] In some optional implementations of this embodiment, the first frame image is any video frame except the first frame in the target image sequence, and the second frame image is a frame image previous to the first frame image in the target image sequence.

[0063] The device provided by the above-mentioned embodiment of the present application first obtains the image features of the first frame image, and then inputs the super-resolution image corresponding to the second frame image, the first frame image and its image features into the weight prediction model to obtain the first weight of each pixel in the first frame image and the second weight of each pixel in the super-resolution image; then, based on the first weight, the first frame image is upsampled to obtain the upsampled image; finally, based on the second weight, the upsampled image and the super-resolution image are fused to generate the super-resolution image corresponding to the first frame image. On the one hand, in the process of generating the super-resolution image corresponding to the first frame image, the weight prediction model is used to combine the image features of the first frame image to predict the first weight of each pixel in the first frame image, so the determination process of the first weight takes into account the image features of the first frame image, so that when upsampling is performed based on the first weight, the detail information of the first frame image can be effectively retained, thereby improving the quality of the upsampled image, thereby improving the quality of the generated super-resolution image. On the other hand, in the process of generating the super-resolution image corresponding to the first frame image, the weight prediction model is used to combine the image features of the first frame image to predict the second weights of each pixel in the super-resolution image corresponding to the second frame image. Therefore, when the super-resolution image corresponding to the second frame image is fused with the upsampled image of the first frame image, it is possible to more accurately and fully reuse the valid information in the super-resolution image corresponding to the second frame image, and filter out or weaken the invalid information in the super-resolution image corresponding to the second frame image, thereby further improving the quality of the generated super-resolution image corresponding to the first frame image.

[0064] The image generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0065] The image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0066] The image generation device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0067] Alternatively, if Figure 5 As shown, an embodiment of the present application further provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores programs or instructions that can be executed on the processor 501, and when the program or instructions are executed by the processor 501, the various steps of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0068] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0069] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0070] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0071] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0072] Among them, the processor 610 is used to obtain image features of the first frame image; input the super-resolution image corresponding to the second frame image, the first frame image and the image features into the weight prediction model to obtain a first weight of each pixel in the first frame image and a second weight of each pixel in the super-resolution image, wherein the second frame image is an image frame before the first frame image; based on the first weight, upsample the first frame image to obtain an upsampled image; based on the second weight, fuse the upsampled image with the super-resolution image to generate a super-resolution image corresponding to the first frame image.

[0073] On the one hand, since in the process of generating the super-resolution image corresponding to the first frame image, the weight prediction model is used in combination with the image features of the first frame image to predict the first weight of each pixel in the first frame image, the process of determining the first weight takes into account the image features of the first frame image, so that when upsampling is performed based on the first weight, the detail information of the first frame image can be effectively retained, thereby improving the quality of the upsampled image, thereby improving the quality of the generated super-resolution image. On the other hand, since in the process of generating the super-resolution image corresponding to the first frame image, the weight prediction model is used in combination with the image features of the first frame image to predict the second weight of each pixel in the super-resolution image corresponding to the second frame image, when the super-resolution image corresponding to the second frame image is fused with the upsampled image of the first frame image, the effective information in the super-resolution image corresponding to the second frame image can be reused more accurately and fully, and the invalid information in the super-resolution image corresponding to the second frame image can be filtered out or weakened, thereby further improving the quality of the generated super-resolution image corresponding to the first frame image.

[0074] In some optional implementations of this embodiment, the processor 610 is further used to obtain image features of the first frame image from the geometric buffer. By performing delayed rendering on the first frame image, the rendering efficiency and image quality of the first frame image can be improved. By obtaining the image features of the first frame image from the geometric buffer, the information in the geometric buffer can be fully utilized to assist in the subsequent weight determination, so that the detail information of the first frame image can be effectively retained when the first frame image is subsequently upsampled, which helps to improve the quality of the upsampled image, and then helps to improve the quality of the super-resolution image finally generated.

[0075] In some optional implementations of the present embodiment, the processor 610 is further used to determine the projection point of each pixel in the upsampled image to be generated in the first frame image; based on the first weight, weighted sum the pixel values ​​of the pixel points in the neighborhood of the projection point to obtain the pixel value of each pixel in the upsampled image; based on the pixel value of each pixel in the upsampled image, generate the upsampled image. Since the previous frame and the first frame image are usually correlated, when the image super-resolution result of the previous frame is fused when the first frame image is subjected to image super-resolution processing, more effective information can be obtained, thereby improving the quality of the generated super-resolution image. Since the determination process of the first weight takes into account the image features of the first frame image, the detail information of the first frame image can be effectively retained when upsampling is performed based on the first weight, thereby improving the quality of the upsampled image, thereby improving the quality of the generated super-resolution image.

[0076] In some optional implementations of this embodiment, the processor 610 is further configured to determine a third weight of each pixel in the upsampled image based on the second weight; and based on the third weight and the second weight, weighted sum the pixel values ​​of the pixel at the same position in the upsampled image and the super-resolution image corresponding to the second frame image to obtain the super-resolution image corresponding to the first frame image. Thus, the setting of the weights of each pixel in the upsampled image and the super-resolution image both takes into account the image features of the first frame image, so that when the super-resolution image and the upsampled image of the first frame image are fused, the effective information in the super-resolution image can be reused more accurately and fully, and the invalid information in the super-resolution image can be filtered out or weakened, thereby further improving the quality of the generated super-resolution image.

[0077] In some optional implementations of this embodiment, the first frame image is any video frame except the first frame in the target image sequence, and the second frame image is a frame image previous to the first frame image in the target image sequence.

[0078] It should be understood that in the embodiment of the present application, the input unit 604 may include a graphics processor (GPU) 6041 and a microphone 6042, and the graphics processor 6041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 607 includes a touch panel 6071 and at least one of other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0079] The memory 609 can be used to store software programs and various data. The memory 609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 609 may include a volatile memory or a non-volatile memory, or the memory 609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0080] The processor 610 may include one or more processing units; optionally, the processor 610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 610.

[0081] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0082] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0083] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0084] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0085] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0086] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0088] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Obtain image features of the first frame image; Inputting the super-resolution image corresponding to the second frame image, the first frame image and the image features into a weight prediction model to obtain a first weight of each pixel in the first frame image and a second weight of each pixel in the super-resolution image, wherein the second frame image is an image frame before the first frame image; Based on the first weight, upsampling the first frame image to obtain an upsampled image; Based on the second weight, the up-sampled image and the super-resolution image are fused to generate a super-resolution image corresponding to the first frame image.

2. The method according to claim 1, characterized in that: The step of acquiring the image features of the first frame of image comprises: The image features of the first frame of image are obtained from the geometry buffer.

3. The method according to claim 1, characterized in that The step of upsampling the first frame image based on the first weight to obtain an upsampled image includes: Determine a projection point of each pixel in the up-sampled image to be generated in the first frame image; Based on the first weight, weighted summing of pixel values ​​of pixel points in the neighborhood of the projection point is performed to obtain a pixel value of each pixel point in the upsampled image; The up-sampled image is generated based on the pixel value of each pixel in the up-sampled image.

4. The method according to claim 1, characterized in that The step of fusing the upsampled image with the super-resolution image based on the second weight to generate a super-resolution image corresponding to the first frame image includes: Determining a third weight of each pixel in the upsampled image based on the second weight; Based on the third weight and the second weight, pixel values ​​of pixels at the same position in the upsampled image and the super-resolution image corresponding to the second frame image are weighted summed to obtain a super-resolution image corresponding to the first frame image.

5. The method according to any one of claims 1 to 4, characterized in that: The first frame image is any video frame except the first frame in the target image sequence, and the second frame image is a frame image previous to the first frame image in the target image sequence.

6. An image generating device, characterized in that: The device comprises: An acquisition unit, used for acquiring image features of a first frame of image; a processing unit, configured to input the super-resolution image corresponding to the second frame image, the first frame image, and the image features into a weight prediction model to obtain a first weight of each pixel in the first frame image and a second weight of each pixel in the super-resolution image, wherein the second frame image is an image frame before the first frame image; an up-sampling unit, configured to up-sample the first frame image based on the first weight to obtain an up-sampled image; A generating unit is used to fuse the up-sampled image with the super-resolution image based on the second weight to generate a super-resolution image corresponding to the first frame image.

7. The device according to claim 6, characterized in that The acquisition unit is further used to acquire image features of the first frame image from the geometric buffer.

8. The device according to claim 6, characterized in that The up-sampling unit is further used for: Determine a projection point of each pixel in the up-sampled image to be generated in the first frame image; Based on the first weight, weighted summing of pixel values ​​of pixel points in the neighborhood of the projection point is performed to obtain a pixel value of each pixel point in the upsampled image; The up-sampled image is generated based on the pixel value of each pixel in the up-sampled image.

9. The device according to claim 6, characterized in that The generating unit is further used for: Determining a third weight of each pixel in the upsampled image based on the second weight; Based on the third weight and the second weight, pixel values ​​of pixels at the same position in the upsampled image and the super-resolution image corresponding to the second frame image are weighted summed to obtain a super-resolution image corresponding to the first frame image.

10. The device according to any one of claims 6 to 9, characterized in that The first frame image is any video frame except the first frame in the target image sequence, and the second frame image is a frame image previous to the first frame image in the target image sequence.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image generation method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image generation method according to any one of claims 1 to 5 are implemented.

Images