Image super-resolution reconstruction method and related device

Abstract

The present application relates to the technical field of coding and decoding, and discloses an image super-resolution reconstruction method and a related device. The method provided by the embodiments of the present application comprises: acquiring input information of a neural network model, wherein the input information comprises a first reconstructed image with a first resolution and a reconstructed image set, the reconstructed image set comprises at least one second reconstructed image, the second reconstructed image is obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image are used for providing image features for the neural network model, and the resolution of the second reconstructed image is different from the resolution of the first reconstructed image; and on the basis of the neural network model and the input information, performing super-resolution reconstruction processing to obtain a third reconstructed image with a second resolution, wherein the second resolution is higher than the first resolution.

Description

Image super-resolution reconstruction method and related equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Chinese patent application No. 202311208490.X filed in China on September 18, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application belongs to the field of coding and decoding technology, and specifically relates to an image super-resolution reconstruction method and related equipment. Background Art

[0004] Related solutions use neural networks to perform super-resolution reconstruction of images to improve image clarity and detail. Specifically, neural networks combine auxiliary information (such as predicted images and quantization parameters) with low-resolution reconstructed images to achieve super-resolution reconstruction. However, this approach struggles to produce high-quality images.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide an image super-resolution reconstruction method and related equipment, which can solve the problem that existing super-resolution reconstruction solutions are difficult to obtain images with high image quality.

[0007] In a first aspect, a method for image super-resolution reconstruction is provided, which is performed by an electronic device, and the method includes:

[0008] Obtaining input information for a neural network model, the input information including a first reconstructed image at a first resolution and a reconstructed image set, the reconstructed image set including at least one second reconstructed image, the second reconstructed image being obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image being used to provide image features for the neural network model, and the resolution of the second reconstructed image being different from that of the first reconstructed image;

[0009] Super-resolution reconstruction processing is performed based on the neural network model and the input information to obtain a third reconstructed image with a second resolution, where the second resolution is higher than the first resolution.

[0010] In a second aspect, an image super-resolution reconstruction device is provided, comprising:

[0011] a first acquisition module, configured to acquire input information of a neural network model, the input information including a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set including at least one second reconstructed image, the second reconstructed image being obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image being used to provide image features for the neural network model, and the resolution of the second reconstructed image being different from that of the first reconstructed image;

[0012] The second acquisition module is used to perform super-resolution reconstruction based on the neural network model and the input information to obtain a third reconstructed image with a second resolution, where the second resolution is higher than the first resolution.

[0013] In a third aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0014] In a fourth aspect, an electronic device is provided, comprising a processor and a communication interface, wherein the processor is used to obtain input information of a neural network model, the input information comprising a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set comprising at least one second reconstructed image, the second reconstructed image being obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image being used to provide image features for the neural network model, and the resolution of the second reconstructed image is different from that of the first reconstructed image; super-resolution reconstruction is performed based on the neural network model and the input information to obtain a third reconstructed image of a second resolution, the second resolution being higher than the first resolution.

[0015] In a fifth aspect, an electronic device is provided, comprising: a memory configured to store video data, and a processing circuit configured to implement the steps of the method described in the first aspect.

[0016] In a sixth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a seventh aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the steps of the method described in the first aspect.

[0018] In an eighth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0019] In an embodiment of the present application, input information of a neural network model is obtained, the input information includes a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set includes at least one second reconstructed image, the second reconstructed image is obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image are used to provide image features for the neural network model, and the resolution of the second reconstructed image is different from the resolution of the first reconstructed image; super-resolution reconstruction is performed based on the neural network model and the input information to obtain a third reconstructed image of a second resolution, the second resolution being higher than the first resolution. In the above scheme, through the above-mentioned reconstructed image set, the neural network model can be provided with image feature information of more reconstructed images of different resolutions on the basis of the first reconstructed image of the first resolution, which is beneficial to improving the reasoning ability of the network model, so that a higher quality (higher resolution) third reconstructed image can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 is a schematic diagram showing the structure of a neural network model used in an embodiment of the present application;

[0021] FIG2 is a schematic diagram showing input information of an existing super-resolution reconstruction scheme;

[0022] FIG3 is a schematic diagram showing the processing of the reconstructed pixel image;

[0023] FIG4 is a schematic diagram showing a flow chart of an image super-resolution reconstruction method according to an embodiment of the present application;

[0024] FIG5 is a schematic diagram showing one of the input information of the image super-resolution reconstruction method according to an embodiment of the present application;

[0025] FIG6 is a second schematic diagram showing input information of the image super-resolution reconstruction method according to an embodiment of the present application;

[0026] FIG7 is a schematic diagram showing a module of an image super-resolution reconstruction device according to an embodiment of the present application;

[0027] FIG8 is a block diagram showing the structure of an electronic device according to an embodiment of the present application;

[0028] FIG9 shows a structural block diagram of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0030] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0031] In order to enable those skilled in the art to better understand the embodiments of the present application, the following description is first given.

[0032] 1. Reference Picture Resampling (RPR);

[0033] The increasing use of high-resolution videos poses a significant challenge to video transmission in bandwidth-constrained environments. Versatile Video Coding (VVC) employs a resampling encoding method to address this issue. RPR adaptively adjusts resolution based on network conditions. When bandwidth is low, it can downsample and encode low-resolution (LR) frames. When bandwidth improves, it can upsample and encode high-resolution (HR) original frames.

[0034] 2. Neural network super resolution (NNSR)

[0035] The neural network-based super-resolution solution refers to the use of neural networks to reconstruct images with super-resolution to improve the clarity and details of the image. This method can learn the characteristics of the image by training the neural network, thereby achieving super-resolution reconstruction of the image. Common neural networks include convolutional neural networks and recurrent neural networks. In the image super-resolution solution, neural networks can be used for single image super-resolution reconstruction, joint super-resolution reconstruction of multiple images, etc. This method has been widely used in image processing, medical image analysis, security monitoring and other fields, and is constantly developing and improving. In the neural network structure shown in Figure 1, in addition to the reconstructed low-resolution image (or low-resolution reconstructed image) Predicted low-resolution image (or predicted image) The slice quantization parameter (QP) and base QP are also input into the above network. The network structure uses several ordinary convolutions, parametric rectified linear units (PReLU) and N residual blocks to obtain effective features from the input. Then, a pixel shuffling layer is used to output a high-resolution residual. Finally, the RPR upsampled by the RPR mechanism is used to reconstruct the To generate high-resolution reconstruction

[0036] As shown in FIG2 , the information in the dotted box is auxiliary input (auxiliary information), and the reconstructed image after RPR serves as the input of the residual connection.

[0037] 3. Residual Connection;

[0038] Residual connections are a type of skip connection commonly used in deep neural networks. Their main idea is to add skip connections to the network, allowing for more direct information transfer. Specifically, residual connections can directly add the output of one layer to the input of a subsequent layer, enabling more direct information transfer within the network. This prevents information loss and distortion within the network, helps address issues like vanishing gradients, and improves network training efficiency and performance.

[0039] 4. Reorganize pixel images (Pixel Shuffle);

[0040] The PixelShuffle layer is a neural network layer used for image super-resolution reconstruction. It converts a low-resolution input image into a high-resolution image while maintaining the image's perceived quality. As shown in Figure 3, the main idea of ​​this layer is to increase image resolution through pixel reorganization. This layer does this by dividing the input image's channels into several small groups, then recombining the pixels in each group into a larger image block according to specific rules, thereby achieving resolution improvement. This method can effectively reduce the number of parameters and improve model training efficiency.

[0041] The image super-resolution reconstruction provided by the embodiments of the present application is described in detail below with reference to some embodiments and their application scenarios in conjunction with the accompanying drawings.

[0042] As shown in FIG4 , an embodiment of the present application provides an image super-resolution reconstruction method, comprising:

[0043] Step 401: Obtain input information of a neural network model, wherein the input information includes a first reconstructed image of a first resolution and a reconstructed image set, wherein the reconstructed image set includes at least one second reconstructed image, wherein the second reconstructed image is obtained by resampling the first reconstructed image, and the first reconstructed image and the second reconstructed image are used to provide image features for the neural network model, and the resolution of the second reconstructed image is different from that of the first reconstructed image.

[0044] The neural network model in the embodiment of the present application is a network model used for super-resolution reconstruction.

[0045] At least one second reconstructed image in the above reconstructed image set is obtained by performing different resampling processes on the first reconstructed image, for example, resampling processes are performed according to different scaling ratios.

[0046] Step 402: Perform super-resolution reconstruction based on the neural network model and the input information to obtain a third reconstructed image with a second resolution, where the second resolution is higher than the first resolution.

[0047] As shown in Figure 5, the first reconstructed image of the first resolution is a low-resolution reconstructed image, and the third reconstructed image is a high-resolution reconstructed image. The above-mentioned reconstructed image set can be used as auxiliary information. Based on the auxiliary information, the neural network model can provide the neural network model with more information of reconstructed images of different resolutions on the basis of the first reconstructed image of the first resolution, which is beneficial to improving the reasoning ability of the network model, thereby obtaining a high-resolution third reconstructed image.

[0048] In an embodiment of the present application, input information of a neural network model is obtained, the input information includes a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set includes at least one second reconstructed image, the second reconstructed image is obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image are used to provide image features for the neural network model, and the resolution of the second reconstructed image is different from the resolution of the first reconstructed image; super-resolution reconstruction is performed based on the neural network model and the input information to obtain a third reconstructed image of a second resolution, the second resolution being higher than the first resolution. In the above scheme, through the above-mentioned reconstructed image set, the neural network model can be provided with image feature information of more reconstructed images of different resolutions on the basis of the first reconstructed image of the first resolution, which is beneficial to improving the reasoning ability of the network model, so that a higher quality (higher resolution) third reconstructed image can be obtained.

[0049] Optionally, performing super-resolution reconstruction based on the neural network model and the input information to obtain a third reconstructed image with a second resolution includes:

[0050] performing convolution processing on the first reconstructed image and the at least one second reconstructed image respectively to obtain a convolved first reconstructed image and a convolved second reconstructed image;

[0051] A third reconstructed image with a second resolution is obtained according to the convolved first reconstructed image and the convolved second reconstructed image.

[0052] As an optional implementation method, based on the neural network structure model shown in Figure 1, the first reconstructed image and at least one second reconstructed image can be convolved respectively to obtain a first reconstructed image after convolution and a second reconstructed image after convolution; then the first reconstructed image after convolution and the second reconstructed image after convolution are feature fused to obtain a fused feature image; the fused feature image is processed using the residual module and the convolution module to obtain a residual image; finally, based on the residual image, the third reconstructed image of the second resolution mentioned above is obtained.

[0053] Optionally, as shown in FIG6 , the input information further includes a fourth reconstructed image, where the fourth reconstructed image is obtained by resampling the first reconstructed image; and the resolution of the fourth reconstructed image is the same as that of the third reconstructed image.

[0054] The step of obtaining a third reconstructed image with a second resolution based on the convolved first reconstructed image and the convolved second reconstructed image comprises:

[0055] Obtaining a residual image according to the convolved first reconstructed image and the convolved second reconstructed image, wherein the resolution of the residual image is the same as the resolution of the fourth reconstructed image;

[0056] The third reconstructed image is obtained according to the residual image and the fourth reconstructed image.

[0057] Exemplarily, the first reconstructed image after convolution and the second reconstructed image after convolution are subjected to feature fusion to obtain a fused feature image; the fused feature image is then processed using a residual module and a convolution module to obtain the residual image. After obtaining the residual image, the third reconstructed image can be obtained by:

[0058] Mode 1: superimposing the residual image and the fourth reconstructed image to obtain the third reconstructed image;

[0059] Mode 2: After superimposing the residual image and the fourth reconstructed image, a target image is obtained, and convolution processing is performed on the target image to obtain the third reconstructed image.

[0060] In the above method 1, the residual image is used as the connection of the residual input, so that the information in the network can be transmitted more directly, avoiding the loss and deformation of information in the network. It also helps to solve problems such as gradient disappearance and improves the network training efficiency and model inference performance.

[0061] In the above-mentioned method 2, convolution processing is added on the basis of method 1, so that a third reconstructed image with higher quality can be obtained.

[0062] In addition, before adopting the above-mentioned method 1 or method 2 to obtain the third reconstructed image, the fourth reconstructed image may be convolved to obtain the above-mentioned third reconstructed image based on the convolved fourth reconstructed image and the residual image.

[0063] Optionally, the resolution of the fourth reconstructed image may be the same as or different from the resolution of the second reconstructed image.

[0064] In an embodiment of the present application, the fourth reconstructed image is connected as the residual input of the neural network model.

[0065] Optionally, the resampling process includes at least one of the following:

[0066] Upsampling processing;

[0067] Downsampling processing;

[0068] Upsampling and pixel reconstructing.

[0069] In an embodiment of the present application, the second reconstructed image and the fourth reconstructed image may adopt the same resampling processing method, such as both adopt upsampling processing or both adopt downsampling processing, or different resampling processing methods may be adopted. For example, the second reconstructed image adopts an upsampling processing method, and the fourth reconstructed image adopts a downsampling processing method; or, the second reconstructed image adopts upsampling processing, and the fourth reconstructed image adopts upsampling processing and pixel reorganization (PixelShuffle) processing, that is, the fourth reconstructed image can be 4 sub-images obtained by the PixelShuffle method; or, the second reconstructed image adopts upsampling processing and PixelShuffle processing, that is, the second reconstructed image includes 4 sub-images, and the fourth reconstructed image adopts upsampling processing.

[0070] Optionally, in the embodiment of the present application, up-sampling processing may be performed through the above-mentioned RPR mechanism, and / or down-sampling processing may be performed through a discrete cosine transform difference filter method.

[0071] Optionally, as shown in FIG5 or FIG6 , the input information further includes at least one of the following:

[0072] a predicted image corresponding to the first reconstructed image;

[0073] Quantization parameter.

[0074] The quantization parameter includes at least one of a frame-level quantization parameter (also described as a slice quantization parameter) and a basic quantization parameter. The predicted image and / or quantization parameter here can be used as auxiliary information in the above-mentioned input information.

[0075] As an optional implementation method, based on the neural network structure model shown in Figure 1, the first reconstructed image and at least one second reconstructed image can be convolved respectively to obtain a convolved first reconstructed image and a convolved second reconstructed image; then the first reconstructed image and the second reconstructed image, the predicted image and the quantization parameter are feature fused to obtain a fused feature image; the fused feature image is processed using the residual module and the convolution module to obtain a residual image; finally, based on the residual image and the fourth reconstructed image, the third reconstructed image of the second resolution mentioned above is obtained.

[0076] In an embodiment of the present application, input information of a neural network model is obtained, the input information includes a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set includes at least one second reconstructed image, the second reconstructed image is obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image are used to provide image features for the neural network model, and the resolution of the second reconstructed image is different from the resolution of the first reconstructed image; super-resolution reconstruction is performed based on the neural network model and the input information to obtain a third reconstructed image of a second resolution, the second resolution being higher than the first resolution. In the above scheme, through the above-mentioned reconstructed image set, the neural network model can be provided with image feature information of more reconstructed images of different resolutions on the basis of the first reconstructed image of the first resolution, which is beneficial to improving the reasoning ability of the network model, so that a higher quality (higher resolution) third reconstructed image can be obtained.

[0077] The image super-resolution reconstruction method provided in the embodiment of the present application can be executed by an image super-resolution reconstruction device. In the embodiment of the present application, the image super-resolution reconstruction device is used as an example to illustrate the image super-resolution reconstruction method provided in the embodiment of the present application.

[0078] As shown in FIG7 , the embodiment of the present application further provides an image super-resolution reconstruction apparatus 700 , comprising:

[0079] A first acquisition module 701 is configured to acquire input information of a neural network model, the input information including a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set including at least one second reconstructed image, the second reconstructed image being obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image being used to provide image features for the neural network model, and the resolution of the second reconstructed image being different from that of the first reconstructed image;

[0080] The second acquisition module 702 is used to perform super-resolution reconstruction processing based on the neural network model and the input information to obtain a third reconstructed image with a second resolution, where the second resolution is higher than the first resolution.

[0081] Optionally, the second acquisition module includes:

[0082] a first acquisition submodule, configured to perform convolution processing on the first reconstructed image and the at least one second reconstructed image, respectively, to obtain a convolved first reconstructed image and a convolved second reconstructed image;

[0083] The second acquisition submodule is used to obtain a third reconstructed image with a second resolution according to the convolved first reconstructed image and the convolved second reconstructed image.

[0084] Optionally, the input information further includes a fourth reconstructed image, where the fourth reconstructed image is obtained by resampling the first reconstructed image; a resolution of the fourth reconstructed image is the same as a resolution of the third reconstructed image;

[0085] The second acquisition submodule includes:

[0086] a first acquiring unit, configured to obtain a residual image according to the convolved first reconstructed image and the convolved second reconstructed image, wherein the resolution of the residual image is the same as the resolution of the fourth reconstructed image;

[0087] The second acquisition unit is configured to obtain the third reconstructed image according to the residual image and the fourth reconstructed image.

[0088] Optionally, the resampling process includes at least one of the following:

[0089] Upsampling processing;

[0090] Downsampling processing;

[0091] Upsampling and pixel reconstructing.

[0092] Optionally, the input information further includes at least one of the following:

[0093] a predicted image corresponding to the first reconstructed image;

[0094] Quantization parameter.

[0095] The device of the embodiment of the present application obtains input information of a neural network model, wherein the input information includes a first reconstructed image of a first resolution and a reconstructed image set, wherein the reconstructed image set includes at least one second reconstructed image, wherein the second reconstructed image is obtained by resampling the first reconstructed image, and the first reconstructed image and the second reconstructed image are used to provide image features for the neural network model, and the resolution of the second reconstructed image is different from the resolution of the first reconstructed image; a super-resolution reconstruction process is performed based on the neural network model and the input information to obtain a third reconstructed image of a second resolution, wherein the second resolution is higher than the first resolution. In the above scheme, through the above-mentioned reconstructed image set, the image feature information of reconstructed images of more different resolutions can be provided to the neural network model on the basis of the first reconstructed image of the first resolution, which is beneficial to improving the reasoning ability of the network model, thereby obtaining a third reconstructed image of higher quality (higher resolution).

[0096] The image super-resolution reconstruction device provided in the embodiment of the present application can implement the various processes implemented in the method embodiments of Figures 4 to 6 and achieve the same technical effects. To avoid repetition, they will not be described here.

[0097] As shown in Figure 8, an embodiment of the present application also provides an electronic device 800, including a processor 801 and a memory 802, and the memory 802 stores a program or instruction that can be run on the processor 801. When the program or instruction is executed by the processor 801, the various steps of the above-mentioned image super-resolution reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0098] An embodiment of the present application further provides an electronic device, comprising: a memory configured to store video data; and a processing circuit configured to implement the various steps of the above-mentioned image super-resolution reconstruction method embodiment.

[0099] The present application also provides an electronic device including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the steps in the method embodiment shown in FIG4 . This device embodiment corresponds to the above-described method embodiment, and each implementation process and implementation method of the above-described method embodiment are applicable to this electronic device embodiment and can achieve the same technical effects.

[0100] The electronic device may be a terminal, or may be other devices other than a terminal, such as a server, a network attached storage (NAS), etc.

[0101] Among them, the terminal can be a mobile phone, tablet personal computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile Internet device (MID), augmented reality (AR), virtual reality (VR) equipment, mixed reality (MR) equipment, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipborne equipment, pedestrian user equipment (PUE), smart home (home appliances with wireless communication function, such as refrigerator, TV, washing machine or furniture, etc.), game console, personal computer (PC), ATM or self-service machine and other terminal-side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, vehicle-mounted devices can also be called vehicle-mounted terminals, vehicle-mounted controllers, vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, or vehicle-mounted units, etc. It should be noted that the specific type of terminal is not limited in the embodiments of this application.

[0102] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), or cloud computing services based on big data and artificial intelligence platforms.

[0103] Taking an electronic device as a terminal as an example, FIG9 is a schematic diagram of the hardware structure of a terminal implementing an embodiment of the present application.

[0104] The terminal 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909 and at least some of the components of the processor 910.

[0105] Those skilled in the art will appreciate that the terminal 900 may also include a power supply (such as a battery) to power various components. The power supply may be logically connected to the processor 910 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The terminal structure shown in FIG9 does not limit the terminal. The terminal may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.

[0106] It should be understood that in an embodiment of the present application, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The graphics processor 9041 processes image data of a static picture or video obtained by an image acquisition device (such as a camera) in a video acquisition mode or an image acquisition mode, or may process the obtained point cloud data. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0107] In the embodiment of the present application, after receiving downlink data from a network-side device, the RF unit 901 may transmit the data to the processor 910 for processing. Furthermore, the RF unit 901 may send uplink data to the network-side device. Typically, the RF unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and the like.

[0108] The memory 909 can be used to store software programs or instructions and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 909 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0109] Processor 910 may include one or more processing units. Optionally, processor 910 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 910.

[0110] Among them, the processor 910 is used to obtain input information of the neural network model, the input information includes a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set includes at least one second reconstructed image, the second reconstructed image is obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image are used to provide image features for the neural network model, and the resolution of the second reconstructed image is different from the resolution of the first reconstructed image; super-resolution reconstruction is performed based on the neural network model and the input information to obtain a third reconstructed image of a second resolution, and the second resolution is higher than the first resolution.

[0111] In an embodiment of the present application, input information of a neural network model is obtained, the input information includes a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set includes at least one second reconstructed image, the second reconstructed image is obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image are used to provide image features for the neural network model, and the resolution of the second reconstructed image is different from the resolution of the first reconstructed image; super-resolution reconstruction is performed based on the neural network model and the input information to obtain a third reconstructed image of a second resolution, the second resolution being higher than the first resolution. In the above scheme, through the above-mentioned reconstructed image set, the neural network model can be provided with image feature information of more reconstructed images of different resolutions on the basis of the first reconstructed image of the first resolution, which is beneficial to improving the reasoning ability of the network model, so that a higher quality (higher resolution) third reconstructed image can be obtained.

[0112] It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of the method embodiment and achieve the same or corresponding technical effects. To avoid repetition, it will not be described here.

[0113] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image super-resolution reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0114] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as ROM, RAM, a magnetic disk, or an optical disk. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0115] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image super-resolution reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0116] It should be understood that the chip mentioned in the embodiments of the present application may include a system-level chip (also known as a system chip, a chip system or a system-on-chip chip), and may also include an independent display chip, etc.

[0117] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-mentioned image super-resolution reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it is not repeated here.

[0118] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0119] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0120] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. A method for image super-resolution reconstruction, performed by an electronic device, the method comprising: Acquire input information of a neural network model, the input information comprising a first reconstructed image of a first resolution and a reconstructed image set, the reconstructed image set comprising at least one second reconstructed image, the second reconstructed image being obtained by resampling the first reconstructed image, the first reconstructed image and the second reconstructed image being used to provide image features for the neural network model, and the resolution of the second reconstructed image being different from that of the first reconstructed image; Super-resolution reconstruction processing is performed based on the neural network model and the input information to obtain a third reconstructed image with a second resolution, where the second resolution is higher than the first resolution.

2. The method according to claim 1, wherein: The performing super-resolution reconstruction processing based on the neural network model and the input information to obtain a third reconstructed image with a second resolution includes: Performing convolution processing on the first reconstructed image and the at least one second reconstructed image respectively to obtain a convolved first reconstructed image and a convolved second reconstructed image; A third reconstructed image with a second resolution is obtained according to the convolved first reconstructed image and the convolved second reconstructed image.

3. The method according to claim 2, wherein: The input information further includes a fourth reconstructed image, where the fourth reconstructed image is obtained by resampling the first reconstructed image; the resolution of the fourth reconstructed image is the same as that of the third reconstructed image; The step of obtaining a third reconstructed image with a second resolution according to the convolved first reconstructed image and the convolved second reconstructed image comprises: Obtaining a residual image according to the first reconstructed image after convolution and the second reconstructed image after convolution, wherein the resolution of the residual image is the same as the resolution of the fourth reconstructed image; The third reconstructed image is obtained according to the residual image and the fourth reconstructed image.

4. The method according to any one of claims 1 to 3, wherein: The resampling process includes at least one of the following: Upsampling processing; Downsampling processing; Upsampling and pixel reconstructing.

5. The method according to any one of claims 1 to 4, wherein: The input information also includes at least one of the following: A predicted image corresponding to the first reconstructed image; Quantization parameter.

6. An image super-resolution reconstruction device, comprising: The first acquisition module is used to obtain input information of the neural network model, wherein the input information includes a first resolution a first reconstructed image and a reconstructed image set, wherein the reconstructed image set includes at least one second reconstructed image, wherein the second reconstructed image is obtained by resampling the first reconstructed image, wherein the first reconstructed image and the second reconstructed image are used to provide image features for the neural network model, and a resolution of the second reconstructed image is different from a resolution of the first reconstructed image; The second acquisition module is used to perform super-resolution reconstruction processing based on the neural network model and the input information to obtain a third reconstructed image with a second resolution, where the second resolution is higher than the first resolution.

7. The device according to claim 6, wherein: The second acquisition module includes: A first acquisition submodule, used for performing convolution processing on the first reconstructed image and the at least one second reconstructed image, respectively, to obtain a convolved first reconstructed image and a convolved second reconstructed image; The second acquisition submodule is used to obtain a third reconstructed image with a second resolution according to the convolved first reconstructed image and the convolved second reconstructed image.

8. The device according to claim 7, wherein: The input information further includes a fourth reconstructed image, where the fourth reconstructed image is obtained by resampling the first reconstructed image; the resolution of the fourth reconstructed image is the same as that of the third reconstructed image; The second acquisition submodule includes: A first acquisition unit, configured to obtain a residual image according to the first reconstructed image after convolution and the second reconstructed image after convolution, wherein the resolution of the residual image is the same as the resolution of the fourth reconstructed image; The second acquisition unit is used to obtain the third reconstructed image according to the residual image and the fourth reconstructed image.

9. The device according to any one of claims 6 to 8, wherein: The resampling process includes at least one of the following: Upsampling processing; Downsampling processing; Upsampling and pixel reconstructing.

10. The device according to any one of claims 6 to 9, wherein: The input information also includes at least one of the following: A predicted image corresponding to the first reconstructed image; Quantization parameter.

11. An electronic device comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image super-resolution reconstruction method according to any one of claims 1 to 5 are implemented.

12. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the image super-resolution reconstruction method according to any one of claims 1 to 5.

13. A chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the image super-resolution reconstruction method according to any one of claims 1 to 5.

Images