Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining the region type of each image region of a current image frame, and enabling the region type of any image region to be used for indicating the pixel change degree between the image region and a corresponding reference region; according to the area category of each image area, the super-division mode of the corresponding image area is determined, and the calculation complexity of the super-division mode of any image area is in direct proportion to the pixel change degree indicated by the area category of the corresponding image area; performing super-division processing on the corresponding image area by adopting a super-division mode of each image area to obtain a super-division result of each image area; integrating the super-division results of the image areas to obtain a super-division image corresponding to the current image frame; according to the invention, image super-division effects such as image super-division efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, specifically to the field of image processing technology, and more particularly to an image processing method, apparatus, device, and storage medium. Background Art

[0002] With the continuous advancement of image processing technology, the demand for image super-resolution (i.e., the need to perform super-resolution processing on images) is increasing. Super-resolution, short for super-resolution, refers to improving the resolution of an original image through hardware or software methods. The process of reconstructing a high-resolution image by performing a series of processing on a low-resolution image is the process of super-resolution. Currently, how to perform super-resolution processing on images to improve their performance has become a hot topic of research. Summary of the Invention

[0003] The embodiments of the present application provide an image processing method, apparatus, device, and storage medium, which can improve image super-resolution effects such as image super-resolution efficiency and accuracy.

[0004] In one aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Acquire multiple image regions of a current image frame, where the current image frame is an image frame to be super-resolved in an image frame sequence;

[0006] Obtaining a region category for each image region, where the region category of any image region indicates a degree of pixel variation between the image region and a corresponding reference region; the reference region corresponding to any image region is a region in a reference image frame that matches the corresponding image region, where the reference image frame is an image frame in the image frame sequence;

[0007] Determining a super-resolution method for each image region according to the region category of each image region; different super-resolution methods have different computational complexities, and the computational complexity of the super-resolution method for any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region;

[0008] The super-resolution method of each image region is adopted to perform super-resolution processing on the corresponding image region to obtain the super-resolution results of each image region; and the super-resolution results of each image region are integrated to obtain the super-resolution image corresponding to the current image frame.

[0009] On the other hand, an embodiment of the present application provides an image processing device, comprising:

[0010] an acquisition unit, configured to acquire a plurality of image regions of a current image frame, wherein the current image frame is an image frame to be super-resolved in an image frame sequence;

[0011] The acquisition unit is further configured to acquire a region category of each image region, where the region category of any image region indicates a degree of pixel change between the image region and a corresponding reference region; the reference region corresponding to any image region is a region in a reference image frame that matches the corresponding image region, where the reference image frame is an image frame in the image frame sequence;

[0012] a processing unit, configured to determine a super-resolution method for each image region according to the region category of each image region; different super-resolution methods have different computational complexities, and the computational complexity of the super-resolution method for any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region;

[0013] The processing unit is further used to adopt the super-resolution method of each image area to perform super-resolution processing on the corresponding image area to obtain the super-resolution results of each image area; and integrate the super-resolution results of each image area to obtain the super-resolution image corresponding to the current image frame.

[0014] In one embodiment, the multiple image regions are obtained by decoding a code stream of a current image frame, where the code stream includes coded data of multiple image blocks and coding information of each image block. The coding information of one image block is used to decode the coded data of the corresponding image block to obtain an image region of the current image frame. Accordingly, when the acquisition unit is used to acquire the region category of each image region, it can be specifically configured to:

[0015] Obtaining target coding parameters for the corresponding image blocks from coding information for the image blocks corresponding to each image region; the target coding parameters for any image block are used to indicate a difference in pixel attributes between the image block and a corresponding reference block; the reference block corresponding to any image block is a block in the reference image frame that matches the corresponding image block, with one block corresponding to one region;

[0016] Based on the target coding parameters of the image blocks corresponding to each image region, category identification is performed on the corresponding image region to obtain the region category of each image region.

[0017] In another embodiment, the multiple image regions are obtained by calling a hardware decoder to decode the code stream. Before obtaining the region category of each image region, the processing unit may further be configured to:

[0018] Call the software decoder to decode the code stream;

[0019] If, during the decoding process of the software decoder, it is detected that the coding information of each image block has been decoded, the software decoder is controlled to end the decoding process and output the coding information of each image block.

[0020] In another embodiment, the target coding parameters of any image block include: a motion vector and residual data; the motion vector is used to indicate: a difference in pixel position between the image block and a corresponding reference block; the residual data is used to indicate: a difference between a pixel value of each pixel point in the image block and a pixel value of a corresponding pixel point in the corresponding reference block; accordingly, when the acquisition unit is used to perform category identification on the corresponding image region based on the target coding parameters of the image block corresponding to each image region to obtain the region category of each image region, it can be specifically used to:

[0021] The image block corresponding to the qth image region is taken as the qth image block, where q is a positive integer and is less than or equal to the number of image regions;

[0022] When the motion vector and the residual data of the qth image block are both zero, determining that the region category of the qth image region is an image invariant region; the image invariant region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is zero;

[0023] When at least one of the motion vector and residual data of the qth image block is not zero, the region category of the qth image region is determined to be an image change region; the image change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero.

[0024] In another embodiment, the image change region includes an image weak change region and an image strong change region; wherein the image weak change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero and less than a preset degree; and the image strong change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than or equal to a preset degree. Accordingly, when the acquisition unit is used to determine that the region category of the qth image region is an image change region when at least one of the motion vector and residual data of the qth image block is non-zero, the acquisition unit may be specifically used to:

[0025] When at least one of the motion vector and residual data of the qth image block is not zero, if the at least one data is residual data and the residual data is less than a preset threshold, the area category of the qth image area is determined to be an image weak change area; if the at least one data includes a motion vector, the area category of the qth image area is determined to be an image strong change area.

[0026] In another embodiment, when the processing unit is configured to perform super-resolution processing on the corresponding image region using the super-resolution method of each image region to obtain the super-resolution result of each image region, it can be specifically configured to:

[0027] If the super-resolution method of the qth image region is a method of reusing the super-resolution result of the reference region, obtaining the super-resolution result of the reference region corresponding to the qth image region, and using the obtained super-resolution result as the super-resolution result of the qth image region;

[0028] If the super-resolution method of the qth image region is an interpolation super-resolution method, an interpolation algorithm is used to perform interpolation processing based on the qth image region to generate a super-resolution result of the qth image region;

[0029] If the super-resolution method of the qth image region is the model super-resolution method, calling the image super-resolution model to perform super-resolution processing on the qth image region to obtain a super-resolution result of the qth image region;

[0030] Wherein, q is a positive integer and is less than or equal to the number of image regions.

[0031] In another embodiment, when the processing unit is configured to perform interpolation processing based on the qth image region using an interpolation algorithm to generate a super-resolution result of the qth image region, it may be specifically configured to:

[0032] Performing interpolation processing on the qth image region using an interpolation algorithm to obtain a super-resolution result of the qth image region;

[0033] Alternatively, residual information between the qth image region and the corresponding reference region is obtained, an interpolation algorithm is used to interpolate the obtained residual information, and the interpolated residual information and the super-resolution result of the reference region corresponding to the qth image region are superimposed to obtain the super-resolution result of the qth image region.

[0034] In another embodiment, the processing unit may be further configured to generate an image super-resolution model, and may be configured to perform the following steps:

[0035] Constructing an initial super-resolution model and obtaining training data; the training data includes: a sample image of a first resolution and a sample image of a second resolution rendered based on the same picture, wherein the first resolution is smaller than the second resolution;

[0036] Calling the initial super-resolution model to perform super-resolution processing on the sample image of the first resolution to obtain a super-resolution image of the second resolution;

[0037] Using the loss function of the initial super-resolution model, performing a loss value calculation based on the sample image of the second resolution and the super-resolution image of the second resolution to obtain a model loss value;

[0038] The model parameters of the initial super-resolution model are optimized in a direction of reducing the model loss value to obtain an image super-resolution model.

[0039] In another embodiment, when the processing unit is used to obtain training data, it can be specifically used to:

[0040] Encode two video streams based on the sample picture data, wherein the two video streams include a video stream with a first resolution and a video stream with a second resolution;

[0041] Transmitting the video stream of the first resolution to a test device via a network, causing the test device to decode the video stream of the first resolution, and after obtaining a first decoding result, transmitting the first decoding result back; the first decoding result includes: at least one sample image of the first resolution;

[0042] Decoding the video stream at the second resolution to obtain a second decoding result; the second decoding result includes: at least one sample image at the second resolution;

[0043] Performing picture alignment on the sample image in the first decoding result and the sample image in the second decoding result, and acquiring training data from the picture alignment result.

[0044] In another embodiment, when the processing unit is used to optimize the model parameters of the initial super-resolution model in a direction of reducing the model loss value to obtain the image super-resolution model, it can be specifically used to:

[0045] Optimizing the model parameters of the initial super-resolution model in a direction of reducing the model loss value to obtain an optimized initial super-resolution model;

[0046] If the optimized initial super-resolution model reaches convergence, the optimized initial super-resolution model is equivalently transformed using the re-parameterization technology to obtain an image super-resolution model; wherein the complexity of the model structure of the image super-resolution model is less than the complexity of the optimized initial super-resolution model, and the model parameters of the image super-resolution model are equivalent to the model parameters of the optimized initial super-resolution model.

[0047] In another aspect, an embodiment of the present application provides a computer device, the computer device including an input interface and an output interface, and the computer device further including:

[0048] processors and computer storage media;

[0049] The processor is suitable for implementing one or more instructions, the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by the processor and executing the above-mentioned image processing method.

[0050] On the other hand, an embodiment of the present application provides a computer storage medium, which stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the above-mentioned image processing method.

[0051] On the other hand, an embodiment of the present application provides a computer program product, which includes one or more instructions; when the one or more instructions in the computer program product are executed by a processor, the above-mentioned image processing method is implemented.

[0052] The embodiment of the present application can obtain multiple image regions of the current image frame and determine the super-resolution method of the corresponding image region according to the region category of each image region, thereby using the super-resolution method of each image region to perform super-resolution processing on the corresponding image region, and integrating the super-resolution results of each image region to achieve super-resolution processing of the current image frame and obtain a super-resolution image corresponding to the current image frame. Since the region category of any image region is used to indicate the degree of pixel change between the image region and the corresponding reference region, and the computational complexity of the super-resolution method of any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region, this can ensure that the super-resolution method assigned to any image region and the corresponding image region have a high degree of adaptability, that is, the super-resolution method assigned to any image region has the ability to efficiently and accurately perform super-resolution processing on the corresponding image region, so that when the super-resolution method of each image region is used to perform super-resolution processing on the corresponding image region, the super-resolution efficiency and accuracy of each image region can be improved, thereby improving the image super-resolution efficiency and accuracy of the current image frame and other image super-resolution effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1a This is a schematic diagram of image super-resolution during the image encoding stage provided by an embodiment of the present application;

[0055] Figure 1b This is a schematic diagram of performing image super-resolution at the image decoding stage provided by an embodiment of the present application;

[0056] Figure 2 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0057] Figure 3ais a schematic diagram of interpolation super-resolution of an image region provided by an embodiment of the present application;

[0058] Figure 3b This is a schematic diagram of obtaining training data provided by an embodiment of the present application;

[0059] Figure 4a This is a schematic diagram of an equivalent conversion of a feature extraction module provided by an embodiment of the present application;

[0060] Figure 4b Schematic diagram of the structure of an image super-resolution model provided in an embodiment of the present application;

[0061] Figure 5a is a schematic diagram of a game screen provided in an embodiment of the present application;

[0062] Figure 5b This is a schematic diagram of image super-resolution in a cloud gaming scenario provided by an embodiment of the present application;

[0063] Figure 5c Schematic diagram of various modules involved in a video super-resolution method provided in an embodiment of the present application;

[0064] Figure 5d This is a flow chart of a video super-resolution method provided in an embodiment of the present application;

[0065] Figure 5e This is a test effect diagram provided by an embodiment of the present application;

[0066] Figure 6 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0067] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0069] The embodiment of the present application proposes an image processing method based on super-resolution technology. The image processing method can be used to perform super-resolution processing on any image frame in an image frame sequence, thereby improving the super-resolution efficiency and accuracy of the corresponding image frame and other image super-resolution effects. Herein, an image frame sequence refers to a sequence composed of multiple image frames displayed in sequence, and the so-called image refers to a picture with visual effects on the display screen. Super-resolution technology is a computer vision technology that reconstructs one or more low-resolution images into images with larger size, more pixels and higher quality. It can be applied to various fields such as medical imaging and Internet entertainment. The so-called computer vision technology is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further performing graphic processing to make computer processing into a technology that is more suitable for human eye observation or transmission to instruments for detection.

[0070] Specifically, the general principle of the image processing method is as follows: for the current image frame in the image frame sequence (i.e., the image frame to be super-resolved), multiple image regions in the current image frame can be determined, and the degree of pixel change between each image region and the corresponding reference region is considered separately, and different super-resolution methods are adaptively used to super-resolve the corresponding image regions, thereby integrating the super-resolution results of each image region to obtain the super-resolved image corresponding to the current image frame, so as to complete the reconstruction process of the current image frame from low resolution to high resolution, and improve the super-resolution efficiency and accuracy of the current image frame. Among them, the reference region corresponding to any image region refers to: the region in the reference image frame (i.e., an image frame in the image frame sequence) that matches the corresponding image region; different super-resolution methods have different computational complexities, and the computational complexity of the super-resolution method adopted for any image region is proportional to the degree of pixel change corresponding to the corresponding image region.

[0071] In a specific implementation, the image processing method proposed in the embodiment of the present application can be applied in the image coding stage. In this case, after the image coding device obtains the low-resolution (i.e., resolution less than a threshold) image frame to be encoded, it can use the image processing method to perform super-resolution processing on the corresponding image frame to obtain a high-resolution (i.e., resolution greater than a threshold) image frame, thereby calling the encoder to use the mainstream video coding standard to encode the high-resolution image frame to obtain a code stream, and transmit the code stream to the image display device through the transmission medium, so that the image display device calls the decoder to decode the received code stream, thereby displaying the high-resolution image frame based on the decoding result, such as Figure 1a As shown in FIG. 1 , it can be seen that by performing super-resolution reconstruction on the image encoding device side, the image display device can directly render and display high-resolution image frames after performing the decoding operation, which can improve the timeliness of image display and reduce the performance requirements of the image display device.

[0072] In another specific implementation, the image processing method proposed in the embodiment of the present application can be applied in the image decoding stage. In this case, after obtaining the low-resolution image frame to be encoded, the image encoding device can call the encoder to use the mainstream video encoding standard to encode the corresponding image frame to obtain a code stream, and transmit the code stream to the image display device through the transmission medium, so that the image display device calls the decoder to decode the received code stream to obtain the corresponding image frame, and then uses the image processing method to perform super-resolution processing on the corresponding image frame to improve the resolution of the corresponding image frame, thereby obtaining a high-resolution image frame, thereby displaying the high-resolution image frame, such as Figure 1b As shown in FIG. 1 , it can be seen that by transmitting low-resolution images between the image encoding device and the image display device and performing super-resolution reconstruction on the image display device side, the resources required for image transmission (such as traffic costs) can be effectively reduced and the efficiency of image transmission can be improved.

[0073] The aforementioned video coding standards may include, but are not limited to, H.264, VVC (Versatile Video Coding), AVS3 (Audio Video Coding Standard 3), and the like. The general process for an encoder to encode any image using a video coding standard is as follows: dividing the corresponding image into multiple image blocks, encoding each image block based on an intra-frame coding method or an inter-frame coding method to obtain coded data for each image block, and generating a code stream for the corresponding image frame using the coded data and coding information of each image block. Intra-frame coding refers to independently encoding each image block, while inter-frame coding utilizes the temporal correlation between frames in a sequence of image frames to find a matching image block in the previous frame as a reference block, thereby encoding the current image block based on the difference between the current image block and the reference block. An image block is the basic unit of video coding. If the video coding standard used is H.264, an image block can also be called a macroblock. An image block corresponds to an image region in an image frame and can include a preset number of pixels (e.g., 16×16). The encoding information of any image block includes the information required to decode the encoded data of the corresponding image block. Accordingly, the decoder decodes the received bitstream in the following manner: decoding the received bitstream to obtain encoding information and encoded data for each image block, then sequentially decoding the encoded data of the corresponding image block based on the encoding information of each image block to obtain multiple image regions, which are then spliced ​​together to obtain the corresponding image.

[0074] In addition, ① the image encoding device mentioned above refers to a device with a built-in encoder, which can be a server or a terminal; the server here can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, etc.; the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, etc. ② The image display device mentioned above refers to a device with a built-in decoder and image display function, which can be, for example, a smartphone, tablet computer, laptop computer, smart watch, or a device that can project images onto a screen for playback, such as a projector. ③ The transmission medium mentioned above refers to the space or entity through which data is transmitted, mainly used to transmit data between the image encoding device and the image display device; it can specifically include but is not limited to: network media such as mobile networks, wireless networks, wired networks, or removable hardware media with read and write functions such as USB flash drives (Universal Serial Bus) and mobile hard drives, etc.

[0075] Based on the above description, the following Figure 2 The method flow chart shown in FIG. 1 illustrates the specific implementation of the image processing method proposed in the embodiment of the present application. In the embodiment of the present application, the image processing method can be executed by a computer device, which can be the image encoding device mentioned above or the image display device mentioned above, without limitation. Figure 2 As shown, the image processing method may include the following steps S201-S205:

[0076] S201, obtaining multiple image regions of a current image frame.

[0077] Among them, the current image frame is the image frame to be super-resolved in the image frame sequence, and each image frame in the image frame sequence is super-resolved in sequence. For example, the image frame sequence includes: image frame a, image frame b, image frame c in sequence; if image frame a in the image frame sequence has been super-resolved when executing step S201, then in this case, the current image frame is the image frame b to be super-resolved. It should be noted that the image frame sequence mentioned in the embodiment of the present application can be any pre-produced video, such as a film and television drama video, a short video (a video with a playback time less than a time threshold), etc.; or, the image frame sequence mentioned in the embodiment of the present application can also be a real-time generated image set, such as an image set composed of each game screen generated in real time in a game scene of any game, an image set composed of each video screen generated in real time in a video conferencing scene, an image set composed of each live screen generated in real time in a live broadcast scene, etc. Further, the game mentioned here can be an ordinary game (i.e., a game running on a terminal device) or a cloud game (i.e., a game running on a cloud game server (referred to as a cloud server)), and this is not limited.

[0078] In one specific implementation, if the computer device is an image display device, that is, step S201 is performed during the image decoding phase, then when executing step S201, the computer device may obtain a code stream for the current image frame, where the code stream includes encoded data for multiple image blocks and encoding information for each image block. The multiple image blocks are obtained by dividing the current image frame during encoding, and the encoding information for each image block is used to decode the encoded data for the corresponding image block to obtain an image region of the current image frame. After obtaining the code stream, the computer device may decode the code stream to obtain multiple image regions of the current image frame. Specifically, the computer device may invoke a hardware decoder or a software decoder to decode the code stream to obtain multiple image regions of the current image frame. A hardware decoder refers to a decoder that uses hardware for decoding, and its output is the multiple decoded image regions; a software decoder refers to a decoder that uses program code for decoding, and its output may include the multiple decoded image regions and other information obtained during the decoding process.

[0079] In another specific implementation, if the computer device is an image encoding device, that is, step S201 is performed during the image encoding phase, then when executing step S201, the computer device may obtain a current image frame and divide the current image frame into regions based on a preset region division method to obtain multiple image regions. It is understood that in other embodiments, if the computer device is an image decoding device, then the computer device may also, after decoding the current image frame, further divide the current image frame into regions based on a preset region division method to obtain multiple image regions.

[0080] S202: Obtain the region category of each image region.

[0081] The region category of any image region indicates the degree of pixel change between the image region and the corresponding reference region. This pixel change can be zero or greater than zero. When the pixel change is zero, it indicates that no pixel changes have occurred between the image region and the corresponding reference region. When the pixel change is greater than zero, it indicates that there has been a pixel change between the image region and the corresponding reference region. The reference region corresponding to any image region is the region in the reference image frame that matches the corresponding image region. The reference image frame mentioned here is an image frame in the image frame sequence.

[0082] In a specific implementation, if multiple image regions are obtained by decoding the encoded data of corresponding image blocks using the encoding information of multiple image blocks in the code stream of the current image frame, and the encoding method of each image block is an inter-frame encoding method (that is, an image block (referred to as block) matching the image block in the current image frame is found in the previous image frame, and the corresponding image block is encoded based on the difference between the corresponding image block and the found matching block), the region category of the image region corresponding to each image block can be determined by comparing the pixel attribute differences (such as pixel position differences and pixel value differences) generated by each image block compared with the matching block in the previous image frame.

[0083] In this specific implementation, the reference image frame may be the previous image frame adjacent to the current image frame in the image frame sequence; for example, if the current image frame is the third image frame in the image frame sequence, the reference image frame may be the second image frame in the image frame sequence. Accordingly, the specific implementation of step S202 may include the following steps: s11, obtaining the target coding parameters of the corresponding image block from the coding information of the image block corresponding to each image region. The target coding parameters of any image block are used to indicate: the difference in pixel attributes between the image block and the corresponding reference block; the reference block corresponding to any image block refers to: the block in the reference image frame that matches the corresponding image block (such as the block with the highest similarity), one block corresponds to one region, and the region corresponding to any reference block can be called a reference region. s12, based on the target coding parameters of the image block corresponding to each image region, the corresponding image region is classified to obtain the region category of each image region.

[0084] Exemplarily, the target coding parameters for any image block may include a motion vector and residual data. The motion vector indicates the difference in pixel position between the image block and the corresponding reference block, and the residual data indicates the difference between the pixel value of each pixel in the image block and the pixel value of the corresponding pixel in the corresponding reference block. In this case, a specific implementation of step s12 may include taking the image block corresponding to the qth image region as the qth image block, and identifying the region category of the qth image region based on the motion vector and residual data of the qth image block, where q is a positive integer less than or equal to the number of image regions. Specifically, when the motion vector and residual data of the qth image block are both zero, this indicates that there is no pixel displacement or change in pixel value between the qth image block and the corresponding reference block. Therefore, in this case, the region category of the qth image region can be determined to be an image-invariant region. The image-invariant region indicates that the degree of pixel change between the image region and the corresponding reference region is zero. When at least one of the motion vector and residual data of the qth image block is not zero, it may indicate that at least one attribute (position and pixel value) of the pixel between the qth image block and the corresponding reference block has changed. Therefore, in this case, the area category of the qth image area can be determined as an image change area; the image change area is used to indicate that the degree of pixel change between the image area and the corresponding reference area is greater than zero.

[0085] Based on the above description, it can be seen that the embodiment of the present application can effectively simplify the region category identification process by using the encoding information of each image block obtained through decoding to identify the region category of each image region, thereby improving the efficiency of region category identification and saving processing resources, thereby improving the efficiency of image super-resolution. It is understandable that in this specific implementation, before the computer device obtains the region category of each image region through the above steps s11-s12, it must also obtain the encoding information of each image block. As mentioned above, the multiple image regions can be obtained by directly calling a software decoder to decode the bitstream; in this case, the software decoder can output the encoding information of each image block at the same time as outputting the multiple image regions, so that the computer device can obtain the encoding information of each image block. Alternatively, the multiple image regions can be obtained by calling a hardware decoder to decode the bitstream; in this case, the computer device can call the software decoder to decode the bitstream. If, during the decoding process of the software decoder, it is detected that the encoding information of each image block has been decoded, the software decoder is controlled to terminate the decoding process and output the encoding information of each image block, so that the computer device can obtain the encoding information of each image block. By combining software and hardware decoding, the coding information of each image block can be obtained while ensuring decoding speed and decoding efficiency, thereby improving processing efficiency.

[0086] In addition, it should be noted that the image change region mentioned above can be a region category. Optionally, the image change region can be further subdivided to increase the number of region categories, so that the recognition granularity of the region category is finer, so that the super-resolution method adapted to each image region can be determined more accurately in the future, thereby improving the image super-resolution effect. For example, the image change region can be set to include an image weak change region (or called an image micro-change region) and an image strong change region; wherein the image weak change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero and less than a preset degree (that is, it indicates that there is a pixel change between the image region and the corresponding reference region and the degree of change is small); the image strong change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than or equal to a preset degree (that is, it indicates that there is a pixel change between the image region and the corresponding reference region and the degree of change is large). In this case, the specific implementation method of the above-mentioned step of determining that the region category of the qth image area is an image change region when at least one data in the motion vector and residual data of the qth image block is not zero can further include: when at least one data in the motion vector and residual data of the qth image block is not zero, if at least one data is residual data and the residual data is less than a preset threshold, it indicates that there is a change in the pixel value of the pixel between the qth image block and the corresponding reference block, but the change is small, so in this case, the region category of the qth image area can be determined as an image weak change region; if at least one data includes a motion vector, it indicates that there is a displacement of pixels between the qth image block and the corresponding reference block, so in this case, the region category of the qth image area can be determined as an image strong change region.

[0087] It should be noted that: ① Based on the above description, it can be seen that the computer device can determine that the region category of the qth image region is an image strong change region as long as it is determined that there is pixel displacement, regardless of whether the pixel value of the pixel has changed; however, in other embodiments, when it is determined that there is pixel displacement (i.e., the motion vector is not zero), it can be further determined whether the residual data is greater than a preset threshold. If so, the region category of the qth image region is determined to be an image strong change region; otherwise, the region category of the qth image region is determined to be an image weak change region. Alternatively, in other embodiments, when it is determined that there is pixel displacement (i.e., the motion vector is not zero), it can be further determined whether the motion vector is greater than a specified value. If so, the region category of the qth image region is determined to be an image strong change region; otherwise, the region category of the qth image region is determined to be an image weak change region.

[0088] ② The above is only an example of the subdivision method of the image change area and does not limit it. For example, in other embodiments, the image change area can be subdivided into at least three categories of areas, and different categories of areas are used to indicate different degrees of pixel change. For example, the image change area is subdivided into an image weak change area, an image medium change area, and an image strong change area. In this case, the image weak change area is used to indicate that the degree of pixel change between the image area and the corresponding reference area is greater than zero and less than a first degree, the image medium change area is used to indicate that the degree of pixel change between the image area and the corresponding reference area is greater than the first degree and less than the second degree, and the image strong change area is used to indicate that the degree of pixel change between the image area and the corresponding reference area is greater than or equal to the second degree, where the second degree is greater than the first degree. In this case, the specific implementation method of identifying the qth image area based on the motion vector and residual data of the qth image block can be adaptively adjusted based on the specific implementation method mentioned above. For example, when at least one of the motion vector and residual data of the qth image block is not zero, if at least one data is residual data and the residual data is less than a preset threshold, the region category of the qth image area is determined to be an image weak change region; if at least one data includes a motion vector and residual data, then when the residual data is less than the preset threshold, the region category of the qth image area is determined to be an image medium change region; when the residual data is greater than the preset threshold, the region category of the qth image area is determined to be an image strong change region, and so on; this is only an illustrative example of a specific implementation method, and is not exhaustive.

[0089] In another specific implementation, if multiple image regions are obtained by dividing the current image frame using a preset region division method, then regardless of whether the encoding method of each image block in the current image frame is inter-frame coding or intra-frame coding (i.e., each image block is independently encoded), the region category of each image region can be determined by comparing the pixel attribute differences between each image region and a matching region in a previous image frame. In this specific implementation, the reference image frame can be a previous image frame adjacent to the current image frame in the image frame sequence. Accordingly, a specific implementation of step S202 can be: dividing the reference image frame using a preset region division method to obtain multiple regions; for the qth image region of the current image frame, a region in the multiple regions having the same region coordinates as the qth image region is used as a reference region for the qth image region. For example, if the region coordinates of the qth image region are (1, 3), the region in the reference image frame having the region coordinates (1, 3) can be used as the reference region for the qth image region. Then, a similarity between the qth image region and the corresponding reference region can be calculated, and the region category of the qth image region is determined based on the calculated similarity.

[0090] For example, when the calculated similarity is greater than a first threshold (e.g., 98%), the region category of the qth image region can be determined to be an image-unchanged region; when the calculated similarity is less than the first threshold (e.g., 98%), the region category of the qth image region can be determined to be an image-changing region. Optionally, if the image-changing region is further subdivided into a weakly changing region and a strongly changing region, then when the calculated similarity is less than the first threshold, the computer device can further determine whether the calculated similarity is less than a second threshold, where the second threshold is less than the first threshold, such as 30%. If the calculated similarity is greater than or equal to the second threshold, i.e., the calculated similarity is between the second threshold and the first threshold, the region category of the qth image region can be determined to be an image-unchanged region; if the calculated similarity is less than the second threshold, the region category of the qth image region can be determined to be an image-strongly changing region. It is understood that if the image-changing region is further subdivided into at least three regions, the logic for identifying the region category of the qth image region based on the calculated similarity can be similarly deduced and will not be further described here.

[0091] Alternatively, when multiple image regions are obtained by dividing the current image frame using a preset region division method, the region category of each image region can be determined by comparing the pixel attribute differences between each image region and a matching region in the first image frame. In this specific implementation, the reference image frame can be the first image frame in the image frame sequence. In this case, the specific implementation of step S202 is the same as the specific implementation corresponding to the reference image frame being the image frame immediately preceding the current image frame, and is not further described here. Alternatively, the region category of each image region can be determined by comparing the pixel attribute differences between each image region and adjacent image regions in the same image frame. In this specific implementation, the reference image frame can be the current image frame. Accordingly, the specific implementation of step S202 can be: for the qth image region of the current image frame, an image region adjacent to the qth image region in the multiple image regions is used as a reference region for the qth image region. Then, a similarity between the qth image region and the corresponding reference region can be calculated, and the region category of the qth image region is determined based on the calculated similarity.

[0092] S203 , determining a super-resolution method for each image region according to the region category of each image region.

[0093] In a specific implementation, multiple super-resolution methods can be preset, and different super-resolution methods have different computational complexities. The computational complexity of the super-resolution method of any image area determined by step S203 is proportional to the degree of pixel change indicated by the area category of the corresponding image area, that is, the greater the degree of pixel change corresponding to the image area, the greater the computational complexity of the super-resolution method determined for the image area. In this way, the determined super-resolution method can cope with the degree of pixel change corresponding to the corresponding image area, so that the determined super-resolution method can efficiently and accurately perform super-resolution processing on the corresponding image area, thereby improving the super-resolution effect.

[0094] Exemplarily, the multiple super-resolution methods may include at least two of the following: a method of reusing the super-resolution results of the reference area, an interpolation super-resolution method, and a model super-resolution method. Among them, the method of reusing the super-resolution results of the reference area refers to a method of directly using the super-resolution results of the reference area corresponding to any image area as the super-resolution results of the corresponding image area to improve the resolution of the corresponding image area; the interpolation super-resolution method refers to a method of performing interpolation processing based on an interpolation algorithm to improve the resolution of the corresponding image area; the model super-resolution method indicates a method of calling a pre-trained image super-resolution model to perform super-resolution processing on the corresponding image area. Based on this, it can be seen that the computational complexity of the method of reusing the super-resolution results of the reference area is less than the computational complexity of the super-resolution method, and the computational complexity of the interpolation super-resolution method is less than the computational complexity of the model super-resolution. It should be noted that the super-resolution result of the reference area can be obtained by selecting a super-resolution method based on actual conditions to perform super-resolution processing on the corresponding reference area. The selection method is similar to the principle of determining the super-resolution method for the image area, which will not be elaborated here. In addition, it can be understood that the super-resolution result obtained by the model super-resolution method is generated by the image super-resolution model by learning the global features of the image area, while the pixel value of a certain pixel in the super-resolution result obtained by the interpolation super-resolution method is obtained by interpolation based on the pixel values ​​of multiple adjacent pixels, and it refers to the pixel values ​​of some pixels. Therefore, the accuracy of the model super-resolution method can be greater than the accuracy of the interpolation super-resolution method.

[0095] Based on the relevant description of the aforementioned step S203, it can be known that the region category of each image region may be an image unchanged region or an image changed region, and the image changed region may not be further subdivided, or may be further subdivided into at least two categories of regions. Based on whether the image changed region is subdivided, the number of region categories involved in the embodiment of the present application may be different; in the case of different numbers of region categories, the multiple super-resolution methods preset in the embodiment of the present application may also be different, so that the super-resolution method determined for each image region is different. Taking the qth image region as an example, the super-resolution method determined for the qth image region in different situations can be referred to the following description:

[0096] (1) If the image change region is not further subdivided, the embodiment of the present application involves two region categories. Then, when the number of region categories is two, and the two region categories include image unchanged regions and image changed regions: if the region category of the qth image region is image unchanged regions, the super-resolution method of the qth image region may be an interpolation super-resolution method or a method of reusing the super-resolution result of the reference region; if the region category of the qth image region is image changed regions, the super-resolution method of the qth image region may be a model super-resolution method. Alternatively, if the region category of the qth image region is image unchanged regions, the super-resolution method of the qth image region may be a method of reusing the super-resolution result of the reference region; if the region category of the qth image region is image changed regions, the super-resolution method of the qth image region may be an interpolation super-resolution method.

[0097] (2) If the image change region is further refined into the image weak change region and the image strong change region, then the embodiment of the present application involves three region categories. Then, when the number of region categories is two, and the two region categories include the image unchanged region, the image weak change region and the image strong change region: if the region category of the qth image region is the image unchanged region, then the super-resolution method of the qth image region is the method of reusing the super-resolution result of the reference region; if the region category of the qth image region is the image weak change region, then the super-resolution method of the qth image region is the interpolation super-resolution method; if the region category of the qth image region is the image strong change region, then the super-resolution method of the qth image region is the model super-resolution method.

[0098] (3) If the image change region is further refined into the image weak change region, the image medium change region and the image strong change region, then the embodiment of the present application involves four region categories. Then, when the number of region categories is four, and the four region categories include the image unchanged region, the image weak change region, the image medium change region and the image strong change region: if the region category of the qth image region is the image unchanged region, then the super-resolution method of the qth image region is the method of reusing the super-resolution result of the reference region; if the region category of the qth image region is the image weak change region or the image medium change region, then the super-resolution method of the qth image region is the interpolation super-resolution method; if the region category of the qth image region is the image strong change region, then the super-resolution method of the qth image region is the model super-resolution method.

[0099] S204 , performing super-resolution processing on the corresponding image region using the super-resolution method of each image region to obtain a super-resolution result of each image region.

[0100] In the embodiments of the present application, the qth image region is used as an example for explanation; the specific method of super-resolution processing of other image regions can refer to the relevant description of the qth image region and is not repeated here. As can be seen from the above, the super-resolution method of the qth image region can be any of the following: a method of reusing the super-resolution result of the reference region, interpolation super-resolution, and model super-resolution; based on this, the specific method of super-resolution processing of the qth image region using the super-resolution method of the qth image region and obtaining the super-resolution result of the qth image region can be referred to as follows:

[0101] (1) If the super-resolution method of the qth image region is a method of reusing the super-resolution result of the reference region, the computer device can obtain the super-resolution result of the reference region corresponding to the qth image region, and use the obtained super-resolution result as the super-resolution result of the qth image region.

[0102] (2) If the super-resolution method of the qth image region is an interpolation super-resolution method, an interpolation algorithm is used to perform interpolation processing based on the qth image region to generate a super-resolution result of the qth image region. The interpolation algorithm may include but is not limited to: a nearest neighbor interpolation algorithm, a bilinear quadratic interpolation algorithm, a bilinear cubic interpolation algorithm, a Lanczos interpolation algorithm (an interpolation algorithm based on a Kaffman filter), etc. In a specific embodiment, the computer device may directly use the interpolation algorithm to perform interpolation processing on the qth image region to obtain a super-resolution result of the qth image region. In another specific embodiment, the computer device may obtain residual information between the qth image region and the corresponding reference region, and the residual information may include the difference between the pixel value of the pth pixel in the qth image region and the pixel value of the pth pixel in the corresponding reference region, where p is a positive integer and is less than or equal to the number of pixels contained in the qth image region. It is understandable that if the qth image region is obtained by decoding the coded data of the qth image block, and the reference region corresponding to the qth image region is the region corresponding to the reference block corresponding to the qth image block, then the residual information mentioned here is the same as the residual data of the qth image block mentioned above. After obtaining the residual information, the computer device can use an interpolation algorithm to interpolate the obtained residual information, and superimpose the interpolated residual information and the super-resolution result of the reference region corresponding to the qth image region to obtain the super-resolution result of the qth image region, such as Figure 3a shown.

[0103] Since the image region and the corresponding reference region are matched, most of the pixel values ​​in the two regions are the same, which makes most of the differences in the residual information zero. Therefore, the time required for interpolation of the residual information can usually be less than the time required for interpolation of the image region. Therefore, by interpolating the residual information and superimposing the super-resolution result of the reference region to achieve super-resolution processing of the qth image region, the efficiency of image super-resolution can be effectively improved. Moreover, when the super-resolution result of the reference region is obtained by a model super-resolution method, based on the accuracy advantage of the model super-resolution method, super-resolution processing of the qth image region by this method can also ensure the accuracy of the super-resolution result of the qth image region.

[0104] (3) If the super-resolution method of the qth image region is the model super-resolution method, the image super-resolution model is called to perform super-resolution processing on the qth image region to obtain the super-resolution result of the qth image region. In one embodiment, the computer device may input the qth image region into the image super-resolution model, so that the image super-resolution model extracts features therefrom, and thereby generates the super-resolution result of the qth image region based on the extracted features. In another embodiment, if the color space of the qth image region is YCbCr, the qth image region includes Y channel data, Cb channel data, and Cr channel data; wherein Y represents brightness, Cb represents blue chrominance component, and Cr represents red chrominance component. In this case, considering that the display effect of the image mainly depends on the Y channel data, in order to reduce the amount of data processed by the model and improve the model super-resolution efficiency, the computer device may input the Y channel data in the qth image region into the image super-resolution model, so that the image super-resolution model extracts features therefrom and generates the super-resolution result of the Y channel data based on the extracted features. In addition, the computer device can also use an interpolation super-resolution method to super-resolve the remaining channel data (i.e., the Cb channel data and the Cr channel data) in the qth image region to obtain a super-resolution result of the remaining channel data, and then integrate the super-resolution result of the remaining channel data with the super-resolution result of the Y channel data to obtain a super-resolution result of the qth image region. By performing model super-resolution on the main Y channel data, performing difference super-resolution on the secondary remaining channel data, and integrating the super-resolution results of each channel data to achieve super-resolution processing of the qth image region, it is possible to improve the super-resolution efficiency of the qth image region and save processing resources consumed by super-resolution while ensuring the super-resolution accuracy of the qth image region.

[0105] The image super-resolution model can be generated through model training using machine learning / deep learning techniques within artificial intelligence (AI). AI refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, artificial intelligence is a comprehensive field of computer science. It primarily aims to understand the essence of intelligence and produce intelligent machines that can react in a manner similar to human intelligence, enabling them to possess diverse capabilities such as perception, reasoning, and decision-making. Machine learning, at the core of AI, is the fundamental path to intelligent computing. It is a multidisciplinary field encompassing probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computing devices can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Deep learning, on the other hand, is a machine learning technique that utilizes deep neural network systems. Deep learning is a branch of machine learning that uses neural network models to learn the inherent patterns and feature representations of sample data (training data). Among them, the neural network is a complex network system formed by a large number of simple processing units (called neurons) that are widely interconnected. It reflects many basic characteristics of the human brain's functions and is a highly complex nonlinear learning system; and the model is a mathematical model used to describe the objective world. The model is abstracted from the data and finds mathematical laws when analyzing a large amount of data. It can predict results based on the input through the data laws.

[0106] In a specific implementation, the method for generating the image super-resolution model may include the following steps s21-s24:

[0107] s21, construct an initial super-resolution model, and obtain training data, the training data may include: a sample image of a first resolution and a sample image of a second resolution rendered based on the same picture, the first resolution is smaller than the second resolution. Specifically, when obtaining training data, the computer device may encode two video streams based on the sample picture, the two video streams including a video stream of a first resolution (LR video stream) and a video stream of a second resolution (HR video stream); decode each video stream separately to obtain two decoding results, any decoding result including at least one sample image of a corresponding resolution; then, align the two decoding results, and obtain training data from the result of the picture alignment. Wherein, picture alignment refers to: aligning the sample images with the same picture content in the two decoding results so that they constitute an image pair; the result of the picture alignment may include an image pair with the same picture content, then when obtaining training data from the result of the picture alignment, any image pair can be selected from the result of the picture alignment as training data. Alternatively, considering that in actual applications, versions of the same video at different resolutions are prone to frame misalignment problems, and the low-resolution images obtained by computer devices are usually transmitted through the network; based on this, in order to improve the authenticity and accuracy of the training data, so that the training data can be more in line with the actual situation, thereby improving the model training effect, the device used to train the model (such as a computer device or a server) can encode two video streams, and then transmit the first resolution video stream (LR video stream) to the test device (such as a client or other device) through the network, so that the test device decodes the first resolution video stream (LR video stream), obtains a first decoding result, and then returns the first decoding result, the first decoding result including: at least one sample image of the first resolution; and, decodes the second resolution video stream (HR video stream) to obtain a second decoding result, the second decoding result including: at least one sample image of the second resolution, such as Figure 3b As shown; then, the sample image in the first decoding result and the sample image in the second decoding result can be aligned, and training data can be obtained from the result of the picture alignment.

[0108] s22, calling the initial super-resolution model, performing super-resolution processing on the sample image of the first resolution, and obtaining a super-resolution image of the second resolution. The super-resolution image of the second resolution refers to an image obtained by super-resolution processing on the sample image of the first resolution. Specifically, the sample image of the first resolution in the training data can be directly input into the initial super-resolution model for processing to obtain the corresponding super-resolution image of the second resolution; or the Y channel data of the sample image of the first resolution can be input into the initial super-resolution model for processing to obtain the corresponding super-resolution image of the second resolution. For the specific implementation method, please refer to the relevant description of the specific implementation method of calling the image super-resolution model to perform super-resolution processing on the qth image area, which will not be repeated here.

[0109] s23, using the loss function of the initial super-resolution model, a loss value calculation is performed based on the sample image of the second resolution and the super-resolution image of the second resolution to obtain a model loss value. Among them, the embodiment of the present application does not limit the loss function of the initial super-resolution model. For example, it can be a cross entropy loss function, or an L2 normal loss function (also known as a minimum square error loss function), etc. For another example, the loss function can be shown in the following formula 1.1:

[0110]

[0111] Among them, L pixel represents the model loss value of the loss function, h represents the height of the sample image, w represents the width of the sample image, h represents the number of channels of the sample image, i is [1, h], j is [1, w], and k is [1, h]; I represents the data of the sample image of the second resolution in the h, w, c dimensions, that is, the data located in the i-th row and j-th column in the c-th channel data corresponding to the sample image of the second resolution; i,j,k The data representing the h, w, and c dimensions of the super-resolved image of the second resolution, that is, the data located at the i-th row and the j-th column in the c-th channel data corresponding to the super-resolved image of the second resolution; express and I i,j,k The absolute value of the difference between them; ∑ represents the summation operation.

[0112] s24, optimize the model parameters of the initial super-resolution model in the direction of reducing the model loss value to obtain an image super-resolution model. Specifically, the model parameters of the initial super-resolution model can be optimized in the direction of reducing the model loss value to obtain an optimized initial super-resolution model; if the optimized initial super-resolution model reaches convergence, the optimized initial super-resolution model can be used as the image super-resolution model. Among them, "in the direction of reducing the model loss value" mentioned here refers to: the model optimization direction with the goal of minimizing the model loss value; through this direction, the model optimization is performed so that the model loss value generated again by the training target after each optimization is less than the model loss value generated by the training target before optimization. For example, the model loss value obtained by this calculation is 0.85. Then, after optimizing the model parameters of the training target in the direction of reducing the model loss value, the model loss value generated by the optimized training target (i.e., the intermediate model corresponding to the pth round) should be less than 0.85. In addition, the optimized initial super-resolution model reaches convergence when: the model parameters of the optimized initial super-resolution model no longer change, or the change of the model parameters is less than the threshold; or the optimized initial super-resolution model reaches convergence when: the model loss value of the optimized initial super-resolution model no longer decreases, or the decrease in the model loss value of the optimized initial super-resolution model is less than the threshold.

[0113] Alternatively, if the optimized initial super-resolution model reaches convergence, the optimized initial super-resolution model is equivalently transformed using a re-parameterization technique to obtain an image super-resolution model; wherein the complexity of the model structure of the image super-resolution model is less than the complexity of the optimized initial super-resolution model, and the model parameters of the image super-resolution model are equivalent to the model parameters of the optimized initial super-resolution model. By using the re-parameterization technique to equivalently transform the optimized initial super-resolution model to obtain the image super-resolution model, a more complex multi-branch convolutional structure can be used during model training so that the model can learn more knowledge to improve its own performance, so that during model inference, the convolutional structure during training can equivalently re-equalize the model parameters to new model parameters, thereby ensuring that while simplifying the model structure of the image super-resolution model to reduce the memory usage of the image super-resolution model, the image super-resolution model can still use the feature expression capability obtained during training for model inference, thereby improving the accuracy of model inference and ensuring that no additional inference delay is introduced during model inference.

[0114] It should be noted that the embodiments of the present application do not limit the model structure of the initial super-resolution model and the image super-resolution model, nor do they limit the specific method of using the re-parameterization technology to perform equivalent conversion on the optimized initial super-resolution model. For example, it is assumed that the optimized initial super-resolution model includes M+2 feature extraction modules, where M is a positive integer; and see Figure 4aAs shown, any feature extraction module may include five parallel branches, the first branch includes conv-3×3 (3×3 convolution), the second branch includes conv-1×1 (1×1 convolution) and conv-3×3 (3×3 convolution), the third branch includes conv-1×1 (1×1 convolution) and Sobel-dx (first edge detection operator), the fourth branch includes conv-3×3 (3×3 convolution) and Sobel-dy (second edge detection operator), and the fifth branch includes conv-1×1 (1×1 convolution) and Laplacian (Laplacian operator). Then, the feature extraction module can be equivalently transformed using the re-parameterization technology to obtain a conv-3×3 (3×3 convolution). The model parameters of the obtained 3×3 convolution (convolution parameters) are equivalent to the model parameters of a feature extraction module. Based on this, assuming that the initial super-resolution model also includes a feature rearrangement (PixelShuffle) network layer, then each feature extraction module in the optimized initial super-resolution model is equivalently transformed, and the feature rearrangement network layer is kept unchanged, and the following can be obtained: Figure 4b The image super-resolution model shown.

[0115] S205 , integrating the super-resolution results of each image region to obtain a super-resolution image corresponding to the current image frame.

[0116] In a specific implementation, the computer device may stitch the super-resolution results of each image region according to their arrangement in the current image frame to obtain a super-resolution image corresponding to the current image frame. Alternatively, the computer device may use the ratio between the size of the super-resolution result of any image region and the size of the corresponding image region to perform geometric amplification on the image size of the current image frame to obtain the amplified image size, and construct a blank image based on the amplified image size; then, based on the position of each image region in the current image frame, the super-resolution result of the corresponding image region is mapped to the blank image to obtain the super-resolution result corresponding to the current image frame.

[0117] The embodiment of the present application can obtain multiple image regions of the current image frame and determine the super-resolution method of the corresponding image region according to the region category of each image region, thereby using the super-resolution method of each image region to perform super-resolution processing on the corresponding image region, and integrating the super-resolution results of each image region to achieve super-resolution processing of the current image frame and obtain a super-resolution image corresponding to the current image frame. Since the region category of any image region is used to indicate the degree of pixel change between the image region and the corresponding reference region, and the computational complexity of the super-resolution method of any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region, this can ensure that the super-resolution method assigned to any image region and the corresponding image region have a high degree of adaptability, that is, the super-resolution method assigned to any image region has the ability to efficiently and accurately perform super-resolution processing on the corresponding image region, so that when the super-resolution method of each image region is used to perform super-resolution processing on the corresponding image region, the super-resolution efficiency and accuracy of each image region can be improved, thereby improving the image super-resolution efficiency and accuracy of the current image frame and other image super-resolution effects.

[0118] Based on the above Figure 2 The relevant description of the method embodiment shown is that the embodiment of the present application is a video super-resolution method based on codec. The video super-resolution method can be applied to the process of low-resolution to high-resolution image reconstruction, which can reduce the complexity of super-resolution calculations and shorten the delay of hardware device inference. Since the video super-resolution method can effectively solve the problem of poor video quality, thereby meeting the needs of users at the playback end for ultimate high-definition image quality; therefore, the video super-resolution method has very important application value in the fields of live broadcast on demand, video codec, mobile phone shooting, medical imaging, digital high-definition and video restoration.

[0119] For example, in the cloud gaming business, since the cloud gaming client is deployed in the cloud server, the cloud server needs to upload the game screen (such as Figure 5a As shown in the figure, the video is transmitted to the mobile terminal in real time for rendering and display. Therefore, in this business model, the traffic cost accounts for a relatively large part. Based on this, in order to reduce the traffic cost of the business, low-resolution game images can be transmitted between the cloud server and the mobile terminal, so that the mobile terminal can reconstruct the low-resolution game images into high-resolution game images by using the video super-resolution method proposed in the embodiment of the present application. For example, see Figure 5bAs shown: the game client process in the cloud server can transmit the code stream of the game screen with a resolution of 540p (i.e., the game video stream) to the mobile terminal, and after receiving the corresponding code stream, the mobile terminal can perform a decoding operation to obtain a 540p game screen, and then use the video super-resolution method proposed in the embodiment of the present application to super-resolution the 540p game screen to obtain a game screen with a resolution of 1080p (i.e., a high-resolution image after super-resolution processing), and render the game screen with a resolution of 1080p to display the 1080p game screen in the application interface.

[0120] In a specific implementation, the video super-resolution method proposed in the embodiment of the present application can realize image super-resolution processing by transmitting and processing data among three modules, namely, a decoding module, a sub-region module and a super-resolution module, and finally output a high-resolution image. Figure 5c As shown in the figure, the functions of each module are as follows:

[0121] (1) Decoding module: This module is used to decode the received source video stream (i.e., the code stream of the current game screen), call the hardware decoder to obtain the super-resolution input stream (i.e., multiple image areas of the current game screen), and call the dedicated software decoder to obtain the encoding information of multiple image blocks of the current game screen. The encoding information may include the type of image block (used to indicate whether the encoding method of the image block is inter-frame encoding or intra-frame encoding), the motion vector of the image block, and the residual data, etc.

[0122] (2) Region division module: This module is used to determine the super-resolution method. For the current image region to be super-resolved, if the type of the image block corresponding to the current image region is used to indicate that the encoding method of the corresponding image block is intra-frame encoding, the super-resolution method of the current image region is directly determined to be the model super-resolution method; if the type of the image block corresponding to the current image region is used to indicate that the encoding method of the corresponding image block is inter-frame encoding, the region category of the current image region is determined based on the residual data and motion vector of the corresponding image block, that is, whether the current image region is an image unchanged region or an image weakly changing region, thereby determining the super-resolution method of the current image region based on the determination result of the region category.

[0123] (3) Super-resolution module: This module performs super-resolution on the input image data. For image areas without pixel movement (i.e., image areas whose region category is image unchanged region), the super-resolution result of the reference region is reused; for image areas with subtle changes (i.e., image areas whose region category is image weak change region), the difference super-resolution method is used; for image areas with pixel movement (i.e., image areas whose region category is image strong change region), the model super-resolution method is used; then, the super-resolution results of each image area can be mapped to the corresponding image area to merge the super-resolution results of each image area, thereby outputting a high-resolution game screen (i.e., the game screen after super-resolution processing).

[0124] Based on the above description, it can be seen that the video super-resolution method proposed in the embodiment of the present application can obtain the relative displacement relationship and image residual between each image area in the current game screen and the corresponding reference area in the previous game screen by parsing the code stream. For image areas with pixel displacement, a deep learning method is used for super-resolution. For image areas without pixel displacement, the super-resolution result of the corresponding reference area of ​​the previous frame is directly referenced or super-resolution is performed using an interpolation super-resolution method. Since the picture changes between adjacent frames in a video are often not particularly drastic, there are a large number of image areas without pixel displacement. Therefore, this video super-resolution method can use the information between frames to reduce the computational complexity of super-resolution, greatly reducing the computational amount of the neural network model, and at the same time will not reduce the quality of the image after super-resolution.

[0125] Furthermore, based on the above Figure 5c The working principle of each module shown in FIG. 1 and the general process of the video super-resolution method proposed in the embodiment of the present application can be exemplified by referring to FIG. Figure 5d As shown, the process is roughly as follows: the mobile terminal receives a low-resolution video stream (i.e., the code stream of the current game screen) and decodes it to obtain encoding information for multiple image regions and image blocks corresponding to each image region. The super-resolution method for the corresponding image region is determined based on the motion vector and residual data in the encoding information of each image block. Specifically, for image regions corresponding to image blocks with zero motion vectors and residual data, the super-resolution result of the corresponding image region is obtained by directly referencing the previous frame and reusing the super-resolution result of the reference region in the previous frame that matches the corresponding image region. For image regions corresponding to image blocks with zero motion vectors but small residual data, the super-resolution result of the corresponding image region is obtained by interpolating the residual data and superimposing the super-resolution result of the reference region in the previous frame. For the remaining image regions with large changes, the image super-resolution model is used to super-resolve the corresponding image regions to obtain the super-resolution result of the corresponding image regions. Then, the super-resolution results of each image region obtained by different branches are integrated into the corresponding position of the corresponding image region to obtain a high-resolution game screen.

[0126] Thus, the embodiment of the present application combines an interpolation algorithm with an image super-resolution model to reconstruct a low-resolution input image into a high-resolution image. During the super-resolution process, the encoding information of each image block obtained by video decoding is used to adaptively determine the super-resolution method for the corresponding image area, which can significantly reduce the computational complexity of the super-resolution process and shorten the super-resolution delay. At the same time, compared with super-resolution using only a neural network model, the video super-resolution method proposed in the embodiment of the present application can reconstruct a high-resolution image without any loss of image quality.

[0127] For example, Figure 5e The test result diagram of the super-resolution effect is shown. It can be clearly seen from the test result diagram that: compared with the low-resolution image that has not been super-resolution processed, the high-resolution image reconstructed after using the video super-resolution method proposed in the embodiment of the present application has significantly better image quality, sharper edges, and clearer fonts; at the same time, it can be seen that compared with the high-resolution image obtained by using only the model super-resolution, which has higher computational complexity, the image effect obtained by using the video super-resolution method proposed in the embodiment of the present application is basically the same, with very close image quality. In addition, the embodiment of the present application reconstructs a frame of 540p (960×540) resolution image into a 1080p (1920×1080) resolution image by testing on the graphics card of a specified hardware device. Without using the video super-resolution method proposed in the embodiment of the present application, the delay required for super-resolution using the model directly is 24 milliseconds, while the delay required for super-resolution using the video super-resolution method proposed in the embodiment of the present application is 10 milliseconds. It can be seen that the video super-resolution method proposed in the embodiment of the present application can significantly reduce the delay of super-resolution while ensuring the quality of the reconstructed image.

[0128] In summary, the video super-resolution method proposed in the embodiment of the present application can use the relationship between the previous and next frames of the video obtained during the decoding process, utilize the relationship between the position change relationship of the image area and the relationship between the degree of change of the image area, combine multiple super-resolution methods such as the model super-resolution method and the interpolation super-resolution method, and adopt different super-resolution methods for different image areas to complete the reconstruction process from low-resolution images to high-resolution images. In this way, while ensuring the quality of the super-resolution image, the computational complexity of the super-resolution of one frame of image can be greatly reduced, and the super-resolution delay can be significantly shortened. In other words, the embodiment of the present application can achieve the purpose of shortening the delay of image super-resolution by making full use of the information redundancy of the video itself to ensure that the computational complexity of the image super-resolution is significantly reduced without losing image quality.

[0129] Based on the description of the above-mentioned image processing method embodiment, the present application also discloses an image processing device; the image processing device can be a computer program (including one or more instructions) running on a computer device, and the image processing device can execute each step in the above-mentioned method flow. Figure 6 , the image processing device can run the following units:

[0130] An acquisition unit 601 is configured to acquire multiple image regions of a current image frame, where the current image frame is an image frame to be super-resolved in an image frame sequence;

[0131] The acquisition unit 601 is further configured to acquire a region category for each image region. The region category of any image region indicates a degree of pixel variation between the image region and a corresponding reference region. The reference region corresponding to any image region is a region in a reference image frame that matches the corresponding image region. The reference image frame is an image frame in the image frame sequence.

[0132] a processing unit 602 configured to determine a super-resolution method for each image region based on the region category of each image region; different super-resolution methods have different computational complexities, and the computational complexity of the super-resolution method for any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region;

[0133] The processing unit 602 is further configured to perform super-resolution processing on the corresponding image regions using a super-resolution method for each image region to obtain super-resolution results for each image region; and to integrate the super-resolution results for each image region to obtain a super-resolution image corresponding to the current image frame.

[0134] In one embodiment, the multiple image regions are obtained by decoding a code stream of a current image frame, where the code stream includes coded data of multiple image blocks and coding information of each image block; the coding information of one image block is used to decode the coded data of the corresponding image block to obtain an image region of the current image frame;

[0135] Accordingly, when the acquisition unit 601 is used to acquire the region category of each image region, it can be specifically used to:

[0136] Obtaining target coding parameters for the corresponding image blocks from coding information for the image blocks corresponding to each image region; the target coding parameters for any image block are used to indicate a difference in pixel attributes between the image block and a corresponding reference block; the reference block corresponding to any image block is a block in the reference image frame that matches the corresponding image block, with one block corresponding to one region;

[0137] Based on the target coding parameters of the image blocks corresponding to each image region, category identification is performed on the corresponding image region to obtain the region category of each image region.

[0138] In another embodiment, the multiple image regions are obtained by calling a hardware decoder to decode the bitstream. Before obtaining the region category of each image region, the processing unit 602 may further be configured to:

[0139] Call the software decoder to decode the code stream;

[0140] If, during the decoding process of the software decoder, it is detected that the coding information of each image block has been decoded, the software decoder is controlled to end the decoding process and output the coding information of each image block.

[0141] In another embodiment, the target coding parameters of any image block include: a motion vector and residual data; the motion vector is used to indicate: a difference in pixel position between the image block and a corresponding reference block; the residual data is used to indicate: a difference between a pixel value of each pixel point in the image block and a pixel value of a corresponding pixel point in the corresponding reference block;

[0142] Accordingly, when the acquisition unit 601 is used to perform category identification on the corresponding image region based on the target coding parameters of the image block corresponding to each image region to obtain the region category of each image region, it can be specifically used to:

[0143] The image block corresponding to the qth image region is taken as the qth image block, where q is a positive integer and is less than or equal to the number of image regions;

[0144] When the motion vector and the residual data of the qth image block are both zero, determining that the region category of the qth image region is an image invariant region; the image invariant region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is zero;

[0145] When at least one of the motion vector and residual data of the qth image block is not zero, the region category of the qth image region is determined to be an image change region; the image change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero.

[0146] In another embodiment, the image change region includes an image weak change region and an image strong change region; wherein the image weak change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero and less than a preset degree; and the image strong change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than or equal to a preset degree.

[0147] Accordingly, when the acquisition unit 601 is configured to determine that the region category of the qth image region is an image change region when at least one of the motion vector and the residual data of the qth image block is non-zero, it may be specifically configured to:

[0148] When at least one of the motion vector and residual data of the qth image block is not zero, if the at least one data is residual data and the residual data is less than a preset threshold, the area category of the qth image area is determined to be an image weak change area; if the at least one data includes a motion vector, the area category of the qth image area is determined to be an image strong change area.

[0149] In another embodiment, when there are two region categories, and the two region categories include image unchanged regions and image changed regions:

[0150] If the region category of the qth image region is an image unchanged region, the super-resolution method of the qth image region is an interpolation super-resolution method or a method of reusing the super-resolution result of the reference region;

[0151] If the region category of the qth image region is the image change region, the super-resolution method of the qth image region is the model super-resolution method;

[0152] Wherein, q is a positive integer and is less than or equal to the number of image regions; the computational complexity of the interpolation super-resolution method and the computational complexity of the method of reusing the super-resolution results of the reference region are both less than the computational complexity of the model super-resolution method.

[0153] In another embodiment, when the number of region categories is three and the three region categories include image unchanged regions, image weakly changed regions, and image strongly changed regions:

[0154] If the region category of the qth image region is an image unchanged region, the super-resolution method of the qth image region is a method of reusing the super-resolution result of the reference region;

[0155] If the region category of the qth image region is the image weak change region, the super-resolution method of the qth image region is the interpolation super-resolution method;

[0156] If the region category of the qth image region is the image strong change region, the super-resolution method of the qth image region is the model super-resolution method;

[0157] Wherein, q is a positive integer and is less than or equal to the number of image regions; the computational complexity of the method of reusing the super-resolution results of the reference region is less than the computational complexity of the interpolation super-resolution method, and the computational complexity of the interpolation super-resolution method is less than the computational complexity of the model super-resolution method.

[0158] In another embodiment, when the processing unit 602 is configured to perform super-resolution processing on the corresponding image region using the super-resolution method of each image region to obtain the super-resolution result of each image region, it can be specifically configured to:

[0159] If the super-resolution method of the qth image region is a method of reusing the super-resolution result of the reference region, obtaining the super-resolution result of the reference region corresponding to the qth image region, and using the obtained super-resolution result as the super-resolution result of the qth image region;

[0160] If the super-resolution method of the qth image region is an interpolation super-resolution method, an interpolation algorithm is used to perform interpolation processing based on the qth image region to generate a super-resolution result of the qth image region;

[0161] If the super-resolution method of the qth image region is the model super-resolution method, calling the image super-resolution model to perform super-resolution processing on the qth image region to obtain a super-resolution result of the qth image region;

[0162] Wherein, q is a positive integer and is less than or equal to the number of image regions.

[0163] In another embodiment, when the processing unit 602 is configured to perform interpolation processing based on the qth image region using an interpolation algorithm to generate a super-resolution result of the qth image region, it may be specifically configured to:

[0164] Performing interpolation processing on the qth image region using an interpolation algorithm to obtain a super-resolution result of the qth image region;

[0165] Alternatively, residual information between the qth image region and the corresponding reference region is obtained, an interpolation algorithm is used to interpolate the obtained residual information, and the interpolated residual information and the super-resolution result of the reference region corresponding to the qth image region are superimposed to obtain the super-resolution result of the qth image region.

[0166] In another embodiment, the processing unit 602 may be further configured to generate an image super-resolution model, and may specifically be configured to perform the following steps:

[0167] Constructing an initial super-resolution model and obtaining training data; the training data includes: a sample image of a first resolution and a sample image of a second resolution rendered based on the same picture, wherein the first resolution is smaller than the second resolution;

[0168] Calling the initial super-resolution model to perform super-resolution processing on the sample image of the first resolution to obtain a super-resolution image of the second resolution;

[0169] Using the loss function of the initial super-resolution model, performing a loss value calculation based on the sample image of the second resolution and the super-resolution image of the second resolution to obtain a model loss value;

[0170] The model parameters of the initial super-resolution model are optimized in a direction of reducing the model loss value to obtain an image super-resolution model.

[0171] In another embodiment, when the processing unit 602 is used to obtain training data, it can be specifically used to:

[0172] Encode two video streams based on the sample picture data, wherein the two video streams include a video stream with a first resolution and a video stream with a second resolution;

[0173] Transmitting the video stream of the first resolution to a test device via a network, causing the test device to decode the video stream of the first resolution, and after obtaining a first decoding result, transmitting the first decoding result back; the first decoding result includes: at least one sample image of the first resolution;

[0174] Decoding the video stream at the second resolution to obtain a second decoding result; the second decoding result includes: at least one sample image at the second resolution;

[0175] Performing picture alignment on the sample image in the first decoding result and the sample image in the second decoding result, and acquiring training data from the picture alignment result.

[0176] In another embodiment, when the processing unit 602 is used to optimize the model parameters of the initial super-resolution model in a direction of reducing the model loss value to obtain the image super-resolution model, it can be specifically used to:

[0177] Optimizing the model parameters of the initial super-resolution model in a direction of reducing the model loss value to obtain an optimized initial super-resolution model;

[0178] If the optimized initial super-resolution model reaches convergence, the optimized initial super-resolution model is equivalently transformed using the re-parameterization technology to obtain an image super-resolution model; wherein the complexity of the model structure of the image super-resolution model is less than the complexity of the optimized initial super-resolution model, and the model parameters of the image super-resolution model are equivalent to the model parameters of the optimized initial super-resolution model.

[0179] According to another embodiment of the present application, Figure 6The various units in the image processing device shown can be individually or all combined into one or several other units to constitute, or one (or some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, other units can also be included based on the image processing device. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0180] According to another embodiment of the present application, a computer program (including one or more instructions) capable of executing each step involved in the above method embodiment can be constructed by running the computer program (including one or more instructions) on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM). Figure 6 The image processing apparatus shown in the embodiment of the present application is used to implement the image processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable storage medium, for example, and loaded into the computing device through the computer-readable storage medium and run therein.

[0181] It is worth noting that, in the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can include a part of the overall module or unit of the module or unit function.

[0182] The embodiment of the present application can obtain multiple image regions of the current image frame and determine the super-resolution method of the corresponding image region according to the region category of each image region, thereby using the super-resolution method of each image region to perform super-resolution processing on the corresponding image region, and integrating the super-resolution results of each image region to achieve super-resolution processing of the current image frame and obtain a super-resolution image corresponding to the current image frame. Since the region category of any image region is used to indicate the degree of pixel change between the image region and the corresponding reference region, and the computational complexity of the super-resolution method of any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region, this can ensure that the super-resolution method assigned to any image region and the corresponding image region have a high degree of adaptability, that is, the super-resolution method assigned to any image region has the ability to efficiently and accurately perform super-resolution processing on the corresponding image region, so that when the super-resolution method of each image region is used to perform super-resolution processing on the corresponding image region, the super-resolution efficiency and accuracy of each image region can be improved, thereby improving the image super-resolution efficiency and accuracy of the current image frame and other image super-resolution effects.

[0183] Based on the description of the above method embodiment and apparatus embodiment, the present application embodiment also provides a computer device. Figure 7 , the computer device at least includes a processor 701, an input interface 702, an output interface 703 and a computer storage medium 704. Among them, the processor 701, input interface 702, output interface 703 and computer storage medium 704 in the computer device can be connected via a bus or other means. The computer storage medium 704 can be stored in the memory of the computer device, and the computer storage medium 704 is used to store a computer program, and the computer program includes one or more instructions. The processor 701 is used to execute one or more instructions in the computer program stored in the computer storage medium 704. The processor 701 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to realize the corresponding method flow or corresponding function.

[0184] In one embodiment, the processor 701 described in the embodiment of the present application can be used to perform a series of super-resolution processing on the current image frame, specifically including: obtaining multiple image areas of the current image frame, wherein the current image frame is an image frame to be super-resolution processed in an image frame sequence; obtaining the area category of each image area, wherein the area category of any image area is used to indicate the degree of pixel change between the image area and the corresponding reference area; the reference area corresponding to any image area refers to: the area in the reference image frame that matches the corresponding image area, wherein the reference image frame is an image frame in the image frame sequence; determining the super-resolution method of the corresponding image area according to the area category of each image area; different super-resolution methods have different computational complexities, and the computational complexity of the super-resolution method of any image area is proportional to the degree of pixel change indicated by the area category of the corresponding image area; using the super-resolution method of each image area, super-resolution processing is performed on the corresponding image area to obtain the super-resolution results of each image area; and integrating the super-resolution results of each image area to obtain the super-resolution image corresponding to the current image frame, and so on.

[0185] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in a computer device for storing computer programs and data. It is understandable that the computer storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer storage medium provides a storage space, which stores the operating system of the computer device. In addition, a computer program is also stored in the storage space, which includes one or more instructions suitable for being loaded and executed by the processor 701, and these instructions can be one or more program codes. It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0186] In one embodiment, a processor may load and execute one or more instructions stored in a computer storage medium to implement the corresponding steps in the above method embodiment. In a specific implementation, the processor may load and execute the following steps:

[0187] Acquire multiple image regions of a current image frame, where the current image frame is an image frame to be super-resolved in an image frame sequence;

[0188] Obtaining a region category for each image region, where the region category of any image region indicates a degree of pixel variation between the image region and a corresponding reference region; the reference region corresponding to any image region is a region in a reference image frame that matches the corresponding image region, where the reference image frame is an image frame in the image frame sequence;

[0189] Determining a super-resolution method for each image region according to the region category of each image region; different super-resolution methods have different computational complexities, and the computational complexity of the super-resolution method for any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region;

[0190] The super-resolution method of each image region is adopted to perform super-resolution processing on the corresponding image region to obtain the super-resolution results of each image region; and the super-resolution results of each image region are integrated to obtain the super-resolution image corresponding to the current image frame.

[0191] In one embodiment, the multiple image regions are obtained by decoding a code stream of a current image frame, where the code stream includes coded data of multiple image blocks and coding information of each image block; the coding information of one image block is used to decode the coded data of the corresponding image block to obtain an image region of the current image frame;

[0192] Accordingly, when obtaining the region category of each image region, the one or more instructions may be loaded and specifically executed by the processor:

[0193] Obtaining target coding parameters for the corresponding image blocks from coding information for the image blocks corresponding to each image region; the target coding parameters for any image block are used to indicate a difference in pixel attributes between the image block and a corresponding reference block; the reference block corresponding to any image block is a block in the reference image frame that matches the corresponding image block, with one block corresponding to one region;

[0194] Based on the target coding parameters of the image blocks corresponding to each image region, category identification is performed on the corresponding image region to obtain the region category of each image region.

[0195] In another embodiment, the multiple image regions are obtained by decoding the bitstream by calling a hardware decoder. Before obtaining the region category of each image region, the one or more instructions may be loaded and specifically executed by the processor:

[0196] Call the software decoder to decode the code stream;

[0197] If, during the decoding process of the software decoder, it is detected that the coding information of each image block has been decoded, the software decoder is controlled to end the decoding process and output the coding information of each image block.

[0198] In another embodiment, the target coding parameters of any image block include: a motion vector and residual data; the motion vector is used to indicate: a difference in pixel position between the image block and a corresponding reference block; the residual data is used to indicate: a difference between a pixel value of each pixel point in the image block and a pixel value of a corresponding pixel point in the corresponding reference block;

[0199] Accordingly, when performing category identification on the corresponding image region based on the target coding parameters of the image block corresponding to each image region to obtain the region category of each image region, the one or more instructions may be loaded and specifically executed by the processor:

[0200] The image block corresponding to the qth image region is taken as the qth image block, where q is a positive integer and is less than or equal to the number of image regions;

[0201] When the motion vector and the residual data of the qth image block are both zero, determining that the region category of the qth image region is an image invariant region; the image invariant region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is zero;

[0202] When at least one of the motion vector and residual data of the qth image block is not zero, the region category of the qth image region is determined to be an image change region; the image change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero.

[0203] In another embodiment, the image change region includes an image weak change region and an image strong change region; wherein the image weak change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero and less than a preset degree; and the image strong change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than or equal to a preset degree.

[0204] Accordingly, when at least one of the motion vector and the residual data of the qth image block is non-zero, and the region category of the qth image region is determined to be an image change region, the one or more instructions may be loaded and specifically executed by the processor:

[0205] When at least one of the motion vector and residual data of the qth image block is not zero, if the at least one data is residual data and the residual data is less than a preset threshold, the area category of the qth image area is determined to be an image weak change area; if the at least one data includes a motion vector, the area category of the qth image area is determined to be an image strong change area.

[0206] In another embodiment, when there are two region categories, and the two region categories include image unchanged regions and image changed regions:

[0207] If the region category of the qth image region is an image unchanged region, the super-resolution method of the qth image region is an interpolation super-resolution method or a method of reusing the super-resolution result of the reference region;

[0208] If the region category of the qth image region is the image change region, the super-resolution method of the qth image region is the model super-resolution method;

[0209] Wherein, q is a positive integer and is less than or equal to the number of image regions; the computational complexity of the interpolation super-resolution method and the computational complexity of the method of reusing the super-resolution results of the reference region are both less than the computational complexity of the model super-resolution method.

[0210] In another embodiment, when the number of region categories is three and the three region categories include image unchanged regions, image weakly changed regions, and image strongly changed regions:

[0211] If the region category of the qth image region is an image unchanged region, the super-resolution method of the qth image region is a method of reusing the super-resolution result of the reference region;

[0212] If the region category of the qth image region is the image weak change region, the super-resolution method of the qth image region is the interpolation super-resolution method;

[0213] If the region category of the qth image region is the image strong change region, the super-resolution method of the qth image region is the model super-resolution method;

[0214] Wherein, q is a positive integer and is less than or equal to the number of image regions; the computational complexity of the method of reusing the super-resolution results of the reference region is less than the computational complexity of the interpolation super-resolution method, and the computational complexity of the interpolation super-resolution method is less than the computational complexity of the model super-resolution method.

[0215] In another embodiment, when super-resolution processing is performed on the corresponding image regions using a super-resolution method for each image region to obtain super-resolution results for each image region, the one or more instructions may be loaded and specifically executed by the processor:

[0216] If the super-resolution method of the qth image region is a method of reusing the super-resolution result of the reference region, obtaining the super-resolution result of the reference region corresponding to the qth image region, and using the obtained super-resolution result as the super-resolution result of the qth image region;

[0217] If the super-resolution method of the qth image region is an interpolation super-resolution method, an interpolation algorithm is used to perform interpolation processing based on the qth image region to generate a super-resolution result of the qth image region;

[0218] If the super-resolution method of the qth image region is the model super-resolution method, calling the image super-resolution model to perform super-resolution processing on the qth image region to obtain a super-resolution result of the qth image region;

[0219] Wherein, q is a positive integer and is less than or equal to the number of image regions.

[0220] In another embodiment, when an interpolation algorithm is used to perform interpolation processing based on the qth image region to generate a super-resolution result of the qth image region, the one or more instructions may be loaded and specifically executed by the processor:

[0221] Performing interpolation processing on the qth image region using an interpolation algorithm to obtain a super-resolution result of the qth image region;

[0222] Alternatively, residual information between the qth image region and the corresponding reference region is obtained, an interpolation algorithm is used to interpolate the obtained residual information, and the interpolated residual information and the super-resolution result of the reference region corresponding to the qth image region are superimposed to obtain the super-resolution result of the qth image region.

[0223] In another embodiment, when generating the image super-resolution model, the one or more instructions may be loaded and specifically executed by the processor:

[0224] Constructing an initial super-resolution model and obtaining training data; the training data includes: a sample image of a first resolution and a sample image of a second resolution rendered based on the same picture, wherein the first resolution is smaller than the second resolution;

[0225] Calling the initial super-resolution model to perform super-resolution processing on the sample image of the first resolution to obtain a super-resolution image of the second resolution;

[0226] Using the loss function of the initial super-resolution model, performing a loss value calculation based on the sample image of the second resolution and the super-resolution image of the second resolution to obtain a model loss value;

[0227] The model parameters of the initial super-resolution model are optimized in a direction of reducing the model loss value to obtain an image super-resolution model.

[0228] In another embodiment, when acquiring training data, the one or more instructions may be loaded and specifically executed by the processor:

[0229] Encode two video streams based on the sample picture data, wherein the two video streams include a video stream with a first resolution and a video stream with a second resolution;

[0230] Transmitting the video stream of the first resolution to a test device via a network, causing the test device to decode the video stream of the first resolution, and after obtaining a first decoding result, transmitting the first decoding result back; the first decoding result includes: at least one sample image of the first resolution;

[0231] Decoding the video stream at the second resolution to obtain a second decoding result; the second decoding result includes: at least one sample image at the second resolution;

[0232] Performing picture alignment on the sample image in the first decoding result and the sample image in the second decoding result, and acquiring training data from the picture alignment result.

[0233] In another embodiment, when optimizing the model parameters of the initial super-resolution model in a direction of reducing the model loss value to obtain the image super-resolution model, the one or more instructions may be loaded and specifically executed by the processor:

[0234] Optimizing the model parameters of the initial super-resolution model in a direction of reducing the model loss value to obtain an optimized initial super-resolution model;

[0235] If the optimized initial super-resolution model reaches convergence, the optimized initial super-resolution model is equivalently transformed using the re-parameterization technology to obtain an image super-resolution model; wherein the complexity of the model structure of the image super-resolution model is less than the complexity of the optimized initial super-resolution model, and the model parameters of the image super-resolution model are equivalent to the model parameters of the optimized initial super-resolution model.

[0236] The embodiment of the present application can obtain multiple image regions of the current image frame and determine the super-resolution method of the corresponding image region according to the region category of each image region, thereby using the super-resolution method of each image region to perform super-resolution processing on the corresponding image region, and integrating the super-resolution results of each image region to achieve super-resolution processing of the current image frame and obtain a super-resolution image corresponding to the current image frame. Since the region category of any image region is used to indicate the degree of pixel change between the image region and the corresponding reference region, and the computational complexity of the super-resolution method of any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region, this can ensure that the super-resolution method assigned to any image region and the corresponding image region have a high degree of adaptability, that is, the super-resolution method assigned to any image region has the ability to efficiently and accurately perform super-resolution processing on the corresponding image region, so that when the super-resolution method of each image region is used to perform super-resolution processing on the corresponding image region, the super-resolution efficiency and accuracy of each image region can be improved, thereby improving the image super-resolution efficiency and accuracy of the current image frame and other image super-resolution effects.

[0237] It should be noted that, according to one aspect of the present application, a computer program product or computer program is also provided, which includes one or more instructions, and the one or more instructions are stored in a computer storage medium. The processor of the computer device reads the one or more instructions from the computer storage medium, and the processor executes the one or more instructions, so that the computer device performs the methods provided in various optional ways in the above-mentioned method embodiments. It should be understood that what is disclosed above is only a preferred embodiment of the present application, and it is certainly not used to limit the scope of the rights of the present application. Therefore, equivalent changes made in accordance with the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: include: Acquire multiple image regions of a current image frame, where the current image frame is an image frame to be super-resolved in an image frame sequence; Obtaining a region category for each image region, where the region category of any image region indicates a degree of pixel variation between the image region and a corresponding reference region; the reference region corresponding to any image region is a region in a reference image frame that matches the corresponding image region, where the reference image frame is an image frame in the image frame sequence; Determining a super-resolution method for each image region according to the region category of each image region; different super-resolution methods have different computational complexities, and the computational complexity of the super-resolution method for any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region; The super-resolution method of each image region is adopted to perform super-resolution processing on the corresponding image region to obtain the super-resolution results of each image region; and the super-resolution results of each image region are integrated to obtain the super-resolution image corresponding to the current image frame.

2. The method according to claim 1, wherein The multiple image regions are obtained by decoding a code stream of a current image frame, wherein the code stream includes coded data of multiple image blocks and coding information of each image block; the coding information of one image block is used to decode the coded data of the corresponding image block to obtain an image region of the current image frame; The obtaining of the region category of each image region includes: Obtaining target coding parameters of the corresponding image blocks from coding information of the image blocks corresponding to each image region; The target coding parameter of any image block is used to indicate the difference in pixel attributes between the image block and the corresponding reference block. The reference block corresponding to any image block is the block in the reference image frame that matches the corresponding image block, with one block corresponding to one region. Based on the target coding parameters of the image blocks corresponding to each image region, category identification is performed on the corresponding image region to obtain the region category of each image region.

3. The method according to claim 2, wherein The multiple image regions are obtained by decoding the code stream by calling a hardware decoder. Before obtaining the region category of each image region, the method further includes: Call the software decoder to decode the code stream; If, during the decoding process of the software decoder, it is detected that the coding information of each image block has been decoded, the software decoder is controlled to end the decoding process and output the coding information of each image block.

4. The method according to claim 2, wherein The target coding parameters of any image block include: a motion vector and residual data; the motion vector is used to indicate the difference in pixel position between the image block and the corresponding reference block; the residual data is used to indicate the difference between the pixel value of each pixel point in the image block and the pixel value of the corresponding pixel point in the corresponding reference block; The performing category identification on the corresponding image region based on the target coding parameters of the image block corresponding to each image region to obtain the region category of each image region includes: The image block corresponding to the qth image region is taken as the qth image block, where q is a positive integer and is less than or equal to the number of image regions; When the motion vector and the residual data of the qth image block are both zero, determining that the region category of the qth image region is an image invariant region; the image invariant region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is zero; When at least one of the motion vector and residual data of the qth image block is not zero, the region category of the qth image region is determined to be an image change region; the image change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero.

5. The method according to claim 4, wherein The image change region includes an image weak change region and an image strong change region; wherein the image weak change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than zero and less than a preset degree; the image strong change region is used to indicate that the degree of pixel change between the image region and the corresponding reference region is greater than or equal to a preset degree; When at least one of the motion vector and the residual data of the qth image block is non-zero, determining that the region category of the qth image region is an image change region includes: When at least one of the motion vector and residual data of the qth image block is not zero, if the at least one data is residual data and the residual data is less than a preset threshold, the area category of the qth image area is determined to be an image weak change area; if the at least one data includes a motion vector, the area category of the qth image area is determined to be an image strong change area.

6. The method according to claim 1, wherein When the number of region categories is two, and the two region categories include image unchanged regions and image changed regions: If the region category of the qth image region is an image unchanged region, the super-resolution method of the qth image region is an interpolation super-resolution method or a method of reusing the super-resolution result of the reference region; If the region category of the qth image region is the image change region, the super-resolution method of the qth image region is the model super-resolution method; Wherein, q is a positive integer and is less than or equal to the number of image regions; the computational complexity of the interpolation super-resolution method and the computational complexity of the method of reusing the super-resolution results of the reference region are both less than the computational complexity of the model super-resolution method.

7. The method according to claim 1, wherein When the number of region categories is three, and the three region categories include image unchanged regions, image weakly changed regions, and image strongly changed regions: If the region category of the qth image region is an image unchanged region, the super-resolution method of the qth image region is a method of reusing the super-resolution result of the reference region; If the region category of the qth image region is the image weak change region, the super-resolution method of the qth image region is the interpolation super-resolution method; If the region category of the qth image region is the image strong change region, the super-resolution method of the qth image region is the model super-resolution method; Wherein, q is a positive integer and is less than or equal to the number of image regions; the computational complexity of the method of reusing the super-resolution results of the reference region is less than the computational complexity of the interpolation super-resolution method, and the computational complexity of the interpolation super-resolution method is less than the computational complexity of the model super-resolution method.

8. The method according to claims 1 to 7, characterized in that The super-resolution method of each image region is adopted to perform super-resolution processing on the corresponding image region to obtain the super-resolution result of each image region, including: If the super-resolution method of the qth image region is a method of reusing the super-resolution result of the reference region, obtaining the super-resolution result of the reference region corresponding to the qth image region, and using the obtained super-resolution result as the super-resolution result of the qth image region; If the super-resolution method of the qth image region is an interpolation super-resolution method, an interpolation algorithm is used to perform interpolation processing based on the qth image region to generate a super-resolution result of the qth image region; If the super-resolution method of the qth image region is the model super-resolution method, calling the image super-resolution model to perform super-resolution processing on the qth image region to obtain a super-resolution result of the qth image region; Wherein, q is a positive integer and is less than or equal to the number of image regions.

9. The method according to claim 8, wherein The adopting an interpolation algorithm to perform interpolation processing based on the qth image region to generate a super-resolution result of the qth image region includes: Performing interpolation processing on the qth image region using an interpolation algorithm to obtain a super-resolution result of the qth image region; Alternatively, residual information between the qth image region and the corresponding reference region is obtained, an interpolation algorithm is used to interpolate the obtained residual information, and the interpolated residual information and the super-resolution result of the reference region corresponding to the qth image region are superimposed to obtain the super-resolution result of the qth image region.

10. The method according to claim 8, wherein: The method for generating the image super-resolution model includes: Constructing an initial super-resolution model and obtaining training data; the training data includes: a sample image of a first resolution and a sample image of a second resolution rendered based on the same picture, wherein the first resolution is smaller than the second resolution; Calling the initial super-resolution model to perform super-resolution processing on the sample image of the first resolution to obtain a super-resolution image of the second resolution; Using the loss function of the initial super-resolution model, performing a loss value calculation based on the sample image of the second resolution and the super-resolution image of the second resolution to obtain a model loss value; The model parameters of the initial super-resolution model are optimized in a direction of reducing the model loss value to obtain an image super-resolution model.

11. The method according to claim 10, wherein The obtaining of training data includes: Encode two video streams based on the sample picture data, wherein the two video streams include a video stream with a first resolution and a video stream with a second resolution; Transmitting the video stream of the first resolution to a test device via a network, causing the test device to decode the video stream of the first resolution, and after obtaining a first decoding result, transmitting the first decoding result back; the first decoding result includes: at least one sample image of the first resolution; Decoding the video stream at the second resolution to obtain a second decoding result; the second decoding result includes: at least one sample image at the second resolution; Performing screen alignment on the sample image in the first decoding result and the sample image in the second decoding result, and acquiring training data from the result of the screen alignment.

12. The method according to claim 10, wherein Optimizing the model parameters of the initial super-resolution model in a direction of reducing the model loss value to obtain an image super-resolution model, including: Optimizing the model parameters of the initial super-resolution model in a direction of reducing the model loss value to obtain an optimized initial super-resolution model; If the optimized initial super-resolution model reaches convergence, the optimized initial super-resolution model is equivalently transformed using the re-parameterization technology to obtain an image super-resolution model; wherein the complexity of the model structure of the image super-resolution model is less than the complexity of the optimized initial super-resolution model, and the model parameters of the image super-resolution model are equivalent to the model parameters of the optimized initial super-resolution model.

13. An image processing device, characterized in that: include: an acquisition unit, configured to acquire a plurality of image regions of a current image frame, wherein the current image frame is an image frame to be super-resolved in an image frame sequence; The acquisition unit is further configured to acquire a region category of each image region, where the region category of any image region indicates a degree of pixel change between the image region and a corresponding reference region; the reference region corresponding to any image region is a region in a reference image frame that matches the corresponding image region, where the reference image frame is an image frame in the image frame sequence; a processing unit, configured to determine a super-resolution method for each image region according to the region category of each image region; different super-resolution methods have different computational complexities, and the computational complexity of the super-resolution method for any image region is proportional to the degree of pixel change indicated by the region category of the corresponding image region; The processing unit is further used to adopt the super-resolution method of each image area to perform super-resolution processing on the corresponding image area to obtain the super-resolution results of each image area; and integrate the super-resolution results of each image area to obtain the super-resolution image corresponding to the current image frame.

14. A computer device comprising an input interface and an output interface, characterized in that: Also includes: processors and computer storage media; The processor is suitable for implementing one or more instructions, the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by the processor and executing the image processing method according to any one of claims 1 to 12.

15. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the image processing method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product includes one or more instructions; when the one or more instructions in the computer program are executed by a processor, the image processing method according to any one of claims 1 to 12 is implemented.

Images