An image super-resolution method, device, server and storage medium

Abstract

The application relates to the field of image processing technology, in particular to an image super-resolution method and device, a server and a storage medium. The method comprises the following steps: obtaining a target resolution of a target video stream and a current resolution of the target video stream; if the current resolution is smaller than the target resolution, determining a processing mode for performing resolution processing on the target video stream; if the determined processing mode comprises super-resolution processing, performing super-resolution processing on the target video stream, so that the resolution of the target video stream after the super-resolution processing is the target resolution. The application has the limitation of the game, and the effect of improving the resolution of the video stream is achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image super-resolution method, apparatus, server, and storage medium. Background Technology

[0002] As image processing technology continues to develop, people are demanding higher and higher image resolutions in order to obtain clearer images. To meet people's needs for images with different resolutions, image resolution adjustment technology has emerged.

[0003] However, the resolution of video frames output by cloud gaming is still limited by the game's limitations. Therefore, how to improve the resolution of video frames has become a key issue. Summary of the Invention

[0004] To address at least one of the above technical problems, this application provides an image super-resolution method, apparatus, server, and storage medium.

[0005] Firstly, this application provides an image super-resolution method, which adopts the following technical solution:

[0006] An image super-resolution method includes:

[0007] Obtain the target resolution of the target video stream and the current resolution of the target video stream;

[0008] If the current resolution is less than the target resolution, then a resolution processing method for the target video stream is determined.

[0009] If the determined processing method includes super-resolution processing, then the target video stream is subjected to super-resolution processing so that the resolution of the target video stream after super-resolution processing is the target resolution.

[0010] By adopting the above technical solution, the target resolution and the current resolution of the target video stream are obtained. Then, the target resolution and the current resolution of the target video stream are compared. If the current resolution is less than the target resolution, the corresponding resolution processing method for the target video stream is determined. If the determined processing method includes super-resolution processing, super-resolution processing can be performed on the target video stream to increase its resolution from the current resolution to the target resolution. Since the resolution adjustment obtained by super-resolution processing of the video stream is not limited to game limitations to improve the resolution of the video stream.

[0011] In one possible implementation, determining the processing method for resolution processing of the target video stream includes:

[0012] If the current resolution and the target resolution satisfy a first preset relationship, then a resolution processing method for the target video stream is determined, wherein the first preset relationship is that the target resolution is greater than the current resolution by a preset multiple;

[0013] The process of performing super-resolution processing on the target video stream further includes, prior to:

[0014] Determine the resolution of the rendered image based on the target resolution;

[0015] Based on the resolution of the rendered image, the target video stream is rendered.

[0016] The super-resolution processing of the target video stream includes:

[0017] Super-resolution processing is performed on the rendered target video stream.

[0018] By adopting the above technical solution, when comparing the current resolution and the target resolution, if the current resolution and the target resolution satisfy a first preset relationship, that is, when the target resolution is greater than a preset multiple of the current resolution, then the resolution processing method for the target video stream is determined. Since the target resolution is significantly different from the current resolution, the resolution processing method for the target video stream can be determined, thereby improving the accuracy of determining the resolution processing method while reducing the computational cost, thus increasing the speed of resolution processing for the target video stream. Furthermore, when performing resolution processing on the target video stream, the image can be rendered to a certain resolution first, and then super-resolution processing can be performed, thereby improving the video stream quality while reducing graphics card resources.

[0019] In one possible implementation, determining the resolution of the rendered image based on the target resolution includes any of the following:

[0020] The resolution of the rendered image is determined based on the target resolution and the preset multiplier.

[0021] Obtain the current usage information of the graphics card, and determine the resolution of the rendered image based on the target resolution and the current usage information of the graphics card.

[0022] By adopting the above technical solution, on the one hand, the target resolution is compared with a preset multiplier, and the resolution after image rendering is determined based on the relationship between the target resolution and the preset multiplier. This allows for a relatively accurate determination of the resolution after image rendering, further reducing graphics card resource consumption without affecting image quality. On the other hand, the current graphics card usage information is obtained, and then the resolution after image rendering is determined based on the relationship between the target resolution and the current graphics card usage information. In other words, the resolution after image rendering can be accurately determined based on the current graphics card usage, improving image quality while minimizing excessive graphics card usage.

[0023] In one possible implementation, the super-resolution processing of the target video stream is performed prior to:

[0024] Determine if the current graphics card supports Deep Learning Super Sampling (DLSS) processing;

[0025] The super-resolution processing of the target video stream includes:

[0026] If the current graphics card supports DLSS processing, then the target video stream is subjected to DLSS processing and super-resolution processing.

[0027] If the current graphics card does not support DLSS processing, then the target video stream is subjected to the super-resolution processing.

[0028] By adopting the above technical solution, if the current graphics card can support DLSS processing during super-resolution processing, DLSS processing can be performed on the target video stream during super-resolution processing. In other words, the resolution of the target video stream can be adjusted through DLSS and super-resolution processing, thereby further reducing the consumption of graphics card resources.

[0029] In one possible implementation, determining the processing method for resolution processing of the target video stream includes:

[0030] If the current resolution and the target resolution satisfy a second preset relationship, then obtain the current usage information of the graphics card;

[0031] The processing method for resolution processing of the target video stream is determined based on the current occupancy information of the graphics card.

[0032] By adopting the above technical solution, when comparing the current resolution with the target resolution, if the current resolution and the target resolution satisfy the second preset relationship, that is, when the target resolution is small, the over-resolution will result in poor image quality, the processing method for resolution processing of the target video stream can be determined based on the current usage information of the graphics card, so that the determined resolution can take into account both the graphics card usage information and the image quality.

[0033] In one possible implementation, the method of obtaining the target resolution of the target video stream includes any of the following:

[0034] Obtain the resolution set by the user through the client, and use the resolution set by the client as the target resolution of the target video stream;

[0035] The resolution set by the user through the client and the resolution corresponding to the display on which the target video stream is to be displayed are obtained, and the target resolution of the target video stream is determined based on the resolution set by the client and the resolution corresponding to the display.

[0036] Obtain the game type corresponding to the target video stream, and determine the target resolution of the target video stream based on the game type corresponding to the target video stream.

[0037] By adopting the above technical solution, on the one hand, the user sets the required resolution through the client, and the resolution set by the user through the client is obtained and used as the target resolution of the target video stream; thus, the user's resolution needs are taken into account to the greatest extent, thereby improving the user experience. On the other hand, the resolution set by the user through the client and the resolution supported by the monitor included in the client are obtained. Subsequently, based on the user's needs and the conditions of the user's hardware, the target resolution of the target video stream is determined, thereby improving the user experience based on the conditions of the hardware used. On the other hand, the game type corresponding to the target video stream is obtained. With the condition that different game types correspond to suitable resolutions, the target resolution of the target video stream is determined based on the game type corresponding to the target video stream, so as to reduce the user's effort in dealing with game resolution issues and improve the quality of game services.

[0038] In one possible implementation, the step of performing super-resolution processing on the target video stream so that the resolution of the super-resolution processed target video stream is the target resolution further includes:

[0039] The target video stream after super-resolution processing is then encoded.

[0040] The encoded target video stream is sent to the client.

[0041] By adopting the above technical solution, the target video stream is subjected to super-resolution processing, and then the super-resolution processed target video stream is encoded to make the target video stream meet the requirements for display on the client. The encoded target video stream is then sent to the client to provide game services to the user.

[0042] Secondly, this application provides an image super-resolution device, which adopts the following technical solution:

[0043] An image super-resolution device includes: a resolution acquisition module, a processing method determination module, and a resolution processing module, wherein...

[0044] A resolution acquisition module is used to acquire the target resolution of the target video stream and the current resolution of the target video stream;

[0045] The processing method determination module is used to determine the processing method for resolution processing of the target video stream if the current resolution is less than the target resolution.

[0046] The resolution processing module is used to perform super-resolution processing on the target video stream if the determined processing method includes super-resolution processing, so that the resolution of the target video stream after super-resolution processing is the target resolution.

[0047] By adopting the above technical solution, the resolution acquisition module obtains the target resolution and the current resolution of the target video stream. Subsequently, the processing method determination module compares the target resolution and the current resolution of the video stream. If the current resolution is less than the target resolution, the processing method for the resolution processing of the target video stream is determined. If the determined processing method includes super-resolution processing, super-resolution processing can be performed on the target video stream, so that the resolution processing module can super-resolution the target video stream from the current resolution to the target resolution. Since the resolution adjustment obtained by super-resolution processing of the video stream is not limited to the game limitation to improve the resolution of the video stream.

[0048] In one possible implementation, the processing method determining module, when determining the processing method for resolution processing of the target video, is specifically used for:

[0049] If the current resolution and the target resolution satisfy a first preset relationship, when determining the processing method for resolution processing of the target video stream, the first preset relationship is that the target resolution is greater than the current resolution by a preset multiple;

[0050] The image super-resolution device further includes: a resolution determination module and an image rendering module, wherein...

[0051] The resolution determination module is used to determine the resolution of the rendered image based on the target resolution;

[0052] The image rendering module is used to perform image rendering on the target video stream based on the resolution of the rendered image;

[0053] Specifically, when performing super-resolution processing on the target video stream, the resolution processing module is used for:

[0054] Super-resolution processing is performed on the rendered target video.

[0055] In one possible implementation, the resolution determination module is specifically used for any of the following:

[0056] The resolution of the rendered image is determined based on the target resolution and the preset multiplier.

[0057] Obtain the current usage information of the graphics card, and determine the resolution of the rendered image based on the target resolution and the current usage information of the graphics card.

[0058] In one possible implementation, the image super-resolution device further includes: a DLSS processing mode determination module, wherein,

[0059] The DLSS processing mode determination module is used to determine whether the current graphics card supports deep learning super sampling DLSS processing.

[0060] The resolution processing module further includes: a first processing unit and a second processing unit, wherein...

[0061] The first processing unit is configured to perform DLSS processing and super-resolution processing on the target video stream if the current graphics card supports DLSS processing.

[0062] The second processing unit is configured to perform super-resolution processing on the target video stream if the current graphics card does not support DLSS processing.

[0063] In one possible implementation, the processing method determination module further includes: a graphics card occupancy information acquisition unit and a processing method determination unit, wherein,

[0064] The graphics card information acquisition unit is used to acquire the current occupancy information of the graphics card if the current resolution and the target resolution satisfy a second preset relationship.

[0065] The processing method determination unit is used to determine the processing method for resolution processing of the target video stream based on the current occupancy information of the graphics card.

[0066] In one possible implementation, the resolution acquisition module is specifically used for any of the following:

[0067] Obtain the resolution set by the user through the client, and use the resolution set by the client as the target resolution of the target video stream;

[0068] The resolution set by the user through the client and the resolution corresponding to the display on which the target video stream is to be displayed are obtained, and the target resolution of the target video stream is determined based on the resolution set by the client and the resolution corresponding to the display.

[0069] Obtain the game type corresponding to the target video stream, and determine the target resolution of the target video stream based on the game type corresponding to the target video stream.

[0070] In one possible implementation, the image super-resolution device further includes: an encoding processing module and a feedback module, wherein,

[0071] The encoding processing module is used to encode the target video stream after super-resolution processing;

[0072] The feedback module is used to send the encoded target video stream to the client.

[0073] Thirdly, this application provides a server-side solution, which adopts the following technical solution:

[0074] A server, comprising:

[0075] At least one processor;

[0076] Memory;

[0077] At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: perform the above-described image super-resolution method.

[0078] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0079] A computer-readable storage medium includes: a computer program stored thereon that can be loaded by a processor and execute the above-described image super-resolution method.

[0080] In summary, this application includes the following beneficial technical effects:

[0081] The system obtains the target resolution and the current resolution of the target video stream. Then, it compares the target resolution and the current resolution. If the current resolution is lower than the target resolution, it determines the resolution processing method for the target video stream. If the determined processing method includes super-resolution processing, it performs super-resolution processing on the target video stream to raise its resolution from the current resolution to the target resolution. The resolution adjustment obtained by super-resolution processing of the video stream is not limited to game-specific limitations to improve the video stream resolution. Attached Figure Description

[0082] Figure 1 This is a schematic flowchart of the image super-resolution method according to an embodiment of this application;

[0083] Figure 2 This is a schematic flowchart of an example image super-resolution method according to an embodiment of this application;

[0084] Figure 3 This is a block diagram of an image super-resolution device according to an embodiment of this application;

[0085] Figure 4 This is a schematic diagram of the server side in an embodiment of this application. Detailed Implementation

[0086] The following is in conjunction with the appendix Figure 1-4 This application will be described in further detail.

[0087] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0088] Currently, the resolution of video frames output by cloud gaming is limited by the game's constraints, resulting in two issues: First, the video frames output by the cloud and the rendered images in the game cannot be made at different resolutions. Second, since the maximum rendering resolution of the game is 1920×1080, the maximum resolution after cloudification is also 1920×1080, which cannot meet the needs of some users. Furthermore, rendering images through the game to obtain higher resolution images consumes more graphics card resources, resulting in higher costs.

[0089] In order to obtain higher resolution images while reducing the graphics card resources and costs, and to further meet the high-resolution needs of some users, this application provides an image super-resolution method, as detailed in the following embodiments.

[0090] It should be noted that the embodiments of this application are used to illustrate the application scenario of providing cloud gaming services to users, but the embodiments of this application are not limited to this application scenario.

[0091] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0092] First, let's introduce and explain several terms used in this application:

[0093] Super-resolution: This refers to improving the resolution of an existing image through hardware or software methods. In this embodiment, super-resolution is achieved by improving the resolution through hardware, specifically the graphics card.

[0094] Deep Learning Super Sampling (DLSS): This technique reduces the rendering resolution within the game while simultaneously stretching the output image using artificial intelligence algorithms and AI-accelerated hardware units, thereby increasing the display resolution.

[0095] The following description of several exemplary embodiments illustrates the technical solutions of this application and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Similar features, identical elements, and similar implementation steps in different embodiments will not be described again.

[0096] This application provides an image super-resolution method, executed by a server, as described above. Figure 1 The method may include steps S101, S102, and S103, wherein,

[0097] Step S101: Obtain the target resolution of the target video stream and the current resolution of the target video stream.

[0098] In the embodiments of this application, the target video stream is the video information that needs to be displayed on the user's client; the target resolution of the target video stream is the resolution when displayed on the user's client; and the current resolution of the target video stream is the resolution of the target video stream when no resolution adjustment is performed.

[0099] Before being sent to the user's client for display, the target video stream is temporarily stored in the storage space. At this time, the target video stream generally has a small resolution for easy storage. If the target video stream needs to be sent to the user's client for display, the resolution of the current target video stream needs to be adjusted so that the target video stream can meet the user's needs and / or be displayed on the user's client at the most suitable resolution. Therefore, whether to adjust the resolution of the video stream can be determined according to the user's needs. Specifically, when the user enters the display request and target resolution of the target video stream through the client, the server obtains the target resolution of the target video stream through the client and retrieves the target video stream from the storage space.

[0100] Furthermore, the storage space used to store the video stream can be a storage module included in the server, such as HDD and RAM; it can also be an independent storage module outside the server. This application embodiment does not make specific limitations.

[0101] Furthermore, the server can directly obtain the resolution corresponding to the temporarily stored target video stream from the storage space and use that resolution as the current resolution; alternatively, the server can obtain the rendered resolution of the target video stream after image rendering and use that resolution as the current resolution; or the server can use the resolution corresponding to the target video stream after DLSS processing as the current resolution.

[0102] Furthermore, whether to adjust the resolution of the video stream can be determined based on the resolution supported by the user's monitor, or based on the resolution recommended by the game type for which the video stream is to be displayed.

[0103] Step S102: If the current resolution is less than the target resolution, determine the processing method for resolution processing of the target video stream.

[0104] In the embodiments of this application, when the server obtains the current resolution and the target resolution, it will determine the resolution processing method corresponding to the target video stream based on the relationship between the current resolution and the target resolution.

[0105] Specifically, the server compares the current resolution with the target resolution. If the current resolution is less than the target resolution, it means that the current resolution of the target video stream does not meet the requirements, and resolution processing is required for the target video stream. That is, the processing method for resolution processing of the target video stream is determined.

[0106] To further explain, there are multiple ways to process the resolution of the target video stream. For example, it can be done by rendering the target video stream using the graphics card included on the server; it can also be done by using the resolution processing program corresponding to the video stream itself; it can also be done by using super-resolution technology; or it can be done by combining at least two of the above methods. Therefore, selecting a processing method that is appropriate for the target video stream's resolution is beneficial in terms of both user experience and video stream adjustment and processing costs.

[0107] Step S103: If the determined processing method includes super-resolution processing, then perform super-resolution processing on the target video stream so that the resolution of the target video stream after super-resolution processing is the target resolution.

[0108] In this embodiment of the application, if the resolution processing method for the target video stream determined by the server includes super-resolution processing, the server performs super-resolution processing on the target video stream. In this embodiment, the determined processing method includes super-resolution processing, which may include: directly performing super-resolution processing on the target video stream so that the resolution of the super-resolution processed target video stream is the target resolution; or, first performing other resolution processing on the target video stream (e.g., image rendering, resolution enhancement through a resolution processing program), and then performing super-resolution processing on the resolution-processed target video stream to obtain the target resolution.

[0109] Furthermore, since the computational cost of super-resolution is less than that of rendering the video stream by the graphics card when processing the target video stream, super-resolution can reduce the resource consumption of the graphics card, thereby saving graphics card resources.

[0110] This application provides an image super-resolution method. The method obtains the target resolution and the current resolution of a target video stream. Then, it compares the target resolution and the current resolution. If the current resolution is less than the target resolution, it determines the corresponding resolution processing method for the target video stream. If the determined processing method includes super-resolution processing, then super-resolution processing can be performed on the target video stream to increase its resolution from the current resolution to the target resolution. Since the resolution adjustment obtained by super-resolution processing of the video stream is not limited to game-specific applications, it is not solely aimed at improving the video stream's resolution.

[0111] Specifically, in step S101, the method for obtaining the target resolution of the target video stream may include any one of the following (method 1, method 2, and method 3):

[0112] Method 1: Obtain the resolution set by the user through the client and use the resolution set by the client as the target resolution of the target video stream.

[0113] Specifically, users can input or select the resolution of the target video stream to be displayed through the client. Subsequently, the server obtains the resolution set by the user's client and uses that resolution as the target resolution of the target video stream.

[0114] Method 2: Obtain the resolution set by the user through the client and the resolution corresponding to the monitor of the target video stream to be displayed, and determine the target resolution of the target video stream based on the resolution set through the client and the resolution corresponding to the monitor.

[0115] Specifically, displays used to show target video streams typically have limitations on the resolution of the target video stream, supporting a certain range of resolutions for display. For example, the target resolution may be between 960x540 and 1920x1080. Therefore, the resolution supported by the display used to show the target video stream is one of the factors that influences the determination of the target resolution of the target video stream. So, if the resolution set by the user through the client is greater than the resolution corresponding to the display on which the target video stream is to be displayed, then the target resolution of the target video stream is the resolution corresponding to the display on which the target video stream is to be displayed. If the resolution set by the user through the client is less than the resolution corresponding to the display on which the target video stream is to be displayed, then the target resolution of the target video stream is the resolution set by the user through the client. If the resolution set by the user through the client is the same as the resolution corresponding to the display on which the target video stream is to be displayed, then the target resolution of the target video stream is either the resolution set by the user through the client or the resolution corresponding to the display on which the target video stream is to be displayed.

[0116] Method 3: Obtain the game type corresponding to the target video stream, and determine the target resolution of the target video stream based on the game type.

[0117] Specifically, different game genres have different resolution requirements for display. For example, games with more detailed scenes require higher resolution to ensure a better user experience; games with simpler scenes have lower resolution requirements, meaning that lower resolution has less impact on the user. In other words, the game genre has a certain influence on determining the target resolution of the target video. Therefore, when obtaining the target resolution of the target video, the server can obtain the game genre corresponding to the target video stream and determine the target resolution of the target video stream based on that game genre.

[0118] Furthermore, after obtaining the target resolution and the current resolution of the target video stream, if the current resolution of the target video stream is less than the target resolution, a resolution processing method for the target video stream is further determined. In this embodiment, determining the resolution processing method for the target video stream may specifically include: if the current resolution and the target resolution satisfy a first preset relationship, then determining the resolution processing method for the target video stream; wherein, before performing super-resolution processing on the target video stream, the method further includes: determining the resolution after image rendering based on the target resolution; performing image rendering on the target video stream based on the resolution after image rendering; wherein, performing super-resolution processing on the target video stream includes: performing super-resolution processing on the rendered target video stream.

[0119] The first preset relationship is that the target resolution is greater than the current resolution by a preset multiple. For example, if the current resolution is 960×540 and the target resolution is 2560×1440, and the preset multiple is 2, then the target resolution is greater than twice the current resolution.

[0120] Furthermore, if the target resolution is greater than a preset multiple of the current resolution, the resolution processing method for the target video stream can be determined to include: super-resolution processing of the target video stream. In this embodiment, when performing super-resolution processing on the target video stream, image rendering of the target video stream can be performed first, and then super-resolution processing can be performed on the rendered video stream. That is, before performing super-resolution processing on the target video stream, the method further includes: determining the resolution after image rendering based on the target resolution; performing image rendering on the target video stream based on the resolution after image rendering; wherein, performing super-resolution processing on the target video stream includes: performing super-resolution processing on the rendered target video stream.

[0121] For example, the current target video stream has a resolution of 960×540, and the target resolution is 2560×1440. During this process, the target video stream needs to be rendered, that is, the target video stream is rendered from a resolution of 960×540 to a resolution of 1920×1080, that is, the current resolution is 1920×1080. After that, the server starts to perform super-resolution processing on the image and video stream until the target video stream has a resolution of 2560×1440.

[0122] Specifically, the resolution of the rendered image is determined based on the target resolution, including any one of the following (Method 1 and Method 2), wherein,

[0123] Method 1: Determine the resolution of the rendered image based on the target resolution and a preset multiplier.

[0124] Specifically, the server determines the resolution after image rendering based on the relationship between the target resolution and the preset multiplier. The resolution after image rendering is the resolution after the target video stream has been rendered first.

[0125] Furthermore, in the embodiments of this application, the resolution after image rendering can be the target resolution / preset multiple, or the resolution after image rendering can be determined by other relationships between the target resolution and the preset multiple, and no limitation is made in the embodiments of this application.

[0126] For example, if the target resolution is 1920×1080 and the preset multiplier is 2, the current resolution can be determined to be 960×540. At this time, the server can determine that the 960×540 resolution is the resolution after image rendering.

[0127] Method 2: Obtain the current usage information of the graphics card, and determine the resolution of the rendered image based on the target resolution and the current usage information of the graphics card.

[0128] Specifically, different resolution processing methods applied to the target video stream will result in different GPU usage. Therefore, the resolution after image rendering can be determined by the current GPU usage. In this embodiment, if the current GPU usage information indicates a high GPU utilization rate, the determined image rendering resolution will be lower. For example, if the target resolution is 1920×1080, the determined image rendering resolution can be 400×400. Conversely, if the current GPU usage information indicates a low GPU utilization rate, the determined image rendering resolution will be higher. For example, if the target resolution is 1920×1080, the determined image rendering resolution can be 960×540.

[0129] In other words, after determining the resolution of the rendered image using the two methods mentioned above, the target video stream is first rendered to achieve the determined resolution. Then, the rendered video stream is processed using super-resolution to achieve the target resolution. For example, the target video stream is first rendered to achieve a resolution of 400×400, and then the rendered video stream is processed using super-resolution to achieve a resolution of 1920×1080.

[0130] Furthermore, super-resolution processing methods also include DLSS processing; some graphics cards now support DLSS processing. That is, before performing super-resolution processing on the target video stream, it may also include determining whether the current graphics card supports deep learning super sampling (DLSS) processing. In other words, after determining that super-resolution processing of the target video stream is needed, it can be first determined whether the current graphics card supports DLSS processing to decide whether to consider performing DLSS processing when performing super-resolution processing on the target video stream.

[0131] In other words, super-resolution processing of the target video stream can specifically include: if the current graphics card supports DLSS processing, then DLSS processing and super-resolution processing are performed on the target video stream; if the current graphics card does not support DLSS processing, then super-resolution processing is performed on the target video stream. In this embodiment, if the current graphics card supports DLSS processing, DLSS can be enabled in the game to perform DLSS processing and super-resolution processing on the target video stream; if the current graphics card does not support DLSS processing, then super-resolution processing is performed on the target video stream. Furthermore, if the current graphics card supports DLSS processing, super-resolution processing can also be performed on the target video stream.

[0132] It should be noted that, in the embodiments of this application, if the current graphics card can support DLSS, then performing DLSS processing and super-resolution processing on the target video stream, or performing DLSS processing on the target video stream, can further save graphics card resources.

[0133] Specifically, while most users seek high-resolution video streams, some users do not require high resolution. When the target resolution is not high, using super-resolution technology to process the target video stream may result in low-quality video streams and less GPU resource saving.

[0134] For example, a resolution of 960×540; if the user's resolution requirement is a smaller resolution, i.e., 960×540, whether to perform super-resolution processing on the target video stream corresponding to the 960×540 resolution requires further judgment. Specifically, the processing method for processing the target video stream's resolution needs to be determined. This can include: if the current resolution and the target resolution satisfy a second preset relationship, then obtaining the current occupancy information of the graphics card; and determining the processing method for processing the target video stream's resolution based on the current occupancy information of the graphics card.

[0135] In this embodiment of the application, the server has a second preset relationship and a preset resolution value. The second preset relationship is that the current resolution is less than the target resolution, and the target resolution is less than the preset resolution value. If the server determines that the current resolution and the target resolution satisfy the second preset relationship, that is, the current resolution is less than the target resolution, and the target resolution is less than the preset resolution value, for example, the preset resolution value is 1000×1000, the current resolution of the target video stream is 400×400, and the target resolution required by the user is 800×800, the current target resolution of the target video stream is relatively small, but the current resolution target is less than the target resolution, so resolution adjustment is still required. Even if only the graphics card is used for image rendering, the resource utilization of the graphics card will not be high. Therefore, in this case, whether to use a resolution processing method other than graphics card rendering can be determined based on the current utilization of the graphics card. That is, the server obtains the current graphics card utilization information and determines the processing method for resolution processing of the target video stream based on the current utilization information of the graphics card.

[0136] Specifically, if the current utilization rate of the graphics card is low, super-resolution processing of the target video stream can be omitted. In this embodiment, if the current utilization rate of the graphics card is low, image rendering can be performed on the target video stream so that the resolution of the rendered video stream is the target resolution; alternatively, the resolution of the target video stream can be adjusted using the resolution processing program built into the game so that the resolution of the rendered video stream is the target resolution; or a combination of the two resolution processing methods can be used to adjust the resolution of the target video stream; of course, at least one of the above two resolution processing methods, and / or DLSS processing method, and / or super-resolution processing method can be combined to perform resolution processing on the target video stream. No limitations are imposed in this embodiment.

[0137] Furthermore, after adjusting the resolution of the target video stream through the above embodiments so that the adjusted target video stream has the target resolution, in order to further enable users to view the target video stream after resolution processing, super-resolution processing is performed on the target video stream so that the resolution of the super-resolution processed target video stream has the target resolution. Then, it may further include: encoding the super-resolution processed target video stream; and sending the encoded target video stream to the client.

[0138] In other words, if super-resolution processing of the target video stream is required, the target video stream after super-resolution processing is encoded, and then the encoded target video stream is pushed to the client so that the client can display the encoded target video stream.

[0139] Furthermore, if super-resolution processing of the target video stream is not required, the resolution of the target video stream can be adjusted through the above embodiments to make the adjusted target video stream resolution the target resolution. Then, the resolution-processed target video stream can be encoded and pushed to the client so that the client can display the encoded target video stream.

[0140] The above embodiments detail an image super-resolution method. The following describes an image super-resolution method using an example, such as... Figure 2 As shown, the user can set the target resolution r1 of the target video stream through the game page on the client, for example, 1920×1080. Other modules can obtain the target resolution of the target video stream, then start the game, set the game resolution to r1 / 2, for example, 960×540, and then render it through the game rendering module. After rendering is complete, the rendered game video stream is sent to the super-resolution module. At this time, the resolution of the video stream is r2, which is r1 / 2. Then, the super-resolution module super-resolutions the game video stream from r2 to r1, and then encodes the super-resolution game video stream and sends the encoded game video stream to the client. The game rendering module and the super-resolution module belong to the game module.

[0141] The above embodiments describe an image super-resolution method from the perspective of process flow. The following embodiments describe an image super-resolution apparatus from the perspective of virtual modules or virtual units. For details, please refer to the following embodiments.

[0142] The image super-resolution device 30 may specifically include: a resolution acquisition module 301, a processing method determination module 302, and a resolution processing module 303, wherein,

[0143] The resolution acquisition module 301 is used to acquire the target resolution of the target video stream and the current resolution of the target video stream;

[0144] The processing method determination module 302 is used to determine the processing method for resolution processing of the target video stream if the current resolution is less than the target resolution.

[0145] The resolution processing module 303 is used to perform super-resolution processing on the target video stream if the determined processing method includes super-resolution processing, so that the resolution of the target video stream after super-resolution processing is the target resolution.

[0146] In one possible implementation of this application embodiment, the processing method determination module 302, when determining the processing method for resolution processing of the target video, is specifically used for:

[0147] If the current resolution and the target resolution satisfy a first preset relationship, the processing method for resolution processing of the target video stream is determined. The first preset relationship is that the target resolution is greater than the current resolution by a preset multiple.

[0148] The image super-resolution device further includes: a resolution determination module and an image rendering module.

[0149] The resolution determination module is used to determine the resolution of the rendered image based on the target resolution.

[0150] The image rendering module is used to render images of the target video stream based on the resolution of the rendered images.

[0151] Specifically, the resolution processing module, when performing super-resolution processing on the target video stream, is used for:

[0152] Perform super-resolution processing on the rendered target video.

[0153] One possible implementation of this application embodiment is that the resolution determination module is specifically used for any of the following:

[0154] Determine the resolution of the rendered image based on the target resolution and the preset magnification.

[0155] Obtain the current usage information of the graphics card, and determine the resolution of the rendered image based on the target resolution and the current usage information of the graphics card.

[0156] In one possible implementation of this application embodiment, the image super-resolution device 30 further includes: a DLSS processing mode determination module, wherein...

[0157] The DLSS processing mode determination module is used to determine whether the current graphics card supports deep learning super sampling DLSS processing.

[0158] The resolution processing module further includes: a first processing unit and a second processing unit, wherein...

[0159] The first processing unit is used to perform DLSS processing and super-resolution processing on the target video stream if the current graphics card supports DLSS processing.

[0160] The second processing unit is used to perform super-resolution processing on the target video stream if the current graphics card does not support DLSS processing.

[0161] In one possible implementation of this application embodiment, the processing method determination module 302 further includes: a graphics card occupancy information acquisition unit and a processing method determination unit, wherein...

[0162] The graphics card information acquisition unit is used to acquire the current usage information of the graphics card if the current resolution and the target resolution satisfy a second preset relationship.

[0163] The processing method determination unit is used to determine the processing method for resolution processing of the target video stream based on the current occupancy information of the graphics card.

[0164] In one possible implementation of this application embodiment, the resolution acquisition module 301 is specifically used for any of the following:

[0165] Obtain the resolution set by the user through the client, and use the resolution set by the client as the target resolution of the target video stream;

[0166] Obtain the resolution set by the user through the client and the resolution corresponding to the monitor of the target video stream to be displayed, and determine the target resolution of the target video stream based on the resolution set through the client and the resolution corresponding to the monitor;

[0167] Obtain the game type corresponding to the target video stream, and determine the target resolution of the target video stream based on the game type.

[0168] In one possible implementation of this application embodiment, the image super-resolution device 30 further includes: an encoding processing module and a feedback module, wherein...

[0169] The encoding processing module is used to encode the target video stream after super-resolution processing;

[0170] The feedback module is used to send the encoded target video stream to the client.

[0171] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0172] This application also describes a server from the perspective of a physical device, such as... Figure 4 As shown, Figure 4 The server 40 shown includes a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, for example, via a bus 402. Optionally, the server 400 may also include a transceiver 404. It should be noted that in practical applications, the transceiver 404 is not limited to one type, and the structure of the server 400 does not constitute a limitation on the embodiments of this application.

[0173] Processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0174] Bus 402 may include a pathway for transmitting information between the aforementioned components. Bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 402 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0175] The memory 403 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0176] The memory 403 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the content shown in the foregoing method embodiments.

[0177] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0178] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image super-resolution method, characterized by, The method comprises the following steps: acquiring a target resolution of a target video stream and a current resolution of the target video stream; if the current resolution is less than the target resolution, determining a processing mode of resolution processing of the target video stream according to a relationship between the current resolution and the target resolution, a resolution supported by a display, a game type corresponding to the target video stream, and GPU occupation information; specifically, if the target resolution is greater than a preset multiple of the current resolution, it is determined that the processing mode of resolution processing of the target video stream comprises super-resolution processing; if it is determined that the processing mode comprises super-resolution processing, performing super-resolution processing on the target video stream, so that a resolution of the target video stream after super-resolution processing is the target resolution; before the super-resolution processing on the target video stream, the method further comprises the following steps: determining a resolution after image rendering according to the target resolution; performing image rendering on the target video stream based on the resolution after image rendering; wherein the super-resolution processing on the target video stream comprises: performing super-resolution processing on the target video stream after rendering.

2. The method of claim 1, wherein, The determination of the resolution after image rendering according to the target resolution comprises any of the following: determining the resolution after image rendering based on the target resolution and the preset multiple; acquiring current GPU occupation information, and determining the resolution after image rendering according to the target resolution and the current GPU occupation information.

3. The method according to claim 1 or 2, characterized in that, Before the super-resolution processing on the target video stream, the method further comprises the following steps: determining whether the current GPU supports deep learning super-sampling (DLSS) processing; wherein the super-resolution processing on the target video stream comprises: if the current GPU supports DLSS processing, performing the DLSS processing and the super-resolution processing on the target video stream; if the current GPU does not support DLSS processing, performing the super-resolution processing on the target video stream.

4. The method of claim 1, wherein, The determination of the processing mode of resolution processing of the target video stream comprises: if the current resolution and the target resolution satisfy a second preset relationship, acquiring current GPU occupation information; determining the processing mode of resolution processing of the target video stream according to the current GPU occupation information.

5. The method of claim 1, wherein, The way of acquiring the target resolution of the target video stream comprises any of the following: acquiring a resolution set by a user through a client, and taking the resolution set through the client as the target resolution of the target video stream; acquiring a resolution set by a user through a client and a resolution corresponding to a display on which the target video stream is to be displayed, and determining the target resolution of the target video stream based on the resolution set through the client and the resolution corresponding to the display; acquiring a game type corresponding to the target video stream, and determining the target resolution of the target video stream based on the game type corresponding to the target video stream.

6. The method of claim 1, wherein, The target video stream is super-resolution processed, so that the resolution of the target video stream after super-resolution processing is the target resolution, and then the method further includes: encoding the target video stream after super-resolution processing; sending the encoded target video stream to the client.

7. An image super-resolution apparatus, characterized by comprising: The method comprises: a resolution acquisition module configured to acquire a target resolution of a target video stream and a current resolution of the target video stream; a processing mode determination module configured to determine a processing mode for performing resolution processing on the target video stream according to a relationship between the current resolution and the target resolution, a resolution supported by a display, a game type corresponding to the target video stream, and GPU occupancy information, if the current resolution is less than the target resolution; specifically, if the target resolution is greater than a preset multiple of the current resolution, the processing mode for performing resolution processing on the target video stream is determined to include super-resolution processing; a resolution processing module configured to perform super-resolution processing on the target video stream, if the determined processing mode includes super-resolution processing, so that the resolution of the target video stream after super-resolution processing is the target resolution; a resolution determination module configured to determine a resolution after image rendering according to the target resolution; an image rendering module configured to perform image rendering on the target video stream based on the resolution after image rendering; the resolution processing module is specifically configured to perform super-resolution processing on the rendered target video stream when performing super-resolution processing on the target video stream.

8. A server, characterized by The server comprises: at least one processor; a memory; at least one application program, wherein the at least one application program is stored in the memory and is configured to be executed by the at least one processor, and the at least one application program is configured to execute the image super-resolution method in any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed in the computer, the computer is caused to execute the image super-resolution method in any one of claims 1 to 6.

Images