Image data generation method and device, electronic equipment and storage medium

Abstract

The invention provides an image data generation method and device, electronic equipment and a storage medium, and the method is realized based on a shader in a graphics processor, and specifically comprises the steps: obtaining a first feature map corresponding to first image data; wherein the resolution of the first image data is a first resolution; for pixels in to-be-generated second image data, a sampling area corresponding to the pixels is inquired in the first feature map, feature extraction is carried out on the sampling area, and pixel values of the pixels are obtained; wherein the resolution of the second image data is a second resolution, and the number of pixels of the second image data and the sampling areas corresponding to the pixels are determined according to a comparison result between the second resolution and the first resolution; and generating second image data according to the pixel value of the pixel. According to the invention, the calculation speed of the image data is greatly improved, and the method can be conveniently applied to terminal equipment with relatively low performance.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method and device for generating image data, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Image data usually refers to a collection of pixel values ​​represented by numerical values. Usually, image data is the result of converting real-world image information into data information. In the field of image processing, in order to flexibly adapt to different display scenarios, the resolution of image data may need to be adjusted.

[0003] In the related art, the resolution adjustment process is usually implemented by a pre-trained neural network. However, the neural network usually contains a large number of parameters, the processing process is complex and time-consuming, and it cannot be flexibly adapted to various application scenarios. Summary of the invention

[0004] The present disclosure provides a method and device for generating image data, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] In a first aspect, the present disclosure provides a method for generating image data, the method being implemented based on a shader in a graphics processor, the method comprising: obtaining a first feature map corresponding to first image data; wherein the resolution of the first image data is a first resolution;

[0006] For a pixel in the second image data to be generated, query the sampling area corresponding to the pixel in the first feature map, perform feature extraction on the sampling area, and obtain a pixel value of the pixel; wherein the resolution of the second image data is a second resolution, and the number of pixels of the second image data and the sampling area corresponding to the pixel are determined according to a comparison result between the second resolution and the first resolution;

[0007] The second image data is generated according to the pixel value of the pixel.

[0008] In an optional implementation, the method is applied to a client device in a cloud game, the first image data is obtained according to a video stream transmitted from a cloud server to the client device in the cloud game, and the first resolution is determined according to a code stream resolution of the cloud game;

[0009] The second image data is used for displaying in the client device, and the second resolution is determined according to the screen resolution of the client device; the second resolution is greater than the first resolution.

[0010] In an optional implementation, the shader includes: a computing shader; the first feature map corresponding to the first image data is obtained in the following manner:

[0011] Converting the first image data into texture data, wherein the texture data includes a plurality of texture blocks divided according to a preset size; wherein each texture block corresponds to N*N pixels; wherein N is a natural number greater than 1;

[0012] The compute shader obtains the texture block to be processed currently from the shared cache, and calculates the feature data corresponding to the texture block to obtain the first feature map; wherein the shared cache is shared by multiple image processing units in the graphics processor, and the compute shader runs in any image processing unit.

[0013] In an optional implementation, the shader further includes: a fragment shader; and the pixel value of the pixel in the second image data is determined by the fragment shader.

[0014] In an optional implementation, the extracting features from the sampling area includes: calling a preset feature detection unit through the shader to extract features from the sampling area;

[0015] The parameter information of the feature detection unit is determined according to a pre-trained super-resolution neural network; and the super-resolution neural network is trained by amplifying the sample data by a first preset multiple; the first preset multiple is an integer multiple;

[0016] Furthermore, a comparison result between the second resolution and the first resolution is a ratio between the second resolution and the first resolution, and the ratio is a non-integer multiple.

[0017] In an optional implementation, the preset feature detection unit includes: a first feature detection unit and a second feature detection unit;

[0018] The calling of a preset feature detection unit to extract features from the sampling area comprises:

[0019] Calling the first feature detection unit to perform feature extraction on the sampling area to obtain an initial pixel value of the pixel;

[0020] The second feature detection unit is called to perform fusion processing on the first image data and the initial pixel value of the pixel to obtain the pixel value of the pixel.

[0021] In an optional implementation, the first image data is in a first image format, and the second image data is in a second image format;

[0022] Furthermore, the first image data includes: single-channel data corresponding to one color channel, and dual-channel data corresponding to two color channels;

[0023] Then the acquiring of the first feature map corresponding to the first image data comprises: performing feature extraction on the single-channel data through a first convolution kernel to obtain a first set of feature data including m channels;

[0024] Performing feature extraction on the dual-channel data through a second convolution kernel to obtain a second set of feature data containing n channels;

[0025] The first feature map is obtained according to the first set of feature data of the m channels and the second set of feature data of the n channels; wherein m and n are natural numbers less than a preset threshold.

[0026] In an optional implementation, the second feature detection unit includes: a 1*1 convolution kernel; the first image format includes a YUV format, and the second image format includes an RGB format; wherein the single-channel data corresponds to the Y channel in the YUV format, and the dual-channel data corresponds to the UV channel in the YUV format.

[0027] In an optional implementation, the super-resolution neural network is trained in the following manner:

[0028] Replacing a first activation function in a super-resolution neural network to be trained with a second activation function, and pre-training the replaced super-resolution neural network to obtain a pre-trained network model;

[0029] Replacing the second activation function in the pre-trained network model with the first activation function, reducing the learning rate of the pre-trained network model, and adjusting the pre-trained network model to obtain a trained super-resolution neural network;

[0030] The computation amount of the second activation function is less than that of the first activation function, and / or the convergence speed of the second activation function is higher than that of the first activation function.

[0031] In an optional implementation, the super-resolution neural network includes: an input processing layer, a close processing layer, and an output layer; wherein the input processing layer is used to calculate multiple channel feature maps corresponding to the first sample image data; the close processing layer is used to amplify the multiple channel feature maps by the first preset multiple, and then calculate the fused feature map corresponding to the amplified multiple channel feature maps through a first convolution filter; the output layer is used to amplify the first sample image data by the first preset multiple, and then fuse the amplified first sample image data and the fused feature map through a second convolution filter to obtain the second sample image data; wherein the first feature detection unit is configured according to the first convolution filter, and the second feature detection unit is configured according to the second convolution filter.

[0032] In a second aspect, the present disclosure provides a device for generating image data, the device being implemented based on a shader in a graphics processor, the device for generating image data comprising: an acquisition module, adapted to acquire a first feature map corresponding to first image data; wherein the resolution of the first image data is a first resolution;

[0033] an extraction module, adapted to query a sampling area corresponding to a pixel in the second image data to be generated in the first feature map, perform feature extraction on the sampling area, and obtain a pixel value of the pixel; wherein the resolution of the second image data is a second resolution, and the number of pixels of the second image data and the sampling area corresponding to the pixel are determined according to a comparison result between the second resolution and the first resolution;

[0034] The generating module is adapted to generate the second image data according to the pixel value of the pixel.

[0035] In a third aspect, the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for generating image data.

[0036] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned method for generating image data when executed by a processor.

[0037] In a fifth aspect, the present disclosure provides a computer program product, which includes a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned method for generating image data.

[0038] The method for generating image data provided by the embodiment of the present disclosure can be implemented based on the shader in the graphics processor, which is convenient for utilizing the parallel processing characteristics of the graphics processor. In order to avoid the problems of large number of parameters and time-consuming calculation when using a neural network to adjust the resolution, the present disclosure adopts a pixel-by-pixel reverse query method in the shader to generate the second image data corresponding to the first image data. Specifically, for the pixel in the second image data to be generated, the sampling area corresponding to the pixel is queried in the first feature map corresponding to the first image data, and the feature extraction is performed on the queried sampling area to obtain the pixel value of the pixel. In which, the number of pixels of the second image data and the sampling area corresponding to the pixel are determined according to the comparison result between the second resolution of the second image data and the first resolution of the first image data. It can be seen that, on the one hand, this method directly performs accurate calculations based on the predetermined number of pixels of the second image data, without calculating redundant pixels, saving calculation time. On the other hand, by utilizing the characteristics of accurate sampling of the shader, the sampling area corresponding to the pixel in the second image data in the first image data can be accurately determined according to the resolution difference before and after processing, so that the calculation object is more accurate and unnecessary calculation processes are avoided. Therefore, this method avoids a large number of parameter calculation processes in the neural network, greatly improves the calculation speed, and is easy to apply to terminal devices with lower performance.

[0039] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0041] Figure 1 A flowchart of a method for generating image data provided by an embodiment of the present disclosure;

[0042] Figure 2 A schematic diagram of a cloud gaming scenario is shown;

[0043] Figure 3 A schematic diagram showing the processing process of the close connection processing layer in the super-resolution neural network is shown;

[0044] Figure 4 A flowchart of a method for generating image data provided for an example of the present disclosure;

[0045] Figure 5 A block diagram of a device for generating image data provided by an embodiment of the present disclosure;

[0046] Figure 6 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0048] In the related art, the resolution adjustment process is usually implemented by a pre-trained neural network. However, the neural network usually contains a large number of parameters, the processing process is complex and time-consuming, and it cannot be flexibly adapted to various application scenarios. In order to solve the above problem, the present disclosure uses the characteristics of precise sampling of the shader to accurately determine the sampling area of ​​each pixel according to the resolution difference before and after processing, thereby making the calculation object more accurate and avoiding unnecessary calculation processes.

[0049] Figure 1 A flowchart of a method for generating image data provided by an embodiment of the present disclosure is shown. The method is implemented based on a shader in a graphics processor and is intended to adjust the image resolution. Figure 1 As shown, the method specifically comprises the following steps:

[0050] Step S110: Obtain a first feature map corresponding to the first image data; wherein the resolution of the first image data is a first resolution.

[0051] The first image data may also be referred to as original image data, which refers to image data to be processed. This embodiment aims to process the first image data to obtain second image data corresponding to the first image data. The resolution of the second image data is different from that of the first image data.

[0052] The first feature map is used to characterize feature information in the first image data. The first feature map can be obtained based on one or more color channel data in the first image data. Accordingly, the first feature map can include a color channel feature map. The first feature map can be flexibly generated in a variety of ways, such as sampling, convolution, filtering, etc. This application does not limit the generation method of the first feature map.

[0053] Step S120: For pixels in the second image data to be generated, query the sampling area corresponding to the pixel in the first feature map, perform feature extraction on the sampling area, and obtain the pixel value of the pixel; wherein the resolution of the second image data is the second resolution, and the number of pixels of the second image data and the sampling area corresponding to the pixel are determined based on the comparison result between the second resolution and the first resolution.

[0054] Among them, the second image data is the image data obtained after processing the first image data. The resolution of the second image data can be set according to the display requirements. For example, in one case, if the first image data is the image data in the received video stream, that is: the first resolution of the first image is determined according to the code stream resolution of the video stream, and the second image data is used to be displayed in the terminal device, then the second resolution of the second image data can be set according to the screen resolution of the terminal device. For example, in another case, the first image data is the original image data to be transmitted, that is: the first resolution of the first image is determined according to the clarity of the original image data, and the second image data is used to be transmitted to the receiving end in the form of a video stream, then the second resolution of the second image data can be set according to the network transmission requirements. It can be seen that the second resolution of the second image data is set according to the application scenario.

[0055] In addition, according to the comparison result between the second resolution and the first resolution, the resolution difference of the second image data relative to the first image data can be determined, and based on the difference, the number of pixels to be included in the second image data (i.e., the image size of the second image data) can be calculated according to the number of pixels included in the first image data. And, based on the difference, according to the mapping relationship between the first image data and the second image data, the sampling area corresponding to the pixels in the second image data in the first feature map can be determined.

[0056] Accordingly, in the disclosed embodiment, for a pixel in the second image data to be generated, the sampling area corresponding to the pixel is inversely queried in the first feature map, thereby performing feature extraction on the sampling area to obtain the pixel value of the pixel. It can be seen that the embodiment utilizes the characteristics of accurate sampling of the shader to query the corresponding sampling area pixel by pixel, thereby determining the pixel value of each pixel one by one in units of pixels.

[0057] The pixel in the second image data to be generated may be any pixel in the second image data, for example, may be every pixel in the second image data, which is not limited in the present disclosure.

[0058] For example, assuming that the second image data contains N*M pixels in total, N and M are natural numbers, then each of the N*M pixels can be used as a pixel in the second image data to be generated. Alternatively, in order to speed up the processing, a part of the main pixels among the N*M pixels can be used as pixels in the second image data to be generated.

[0059] It can be seen that for any pixel in the second image data to be generated, the sampling area corresponding to the any pixel is queried in the first feature map, and the feature extraction is performed on the sampling area to obtain the pixel value of the any pixel; wherein the resolution of the second image data is the second resolution, and the number of pixels of the second image data and the sampling area corresponding to the any pixel are determined according to the comparison result between the second resolution and the first resolution. In short, the present disclosure utilizes the characteristic of pixel-by-pixel output of the shader, takes the pixels in the second image data as units, and obtains the second image data by pixel-by-pixel reverse query. wherein, each pixel in the second image data is generated in the same way, therefore, only any pixel therein is used as an example for explanation, and the processing methods of other pixels are similar.

[0060] Step S130: Generate second image data according to the pixel value of the pixel.

[0061] The second image data to be generated can be obtained by combining the pixel values ​​of the multiple pixels according to the positional relationship between the multiple pixels.

[0062] It can be seen that in order to avoid the problems of large number of parameters and time-consuming calculation when using a neural network to adjust the resolution, the embodiment of the present disclosure adopts a pixel-by-pixel reverse query method in the shader to generate the second image data corresponding to the first image data. Specifically, for the pixel in the second image data to be generated, the sampling area corresponding to the pixel is queried in the first feature map corresponding to the first image data, and the feature extraction is performed on the queried sampling area to obtain the pixel value of the pixel. Among them, the number of pixels of the second image data and the sampling area corresponding to the pixel are determined according to the comparison result between the second resolution of the second image data and the first resolution of the first image data. It can be seen that, on the one hand, this method directly performs accurate calculations based on the predetermined number of pixels of the second image data, without calculating redundant pixels, saving calculation time. On the other hand, by utilizing the characteristics of accurate sampling of the shader, the sampling area corresponding to each pixel in the second image data in the first image data can be accurately determined according to the resolution difference before and after processing, so as to make the calculation object more accurate and avoid generating redundant calculation processes. Therefore, this method avoids a large number of parameter calculation processes in the neural network, greatly improves the calculation speed, and is convenient for application in terminal devices with lower performance.

[0063] In addition, in order to achieve better technical effects, those skilled in the art may also make various changes and modifications to the above embodiments:

[0064] In an optional implementation, the above method is used to implement super-resolution processing in cloud gaming applications. Considering that the device performance of client devices in cloud gaming is usually low and the computing resources are limited, in order to utilize the limited computing resources in the client device to generate a game image with higher clarity, super-resolution processing is implemented with the help of a shader, thereby improving the resolution of the first image data and obtaining the second image data. It can be seen that the above method can be applied to client devices in cloud gaming, and the first image data is obtained according to the video stream transmitted from the cloud server to the client device in the cloud game, and accordingly, the first resolution is determined according to the code stream resolution of the cloud game. The second image data is used to be displayed in the client device, and the second resolution is determined according to the screen resolution of the client device. Typically, the second resolution is greater than the first resolution to achieve the effect of improving image quality.

[0065] In an optional implementation, the shader includes: a computational shader. Accordingly, the first feature map corresponding to the first image data can be obtained by:

[0066] First, the first image data is converted into texture data, and the texture data includes a plurality of texture blocks divided according to a preset size. The first feature map corresponding to the first image data can be converted into texture data. In the case where the first feature map includes a plurality of color channel feature maps, each color channel feature map can be converted into corresponding texture data respectively. The texture data has the advantage of being easy to process by a shader. In addition, in order to facilitate the allocation of computing tasks to the shader, the texture data is further divided into a plurality of texture blocks of preset sizes.

[0067] Then, the calculation shader obtains the texture block to be processed currently from the shared cache, and calculates the feature data corresponding to the texture block to obtain the first feature map. It can be seen that the calculation shader processes each texture block one by one in units of texture blocks. Among them, the shared cache can be shared by multiple image processing units in a graphics processing unit (Graphics Processing Unit, GPU), and the calculation shader can run in any image processing unit. For example, the shared cache can be the L2-cache in the GPU. Among them, the L1-cache is a cache exclusively occupied by the processing unit, and the L2-cache is a cache shared between multiple GPU processing units, and the content access speed corresponding to the L2-cache is faster than the content access speed corresponding to the video memory. It can be seen that in the above method, the shader programs running in multiple GPU processing units can share the texture data stored in the L2-cache, thereby improving the access and processing speed. It can be seen that the shader program in the present disclosure can be multiple, and the processing efficiency can be greatly improved by running multiple programs in parallel.

[0068] Among them, the inventor found in the process of implementing the present invention that in the traditional dispatching method, the shader takes pixels as units, and sequentially obtains the data of each pixel from the L2-cache and processes it. However, when the length of a pixel row is long, it may cause the two adjacent pixel rows above and below to not be stored in the L2-cache at the same time. For example, suppose that pixel row 1 contains 1080 pixels arranged from left to right, and pixel row 2 also contains 1080 pixels arranged from left to right, and pixel row 2 is located below pixel row 1. Moreover, only the content of pixel row 1 is stored in the L2-cache, and the content of pixel row 2 is not stored. At this time, the following problem may arise: in the process of processing a certain pixel in pixel row 1, it may be necessary to combine the adjacent pixels located above, below, left, and right of the pixel for processing. However, since the adjacent pixels located below the pixel belong to pixel row 2, they are not stored in the L2-cache, which will cause the failure of accessing data from the cache, that is, the problem of cache failure, thereby affecting the processing efficiency. It can be seen that in the method of allocating computing tasks to shaders in units of pixels, cache failure problems may occur. In order to solve the above problem, in this embodiment, the texture data is divided into multiple texture blocks (also called tiles), so that the calculation is performed in units of texture blocks. Since each texture block includes a set of pixels composed of multiple pixels, it can be ensured that the processing efficiency will not be affected by cache failure during the processing of the set of pixels.

[0069] The texture blocks can be set to various forms such as square blocks and rectangular blocks. Preferably, for the convenience of calculation, each texture block corresponds to N*N pixels; wherein N is a natural number greater than 1. In addition, the number of N can be set according to the parallel computing capability of the GPU so that the division size of the texture blocks matches the parallel computing resources of the GPU. If the texture blocks are divided too small, the parallelism of the processing process will be reduced, thereby affecting the processing efficiency; if the texture blocks are divided too large, it may still cause multiple pixels inside the texture blocks to not be stored in the L2-cache at the same time, thereby causing the above-mentioned cache failure problem. Therefore, in this embodiment, the division size of the texture blocks is matched with the parallel computing resources of the GPU, thereby taking into account the problems of parallel efficiency and cache effectiveness. The pixel area of ​​each texture block can be less than or equal to the processing capacity of the maximum dispatch group supported by the GPU. For example, assuming that the processing capacity supported by the maximum dispatch group supported by the GPU is 16*16 pixels, the size of the texture block can also be set to 16*16 pixels. The dispatch group can also be called a work group or a thread block, etc., which is used to organize and manage tasks executed in parallel. By matching the texture block size with the maximum dispatch group supported by the GPU, a better balance can be achieved between parallel efficiency and cache effectiveness.

[0070] In an optional implementation, the shader may also include: a fragment shader. Accordingly, the pixel value of the pixel in the second image data is determined by the fragment shader. It can be seen that in this method, a two-stage shader is designed. First, the first feature map corresponding to the first image data is calculated by the calculation shader; then, the first feature map is queried and calculated by the fragment shader to obtain the second image data. Among them, the calculation shader has a more powerful computing power and can complete the calculation process of the first feature map in a highly parallel manner. The fragment shader has the characteristics of precise sampling and pixel-by-pixel output, and can generate multiple pixels in the second image data one by one.

[0071] In an optional implementation, when extracting features from a sampling area, a preset feature detection unit can be called through a shader to extract features from the sampling area; wherein the parameter information of the feature detection unit is determined according to a pre-trained super-resolution neural network; and the super-resolution neural network is trained by amplifying the sample data by a first preset multiple; the first preset multiple is usually an integer multiple; and the comparison result between the second resolution and the first resolution can be represented by the ratio between the second resolution and the first resolution, and the ratio is usually a non-integer multiple. The feature detection unit includes various functional modules that can realize feature detection, such as convolution kernels, filters, feature extractors, etc.

[0072] It can be seen that the relevant parameters of the shader can be configured according to the parameters in the pre-trained neural network. For example, the relevant parameters of the feature detection unit can be determined in the training process of the super-resolution neural network, and then the feature detection unit with the relevant parameters can be configured in the shader through the shader language, so as to realize super-resolution processing by using the precise sampling characteristics of the shader. In actual scenarios, the ratio between the second resolution and the first resolution is usually not exactly an integer multiple, and the neural network usually first enlarges the sample data to an integer multiple (such as 2 times, 4 times, etc.) and then reduces it during the training process. For example, assuming that the ratio between the second resolution and the first resolution is 1.6, the neural network first enlarges the sample data to 2 times and then reduces it during the training process. However, this method of enlarging and then reducing will lead to a lot of unnecessary calculations. For neural networks with large parameter magnitudes running in high-performance server-side devices, the increase in the amount of calculation may have a small impact, but for client devices with limited computing resources, a large amount of redundant calculations will seriously affect the performance of the device, thereby causing a series of problems such as device freezes and display abnormalities. When calculating each pixel in the second image data, the fragment shader in the disclosed embodiment does so by inversely querying the corresponding sampling area in the first feature map, thereby directly generating second image data with a resolution that meets the requirements, avoiding the disadvantage of excessive computational complexity caused by first generating intermediate result data with an integer multiple of the magnification and then downsampling the intermediate result data, thereby optimizing the performance of the super-resolution processing process.

[0073] In an optional implementation, the preset feature detection unit further includes: a first feature detection unit and a second feature detection unit. Accordingly, the preset feature detection unit can be called to perform feature extraction on the sampling area in the following manner: First, call the first feature detection unit to perform feature extraction on the sampling area to obtain the initial pixel value of the pixel. The first feature detection unit can be implemented by a convolution filter. Then, call the second feature detection unit to fuse the first image data and the initial pixel value of the pixel to obtain the pixel value of the pixel. The data content corresponding to the pixel is obtained from the first image data (such as the color channel data of the first image data) by the second feature detection unit, so that the data content corresponding to the pixel is fused with the initial pixel value of the pixel to obtain the pixel value of the pixel (i.e., the final output value). It can be seen that in this method, on the basis of the first feature detection unit, a second feature detection unit is further added. The second feature detection unit can fuse the initial image data (i.e., the first image data) and the image data after feature extraction (i.e., the initial pixel value) through cross-layer connection. Through the fusion processing, it can not only retain the original image information to the greatest extent according to the initial image data, but also make full use of the feature extraction results according to the initial pixel value, so as to obtain a more accurate final result by combining the two parts.

[0074] Among them, the first feature detection unit and the related configuration parameters of the first feature detection unit can be determined according to the training results of the super-resolution neural network. Accordingly, in the case where the super-resolution neural network includes an input layer (for calculating the first feature map), a close processing layer (for calculating the initial pixel value of each pixel in the second image data) and an output layer (for calculating the final pixel value of each pixel in the second image data), the first feature detection unit and the second feature detection unit can correspond to the close processing layer in the super-resolution neural network and the convolution filter in the output layer, respectively. Among them, in this embodiment, in order to reduce the amount of calculation and improve the processing speed of the client device, a feature extraction method with a small number of channels is usually adopted to reduce the amount of calculation as much as possible. However, in the case of a small number of channels or a small number of parameters, the network model will not converge easily. Therefore, in order to speed up the convergence speed of the network model and improve the training effect, the output layer in the super-resolution neural network is connected across layers (that is, the input end of the output layer is simultaneously connected to the input layer and the close processing layer), which is conducive to the rapid convergence of the model. Correspondingly, when the output layer in the super-resolution neural network adopts cross-layer connection, the second feature detection unit in the present application also fuses the initial image data and the image data after feature extraction through cross-layer connection.

[0075] In an optional implementation, second image data having an image format different from that of the first image data can be directly generated. For example, the first image data is in the first image format, and the second image data is in the second image format; and the first image data includes: single-channel data corresponding to one color channel, and dual-channel data corresponding to two color channels. Accordingly, the first feature map corresponding to the first image data can be obtained in the following manner: feature extraction is performed on the single-channel data through the first convolution kernel to obtain a first set of feature data containing m channels; feature extraction is performed on the dual-channel data through the second convolution kernel to obtain a second set of feature data containing n channels; a first feature map is obtained according to the first set of feature data of m channels and the second set of feature data of n channels; wherein m and n are natural numbers less than a preset threshold. It can be seen that, in the case where the original image and the processed image formats are different, if the original image contains two sets of color channels, the corresponding convolution kernel can be configured for each set of color channels respectively, and feature extraction is performed for each set of color channels respectively. By processing the two sets of color channels separately, there is no need to align the two sets of color channels, thereby simplifying the amount of calculation. Furthermore, in this embodiment, by controlling the number of channels, the number of channels of feature data corresponding to each group of color channels is made smaller than a preset threshold, which helps to simplify the amount of calculation and improve the calculation speed.

[0076] In an optional implementation, the second feature detection unit includes: a 1*1 convolution kernel. Through the 1*1 convolution kernel, channel fusion and other processing can be achieved. The first image format includes a YUV format, and the second image format includes an RGB format; accordingly, the single-channel data mentioned above corresponds to the Y channel in the YUV format, and the dual-channel data corresponds to the UV channel in the YUV format. Usually, the sizes of the Y channel and the UV channel are different. In the related art, most of them need to align the sizes of the Y channel and the UV channel first, so that the small-sized color channel is expanded to the same size as the large-sized color channel, thereby increasing the amount of calculation. In addition, the data format before and after processing in the related art must be the same. For example, it is necessary to convert the YUV format into the RGB format before processing, or convert the YUV format into the RGB format after processing. In short, the format conversion process has the defects of being cumbersome and time-consuming. This embodiment can directly convert the first image data in the YUV format into the second image data in the RGB format by setting two sets of convolution kernels, thereby avoiding the above problems.

[0077] In an optional implementation, in order to improve the training efficiency of the neural network, the super-resolution neural network can be trained in the following manner: replace the first activation function in the super-resolution neural network to be trained with the second activation function, pre-train the replaced super-resolution neural network, and obtain a pre-trained network model; replace the second activation function in the pre-trained network model with the first activation function, and reduce the learning rate of the pre-trained network model, adjust the pre-trained network model, and obtain the trained super-resolution neural network; wherein the calculation amount of the second activation function is less than the calculation amount of the first activation function, and / or the convergence speed of the second activation function is higher than the convergence speed of the first activation function. By first using the second activation function with a smaller amount of calculation for pre-training and selecting a relatively large learning rate, the network model can converge quickly and learn a better initial weight distribution. In addition, by replacing the first activation function and reducing the learning rate, the model can be fine-tuned near the learned weights to improve the final performance.

[0078] For ease of understanding, an example is used below to introduce the implementation of the above embodiment. This example is described using a cloud gaming application scenario as an example.

[0079] In conventional game solutions, games are usually run locally on client devices. Taking mobile phones as an example, players need to install the game package (i.e. the game installation package and other game-related resource data required for game operation) locally on their mobile phones, and the mobile phone then renders the game screen in real time based on the player's operation. Therefore, all computing operations are concentrated on the player's mobile device.

[0080] However, the above method has at least the following problems: In order to improve the quality of the game screen, more computing power is required, which leads to a series of problems in the hardware configuration of terminal devices such as mobile phones because they cannot meet the computing requirements. For example, the mobile phone may become hot, have insufficient memory, freeze, and other problems. It can be seen that if the user cannot replace the game client device with a higher configuration, they will not be able to experience better game performance. Therefore, how to improve the quality and smoothness of the game screen as much as possible under limited hardware configuration has become a technical problem that needs to be solved in the game field.

[0081] In order to solve the above problems, this example adds a processing link to install the game package on a dedicated cloud game server, and then transmit the game screen to mobile phones and other terminal devices in real time. In this way, the hardware configuration requirements for mobile terminals in the game scene are greatly reduced. It can be seen that by installing the game package on the cloud server and transferring the calculation process to the cloud server, the hardware pressure of the client device can be transferred to the cloud server, so that users can experience better game performance without upgrading to a better-performing client device.

[0082] Figure 2 A schematic diagram of a cloud gaming scenario is shown, Figure 2 As shown, the player user transmits various types of user operation instructions triggered by the user through the touch device, keyboard and mouse device, handle device and / or gyroscope device to the cloud server through the client device. The cloud server performs calculations corresponding to the user operation instructions in the cloud according to the instruction parameters (specifically including instruction type, text / picture corresponding to the instruction, etc.) and user information (such as user account, etc.) contained in the received user operation instructions. For example, rendering calculations, compression processing, resource management, game operation, etc. can all be implemented by the cloud server, thereby greatly reducing the pressure on the client. Correspondingly, the cloud server transmits data such as audio and video streams calculated according to the user operation instructions to the client, and the client obtains the game screen to be displayed through decoding, decompression, etc., and displays it directly on the client. It can be seen that in the above method, the main computing overhead in the game is realized by the cloud server, and the client only needs to transmit the operation instructions and receive the game screen calculated by the cloud server, thereby greatly reducing the hardware requirements for the game client, so that users can experience better game effects without upgrading the hardware configuration.

[0083] The following is an explanation of some of the concepts that may be involved in this example:

[0084] Super-resolution: Through models or algorithms such as convolutional neural networks, the resolution of the image is improved, the details of the image are restored, and a sharper image is obtained. Generally, the magnification is 2 times. Super-resolution algorithms can be applied to medical imaging, image processing and other fields.

[0085] GPU: (Graphics Processing Unit) is a processor that is specifically designed to quickly process and generate graphics. Mobile devices such as mobile phones and tablets usually have GPUs integrated into them.

[0086] GPU L2-cache: a cache shared between multiple GPU processing units (L1-cache is a cache exclusive to a processing unit). Access to content in the L2-cache is faster than access to video memory.

[0087] Shader: A program that runs on the GPU. Graphics pipeline APIs such as DirectX, OpenGL, Vulkan, and Metal generally support shader programming with similar functions.

[0088] Fragment Shader: A type of shader that runs code executed for each output pixel and can be executed in parallel in the GPU according to the output pixels.

[0089] Compute Shader: A type of shader used for calculations. It has a more flexible way of assigning work tasks and can assign tasks in one to three dimensions.

[0090] Convolutional Neural Network (CNN): A deep learning model whose convolutional layer calculations can extract local features and is particularly suitable for processing image data.

[0091] RELU: It is a commonly used activation function in deep neural networks. It is simple to calculate and the output is equal to the maximum value of the input and 0.

[0092] SELU: It is a commonly used activation function in deep neural networks. It is similar to RELU but has a flat curve at the turning point, which makes training easier.

[0093] MAC: Multiply-Accumulate. The main calculation of neural networks, especially convolutional neural networks, is weighted sum. Each weighted sum is a MAC. The amount of MAC calculation for each pixel is close to its parameter amount.

[0094] YUV420: A color output format for video decoding, where the Y channel corresponds to brightness, and the U and V channels correspond to chrominance. Each Y value corresponds to one pixel, and each U / V value corresponds to two pixels. After video decoding, YUV is converted to RGB format to achieve on-screen rendering.

[0095] In the related art, super-resolution algorithms are usually implemented through convolutional neural networks. For example, models such as FSRCNN and ESRGAN can increase the amount of feature information obtained by increasing the width (i.e., the number of channels), and increase the generalization ability of the network by increasing the depth. It can be seen that whether it is to increase the number of channels or increase the depth of the model, a large number of parameters will be involved. Although the convolutional neural network can be rewritten into a shader operation form through encoding, it can be run in the GPU of the mobile device. However, if the convolutional neural network in the related art is directly deployed in the shader, it will lead to a huge amount of calculation, which will exceed the carrying capacity of the mobile device, and a series of problems such as device jamming will occur. For example, the neural network commonly used for super-resolution has tens of thousands to millions of parameters, which means that each pixel has to perform tens of thousands to millions of MAC operations. Assuming a neural network with 1 trillion parameters, in the cloud gaming scenario, the commonly used resolution is 1080p (i.e., each screen has two million pixels). Assuming 60 frames are rendered per second, this requires about 120T MAC, which far exceeds the computing power of the current mobile GPU with the strongest computing power.

[0096] It can be seen that in cloud gaming applications, on the one hand, a higher image resolution is required to present a clearer picture; on the other hand, the computing power of mobile terminals is limited and cannot match the computing performance of the cloud. Therefore, if the super-resolution neural network with a huge number of parameters in the related technology is directly deployed in the shader, it will cause abnormal operation of the device.

[0097] To solve the above problems, this example has made at least the following improvements:

[0098] (1) A mini (i.e., small number of parameters) image quality enhancement neural network (also called super-resolution neural network) was designed and trained, and its parameter number is much smaller than the super-resolution model in the related art. The number of channels in the neural network is small to achieve the effect of reducing the number of parameters. In addition, in order to solve the problem that the network with small parameters is not easy to converge, the activation function replacement method is used in the training process of the neural network (pre-training with an activation function that is easy to converge to accelerate the convergence process). In addition, the cross-layer connection (also called skip connection) method is also used in the neural network to enable the output layer to combine more information for calculation, thereby compensating for the accuracy loss that may be caused by the small number of parameters.

[0099] (2) A neural network is implemented using a shader, and a calculation dispatch order that is friendly to the GPU L2-cache is designed, thereby reducing the calculation bottleneck of the GPU. In particular, the inventors discovered in the process of implementing the present invention that the GPU execution efficiency is not only affected by the GPU execution frequency, the number of execution units, and the pipeline design, but also by the L2-cache. If the access to the L2-cache often fails, the execution efficiency will be greatly reduced. This example can be dispatched in units of tiles by dividing the tiles. Therefore, by reasonably designing the calculation dispatch order, the problem of L2-cache failure can be avoided.

[0100] (3) The screen resolution of a general mobile device is larger than the bitstream resolution, but less than twice the bitstream resolution. In order to avoid the problem of excessive computational complexity caused by first amplifying the bitstream resolution by two times and then reducing it in the related art, this example is based on a two-stage shader. In the second-stage fragment shader, the closest output is searched point by point on the screen pixel to avoid generating an intermediate result of a twice-bitstream image (four times the number of pixels), thereby greatly improving computational efficiency.

[0101] (4) The model is simplified for the YUV output of video decoding. The model can directly output the RGB format that can be displayed on the screen, reducing redundant calculations.

[0102] Based on the above improvements, this example can achieve real-time super-resolution image quality enhancement for 1080p 60 frames per second video streaming images on mainstream mobile devices.

[0103] It can be seen that this example first designs and trains a super-resolution neural network with a small number of parameters, and configures the shader program in the GPU according to the parameter information of the super-resolution neural network to achieve super-resolution processing through the shader.

[0104] The super-resolution neural network in this example can be a three-layer structure, namely, an input processing layer, a close processing layer, and an output layer. The input processing layer is used to calculate multiple channel feature maps corresponding to the first sample image data; the close processing layer is used to amplify the multiple channel feature maps by a first preset multiple, and then calculate the fusion feature map corresponding to the amplified multiple channel feature maps through a first convolution filter; the output layer is used to amplify the first sample image data by a first preset multiple, and then fuse the amplified first sample image data and the fusion feature map through a second convolution filter to obtain the second sample image data. The first feature detection unit is configured according to the first convolution filter, and the second feature detection unit is configured according to the second convolution filter.

[0105] Specifically, the input processing layer is used to perform the following processing: for the Y channel image data in the first sample image data, a first group of 3*3 convolution kernels (i.e., the first convolution kernel) is used to perform convolution operations, and the features FY of m channels are calculated using the RELU activation function. Among them, the feature FY is the Y channel feature map corresponding to the above Y channel image data. Similarly, for the UV channel image data in the first sample image data, a second group of 3x3 convolution kernels (i.e., the second convolution kernel) is used to perform convolution operations, and the features FUV of n channels are calculated using the RELU activation function. Among them, the feature FUV is the UV channel feature map corresponding to the above UV channel image data. Usually, the size of the Y channel image data and the UV channel image data is different. Therefore, in order to avoid the problem of increased computational complexity caused by the size alignment process, this example designs two groups of convolution kernels with different parameters, so that the two groups of channel data can be processed separately to reduce the amount of computation and simplify the computational process. It can be seen that the input processing layer is used to obtain the channel feature map corresponding to each color channel through convolution operations.

[0106] The close processing layer is used to further process the multiple channel feature maps obtained by the input processing layer to obtain a fused feature map of multi-channel fusion. Specifically, in order to achieve an improvement in resolution, the close processing layer can first amplify FY and FUV by a first preset multiple (i.e., an integer multiple, such as 2 times). Then, a 3x3 convolution operation is performed through a 3x3 convolution kernel (i.e., the first convolution filter, corresponding to the first feature detection unit mentioned above), and the result F of p channels is calculated using the RELU activation function. Among them, F is used to characterize the fused feature map after multi-channel fusion.

[0107] Among them, the number of channels p of the fusion feature map F, the number of channels m of the Y channel feature map FY, and the number of channels n of the UV channel feature map FUV can be configured according to the hyperparameters in the neural network. In this example, in order to reduce the number of parameters of the neural network and reduce the amount of calculation, the values ​​of p, m, and n are all less than the preset threshold, which can be a natural number such as 8 or 10. Among them, different values ​​corresponding to m, n, and p can obtain different performance and capabilities. Preferably, the following value method can be adopted: m=2, n=2, p=3.

[0108] The output layer is used to perform channel fusion processing on the data of the input processing layer and the close processing layer, so as to obtain the final pixel calculation result. Specifically, in order to improve the resolution, the output layer can first amplify the input data Y (i.e., Y channel image data) and UV (i.e., UV channel image data) of the input processing layer by a first preset multiple (i.e., an integer multiple, for example, 2 times) to obtain Y' and UV', and then splice Y', UV' and F, and fuse the spliced ​​result through a 1*1 convolution kernel (i.e., a second convolution filter, corresponding to the second feature detection unit mentioned above) to obtain the final pixel calculation result (i.e., the second sample image data). Wherein, when the input data is in YUV format, the output layer can directly output the final result of the three RGB channels by adjusting the annotation format of the training sample, so as to obtain the second image data in RGB format. It can be seen that the second feature detection unit not only realizes the fusion processing for Y', UV' and F to make the final result more accurate, but also realizes the conversion processing from YUV format to RGB format.

[0109] It can be seen that through the above processing of the input processing layer, the close connection processing layer and the output layer, a training process for the sample data can be completed. Among them, the loss value between the final result of the three RGB channels output by the output layer and the pre-acquired sample annotated image can be calculated through a preset loss function, and the training process is completed when the model converges according to the convergence condition.

[0110] In specific implementation, the above-mentioned super-resolution neural network model can be trained using pre-recorded game screens as a training set. That is, the above-mentioned sample image data is the game screen data in the training set. In addition, in order to speed up convergence, the training process can be implemented in the following ways: first, the first activation function (RELU function) in the super-resolution neural network to be trained is replaced with the second activation function (SELU function), and the replaced super-resolution neural network is pre-trained to obtain a pre-trained network model; then, the SELU function in the pre-trained network model is replaced back to the RELU function, and the learning rate of the pre-trained network model is reduced, and the pre-trained network model is adjusted to obtain a trained super-resolution neural network. Since networks with smaller parameter magnitudes are not easy to converge, the method of pre-training using the SELU function with small computational complexity and fast convergence speed can help speed up model convergence, thereby solving the training problem of small parameter networks.

[0111] In order to improve the training effect, the above training process can be repeated multiple times, so as to select the model weight with the highest picture sharpness and the most accurate color restoration from the results of multiple trainings as the final model weight. Finally, the selected model weight is exported as the parameter definition of the shader, so as to configure the trained super-resolution neural network model into the shader.

[0112] For ease of understanding, Figure 3 Figure 2 shows a schematic diagram of the processing process of the close connection processing layer in the super-resolution neural network. Figure 3 As shown in (a) of FIG, the input of the close contact processing layer is the output results FY and FUV of the input processing layer. Figure 3 As shown in (b) of FIG. 1 , the close contact treatment layer is first linearly enlarged by 2 times for FY and FUV to obtain enlarged FY and FUV. Then, as Figure 3 As shown in (c) in FIG. 1 , the close processing layer performs feature extraction on the enlarged FY and FUV through a set of 3*3 convolution kernels, thereby obtaining each pixel in the fused feature map F. Optionally, when the resolution ratio between the second image data and the first image data is not an integer multiple, the fused feature map F can be further downsampled to obtain Figure 3 The feature map shown in (d) in . Figure 3 The feature map shown in (d) shows the feature map pixels required for the final calculation of screen pixels. It can be seen that in the training process of the neural network, the channel feature map of the sample image data needs to be enlarged by an integer multiple and then reduced to obtain an accurate resolution result.

[0113] After the super-resolution neural network is trained, the relevant codes and parameters in the shader program can be configured according to the weight data of the super-resolution neural network so that the parameters defined in the shader code match the weight data of the super-resolution neural network, thereby enabling the shader to realize the corresponding functions of the super-resolution neural network.

[0114] Among them, the convolution layer in the neural network generally uses a convolution kernel size of 3x3, that is, the 3x3 neighborhood of each input channel is operated, the number of output channels is calculated, and an activation function such as RELU is used for each output value to obtain the result.

[0115] Specifically, in this example, the parameter information and weight information corresponding to the first group of 3*3 convolution kernels (used to calculate the Y channel feature map) and the second group of 3*3 convolution kernels (used to calculate the UV channel feature map) contained in the input processing layer of the super-resolution neural network need to be configured in the calculation shader, so that the calculation shader can realize the calculation of the channel feature map based on the two groups of convolution kernels. In addition, the parameter information (i.e., the parameters of the first convolution filter) and weight information corresponding to the first feature detection unit contained in the close processing layer of the super-resolution neural network need to be configured in the fragment shader, so that the fragment shader also has the first feature detection unit with the same function, thereby calculating the initial pixel value of the pixel in the second image data based on the first feature detection unit. In addition, the parameter information (i.e., the parameters of the second convolution filter) and weight information corresponding to the second feature detection unit contained in the output layer of the super-resolution neural network need to be configured in the fragment shader, so that the fragment shader also has the second feature detection unit with the same function, thereby calculating the final pixel value of the pixel in the second image data based on the second feature detection unit.

[0116] Figure 4 A flowchart showing a method for generating image data in this example is shown in FIG. Figure 4 As shown, the method comprises the following steps:

[0117] Step S301: Generate a first feature map corresponding to first image data by computing a shader.

[0118] In this example, the first image data is image data in a video code stream transmitted from a cloud server to a game client, and therefore, the resolution of the first image data is usually equal to the code stream resolution.

[0119] The first feature map specifically includes: a channel feature map FY corresponding to the Y channel, and a channel feature map FUV corresponding to the UV channel.

[0120] In order to facilitate processing, the Y channel image data and the UV channel image data contained in the first image data need to be converted into corresponding texture data respectively. For example, the input Y and UV are respectively bound to two textures, the coordinates of UV (i.e., the coordinates of the texture corresponding to the UV channel image data) are traversed, and the calculation process corresponding to the input processing layer in the super-resolution neural network is performed, so as to obtain the output results FY and FUV of the calculation shader.

[0121] Among them, in order to avoid the problem of L2-cache invalidation, when traversing the UV coordinates, the UV image is divided into multiple square tiles of the same size (i.e., a specific implementation form of texture blocking), the pixel area of ​​each tile is less than or equal to the maximum dispatch group supported by the GPU, and the order of computing shader dispatch is re-specified according to the tile, and the next tile is calculated after one tile is calculated. By dispatching computing tasks in units of tiles, it can be ensured that all pixels required for a computing task are located in the same tile, and all pixels of the tile are stored in the L2-cache, thereby avoiding the cache invalidation problem caused by pixel-by-pixel dispatch.

[0122] Step S302: by using the fragment shader, inversely query the sampling area corresponding to the pixel in the second image data in the first feature map, and call the first feature detection unit to obtain the initial pixel value of the pixel.

[0123] The second image data is screen image data that needs to be displayed on the game client, so the resolution of the second image data is usually the same as the screen resolution.

[0124] The first feature detection unit corresponds to a close-fitting processing layer in a super-resolution neural network. Therefore, this step aims to implement a processing process corresponding to a close-fitting processing layer in a super-resolution neural network.

[0125] As mentioned above, the processing process of the close connection processing layer in the super-resolution neural network is as follows Figure 4 As shown in (b) and (c) in Fig. 1, the feature map is first enlarged by an integer multiple, and the fused feature map F is generated based on the result after enlargement by an integer multiple. For example, the pixel "c3" in the feature map F is Figure 4 The enlarged FY shown in (b) of FIG. 1 is obtained by the gray sampling area containing 9 sampling positions in FUV. However, since the screen resolution is usually not an integer multiple of the bitstream resolution, it is necessary to downsample the fused feature map to obtain Figure 4By comparison, we can see that pixels a2, a5, b1-b6, c2, c5, d2, d5, e1-e6, f2, and f5 in the fusion feature map F are not needed when calculating screen pixels. Therefore, it can be seen that the feature extraction process of the pixels at these positions during the training of the neural network is unnecessary, which will lead to a waste of computing resources.

[0126] To avoid the above problems, the fragment shader in this example does not generate Figure 4 Instead of directly querying the sampling area corresponding to the pixel in the second image data in the first feature map, the first feature detection unit is called to directly obtain the initial pixel value of the pixel. Figure 4 As shown, the fragment shader is directly based on Figure 4 The content shown in (a') in the above equation is obtained. Figure 4 For example, the pixel “c3” is directly based on Figure 3 The gray sampling area containing 4 sampling positions in FY and FUV is obtained by showing the unamplified FY in (a'). Figure 4 The content shown in (a') is Figure 4 The first characteristic graph shown in (a) is Figure 4 The content shown in (d') is Figure 4 (d) shows the pixels in the second image data that ultimately need to be output to the screen.

[0127] It can be seen that since the sampling ratio of the feature detection unit (i.e., the convolution kernel) in the shader can be flexibly configured to achieve precise sampling, this example directly checks the sampling area corresponding to each pixel that should be included in the second image data to be generated in the first feature map according to the number of pixels that should be included in the second image data to be generated (specifically obtained according to the ratio between the resolution of the second image data and the resolution of the first image data), and then uses the pre-trained first feature detection unit to directly calculate the sampling area, thereby obtaining the pixels in the second image data whose resolution meets the requirements.

[0128] In summary, this example uses the precise sampling feature of the shader to avoid generating Figure 4The unnecessary computational overhead and intermediate storage overhead caused by (b) and (c) in the above figure speed up the generation of the second image data and significantly reduce the amount of calculation in the generation process. For example, since pixels a2, a5, b1-b6, c2, c5, d2, d5, e1-e6, f2, and f5 in the fusion feature map F are not needed when calculating screen pixels, they do not need to be calculated in the shader, and the pixels required for calculating screen pixels can be directly obtained, so that complex super-resolution algorithms can run on client devices with limited computing resources, improving the image quality and processing speed of client devices.

[0129] Step S303: through the fragment shader, call the second feature detection unit to perform fusion processing on the channel data in the first image data and the initial pixel value of any pixel to obtain the final pixel value of any pixel.

[0130] The second feature detection unit corresponds to the output layer in the super-resolution neural network. Therefore, this step is intended to implement the processing corresponding to the output layer in the super-resolution neural network.

[0131] Similar to the processing of the output layer, this step aims to perform channel fusion processing on the first image data and the initial pixel value to obtain the final pixel calculation result. Specifically, in order to achieve an improvement in resolution, this step can first amplify the Y channel image data and the UV channel image data in the first image data by a specified multiple (the specified multiple is usually not an integer multiple, but is obtained based on the ratio between the screen resolution and the code stream resolution to ensure accurate amplification), and then fuse the amplified Y channel image data, UV channel image data, and the initial pixel values ​​of the relevant pixels through the second feature detection unit to obtain the final pixel calculation result. Wherein, when the first image data is in YUV format, the second image data in RGB format can be directly generated.

[0132] Step S304: Generate second image data according to the final pixel value of any pixel.

[0133] It can be seen that in this example, using the fragment shader, the corresponding FY and FUV can be inversely queried for each screen pixel (i.e., the pixel in the second image data), and the results of the close processing layer and the output layer can be combined as output to be submitted to the screen.

[0134] In summary, this example pre-trains a super-resolution neural network based on a sample training set, and exports the network weights of the super-resolution neural network to the shader. During the training process, by replacing the activation function, limiting the number of channels, and connecting the output layer across layers, it is ensured that the neural network can converge quickly and achieve accurate processing with a small number of parameters. In addition, the neural network model is only used in the training process, and the reasoning process of the model is converted to the shader for implementation. Moreover, when the shader implements the reasoning process, it is optimized, and there is no need to generate intermediate results that are magnified integer times, but directly output each pixel accurately through the reverse query method, thereby greatly reducing the amount of calculation. In addition, by dividing the texture into blocks (i.e., blocks), the task dispatching method of the shader is improved, and the problem of L2-cache failure is effectively avoided. Moreover, this example can directly realize the conversion between the code stream data in YUV format and the screen data in RGB format, without the need to perform additional format conversion operations, thereby simplifying the image processing process and improving the processing efficiency.

[0135] Figure 5 A schematic diagram of a device 40 for generating image data provided by another embodiment of the present disclosure is shown. The device is implemented based on a shader in a graphics processor, and the device includes:

[0136] An acquisition module 41 is adapted to acquire a first feature map corresponding to first image data; wherein the resolution of the first image data is a first resolution;

[0137] an extraction module 42, adapted to query a sampling area corresponding to a pixel in the second image data to be generated in the first feature map, perform feature extraction on the sampling area, and obtain a pixel value of the pixel; wherein the resolution of the second image data is a second resolution, and the number of pixels of the second image data and the sampling area corresponding to the pixel are determined according to a comparison result between the second resolution and the first resolution;

[0138] The generating module 43 is adapted to generate the second image data according to the pixel value of the pixel.

[0139] In an optional implementation, the apparatus is applied to a client device in a cloud game, the first image data is obtained according to a video stream transmitted from a cloud server to the client device in the cloud game, and the first resolution is determined according to a code stream resolution of the cloud game;

[0140] The second image data is used for displaying in the client device, and the second resolution is determined according to the screen resolution of the client device; the second resolution is greater than the first resolution.

[0141] In an optional implementation, the shader includes: a computing shader; the first feature map corresponding to the first image data is obtained in the following manner:

[0142] Converting the first image data into texture data, wherein the texture data includes a plurality of texture blocks divided according to a preset size; wherein each texture block corresponds to N*N pixels; wherein N is a natural number greater than 1;

[0143] The compute shader obtains the texture block to be processed currently from the shared cache, and calculates the feature data corresponding to the texture block to obtain the first feature map; wherein the shared cache is shared by multiple image processing units in the graphics processor, and the compute shader runs in any image processing unit.

[0144] In an optional implementation, the shader further includes: a fragment shader; and the pixel value of the pixel in the second image data is determined by the fragment shader.

[0145] In an optional implementation, the extraction module is specifically adapted to: call a preset feature detection unit to extract features from the sampling area through the shader;

[0146] The parameter information of the feature detection unit is determined according to a pre-trained super-resolution neural network; and the super-resolution neural network is trained by amplifying the sample data by a first preset multiple; the first preset multiple is an integer multiple;

[0147] Furthermore, a comparison result between the second resolution and the first resolution is a ratio between the second resolution and the first resolution, and the ratio is a non-integer multiple.

[0148] In an optional implementation, the preset feature detection unit includes: a first feature detection unit and a second feature detection unit;

[0149] The extraction module is specifically suitable for:

[0150] Calling the first feature detection unit to perform feature extraction on the sampling area to obtain an initial pixel value of the pixel;

[0151] The second feature detection unit is called to perform fusion processing on the first image data and the initial pixel value of the pixel to obtain the pixel value of the pixel.

[0152] In an optional implementation, the first image data is in a first image format, and the second image data is in a second image format;

[0153] Furthermore, the first image data includes: single-channel data corresponding to one color channel, and dual-channel data corresponding to two color channels;

[0154] Then the acquisition module is specifically adapted to: perform feature extraction on the single-channel data through a first convolution kernel to obtain a first set of feature data including m channels;

[0155] Performing feature extraction on the dual-channel data through a second convolution kernel to obtain a second set of feature data containing n channels;

[0156] The first feature map is obtained according to the first set of feature data of the m channels and the second set of feature data of the n channels; wherein m and n are natural numbers less than a preset threshold.

[0157] In an optional implementation, the second feature detection unit includes: a 1*1 convolution kernel; the first image format includes a YUV format, and the second image format includes an RGB format; wherein the single-channel data corresponds to the Y channel in the YUV format, and the dual-channel data corresponds to the UV channel in the YUV format.

[0158] In an optional implementation, the super-resolution neural network is trained in the following manner:

[0159] Replacing a first activation function in a super-resolution neural network to be trained with a second activation function, and pre-training the replaced super-resolution neural network to obtain a pre-trained network model;

[0160] Replacing the second activation function in the pre-trained network model with the first activation function, reducing the learning rate of the pre-trained network model, and adjusting the pre-trained network model to obtain a trained super-resolution neural network;

[0161] The computation amount of the second activation function is less than that of the first activation function, and / or the convergence speed of the second activation function is higher than that of the first activation function.

[0162] Figure 6 A schematic structural diagram of an electronic device according to another embodiment of the present disclosure is shown. The specific embodiment of the present disclosure does not limit the specific implementation of the electronic device.

[0163] like Figure 6 As shown, the electronic device may include: a processor (processor) 502 , a communication interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0164] in:

[0165] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0166] The communication interface 504 is used to communicate with other devices such as clients or other servers.

[0167] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above-mentioned video image detection method embodiment.

[0168] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0169] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present disclosure. The one or more processors included in the electronic device may be processors of the same type, such as one or more CPUs. They may also be processors of different types, such as one or more CPUs and one or more ASICs.

[0170] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0171] The program 510 can be specifically used to enable the processor 502 to execute the corresponding operations in the above-mentioned method embodiment for generating image data.

[0172] The algorithms and displays provided herein are not inherently related to any particular computer, virtual device or other device. Various general purpose devices may also be used together with the teachings based thereon. According to the above description, it is apparent that the structure required for constructing such devices is. In addition, the present disclosure is not directed to any particular programming language either. It should be understood that various programming languages ​​may be utilized to implement the disclosure described herein, and the description of the specific languages ​​above is intended to disclose the best mode of implementation of the present disclosure.

[0173] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0174] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various disclosed aspects, in the above description of exemplary embodiments of the present disclosure, various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed disclosure requires more features than are expressly recited in each claim.

[0175] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0176] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present disclosure and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0177] The various component embodiments of the present disclosure may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the device according to the embodiments of the present disclosure. The present disclosure may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present disclosure may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0178] It should be noted that the above embodiments illustrate the present disclosure rather than limit the present disclosure, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present disclosure may be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

Claims

1. A method for generating image data, the method being implemented based on a shader in a graphics processor, the method comprising: Acquire a first feature map corresponding to first image data; wherein the resolution of the first image data is a first resolution; For a pixel in the second image data to be generated, query the sampling area corresponding to the pixel in the first feature map, perform feature extraction on the sampling area, and obtain a pixel value of the pixel; wherein the resolution of the second image data is a second resolution, and the number of pixels of the second image data and the sampling area corresponding to the pixel are determined according to a comparison result between the second resolution and the first resolution; The second image data is generated according to the pixel value of the pixel.

2. The method according to claim 1, wherein: The method is applied to a client device in a cloud game, the first image data is obtained according to a video stream transmitted from a cloud server to the client device in the cloud game, and the first resolution is determined according to a code stream resolution of the cloud game; The second image data is used for displaying in the client device, and the second resolution is determined according to the screen resolution of the client device; the second resolution is greater than the first resolution.

3. The method according to claim 1 or 2, wherein: The shader includes: a computing shader; the first feature map corresponding to the first image data is obtained by: Converting the first image data into texture data, wherein the texture data includes a plurality of texture blocks divided according to a preset size; wherein each texture block corresponds to N*N pixels; wherein N is a natural number greater than 1; The compute shader obtains the texture block to be processed currently from the shared cache, and calculates the feature data corresponding to the texture block to obtain the first feature map; wherein the shared cache is shared by multiple image processing units in the graphics processor, and the compute shader runs in any image processing unit.

4. The method according to claim 3, wherein: The shader further includes: a fragment shader; and the pixel value of the pixel in the second image data is determined by the fragment shader.

5. The method according to any one of claims 1 to 4, wherein: The extracting features from the sampling area comprises: calling a preset feature detection unit to extract features from the sampling area through the shader; The parameter information of the feature detection unit is determined according to a pre-trained super-resolution neural network; and the super-resolution neural network is trained by amplifying the sample image data by a first preset multiple; the first preset multiple is an integer multiple; Furthermore, a comparison result between the second resolution and the first resolution is a ratio between the second resolution and the first resolution, and the ratio is a non-integer multiple.

6. The method according to claim 5, wherein: The preset feature detection unit includes: a first feature detection unit and a second feature detection unit; The calling of a preset feature detection unit to extract features from the sampling area comprises: Calling the first feature detection unit to perform feature extraction on the sampling area to obtain an initial pixel value of the pixel; The second feature detection unit is called to perform fusion processing on the first image data and the initial pixel value of the pixel to obtain the pixel value of the pixel.

7. The method according to claim 6, wherein: The first image data is in a first image format, and the second image data is in a second image format; Furthermore, the first image data includes: single-channel data corresponding to one color channel, and dual-channel data corresponding to two color channels; Then the acquiring of the first feature map corresponding to the first image data comprises: performing feature extraction on the single-channel data through a first convolution kernel to obtain a first set of feature data including m channels; Performing feature extraction on the dual-channel data through a second convolution kernel to obtain a second set of feature data containing n channels; The first feature map is obtained according to the first set of feature data of the m channels and the second set of feature data of the n channels; wherein m and n are natural numbers less than a preset threshold.

8. The method according to claim 7, wherein: The first image format includes a YUV format, and the second image format includes an RGB format; wherein the single-channel data corresponds to a Y channel in the YUV format, and the dual-channel data corresponds to a UV channel in the YUV format.

9. The method according to any one of claims 5 to 8, wherein: The super-resolution neural network is trained in the following way: Replacing a first activation function in a super-resolution neural network to be trained with a second activation function, and pre-training the replaced super-resolution neural network to obtain a pre-trained network model; Replacing the second activation function in the pre-trained network model with the first activation function, reducing the learning rate of the pre-trained network model, and adjusting the pre-trained network model to obtain a trained super-resolution neural network; The computation amount of the second activation function is less than that of the first activation function, and / or the convergence speed of the second activation function is higher than that of the first activation function.

10. The method according to any one of claims 5 to 9, wherein: The super-resolution neural network includes: an input processing layer, a close connection processing layer, and an output layer; wherein, The input processing layer is used to calculate multiple channel feature maps corresponding to the first sample image data; The close connection processing layer is used to amplify the multiple channel feature maps by the first preset multiple, and then calculate the fusion feature map corresponding to the amplified multiple channel feature maps through the first convolution filter; The output layer is used to amplify the first sample image data by the first preset multiple, and then fuse the amplified first sample image data and the fusion feature map through a second convolution filter to obtain second sample image data; Wherein, the first feature detection unit is configured according to the first convolution filter, and the second feature detection unit is configured according to the second convolution filter.

11. A device for generating image data, the device being implemented based on a shader in a graphics processor, the device comprising: An acquisition module, adapted to acquire a first feature map corresponding to first image data; wherein the resolution of the first image data is a first resolution; an extraction module, adapted to query a sampling area corresponding to a pixel in the second image data to be generated in the first feature map, perform feature extraction on the sampling area, and obtain a pixel value of the pixel; wherein the resolution of the second image data is a second resolution, and the number of pixels of the second image data and the sampling area corresponding to the pixel are determined according to a comparison result between the second resolution and the first resolution; The generating module is adapted to generate the second image data according to the pixel value of the pixel.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the method for generating image data according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method for generating image data according to any one of claims 1 to 10.

14. A computer program product, characterized in that The invention comprises a computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the method for generating image data as described in any one of claims 1 to 10.

Images