Data processing method and device, equipment, storage medium and computer program product

Abstract

The application discloses a data processing method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: receiving a first texture feature of a first image sent by a cloud computer server; receiving a second image sent by the cloud computer server and decoding the second image to obtain a third image; the second image is an image obtained by encoding the first image; processing the first texture feature and the third image through a first model to obtain a fourth image; the third image is super-resolution processed through a first network of the first model to obtain a fifth image, and the fifth image and the first texture feature are fused and processed through a second network of the first model to obtain the fourth image; rendering the fourth image, and displaying the rendered fourth image.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] In cloud computing applications, cloud servers encode images and transmit the encoded images to cloud terminals via transmission protocols. The cloud terminals then decode and process the encoded images before displaying them. For ease of explanation, the images displayed on the cloud terminal are referred to as cloud computer images. However, due to factors such as codec type, image compression method, and network environment, cloud computer images may suffer from distortion or reduced resolution. Therefore, improving the display quality of cloud computer images has become a pressing issue. Summary of the Invention

[0003] To address the related technical problems, embodiments of this application provide a data processing method, apparatus, device, storage medium, and computer program product.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method, the method comprising:

[0006] The first texture feature of the first image sent by the cloud computer server;

[0007] The system receives a second image sent by the cloud computer server and decodes the second image to obtain a third image; wherein the second image is an image obtained by encoding the first image.

[0008] The first model processes the first texture feature and the third image to obtain a fourth image. The first network of the first model performs super-resolution processing on the third image to obtain a fifth image. The second network of the first model fuses the fifth image and the first texture feature to obtain the fourth image.

[0009] The fourth image is rendered and the rendered fourth image is displayed.

[0010] In the above scheme, the first network of the first model includes multiple first convolutional layers, and the stride of each of the multiple first convolutional layers is different.

[0011] In the above scheme, the super-resolution processing of the third image through the first network of the first model includes:

[0012] The third image is processed by each of the plurality of first convolutional layers to obtain the first data output by each of the first convolutional layers;

[0013] The first data output from each of the first convolutional layers is upsampled to obtain multiple first feature matrices, wherein each feature matrix in the multiple first feature matrices corresponds to a first data output from a first convolutional layer;

[0014] The plurality of first feature matrices are subjected to a first process to obtain the fifth image. The first process includes stitching, encoding, and decoding.

[0015] In the above scheme, the first data includes one or more of the following:

[0016] The visual features of the third image;

[0017] The global information of the third image;

[0018] Local information of the third image.

[0019] In the above scheme, the first network further includes a second convolutional layer. Correspondingly, the first processing of the plurality of first feature matrices includes:

[0020] The plurality of first feature matrices are concatenated to obtain a second feature matrix;

[0021] The second feature matrix is ​​encoded by the second convolutional layer to obtain the encoded second feature matrix;

[0022] The encoded second feature matrix is ​​concatenated with the first texture feature and bilinear upsampling is performed to obtain the fifth image.

[0023] In the above scheme, the second network includes a third convolutional layer, and the fusion processing of the fifth image and the first texture features through the second network of the first model includes:

[0024] The fifth image is magnified to obtain the magnified fifth image;

[0025] Based on the first texture feature, the magnified fifth image is subjected to texture synthesis and texture reconstruction to obtain the sixth image;

[0026] The sixth image is processed by the third convolutional layer to obtain the seventh image;

[0027] The seventh image is upsampled to obtain the fourth image.

[0028] In the above scheme, the training data of the first model includes an eighth image and the second texture features of the eighth image. The eighth image is obtained by preprocessing a ninth image, which is an image sent by the cloud computer server to the cloud computer terminal.

[0029] This application also provides a data processing apparatus, including:

[0030] The first receiving unit is used to receive the first texture feature of the first image sent by the cloud computer server;

[0031] The second receiving unit is used to receive the second image sent by the cloud computer server and decode the second image to obtain a third image; wherein the second image is an image obtained by encoding the first image;

[0032] The first processing unit is configured to process the first texture feature and the third image using a first model to obtain a fourth image, wherein the third image is super-division processed using a first network of the first model to obtain a fifth image, and the fifth image and the first texture feature are fused using a second network of the first model to obtain the fourth image.

[0033] A rendering unit is used to render the fourth image and display the rendered fourth image.

[0034] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0035] The processor is used to execute the steps of any of the above-mentioned technical solutions when running the computer program.

[0036] This application also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0037] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0038] In this embodiment, a first texture feature of a first image and a second image are received from a cloud computer server. The second image is an image obtained by encoding the first image. The second image is decoded to obtain a third image. The first texture feature and the third image are processed by a first model to obtain a fourth image. Specifically, during the processing of the first model, the third image is subjected to super-resolution processing. The fifth image obtained by super-resolution processing is also fused with the third image to obtain the fourth image. Then, the fourth image is rendered and displayed. As can be seen, in the embodiments of this application, when performing super-resolution processing on the second image sent by the cloud computer server, texture features are considered. Furthermore, the embodiments of this application introduce a first model. On the one hand, the first model can perform super-resolution on the second image; on the other hand, the first model can fuse the texture features with the super-resolution image. After rendering the fused image, the embodiments of this application will display the rendered image, i.e., the cloud computer image. In the embodiments of this application, image super-resolution can improve the resolution of the final rendered cloud computer image, thereby improving the display quality of the cloud computer image. Moreover, the embodiments of this application fuse the texture features with the super-resolution image, thus reducing the loss of key information in the cloud computer image and avoiding poor cloud computer image reconstruction results due to the loss of this key information, thereby improving the display quality of the cloud computer image. Attached Figure Description

[0039] Figure 1 Flowchart of the data processing method provided for the application embodiments of this application Figure 1 ;

[0040] Figure 2 Flowchart of the data processing method provided for the application embodiments of this application Figure 2 ;

[0041] Figure 3 A schematic diagram of the network architecture of the first model provided for the application embodiments of this application;

[0042] Figure 4 A flowchart illustrating Embodiment 1 provided for application of this application;

[0043] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0044] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0046] It should be understood that the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "one or more" in this document is an exemplary expression and can be replaced with any possible expression, such as one or more, at least one, or at least one item.

[0047] It should also be understood that the term "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of a relationship. For example, A instructing B can mean that A directly instructs B, such as B being able to obtain information through A; it can also mean that A indirectly instructs B, such as A instructing C, so B can obtain information through C; or it can mean that there is a relationship between A and B.

[0048] It should also be understood that the term "correspondence" mentioned in the embodiments of this application may indicate a direct or indirect correspondence between the two, or an association between the two, or a relationship of instruction and being instructed, configuration and being configured, etc.

[0049] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0050] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0051] The relevant technologies of the embodiments of this application are described below:

[0052] With the rapid development of artificial intelligence and deep learning technologies, image super-resolution technology is being used more and more widely in fields such as image recognition, object detection, and image editing. Image super-resolution technology can upscale images from low resolution to high resolution, thereby improving image quality and visual appeal. High-resolution images can provide more detailed information, helping to improve the accuracy and effectiveness of these tasks. High-resolution images have more pixels and finer image quality, which can more clearly show the details and textures of objects, making the images more realistic and vivid.

[0053] Currently, image super-resolution technology is widely used in many practical applications such as remote sensing images, medical imaging, and mobile phone photography. In cloud computing scenarios, images are encoded on cloud servers and decoded on cloud terminal devices via transmission protocols. However, factors such as codec type, image compression method, and network environment can cause image distortion or resolution degradation. Therefore, improving the image display quality on cloud computers has become an urgent problem to be solved.

[0054] With the continuous development of deep learning technology, various image super-resolution optimization methods based on deep learning have been designed in related technologies.

[0055] A mobile image compression method based on edge feature fusion and super-resolution is proposed in related technologies. In this method, a lightweight encoder is proposed, and bandwidth is saved by introducing a decoding module based on super-resolution algorithm at the decoder end. The image is reduced in size at the source, transmitted with less bandwidth, and decoded and reconstructed at the receiving point.

[0056] A related technology proposes an image super-resolution reconstruction method based on a compressed multi-scale feature fusion network. In this method, a multi-scale feature fusion layer in the multi-scale feature fusion network is used to extract features of multiple scales in low-resolution images to characterize richer detail information and more complete structural information in low-resolution images. Furthermore, multiple multi-scale feature fusion layers are stacked and connected in sequence to further improve the network's ability to fit the nonlinear mapping relationship between features of high and low resolution images.

[0057] A compressed image super-resolution method based on an enhanced attention network is proposed in related technologies. This method uses deep learning to construct an end-to-end convolutional neural network and adds an enhanced attention module to the network.

[0058] Among the aforementioned technologies, no mature computer image optimization dataset has been established, nor are there any targeted cloud computer image super-resolution algorithms. Specifically, image super-resolution technology aims to improve image resolution, but cloud computer images are a type of image with special characteristics, and related super-resolution algorithms cannot reconstruct these images well.

[0059] In the aforementioned related technologies, directly compressing the original image and extracting low-resolution image features without considering the texture features of the original image will result in the loss of some key information of the original image, which leads to poor image reconstruction results during feature reconstruction.

[0060] The aforementioned technologies suffer from problems such as excessive network bandwidth consumption and low transmission efficiency. Since images consume significantly more bandwidth than text or other information during transmission, large deep network training models, such as the Visual Geometry Group Network (VGG), suffer from excessively large parameter counts, resulting in slow training speeds and higher computational resource consumption.

[0061] The aforementioned related technologies neglect the importance of local information in image super-resolution tasks. For example, the focus of traditional multi-layer stacked convolutional layers and pyramid pooling modules is on the global information of the image, while image super-resolution tasks only focus on global information and ignore local information, which will lead to incomplete image super-resolution. The related technologies do not have targeted methods to process local information.

[0062] To address the aforementioned issues, this application proposes a data processing method. In this embodiment, a first texture feature of a first image and a second image are received from a cloud computer server. The second image is an image obtained by encoding the first image. The second image is decoded to obtain a third image. The first texture feature and the third image are processed by a first model to obtain a fourth image. Specifically, during the processing of the first model, the third image undergoes super-resolution processing. The fifth image obtained by the super-resolution processing is then fused with the third image to obtain the fourth image. Finally, the fourth image is rendered and displayed. As can be seen, in the embodiments of this application, when performing super-resolution processing on the second image sent by the cloud computer server, texture features are considered. Furthermore, the embodiments of this application introduce a first model. On the one hand, the first model can perform super-resolution on the second image; on the other hand, the first model can fuse the texture features with the super-resolution image. After rendering the fused image, the embodiments of this application will display the rendered image, i.e., the cloud computer image. In the embodiments of this application, image super-resolution can improve the resolution of the final rendered cloud computer image, thereby improving the display quality of the cloud computer image. Moreover, the embodiments of this application fuse the texture features with the super-resolution image, thus reducing the loss of key information in the cloud computer image and avoiding poor cloud computer image reconstruction results due to the loss of this key information, thereby improving the display quality of the cloud computer image.

[0063] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0064] This application provides a data processing method, see [link to relevant documentation] Figure 1 The method includes:

[0065] Step 101: Receive the first texture features of the first image sent by the cloud computer server.

[0066] For example, embodiments of this application can be applied to cloud computer terminals and / or modules configured in cloud computer terminals.

[0067] It should be noted that the cloud computer server in this embodiment is an exemplary term and can be replaced with any possible term, such as cloud computer server or server, etc.

[0068] It should be noted that the term "cloud computer terminal" in this application embodiment is an exemplary term and can be replaced with any possible term, such as cloud computer client, cloud terminal, or cloud computer.

[0069] It should be noted that steps 101 to 104 are examples of this application applied to a cloud computer scenario. This application embodiment is not limited to cloud computer scenarios; it can also be used in other scenarios where virtual desktops are built or other scenarios where cloud resource services are accessed. This application embodiment can also be used in scenarios involving the processing of other images, such as processing images sent to the terminal by a third-party application's server.

[0070] For example, the first image may include one or more image frames.

[0071] For example, the first image may be the original image corresponding to the image that the cloud computer server needs to send to the cloud computer terminal, and / or the first image may be an image obtained after preprocessing the original image.

[0072] For example, the first image may be an image captured from a cloud computer server.

[0073] For example, before receiving the first texture features of the first image sent by the cloud computer server, the cloud computer server can preprocess the original image to obtain the first image. The preprocessing includes denoising, edge enhancement, standardization, and / or normalization. Denoising employs mean filtering and median filtering to remove noise from the original image. If the original image is a color image, it can be converted to grayscale through grayscale conversion, and image noise can be reduced using smoothing techniques such as Gaussian filters.

[0074] Understandably, the purpose of preprocessing is to improve image quality and provide better input for subsequent texture feature extraction and super-resolution processing.

[0075] For example, before receiving the first texture features of the first image sent by the cloud computer server, the cloud computer server extracts the first texture features of the first image. For instance, the cloud computer server may use methods such as Local Binary Patterns (LBP) or Texture Orientation Distribution (TOD) to extract the texture features of the first image. The cloud computer server may use the Sobel operator or the Canny operator to enhance the edge information of the first image.

[0076] Understandably, the purpose of cloud computer servers performing texture feature extraction is to capture texture information in images, providing a basis for subsequent texture feature analysis and image optimization processing.

[0077] For feature analysis of the first texture feature, this application provides several optional examples:

[0078] Example 1: The cloud computer server performs feature analysis on the first texture feature.

[0079] For example, the cloud computer server can use statistical analysis or pattern recognition methods to perform feature analysis on the first texture feature to obtain texture information, wherein the texture information includes information such as the distribution, intensity and direction of the texture features, which can provide a reference for subsequent texture feature preservation and image optimization processing.

[0080] For example, when the cloud computer server performs feature analysis on the first texture feature, the cloud computer server can send the texture information to the cloud computer terminal. Accordingly, step 101 can be replaced by: receiving the texture information of the first image sent by the cloud computer server.

[0081] Example 2: The cloud computer terminal performs feature analysis on the first texture feature.

[0082] For example, the cloud computer terminal can use methods such as statistical analysis or pattern recognition to perform feature analysis on the first texture feature to obtain texture information.

[0083] Step 102: Receive the second image sent by the cloud computer server and decode the second image to obtain the third image.

[0084] The second image is an image obtained by encoding the first image.

[0085] For example, before receiving the second image sent by the cloud computer server, the cloud computer server encodes the first image to obtain the second image. For instance, the second image can be bitstream data; please refer to [link to relevant documentation]. Figure 2The cloud computer server can encode the first image into a bitstream data, and then send the bitstream data to the cloud computer terminal through end-to-end transmission.

[0086] For example, when encoding the first image, the cloud computer server can use standard encoding algorithms such as H.264 and H.265 to improve the transmission efficiency and quality of the image.

[0087] For example, in this embodiment of the application, after the cloud computer server encodes the first image to obtain the second image, it can send the second image to the cloud computer server based on the transmission protocol. Correspondingly, the cloud computer terminal receives the second image sent by the cloud computer server.

[0088] Step 103: Process the first texture features and the third image using the first model to obtain the fourth image.

[0089] Specifically, the third image is super-resolution processed by the first network of the first model to obtain the fifth image, and the fifth image and the first texture features are fused by the second network of the first model to obtain the fourth image.

[0090] For example, the first model is a neural network model, which includes an encoding structure and a decoding structure.

[0091] For example, the first model is used to fuse the first texture features with the third image and perform upsampling reconstruction.

[0092] In practical applications, the first network of the first model includes multiple first convolutional layers, each with a different stride.

[0093] Understandably, multiple convolutional layers of different lengths can be used to extract information at different scales, which is beneficial for more comprehensive image reconstruction and improves the image reconstruction effect.

[0094] For example, the plurality of first convolutional layers are four convolutional layers.

[0095] For example, the stride of these multiple first convolutional layers is a multiple of 1 or n, where n is the adaptive network stride and n is a positive integer.

[0096] For example, when the number of first convolutional layers is 4, the strides of these 4 first convolutional layers are set to 1, n, 2n, and 3n, respectively.

[0097] For example, the value of n can be adaptively adjusted according to the requirements of the image processing task and / or the network environment.

[0098] For example, the first network also includes a global pooling layer.

[0099] For example, the global pooling layer and multiple first convolutional layers are both part of the encoding portion of the first model.

[0100] In practical applications, the third image is super-resolution processed using the first network of the first model, including:

[0101] The third image is processed by each of the multiple first convolutional layers to obtain the first data output by each first convolutional layer;

[0102] The first data output from each first convolutional layer is upsampled to obtain multiple first feature matrices, wherein each feature matrix in the multiple first feature matrices corresponds to the first data output from a first convolutional layer;

[0103] A first process is performed on multiple first feature matrices to obtain a fifth image. The first process includes stitching, encoding, and decoding.

[0104] It is understood that the embodiments of this application have improved the upsampling of the encoding process by upsampling the data output by the first convolutional layer of different phase lengths to reduce the loss of semantic information.

[0105] To help understand "upsampling the first data output by each first convolutional layer to obtain multiple first feature matrices", let's take an example. For instance, there are two first convolutional layers, convolutional layer 1 and convolutional layer 2. Upsampling the first data output by convolutional layer 1 yields first feature matrix 1, and upsampling the first data output by convolutional layer 2 yields first feature matrix 2. That is, there is a one-to-one correspondence between the first data output by each first convolutional layer and the first feature matrix.

[0106] It is understood that the aforementioned upsampling and first processing are both performed by the first network. For example, the first network of the first model includes multiple first convolutional layers and other network structures. For instance, the other network structures can be upsampling modules, which can upsample the first data output by each first convolutional layer to obtain multiple first feature matrices. For example, other network structures can include concatenation modules, which can concatenate multiple first feature matrices, etc.

[0107] In practical applications, the first data includes one or more of the following:

[0108] Visual features of the third image;

[0109] Global information of the third image;

[0110] Local information in the third image.

[0111] It is understood that, in this embodiment of the application, since there are multiple convolutional layers with different strides in the first network, the third image can be processed by these multiple convolutional layers with different strides, which can extract both global information and local information of the third image. For example, local information can be obtained by convolutional layers with small strides, and global information can be obtained by convolutional layers with large strides.

[0112] For example, the visual features of the third image can be its color features and / or shape features, etc.

[0113] For example, global information could be the overall image attributes of the third image.

[0114] For example, the aforementioned local information could be features of local details such as image edges.

[0115] For example, the aforementioned visual features can be multi-scale visual features.

[0116] In practical applications, the first network also includes a second convolutional layer, which performs a first process on multiple first feature matrices, including:

[0117] The second feature matrix is ​​obtained by concatenating multiple first feature matrices.

[0118] The second feature matrix is ​​encoded by the second convolutional layer to obtain the encoded second feature matrix;

[0119] The encoded second feature matrix is ​​concatenated with the first texture feature and bilinear upsampling is performed to obtain the fifth image.

[0120] It is understandable that the first processing described above is all performed by the first model. For example, "concatenating the encoded second feature matrix with the first texture feature and performing bilinear upsampling" can be performed by the decoding module of the first model.

[0121] It is understood that the bilinear upsampling method used in the embodiments of this application can generate more natural images, and the bilinear upsampling method can improve computational efficiency and speed.

[0122] For example, the second convolutional layer described above can be a 1×1 convolutional layer.

[0123] For example, multiple first feature matrices obtained by upsampling are concatenated and encoded after passing through a 1×1 convolutional layer; in the decoding stage, the encoded first feature matrix is ​​concatenated with the texture features and bilinearly upsampled to finally obtain the corresponding fifth image.

[0124] For example, the fifth image has a higher resolution than the third image.

[0125] In practical applications, the second network includes a third convolutional layer. The second network of the first model fuses the fifth image and the first texture features, including:

[0126] The fifth image is enlarged to obtain the enlarged fifth image;

[0127] Based on the first texture features, texture synthesis and texture reconstruction are performed on the magnified fifth image to obtain the sixth image;

[0128] The sixth image is processed by the third convolutional layer to obtain the seventh image;

[0129] The seventh image is upsampled to obtain the fourth image.

[0130] It is understood that the embodiments of this application can reconstruct low-resolution images through the first model, and can fuse the extracted texture features with the reconstructed high-resolution image, and reconstruct and output a super-resolution image from the encoded lossy low-resolution image through the first model.

[0131] For example, the third convolutional layer can be a 3×3 convolutional layer.

[0132] For example, enlarging the fifth image includes: enlarging the fifth image using a first interpolation algorithm.

[0133] For example, the first interpolation algorithm includes bilinear interpolation, nearest neighbor interpolation, or bicubic interpolation, etc.

[0134] For example, the first interpolation algorithm is determined based on the distribution and intensity of the first texture feature. That is, the first interpolation algorithm used can be adaptively adjusted according to the distribution and intensity of the first texture feature. For example, when the texture feature is relatively smooth and is a weak texture, a bilinear interpolation algorithm can be used. For example, when the texture feature is rich in detail and has a high frequency, a bilinear interpolation algorithm can be used.

[0135] For example, performing texture synthesis on the magnified fifth image based on the first texture feature includes: mapping the first texture feature to the magnified fifth image, and / or synthesizing the first texture feature into the magnified fifth image.

[0136] For example, texture reconstruction of the magnified fifth image based on the first texture features includes: reconstructing the texture information of the magnified fifth image using sparse representation and / or dictionary learning methods.

[0137] For example, please refer to Figure 3 , Figure 3This is a schematic diagram of the network architecture of the first model provided in an embodiment of this application. The first network can be a network consisting of multiple convolutional layers of different lengths up to the first upsampling section. The fifth image, i.e., the super-resolution image, can be obtained through the first network. The second network includes a fusion module, as well as a 3×3 convolutional layer and the second upsampling section. The output of the second network is the fourth image, i.e., the reconstructed image, which is the image obtained by reconstructing the low-resolution third image.

[0138] Step 104: Render the fourth image and display the rendered fourth image.

[0139] For example, displaying the rendered fourth image includes displaying the rendered fourth image on the desktop of a cloud computer terminal.

[0140] For example, a user can trigger a first instruction by operating a cloud computer terminal. The cloud computer terminal sends the first instruction to a cloud computer server. The first instruction triggers the cloud computer server to generate a first image or capture a first graphic. Then, the cloud computer server sends the first texture feature of the first image to the cloud computer terminal and sends a second image.

[0141] For example, the above operations can be user operation of the mouse, keyboard, or user touch screen, etc.

[0142] For example, a user can click the play button for the first video on the cloud computer terminal. At this time, the cloud computer server can capture the first image from the video stream corresponding to the first video, encode the first image to obtain the second image, extract the texture features of the first image to obtain the first texture feature, and send the second image and the first texture feature to the cloud computer terminal. The cloud computer terminal then executes... Figure 1 The method shown generates a fifth image, which can be one or more high-resolution image frames, and then displays the fifth image to the user.

[0143] To better train the first model, this application embodiment creates a cloud computer image dataset, which includes multiple eighth images, as detailed in the following practical application:

[0144] In practical applications, the training data of the first model includes an eighth image and the second texture features of the eighth image. The eighth image is obtained by preprocessing a ninth image, which is an image sent from the cloud computer server to the cloud computer terminal.

[0145] It is understood that the embodiments of this application construct a model training dataset specifically for the scenario of cloud computer image super-resolution optimization, namely the eighth image and the second texture features of the eighth image. Compared to training the model using images such as natural landscape images, training the model using the eighth image, which is actually involved in the cloud computer image super-resolution optimization scenario, is more targeted and better suited to the cloud computer image super-resolution optimization scenario, thereby improving the model's performance in super-resolution optimization of cloud computer images.

[0146] For example, the ninth image is a cloud computer image, which may include an image sent by the cloud computer server to the cloud computer terminal and / or an image displayed on the cloud computer terminal. When the ninth image is an image sent by the cloud computer server to the cloud computer terminal, the cloud computer terminal processes the ninth image and then displays the ninth image.

[0147] For example, in this embodiment of the application, the term "cloud computer image" is an exemplary term and can be replaced with any possible term, such as "cloud computer desktop display image," etc.

[0148] For example, the ninth image may include cloud computer images in various user scenarios, such as cloud computer images in multiple actual user scenarios such as office scenarios, video scenarios, and idle scenarios.

[0149] For example, the ninth image can be collected multiple times, such as 10,000 ninth images, to ensure that the ninth images cover multiple actual user scenarios such as office scenarios, video scenarios, and idle scenarios.

[0150] For example, preprocessing the ninth image includes: denoising, edge enhancement, standardization, and / or normalization. Denoising employs mean filtering, median filtering, etc., to remove noise from the image. Converting the color image to grayscale through grayscale conversion and reducing image noise through smoothing techniques such as Gaussian filters are also performed. It is understood that the purpose of preprocessing is to improve image quality and provide better input for subsequent texture feature extraction and super-resolution processing.

[0151] As can be seen from the above, this application proposes a cloud computer image optimization method based on texture features and super-resolution. In the field of video coding communication, super-resolution technology can improve image resolution, thereby improving the quality and efficiency of video communication and reducing transmission bandwidth requirements. Specifically, the application of this application in video coding communication is mainly reflected in the following aspects: First, improving video resolution: The technology used in this application can upscale low-resolution videos to high-resolution videos, making the video images clearer and more detailed. This helps improve the display quality of cloud computer clients and provides users with a better viewing experience. Second, reducing bitrate: The technology used in this application can reduce the video bitrate while maintaining image quality. This helps reduce the amount of data during video transmission, lower communication costs, and improve transmission efficiency. Third, improving image quality: The technology used in this application can perform noise reduction, sharpening, and other processing on images to further improve video image quality. This helps reduce video loss during transmission and playback, while improving the user's viewing experience.

[0152] Example 1

[0153] The process involved in Example 1 includes: preparing a cloud computer image dataset, extracting the texture features of the images, fusing and reconstructing the images based on the texture features and after encoding, compression, transmission and decoding, and rendering the cloud images to the client, etc.

[0154] Please see Figure 4 , Figure 4 Here is an exemplary flowchart of Example 1. Figure 4 The process includes the following steps:

[0155] S401: Prepare the cloud computer image dataset and preprocess the cloud computer image dataset.

[0156] Since there are no traditional publicly available datasets for cloud computer images, the images in this embodiment are derived from cloud computer image data. Multiple collections of cloud computer images, for example, collecting 10,000 images, yield a cloud computer image dataset.

[0157] It should be noted that the cloud computer image in Example 1 is an example of the ninth image in the above method embodiments.

[0158] For example, cloud computer images are collected in multiple scenarios, so that the cloud computer image dataset covers multiple actual user usage scenarios such as office scenarios, video scenarios, and idle scenarios.

[0159] For example, preprocessing is performed on cloud computer images in the cloud computer image dataset, including denoising and edge enhancement, standardization, and normalization. Denoising employs mean filtering and / or median filtering to remove noise from the image. Color images are converted to grayscale images through grayscale conversion, and image noise is reduced using smoothing techniques such as Gaussian filters. The purpose of preprocessing is to improve image quality and provide better input for subsequent texture feature extraction and super-resolution processing.

[0160] S402: Extract texture features from cloud computer images.

[0161] For example, methods such as LBP or TOD can be used to extract texture features from the preprocessed cloud computer image. The Sobel and Canny operators are then used to enhance the edges of the preprocessed cloud computer image, thereby improving edge information. The purpose of texture feature extraction is to capture texture information in the preprocessed cloud computer image, providing a basis for subsequent texture feature analysis and image optimization.

[0162] For example, statistical analysis and pattern recognition methods are used to perform feature analysis on the extracted texture features. The purpose of feature analysis is to analyze information such as the distribution, intensity, and orientation of texture features, providing a reference for subsequent texture feature preservation and image optimization processing. The optimized image is then video encoded and transmitted. Video encoding uses standard encoding algorithms such as H.264 and H.265 to improve image transmission efficiency and quality.

[0163] To better perform super-resolution reconstruction of cloud computer images, this application proposes a Texture Feature Super-Resolution Network (TFSR) model, which is an example of the first model in the above method embodiments.

[0164] S403: Train the TFSR model.

[0165] For example, the preprocessed cloud computer image obtained in step S401 and the texture features extracted in step S402 can be used to train the TFSR model.

[0166] For example, after training, the TFSR model can be used to fuse texture features with images and perform upsampling reconstruction. For instance, a cloud computer server encodes and compresses cloud computer images, then transmits the encoded and compressed cloud computer images to a cloud computer terminal. The cloud computer terminal decodes the encoded and compressed cloud computer images, and then uses the TFSR model to fuse texture features with the decoded images and perform upsampling reconstruction.

[0167] Exemplarily, the TFSR model in this application uses an encoding-decoding structure. The encoding part uses four multi-scale dilated convolutional layers and one global pooling layer. The stride of the convolutional layers is set to 1, n, 2n, and 3n, respectively, where n is the adaptive network stride and n is a positive integer. These four convolutional layers are used to capture multi-scale visual features, global information, and local information in cloud computer images. This application improves the upsampling process in the encoding process by upsampling the outputs of convolutional layers with different strides, minimizing the loss of semantic information. Then, the upsampled multi-scale feature matrices are concatenated and encoded after passing through a 1×1 convolutional layer. In the decoding stage, the encoded multi-scale feature matrix is ​​concatenated with the input texture features and bilinearly upsampled to finally obtain the corresponding super-resolution image. The network architecture of the TFSR model is as follows: Figure 3 As shown.

[0168] The above example is an example of the first network of the TFSR model in the embodiments of this application.

[0169] S404: Integrates texture features with the image and performs super-resolution reconstruction.

[0170] For example, a super-resolution reconstruction algorithm is used to fuse texture features with the image and perform super-resolution reconstruction, while a bilinear interpolation algorithm is used to enlarge the image. The choice of interpolation algorithm can be adaptively adjusted according to the distribution and intensity of texture features. Combining the texture features, texture synthesis and reconstruction are performed on the enlarged image. Texture synthesis uses texture mapping and texture synthesis methods to map texture features onto the enlarged image region; texture reconstruction uses sparse representation and dictionary learning methods to reconstruct the texture information of the enlarged image.

[0171] The above example demonstrates how to reconstruct low-resolution images. The extracted texture features are fused with the reconstructed high-resolution image, and a super-resolution reconstruction function is used to reconstruct the coded, lossy low-resolution image, outputting a super-resolution image.

[0172] S405: Evaluation and demonstration of super-resolution effects.

[0173] This application evaluates and demonstrates the super-resolution effect of the above-described method. To more clearly demonstrate the super-resolution effect of the method involved in this application embodiment, the images before and after super-resolution reconstruction are compared. The comparison shows that the original image is relatively blurry in terms of image and text edge processing, while the image processed by the technology of this application embodiment has improved overall clarity, and the edges of text and images are neater.

[0174] Compared to directly transmitting the original image, the data processing method in this embodiment can achieve better image display. The quality of the optimized image is evaluated using Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) to assess the super-resolution results. The calculated values ​​of the super-resolution reconstructed image using this embodiment are significantly greater than those of the original image compared to the compressed, lossy image. Higher PSNR and SSIM scores indicate more accurate prediction results. Parameter adjustments and optimizations can be made based on the optimization effect, providing direction and theoretical basis for subsequent cloud computer image optimization, thereby bringing a better user experience.

[0175] As can be seen from the above, this application proposes a cloud computer image dataset and a texture feature super-resolution network, as well as a cloud computer image optimization method based on texture features and super-resolution. Compared to directly using a lightweight encoder with mobile image compression methods, this application uses an encoder that supports personal computer (PC) applications. While ensuring encoding speed, it can encode richer and more faithfully reconstruct image content, eliminating the heavy computational burden of the encoding process and reducing task complexity while maintaining reconstruction quality. Compared to directly compressing the image and extracting low-resolution image features, this application considers extracting the original image texture features, greatly reducing the risk of losing key information after compression and avoiding poor reconstruction results due to the loss of image super-resolution information. Compared to compressing image super-resolution using an enhanced attention network, this scheme focuses on the global information of the image. However, image super-resolution evaluation tasks only focus on global information and ignore local information, leading to incomplete image super-resolution. This scheme does not have a specific method for processing local information. This application embodiment not only uses an adaptive encoding module, but also adds a decoding module. After concatenating the encoded feature matrix with the texture features and performing bilinear upsampling, it captures both global and local information, ultimately obtaining a more accurate image quality evaluation result.

[0176] Based on the embodiments described above, this application also provides a data processing apparatus, see [link to previous document]. Figure 5 The data processing device includes:

[0177] The first receiving unit 501 is used to receive the first texture feature of the first image sent by the cloud computer server;

[0178] The second receiving unit 502 is used to receive the second image sent by the cloud computer server and decode the second image to obtain a third image; wherein the second image is an image obtained by encoding the first image;

[0179] The first processing unit 503 is configured to process the first texture feature and the third image through a first model to obtain a fourth image, wherein the third image is super-resolution processed through the first network of the first model to obtain a fifth image, and the fifth image and the first texture feature are fused through the second network of the first model to obtain the fourth image.

[0180] The rendering unit 504 is used to render the fourth image and display the rendered fourth image.

[0181] In one embodiment, the first network of the first model includes a plurality of first convolutional layers, each with a different stride.

[0182] In one embodiment, the first processing unit 503 performs super-resolution processing on the third image through the first network of the first model, including:

[0183] The third image is processed by each of the plurality of first convolutional layers to obtain the first data output by each of the first convolutional layers;

[0184] The first data output from each of the first convolutional layers is upsampled to obtain multiple first feature matrices, wherein each feature matrix in the multiple first feature matrices corresponds to a first data output from a first convolutional layer;

[0185] The plurality of first feature matrices are subjected to a first process to obtain the fifth image. The first process includes stitching, encoding, and decoding.

[0186] In one embodiment, the first data includes one or more of the following:

[0187] The visual features of the third image;

[0188] The global information of the third image;

[0189] Local information of the third image.

[0190] In one embodiment, the first network further includes a second convolutional layer. Correspondingly, the first processing unit 503 performs a first processing on the plurality of first feature matrices, including:

[0191] The plurality of first feature matrices are concatenated to obtain a second feature matrix;

[0192] The second feature matrix is ​​encoded by the second convolutional layer to obtain the encoded second feature matrix;

[0193] The encoded second feature matrix is ​​concatenated with the first texture feature and bilinear upsampling is performed to obtain the fifth image.

[0194] In one embodiment, the second network includes a third convolutional layer, and the first processing unit 503 performs fusion processing on the fifth image and the first texture features through the second network of the first model, including:

[0195] The fifth image is magnified to obtain the magnified fifth image;

[0196] Based on the first texture feature, the magnified fifth image is subjected to texture synthesis and texture reconstruction to obtain the sixth image;

[0197] The sixth image is processed by the third convolutional layer to obtain the seventh image;

[0198] The seventh image is upsampled to obtain the fourth image.

[0199] In one embodiment, the training data of the first model includes an eighth image and a second texture feature of the eighth image. The eighth image is obtained by preprocessing a ninth image, which is an image sent from the cloud computer server to the cloud computer terminal.

[0200] In practical applications, the first receiving unit 501, the second receiving unit 502, the first processing unit 503, and the rendering unit 504 can be implemented by the processor in the data processing device.

[0201] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0202] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, this application also provides an electronic device, see [link to relevant documentation]. Figure 6 The electronic device includes:

[0203] The first communication interface 1 is used to receive the first texture feature of a first image sent by the cloud computer server; receive the second image sent by the cloud computer server, and decode the second image to obtain a third image; wherein the second image is an image obtained by encoding the first image;

[0204] The first processor 2 is configured to process the first texture feature and the third image using a first model to obtain a fourth image, wherein the third image is super-resolution processed using a first network of the first model to obtain a fifth image, and the fifth image and the first texture feature are fused using a second network of the first model to obtain the fourth image; the fourth image is then rendered and displayed.

[0205] In one embodiment, the first network of the first model includes a plurality of first convolutional layers, each with a different stride.

[0206] In one embodiment, the first processor 2 performs super-resolution processing on the third image through the first network of the first model, including:

[0207] The third image is processed by each of the plurality of first convolutional layers to obtain the first data output by each of the first convolutional layers;

[0208] The first data output from each of the first convolutional layers is upsampled to obtain multiple first feature matrices, wherein each feature matrix in the multiple first feature matrices corresponds to a first data output from a first convolutional layer;

[0209] The plurality of first feature matrices are subjected to a first process to obtain the fifth image. The first process includes stitching, encoding, and decoding.

[0210] In one embodiment, the first data includes one or more of the following:

[0211] The visual features of the third image;

[0212] The global information of the third image;

[0213] Local information of the third image.

[0214] In one embodiment, the first network further includes a second convolutional layer, and correspondingly, the first processor 2 performs a first processing on the plurality of first feature matrices, including:

[0215] The plurality of first feature matrices are concatenated to obtain a second feature matrix;

[0216] The second feature matrix is ​​encoded by the second convolutional layer to obtain the encoded second feature matrix;

[0217] The encoded second feature matrix is ​​concatenated with the first texture feature and bilinear upsampling is performed to obtain the fifth image.

[0218] In one embodiment, the second network includes a third convolutional layer, and the first processor 2 performs fusion processing on the fifth image and the first texture features through the second network of the first model, including:

[0219] The fifth image is magnified to obtain the magnified fifth image;

[0220] Based on the first texture feature, the magnified fifth image is subjected to texture synthesis and texture reconstruction to obtain the sixth image;

[0221] The sixth image is processed by the third convolutional layer to obtain the seventh image;

[0222] The seventh image is upsampled to obtain the fourth image.

[0223] In one embodiment, the training data of the first model includes an eighth image and a second texture feature of the eighth image. The eighth image is obtained by preprocessing a ninth image, which is an image sent from the cloud computer server to the cloud computer terminal.

[0224] It should be noted that the specific processing procedure of the first communication interface 1 can be understood by referring to the above method.

[0225] Of course, in practical applications, the various components in an electronic device are coupled together through bus system 4. It can be understood that bus system 4 is used to achieve communication and connection between these components. In addition to the data bus, bus system 4 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general will label all buses as Bus System 4.

[0226] The first memory 3 in this embodiment is used to store various types of data to support operation in the electronic device. Examples of such data include any computer program used to operate on the electronic device.

[0227] The methods disclosed in the embodiments of this application can be applied to the first processor 2, or implemented by the first processor 2. The first processor 2 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the first processor 2. The first processor 2 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 2 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the first memory 3. The first processor 2 reads the information in the first memory 3 and completes the steps of the aforementioned method in combination with its hardware.

[0228] In an exemplary embodiment, the electronic device may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0229] It is understood that the first memory 3 in the embodiments of this application can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0230] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory 3 storing a computer program, which can be executed by a first processor 2 to complete the steps described in the aforementioned method.

[0231] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a first processor 2 to perform the steps described in the foregoing method.

[0232] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data processing method, characterized in that, Applied to cloud computer terminals, the data processing method includes: The first texture feature of the first image sent by the cloud computer server; The system receives a second image sent by the cloud computer server and decodes the second image to obtain a third image; wherein the second image is an image obtained by encoding the first image. The first model processes the first texture feature and the third image to obtain a fourth image; wherein, the first network of the first model performs super-resolution processing on the third image to obtain a fifth image, and the second network of the first model performs fusion processing on the fifth image and the first texture feature to obtain the fourth image; The fourth image is rendered and the rendered fourth image is displayed; The second network includes a third convolutional layer, and the fusion processing of the fifth image and the first texture features through the second network of the first model includes: The fifth image is magnified to obtain the magnified fifth image; The magnified fifth image is subjected to texture synthesis and texture reconstruction based on the first texture feature to obtain a sixth image. The texture synthesis of the magnified fifth image based on the first texture feature includes: mapping the first texture feature onto the magnified fifth image; and / or, synthesizing the first texture feature into the magnified fifth image. The sixth image is processed by the third convolutional layer to obtain the seventh image; The seventh image is upsampled to obtain the fourth image.

2. The method according to claim 1, characterized in that, The first network of the first model includes multiple first convolutional layers, each with a different stride.

3. The method according to claim 2, characterized in that, The process of processing the first texture feature and the third image using the first model includes: The third image is processed by each of the plurality of first convolutional layers to obtain the first data output by each of the first convolutional layers; The first data output from each of the first convolutional layers is upsampled to obtain multiple first feature matrices, wherein each feature matrix in the multiple first feature matrices corresponds to a first data output from a first convolutional layer; The plurality of first feature matrices are subjected to a first process to obtain the fifth image. The first process includes stitching, encoding, and decoding.

4. The method according to claim 3, characterized in that, The first data includes one or more of the following: The visual features of the third image; The global information of the third image; Local information of the third image.

5. The method according to claim 3, characterized in that, The first network further includes a second convolutional layer, and correspondingly, the first processing of the plurality of first feature matrices includes: The plurality of first feature matrices are concatenated to obtain a second feature matrix; The second feature matrix is ​​encoded by the second convolutional layer to obtain the encoded second feature matrix; The encoded second feature matrix is ​​concatenated with the first texture feature and bilinear upsampling is performed to obtain the fifth image.

6. The method according to claim 1, characterized in that, The training data for the first model includes an eighth image and the second texture features of the eighth image. The eighth image is obtained by preprocessing a ninth image, which is an image sent from the cloud computer server to the cloud computer terminal.

7. A data processing device, wherein the data processing device is a cloud computer terminal, characterized in that, include: The first receiving unit is used to receive the first texture feature of the first image sent by the cloud computer server; The second receiving unit is used to receive the second image sent by the cloud computer server and decode the second image to obtain a third image; wherein the second image is an image obtained by encoding the first image; The first processing unit is configured to process the first texture feature and the third image using a first model to obtain a fourth image; wherein, the third image is super-division processed using a first network of the first model to obtain a fifth image, and the fifth image and the first texture feature are fused using a second network of the first model to obtain the fourth image; A rendering unit is used to render the fourth image and display the rendered fourth image. The second network includes a third convolutional layer. The first processing unit performs fusion processing on the fifth image and the first texture features through the second network of the first model, including: The fifth image is magnified to obtain the magnified fifth image; The magnified fifth image is subjected to texture synthesis and texture reconstruction based on the first texture feature to obtain a sixth image. The texture synthesis of the magnified fifth image based on the first texture feature includes: mapping the first texture feature onto the magnified fifth image; and / or, synthesizing the first texture feature into the magnified fifth image. The sixth image is processed by the third convolutional layer to obtain the seventh image; The seventh image is upsampled to obtain the fourth image.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images