Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium, and relates to the application of artificial intelligence technology in the field of computer vision technology. The method comprises the following steps: receiving a target game image; performing enhancement processing on the target game image through a target neural network model, wherein the target neural network model is obtained by optimizing shared parameters and target unique parameters, the shared parameters are jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to a neural network model associated with the target neural network model, and the target unique parameters are optimized based on sample data corresponding to the target neural network model; obtaining the target game image enhanced by the target neural network model; and displaying the target game image enhanced by the target neural network model. The method can improve the display effect of the game image in the cloud game scene when the game image is displayed, and improve the user experience quality.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing. It enables thin clients with relatively limited graphics processing and data processing capabilities to run high-quality games. In cloud gaming, the game does not reside on the player's terminal but runs on a cloud gaming server. The server renders the game scene as a video and audio stream, which is then transmitted to the player's terminal via the network. The player's terminal does not need powerful graphics processing and data processing capabilities; it only needs basic streaming media playback capabilities and the ability to receive player input commands and send them to the cloud gaming server.

[0003] In cloud gaming scenarios, a high quality of experience (QoE) is often required. Currently, to improve the quality of experience, the common practice is to optimize game images from the perspective of streaming media quality, such as image latency compensation, transmission frame rate adjustment, and video stream encoding adjustment. However, these methods of optimizing game images from the perspective of streaming media quality can only reduce the impact of external factors on game images, and the actual display quality still cannot meet the needs of users. Summary of the Invention

[0004] In view of the above problems, embodiments of this application propose an image processing method, apparatus, electronic device, and storage medium to improve the above problems.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising: receiving a target game image; enhancing the target game image through a target neural network model, wherein the target neural network model is obtained by optimizing shared parameters and target-specific parameters, the shared parameters being jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to a neural network model associated with the target neural network model, and the target-specific parameters being optimized based on sample data corresponding to the target neural network model.

[0006] Secondly, embodiments of this application provide an image processing method, the method comprising: obtaining a target image rendering strategy corresponding to a target client; using the target image rendering strategy to render game data to generate a rendered game image; sending the rendered game image to the target client so that the target client obtains a target game image corresponding to the rendered game image, and obtaining an enhanced target game image corresponding to the target game image through a target neural network model, wherein the target neural network model is obtained by optimizing shared parameters and target-specific parameters, the shared parameters being jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to a neural network model associated with the target neural network model, and the target-specific parameters being optimized based on sample data corresponding to the target neural network model.

[0007] Thirdly, embodiments of this application provide an image processing apparatus, comprising: an acquisition module, an extraction module, a target event type determination module, and a target risk value calculation module. The acquisition module includes a target game image receiving module for receiving a target game image; a target game image enhancement module for enhancing the target game image using a target neural network model, wherein the target neural network model is obtained by optimizing shared parameters and target-specific parameters, the shared parameters being jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to neural network models associated with the target neural network model, and the target-specific parameters being optimized based on sample data corresponding to the target neural network model. An enhanced target game image acquisition module is used to acquire the enhanced target game image output by the target network model. A display module is used to display the enhanced target game image.

[0008] Optionally, the device further includes a degradation detection module and a restoration module. The degradation detection module is used to perform degradation detection on the target game image and obtain the degradation detection result. The restoration module is used to call the image restoration algorithm corresponding to the degradation detection result to perform image restoration processing on the target game image and obtain the restored target game image.

[0009] Optionally, the target game image enhancement module is also used to enhance the recovered target game image through a target neural network model.

[0010] Optionally, the degradation detection result includes the predicted degradation type and the probability corresponding to the predicted degradation type. Accordingly, the restoration module is also used to determine the predicted degradation type with a probability greater than the target probability threshold as the target degradation type; and to call the image restoration algorithm corresponding to the target degradation type to perform image restoration processing on the target game image to obtain the restored target game image.

[0011] Optionally, the target neural network model includes an attention subnetwork and an enhancement subnetwork. Accordingly, the target game image enhancement module is also used to determine the enhancement level of each image region in the target game image through the attention subnetwork using the attention weight matrix; and to perform enhancement processing on the target game image based on the enhancement level of each image region in the target game image and the enhancement subnetwork. Accordingly, the enhanced target game image acquisition module is also used to acquire the enhanced target game image output by the enhancement subnetwork.

[0012] Optionally, the device further includes a score detection module and a target score sending module. The score detection module is used to perform score detection on the target game image to obtain the target score corresponding to the target game image. The target score sending module is used to send the target score to the cloud gaming server, so that the cloud gaming server can determine the target image rendering strategy for the game data based on the target score.

[0013] Optionally, the rating detection module is also used to perform aesthetic dimension rating detection on the target game image to obtain the rating of the target game image under each aesthetic dimension; perform degradation detection on the target game image to obtain the predicted degradation type of the target game image and the probability of each predicted degradation type; based on the rating of the target game image under each aesthetic dimension and the target weight matrix, obtain the score of the target game image under the predicted degradation type; and based on the score of the target game image under the predicted degradation type and the probability of each predicted degradation type, obtain the target rating of the target game image.

[0014] Optionally, the device further includes: a sample dataset acquisition module, a first transfer module, a second transfer module, and a third transfer module. The sample dataset acquisition module is used to acquire a first sample dataset consisting of multiple degraded first game images carrying degradation type labels, a second sample dataset consisting of multiple second game images carrying subjective rating labels, multiple degraded second game images carrying degradation type labels, and multiple enhanced second game images, a third sample dataset consisting of multiple first natural images carrying subjective rating labels, and a fourth sample dataset consisting of multiple second natural images and multiple enhanced second natural images. The first transfer module is used to perform transfer learning on a first initial neural network model and a second initial neural network model using the first, second, and third sample datasets to obtain an optimized second intermediate neural network model, the optimized second intermediate neural network model including shared parameters. The second transfer module is used to transfer knowledge to the initial target neural network model through the shared parameters to obtain a knowledge-transferred initial target neural network model. The third transfer module is used to perform transfer learning on the initial target neural network model after knowledge transfer and the optimized second intermediate neural network model using the second and fourth sample data sets to obtain the target neural network model.

[0015] Optionally, the device further includes a first training module, a second training module, a third training module, a parameter update module, a fourth training module, and a fifth training module. The first training module is used to train a first initial neural network model using a first sample data set to obtain a first intermediate neural network model, which includes shared parameters and first parameters. The second training module is used to update a second initial neural network model using the shared parameters and to train the updated second initial neural network model using a third sample data set to obtain a second intermediate neural network model, which includes the shared parameters updated in the first iteration and second parameters. The third training module is used to perform a first update on the first intermediate neural network model using the shared parameters updated in the first iteration, and to jointly train the first and second intermediate neural network models using the second sample data set to obtain an optimized second intermediate neural network model, which includes the shared parameters updated in the second iteration and the second parameters updated in the first iteration. The parameter update module is used to update the target initial neural network model using the shared parameters updated in the second iteration to obtain the initial target neural network model after knowledge transfer. The fourth training module uses the fourth sample dataset to train the initial target neural network model after knowledge transfer, obtaining the target intermediate neural network model. This target intermediate neural network model includes the shared parameters updated in the third iteration and the third parameter. The fifth training module updates the optimized second intermediate neural network model using the shared parameters updated in the third iteration, and then uses the second sample dataset to jointly train the updated and optimized second intermediate neural network model and the target intermediate neural network model, obtaining the target neural network model. This target neural network model includes the shared parameters updated in the fourth iteration and the third parameter updated in the first iteration.

[0016] Optionally, the first natural image carrying a subjective rating label includes at least one of a first natural image carrying an image quality rating, a first natural image carrying an overall image aesthetic rating, or a first natural image carrying a multi-dimensional aesthetic rating label.

[0017] Fourthly, embodiments of this application provide an image processing apparatus, comprising: a target image rendering strategy acquisition module, a rendering module, a game image sending module, a target score receiving module, and a target image rendering strategy updating module. The target image rendering strategy acquisition module is used to acquire a target image rendering strategy corresponding to a target client. The rendering module is used to render game data using the target image rendering strategy to generate a rendered game image. The game image sending module is used to send the rendered game image to the target client, so that the target client obtains a target game image corresponding to the rendered game image, and obtains an enhanced target game image corresponding to the target game image through a target neural network model. The target neural network model is obtained by optimizing shared parameters and target-specific parameters. The shared parameters are jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to a neural network model associated with the target neural network model. The target-specific parameters are optimized based on sample data corresponding to the target neural network model. The target score receiving module is used to receive a target score sent by the target client, wherein the target score is obtained by the target client performing a score detection on the target game image. The target image rendering strategy updating module is used to update the target image rendering strategy according to the target score.

[0018] Optionally, the device further includes a detection module, a first target image rendering strategy determination module, and a second target image rendering strategy determination module. The detection module is used to detect whether a historical image rendering strategy corresponding to the target client exists when a connection is established with the target client. The first target image rendering strategy determination module is used to use the historical image rendering strategy as the target image rendering strategy if a historical image rendering strategy exists. The second target image rendering strategy determination module is used to select an initial image rendering strategy as the target image rendering strategy if no historical image rendering strategy exists.

[0019] Optionally, the device further includes: a sample dataset acquisition module, a first transfer module, a second transfer module, and a third transfer module. The sample dataset acquisition module is used to acquire a first sample dataset consisting of multiple degraded first game images carrying degradation type labels, a second sample dataset consisting of multiple second game images carrying subjective rating labels, multiple degraded second game images carrying degradation type labels, and multiple enhanced second game images, a third sample dataset consisting of multiple first natural images carrying subjective rating labels, and a fourth sample dataset consisting of multiple second natural images and multiple enhanced second natural images. The first transfer module is used to perform transfer learning on a first initial neural network model and a second initial neural network model using the first, second, and third sample datasets to obtain an optimized second intermediate neural network model, the optimized second intermediate neural network model including shared parameters. The second transfer module is used to transfer knowledge to the initial target neural network model through the shared parameters to obtain a knowledge-transferred initial target neural network model. The third transfer module is used to perform transfer learning on the initial target neural network model after knowledge transfer and the optimized second intermediate neural network model using the second and fourth sample data sets to obtain the target neural network model.

[0020] In one implementation, the device further includes a first training module, a second training module, a third training module, a parameter update module, a fourth training module, and a fifth training module. The first training module is used to train a first initial neural network model using a first sample data set to obtain a first intermediate neural network model, which includes shared parameters and first parameters. The second training module is used to update a second initial neural network model using the shared parameters and to train the updated second initial neural network model using a third sample data set to obtain a second intermediate neural network model, which includes the shared parameters updated in the first iteration and second parameters. The third training module is used to perform a first update on the first intermediate neural network model using the shared parameters updated in the first iteration, and to jointly train the first and second intermediate neural network models using the second sample data set to obtain an optimized second intermediate neural network model, which includes the shared parameters updated in the second iteration and the second parameters updated in the first iteration. The parameter update module is used to update the target initial neural network model using the shared parameters updated in the second iteration to obtain an initial target neural network model after knowledge transfer. The fourth training module uses the fourth sample dataset to train the initial target neural network model after knowledge transfer, obtaining the target intermediate neural network model. This target intermediate neural network model includes the shared parameters updated in the third iteration and the third parameter. The fifth training module updates the optimized second intermediate neural network model using the shared parameters updated in the third iteration, and then uses the second sample dataset to jointly train the updated and optimized second intermediate neural network model and the target intermediate neural network model, obtaining the target neural network model. This target neural network model includes the shared parameters updated in the fourth iteration and the third parameter updated in the first iteration.

[0021] Fifthly, embodiments of this application provide an electronic device, including a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above-described method.

[0022] Sixthly, embodiments of this application provide a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a processor.

[0023] In a seventh aspect, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described above.

[0024] This application provides an image processing method, apparatus, electronic device, and storage medium that can directly enhance and display game images on a client-side through a target neural network model. Because the enhancement is direct, it improves the user's subjective visual experience of the displayed game image, making it more in line with human visual aesthetic preferences. This enhances the display effect of the game image and improves the user experience quality in cloud gaming scenarios. Furthermore, since the target neural network model is obtained by optimizing shared parameters and target-specific parameters, and the shared parameters are jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to neural network models associated with the target neural network model, while the target-specific parameters are optimized based on sample data corresponding to the target neural network model, the target neural network can learn information from different data domains, improving the generalization ability of the target neural network model. Simultaneously, it reduces the difficulty of obtaining sample data. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram of the application environment of a cloud gaming system proposed in an embodiment of this application is shown.

[0027] Figure 2 This illustration shows a schematic diagram of image processing between a cloud gaming server and a game client according to an embodiment of this application.

[0028] Figure 3 A schematic diagram of the overall structure of an image processing method proposed in an embodiment of this application is shown.

[0029] Figure 4 A flowchart of an image processing method proposed in an embodiment of this application is shown.

[0030] Figure 5A flowchart of another image processing method proposed in an embodiment of this application is shown.

[0031] Figure 6 This paper shows a comparison diagram of a target game image and an enhanced target game image according to an embodiment of this application.

[0032] Figure 7 This illustration shows a target control selection diagram in the settings interface of a cloud game according to an embodiment of this application.

[0033] Figure 8 A flowchart of another image processing method proposed in an embodiment of this application is shown.

[0034] Figure 9 It shows Figure 8 The flowchart illustrates one implementation of S230 in an image processing method proposed in the illustrated embodiment.

[0035] Figure 10 A flowchart of another image processing method proposed in an embodiment of this application is shown.

[0036] Figure 11 A flowchart of another image processing method proposed in an embodiment of this application is shown.

[0037] Figure 12 A flowchart of another image processing method proposed in an embodiment of this application is shown.

[0038] Figure 13 A flowchart illustrating the training process of a multi-task learning model proposed in an embodiment of this application is shown.

[0039] Figure 14 It shows Figure 13 The flowchart illustrates one implementation of S602 in an image processing method proposed in the illustrated embodiment.

[0040] Figure 15 It shows Figure 13 The flowchart illustrates one implementation of S604 in an image processing method proposed in the illustrated embodiment.

[0041] Figure 16 A schematic diagram of an exemplary image processing flow proposed in an embodiment of this application is shown.

[0042] Figure 17 A block diagram of an image processing apparatus according to an embodiment of this application is shown.

[0043] Figure 18 A block diagram of another image processing apparatus proposed in an embodiment of this application is shown.

[0044] Figure 19 A structural block diagram of an electronic device for performing an image processing method according to an embodiment of this application is shown. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0046] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0048] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0049] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0050] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0051] With the development of machine learning technology, it has been widely researched and applied in various fields. The technical solution provided in this application relates to the application of machine learning technology in the field of computer vision technology. Specifically, it relates to an image processing method.

[0052] Before providing a further detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application are explained. The nouns and terms involved in the embodiments of this application shall be interpreted as follows:

[0053] 1) CG images: Computer Graphics, which is the general term for all graphics drawn using computer software. Internationally, the field of visual design and production using computer technology is commonly referred to as CG.

[0054] 2) Natural Image: Natural images are different from randomly generated noisy images, distorted images, and a large part of computer-generated images. They refer to clear images that can be observed or captured by the human eye or a camera in nature.

[0055] 3) Quality of Experience (QoE): This is an indicator that measures a customer's level of enjoyment or annoyance with a service (e.g., web browsing, telephone, video broadcasting). QoE focuses on the overall service experience; it is a holistic concept, similar to the field of user experience.

[0056] 4) Transfer Learning: This is a machine learning method that transfers knowledge from one domain (i.e., the source domain) to another domain (i.e., the target domain) to achieve better learning results in the target domain.

[0057] 5) Attention Mechanism: Essentially, it is a mechanism that uses a set of weight coefficients learned autonomously by the network to emphasize the areas of interest to the user in a "dynamically weighted" manner, while suppressing irrelevant background areas.

[0058] To address the aforementioned problems in cloud gaming scenarios, the inventors have proposed the image processing method, apparatus, electronic device, and storage medium provided in this application. These methods enable direct enhancement and display of game images on the client side via a target neural network model. Because the enhancement is direct, it improves the user's subjective visual experience of the displayed game image, making it more aligned with human visual aesthetic preferences. This enhances the display effect of the game image and improves the user experience quality in cloud gaming scenarios. Furthermore, since the target neural network model is obtained by optimizing shared parameters and target-specific parameters, and the shared parameters are jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to neural network models associated with the target neural network model, while the target-specific parameters are optimized based on sample data corresponding to the target neural network model, the target neural network can learn information from different data domains, improving the generalization ability of the target neural network model. Simultaneously, it reduces the difficulty of obtaining sample data.

[0059] Before providing a more detailed description of the embodiments of this application, an application environment related to the embodiments of this application will be introduced.

[0060] like Figure 1 As shown, Figure 1 The diagram shown illustrates the application environment of the cloud gaming system described in this application embodiment. Figure 1 As shown, to support the operation of a cloud game, the cloud gaming system includes a game client 110 and a cloud gaming server 120. The game client 110 and the cloud gaming server 120 can be connected directly or indirectly via wired or wireless communication, which is not limited herein. The type of cloud game can be a multiplayer online game. In this case, the cloud gaming server 120 can connect to multiple game clients 110 simultaneously; that is, there can be multiple game clients 110. Figure 1 We will use only 3 examples as examples.

[0061] It should be noted that the cloud gaming server 120 can be understood as a server for deploying or installing cloud games. It can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0062] The electronic device hosting the game client 110 can be a smartphone, tablet, laptop, desktop computer, smart speaker, wearable device, smart robot, in-vehicle terminal, etc., but is not limited to these. In this embodiment, the application corresponding to the game client can be any application capable of providing cloud gaming to users. Users can start cloud gaming through the application corresponding to the game client, and the application game client can include one or more cloud games. Optionally, the aforementioned application can be a browser displaying the application platform in web page form, or it can be a client of the application platform.

[0063] The following is combined Figure 1 The cloud gaming system shown illustrates the operation process of cloud gaming according to embodiments of this application.

[0064] After creating the cloud game image file, it can be deployed or installed on the cloud game server, allowing the server to launch the corresponding cloud game by loading the image file. When the cloud game server receives a connection request from a game client, it can establish a connection with the client. Once the connection is successfully established, the cloud game server can run the cloud game.

[0065] like Figure 2As shown, after cloud gaming is started, during its operation, the game application on the cloud gaming server can call the rendering module within the server to render the game data based on the target rendering strategy, thus obtaining the game screen / image in real time. Secondly, the game application on the cloud gaming server can also call the encoding module to perform image compression encoding on the rendered game screen according to video encoding standards (such as H.264, H.265, etc.) to save network bandwidth used in subsequent image transmission. Then, the encoded data (i.e., compressed image) obtained from the image compression encoding can be transmitted to the game client via a video stream over the network. Correspondingly, after receiving the encoded data, the game client can perform decoding operations on the encoded data through the decoding module to restore the game screen. At this point, the game client can receive the target game image. After receiving the target game image, the game client can enhance the target game image using a target neural network model obtained through transfer learning of sample data, obtaining an enhanced target game image. Finally, the game client can display the enhanced game image.

[0066] In the aforementioned cloud gaming process, a target neural network model is used to process game images. This target neural network model is obtained by optimizing shared parameters and target-specific parameters. The shared parameters are jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to neural network models associated with the target neural network model. The target-specific parameters are also optimized based on sample data corresponding to the target neural network model. Therefore, the system also includes electronic devices for training the target neural network model. These devices can be servers or clients. Optionally, the server used to train the target neural network model can be the aforementioned cloud gaming server, or it can be a dedicated server for training neural network models, distinct from cloud gaming servers.

[0067] Based on the above, the following will be combined with... Figure 3 The overall structure of the image processing method according to the embodiments of this application will be described.

[0068] like Figure 3 As shown, this application mainly includes four parts: sample data collection, feature processing, model application, and interaction between the game client and the cloud game server.

[0069] First, in the data collection phase, four sample datasets need to be collected. The first sample dataset includes a degraded first game image with a degradation type label; the second sample dataset includes a degraded second game image with a degradation type label and an enhanced second game image; the third sample dataset includes a first natural image with a subjective rating label; and the fourth sample dataset includes a second natural image and an enhanced second natural image as sample data. The feature processing and model training phases can be executed either on the cloud gaming server or on the game client.

[0070] Secondly, in the feature processing stage, a multi-task neural network model is obtained by performing transfer learning and data feature extraction on the aforementioned sample data. This model includes a first neural network model for degradation type detection, a second neural network model for score detection, and a target neural network model for game image enhancement. The feature processing and model training stages can be executed either on a cloud gaming server or on the game client.

[0071] Then, in the model application stage, the game client calls the multi-task neural network model to perform image enhancement processing on the game image, obtain the enhanced game image, and performs score detection to obtain the target score corresponding to the game image.

[0072] Finally, during the interaction between the game client and the cloud gaming server, the game client sends the target score to the cloud gaming server, establishing an information feedback mechanism that allows the server to adjust its resources.

[0073] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0074] Please see Figure 4 , Figure 4 The diagram shown is a flowchart of an image processing method according to an embodiment of this application. This method can be applied to a client (e.g., Figure 1 The method includes: (in the game client)

[0075] S110, receives the target game image.

[0076] In this embodiment, the target game image refers to the game image obtained by the game client decoding the received compressed and encoded data (compressed image). As described above, during cloud gaming operation, the cloud gaming server can render game data based on a target rendering strategy to obtain a rendered game image. Then, it performs image compression encoding on the rendered game image and transmits the compressed and encoded data (i.e., the compressed image) to the game client via a video stream through the network environment connecting the cloud gaming server and the game client. After receiving the encoded data, the game client performs decoding to obtain the game image, which is the target game image.

[0077] S120 enhances the target game image using a target neural network model.

[0078] The target neural network model is obtained by optimizing shared parameters and target-specific parameters. The shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model. The target-specific parameters are optimized based on the sample data corresponding to the target neural network model.

[0079] Understandably, cloud gaming presents unique challenges. While there are numerous game images, those with subjective human rating labels are extremely scarce. This results in limited supervised data for image enhancement tasks targeting CG content. Directly using game images with subjective rating labels for deep learning would require significant human and material resources for manual annotation, increasing the cost of enhancing the target game images. Therefore, a possible solution is to combine a limited number of game images with subjective rating labels with other readily available sample data. By sharing network parameters, a target neural network model can be obtained, simplifying the sample acquisition process for the target neural network used to enhance game images. This involves training the neural network model by optimizing both shared and unique parameters.

[0080] In this embodiment, the neural network model associated with the target neural network model can be understood as a model that shares parameters with the target neural network model during the training process of the target neural network model.

[0081] Target-specific parameters can be understood as parameters unique to the target neural network that need to be optimized. Shared parameters can be understood as parameters shared between the target neural network model and related neural network models.

[0082] It is understood that in this embodiment, the target neural network model needs to continuously optimize the shared parameters and target-specific parameters during the training process to obtain the trained target neural network model.

[0083] As one implementation method, the shared parameters in this embodiment can be jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model. That is, the target neural network model can learn the sample data features of the associated neural network model during the training process, that is, learn information from different data domains, thereby improving the generalization ability of the target neural network model.

[0084] In one implementation, the sample data used in optimizing shared parameters and target-specific parameters may include a degraded first game image carrying a degradation type label, a second game image carrying a subjective rating label, a degraded second game image carrying a degradation type label, an enhanced second game image, a first natural image carrying a subjective rating label, a second natural image, and an enhanced second natural image.

[0085] The first game image can be a large amount of CG content game image data without obvious degradation, collected by game enthusiasts or legitimate web crawlers, without the need for manual annotation of subjective rating tags. In some implementations, after obtaining the first game image, it can be subjected to degradation processing to obtain a game image carrying a degradation type tag.

[0086] In some implementations, the second game image can be a large amount of CG content game image data without significant degradation, collected by game enthusiasts or legitimate web crawlers. In some implementations, after obtaining the second game image, it can be manually and subjectively rated and labeled to obtain a second game image carrying a subjective rating label. Furthermore, the second game image can be degraded to obtain a second game image carrying a degradation type label. Additionally, the second game image can be processed using an enhancement algorithm to obtain an enhanced second game image.

[0087] In some implementations, the first natural image carrying a subjective rating label includes at least one of a first natural image carrying an image quality rating, a first natural image carrying an overall aesthetic rating, or a first natural image carrying a multi-dimensional aesthetic rating label. For example, the first natural image carrying the subjective rating label may consist entirely of first natural images carrying image quality ratings, entirely of first natural images carrying overall aesthetic ratings, entirely of first natural images carrying multi-dimensional aesthetic rating labels, or partially of first natural images carrying image quality ratings and partially of first natural images carrying overall aesthetic ratings, or partially of first natural images carrying image quality ratings and partially of first natural images carrying multi-dimensional aesthetic rating labels. The first natural image carrying the image quality rating can be collected through legitimate internet web crawlers or directly obtained from existing natural image databases, such as the existing natural image quality assessment databases TID2013 or TID2008. The first natural image carrying multi-dimensional aesthetic rating labels can be collected through legitimate internet web crawlers or directly obtained from existing image aesthetic evaluation databases, such as the existing image aesthetic evaluation dataset AVA (aesthetic visual analysis).

[0088] In some implementations, the second natural image can be collected through legitimate web crawlers or directly obtained from existing databases of images to be enhanced, such as the MIT-Adobe Five-K dataset, which contains a large number of images to be enhanced. MIT-Adobe Five-K is a database frequently used by researchers in image enhancement and retouching. After obtaining the second natural image, it can be processed using enhancement algorithms to obtain the enhanced second natural image.

[0089] It should be noted that the first game image and the second game image can be the same image or different images, and the first natural image and the second natural image can be the same image or different images.

[0090] In one implementation, enhancing the second game image can involve applying an HDR+ (High Dynamic Range+) algorithm and a detail enhancement algorithm sequentially to the second game image. Similarly, enhancing the second natural image can also involve applying an HDR+ (High Dynamic Range+) algorithm and a detail enhancement algorithm sequentially to the second natural image. Optionally, the detail enhancement algorithm can be a local laplacian filter algorithm.

[0091] Specifically, enhancing the target game image using a target neural network model refers to using this model to enhance the image in a way that aligns with human visual aesthetics. It's understandable that, in some cases, due to network conditions or the game client's hardware limitations, the game image received and decoded by the game client may differ from the image rendered by the cloud gaming server. Therefore, to ensure that the game image seen by the user on the client side better conforms to human visual aesthetic preferences, the client can invoke a target neural network model to enhance the target game image.

[0092] Because the sample data optimized for shared parameters and target-specific parameters includes a degraded first game image with a degradation type label, a second game image with a subjective rating label, a degraded second game image with a degradation type label, an enhanced second game image, a first natural image with a subjective rating label, a second natural image, and an enhanced second natural image, the target neural network model can learn (transfer) information from different data domains during training, thereby improving the generalization ability of the target neural network and enabling the target neural network to further enhance the target game image in a direction that conforms to human visual aesthetic preferences.

[0093] It should be noted that the target neural network model used by the game client in this embodiment of the application can be deployed on the game client after the server has completed training, or it can be trained and deployed on the game client.

[0094] S130: Obtain the enhanced target game image output by the target network model.

[0095] Understandably, after the game client enhances the target game image by calling the target neural network model, the target neural network model can output a processing result, namely, an enhanced target game image, which is more in line with human visual aesthetic preferences than the original target game image.

[0096] S140 displays the enhanced target game image.

[0097] Furthermore, the game client is able to display the enhanced game graphics.

[0098] In some implementations, such as Figure 5 As shown, after receiving the target game image, the game client can also execute steps S150-S180.

[0099] S150, perform score detection on the target game image to obtain the target score corresponding to the target game image.

[0100] Rating detection refers to the rating of a target game image based on human visual aesthetic preferences. A higher target rating indicates that the target game image better aligns with human visual aesthetic preferences, resulting in a higher quality user experience.

[0101] In some implementations, a score detection of the target game image can directly yield the target score corresponding to the target game image.

[0102] Optionally, a neural network model can be used to detect the rating of the target game image. This neural network model, used to detect the rating of the target game image and directly obtain the target rating, can be a first rating detection model. The first rating detection model can be obtained by optimizing shared parameters and first unique parameters. The shared parameters are jointly optimized based on the sample data corresponding to the first rating detection model and the sample data corresponding to the neural network model associated with the first rating detection model. The first unique parameters are optimized based on the sample data corresponding to the first rating detection model. Here, the first unique parameters refer to parameters specific to the first rating detection model that need to be optimized. In this case, the sample data used by the first rating detection model can include a degraded first game image carrying a degradation type label, a second game image carrying a subjective rating label, a degraded second game image carrying a degradation type label, an enhanced second game image, a first natural image carrying a subjective rating label, a second natural image, and an enhanced second natural image. The first natural image carrying a subjective rating label includes a first natural image carrying an image quality rating.

[0103] Optionally, in some cases, where the cost of manual annotation is not considered, and multiple sample game images carrying human visual aesthetic preference rating labels can be easily obtained, the initial rating detection model can be trained directly from these sample game images. During training, each sample game image carrying a human visual aesthetic preference rating label can be input into the initial rating detection model. The initial rating detection model can output a predicted rating for that game image and calculate the loss between the predicted rating and the rating label information of the game image. The model parameters of the initial rating detection model are then adjusted based on the loss to obtain the final rating detection model.

[0104] It is understandable that first natural images carrying image quality scores and first natural images carrying aesthetic dimension score labels are easier to obtain than sample game images carrying human visual aesthetic preference score labels, which can simplify the difficulty of obtaining sample data.

[0105] In other implementations, rating detection of the target game image does not directly yield the target rating. In such cases, a neural network model can be used to perform aesthetic dimension rating detection on the target game image, obtaining the rating corresponding to each aesthetic dimension. A neural network model can also be used to perform degradation detection on the target game image, obtaining the predicted degradation type and the probability of each predicted degradation type. Then, based on the rating of the target game image in each aesthetic dimension and the target weight matrix, the score of the target game image under the predicted degradation type is obtained. Finally, based on the score of the target game image under the predicted degradation type and the probability of each predicted degradation type, the target rating of the target game image is obtained.

[0106] Optionally, the target score corresponding to the target game image can be obtained based on the score of the target game image under the predicted degradation type and the probability corresponding to each predicted degradation type by weighted summation of the score of the target game image under the predicted degradation type and the probability corresponding to each predicted degradation type, and the weighted summation result can be used as the target score corresponding to the target game image.

[0107] The neural network model used to perform degradation detection on the target game image, obtaining the predicted degradation type and the probability of each predicted degradation type, can be a first neural network model. The neural network model used to perform aesthetic dimension rating detection on the target game image, obtaining the rating of the target game image under each aesthetic dimension, and then, based on the rating of the target game image under each aesthetic dimension and the target weight matrix, obtaining the score of the target game image under the predicted degradation type, and the neural network model used to obtain the target rating of the target game image based on the score of the target game image under the predicted degradation type and the probability of each predicted degradation type, can be a second rating detection model. In the above process, the second rating detection model uses the output of the first neural network model to obtain the target rating of the target game image.

[0108] The first neural network model is obtained by optimizing shared parameters and second specific parameters. The second scoring detection model is obtained by optimizing shared parameters and third specific parameters. The shared parameters are jointly optimized based on sample data corresponding to the first and second neural network models, as well as sample data corresponding to neural network models associated with the first and second neural network models. The second specific parameters are optimized based on sample data corresponding to the first neural network model, and the third specific parameters are optimized based on sample data corresponding to the second scoring detection model. The second specific parameters refer to parameters specific to the first neural network model that require optimization, and the third specific parameters refer to parameters specific to the second scoring detection model that require optimization.

[0109] In one implementation, the sample data used in optimizing the shared parameters, the second specific parameters, and the third specific parameters may include transfer learning of sample data from a degraded first game image carrying a degradation type label, a second game image carrying a subjective rating label, an enhanced second game image carrying a degradation type label, a first natural image carrying an image quality rating, and a first natural image, a second natural image, and an enhanced second natural image carrying a multi-dimensional aesthetic rating label. The first natural image carrying a subjective rating label includes at least one of a first natural image carrying an overall image aesthetic rating or a first natural image carrying a multi-dimensional aesthetic rating label.

[0110] It is understandable that first natural images carrying image quality scores and first natural images carrying multi-dimensional aesthetic score labels are easier to obtain than sample game images carrying human visual aesthetic preference score labels, which can simplify the difficulty of obtaining sample data.

[0111] like Figure 6As shown, the image illustrates the target game image A1 and its enhanced version A2. The enhanced version is obtained by using a target neural network model to enhance the target game image. Furthermore, images A1 and A2 are labeled with corresponding scores, obtained by performing score detection on both images. That is, in addition to performing score detection on image A1, score detection can also be performed on image A2. These scores can reflect human visual aesthetic preferences. Figure 6 As can be seen, the enhanced target game image A2 has more detailed texture and clearer details than the target game image A1. The enhanced target game image A2 looks more beautiful and is more in line with human visual aesthetic preferences. In addition, the score of image A2 is higher than that of image A1. Therefore, it can be seen that the enhanced target game image is more in line with human visual aesthetic preferences. The image enhancement method of this embodiment can improve the display effect of game images and enhance the user experience quality in cloud gaming scenarios.

[0112] S160 sends the target score to the cloud gaming server so that the cloud gaming server can determine the target image rendering strategy for the game data based on the target score.

[0113] After obtaining the target score, the game client can send the target score to the cloud gaming server to establish a feedback mechanism for the rendered game image transmitted by the cloud gaming server. This allows the cloud gaming server to determine the target image rendering strategy for the game data based on the target score, and then render the subsequent game data according to the determined target image rendering strategy to obtain the rendered game image.

[0114] In some implementations, considering that the client's invocation of the target neural network model to process the target game image requires memory and processing resources, and that some low-configuration terminal devices may experience insufficient memory or processing resources, it is possible to detect whether the function of enhancing the target game image through the target neural network model is enabled before performing enhancement processing on the target game image. If it is enabled, the step of enhancing the target game image through the target neural network model is executed. If it is disabled, no enhancement processing is performed on the target game image.

[0115] Optionally, in some implementations, users of the game client can decide whether to enable the function of enhancing the target game image through the target neural network model via the target control provided in the settings interface of the cloud gaming client.

[0116] Please see Figure 7The diagram illustrates the target control selection in the settings interface of a cloud gaming system. When the user selects the image settings, the interface displays the "Image Enhancement" setting, which includes "On" and "Off". When the user selects "On", the function of the target neural network model to enhance the target game image is enabled. When the user selects "Off", the function of the target neural network model to enhance the target game image is disabled.

[0117] The image processing method provided in this embodiment can directly enhance and display game images on the client side through a target neural network model. Since it directly enhances the image, the enhanced target game image can improve the user's subjective visual experience of the displayed game screen, making the enhanced target game image more in line with human visual aesthetic preferences. This improves the display effect of the game image and enhances the user experience quality in cloud gaming scenarios. Furthermore, since the target neural network model is obtained by optimizing shared parameters and target-specific parameters, and the shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model, while the target-specific parameters are optimized based on the sample data corresponding to the target neural network model, the target neural network can learn information from different data domains, improve the generalization ability of the target neural network model, and reduce the difficulty of obtaining sample data.

[0118] Please see Figure 8 , Figure 8 The diagram shown is a flowchart of an image processing method according to another embodiment of this application. This method can be applied to a client (e.g., Figure 1 (game client in the middle), the method includes:

[0119] S210, receives the target game image.

[0120] S220, Degradation detection is performed on the target game image to obtain the degradation detection results.

[0121] Degradation detection refers to the detection of the degradation type of a target game image, and the degradation detection result indicates the degradation type of the target game object.

[0122] As can be seen from the foregoing, due to network environment or hardware conditions of the game client, the game image actually received and decoded by the game client may differ from the game image rendered by the cloud gaming server. That is, the game image rendered by the cloud gaming server will degrade after reaching the game client and being decoded. Therefore, in order to further improve the display effect of the enhanced target game image displayed on the client and further improve the user experience quality, degradation detection can be performed on the target game image first to obtain the degradation detection results.

[0123] It is understandable that the target game image decoded by the game client may have one type of degradation compared to the game image rendered by the cloud gaming server, or it may have more than one type of degradation, or it may not have any degradation at all, i.e., no degradation type exists.

[0124] As one implementation method, a neural network model can be used to detect degradation in the target game image.

[0125] Optionally, the neural network model used for degradation detection of the target game image, which can be simply referred to as the degradation detection model, can be obtained by training multiple sample game images and the corresponding degraded game images carrying degradation type labels.

[0126] During training, each degraded game image carrying a degradation type label can be input into the initial degradation detection model. The initial degradation detection model can output the predicted degradation type information for the game image and calculate the loss between the predicted type and the degradation type label information of the game image. The model parameters of the initial degradation detection model are adjusted according to the loss to obtain the degradation detection model.

[0127] Alternatively, the degradation detection model may also be the first neural network model in the foregoing embodiments.

[0128] It is understandable that, in some implementations, after performing degradation detection on the target game image using a degradation detection model, the output of the degradation detection model may only include the degradation type to which the target game object belongs. For example, the degradation type of the target game image may be one of the degradation types such as blur degradation, compression degradation, or noise degradation. In this case, the degradation detection result is the degradation type output by the degradation detection model.

[0129] In other implementations, after performing degradation detection on the target game image using a degradation detection model, the output of the degradation detection model can include the degradation type and the probability corresponding to the degradation type. For example, the output of the degradation detection model can be a probability of 0.5 for blur degradation, a probability of 0.4 for compression degradation, and a probability of 0.1 for noise degradation. In this case, the degradation detection result includes the degradation type and the probability corresponding to the degradation type.

[0130] S230: Call the image restoration algorithm corresponding to the degradation detection result to perform image restoration processing on the target game image and obtain the restored target game image.

[0131] Understandably, since the target game image may be degraded, directly enhancing it might not result in a satisfactory display. Therefore, to improve the display quality of the enhanced target game image and enhance the user experience, after obtaining the degradation detection results, the target game image can be restored first. This involves calling an image restoration algorithm corresponding to the degradation type to obtain an image that is as close as possible to the game image rendered by the server. Then, the restored target game image can be enhanced to improve its display quality.

[0132] Based on the foregoing, degradation detection results can include either only degradation type results or degradation type and the probability corresponding to the degradation type results.

[0133] In some implementations, when the degradation detection results only include this type of degradation, the degradation detection results can directly represent the degradation type. In this case, the image restoration algorithm corresponding to the degradation detection results can be directly called to perform image restoration processing on the target game image to obtain the restored target game image.

[0134] In other implementations, when the degradation detection results include degradation type and the probability corresponding to the degradation type, such as... Figure 9 As shown, calling the image restoration algorithm corresponding to the degradation detection result to perform image restoration processing on the target game image to obtain the restored target game image may include the following steps:

[0135] S231, determine the degradation type whose probability satisfies the target probability threshold as the target degradation type.

[0136] In this embodiment, a target probability threshold for determining the target degradation type can be pre-configured on the game client. Then, the probability corresponding to the degradation type output by the degradation detection model is compared with the target probability threshold. If the probability corresponding to the degradation type is greater than or equal to the target probability threshold, it indicates that the degradation type corresponding to the target game image is likely a degradation type that meets the target probability threshold. Therefore, the degradation type whose probability meets the target probability threshold can be designated as the target degradation type. Optionally, the target probability threshold can be configured based on experience or actual needs; for example, the target probability threshold can be configured to 0.4.

[0137] S232, invoke the image restoration algorithm corresponding to the target degradation type to perform image restoration processing on the target game image, and obtain the restored target game image.

[0138] Since the target degradation type represents the degradation type corresponding to the target game image, the image restoration algorithm corresponding to the target degradation type can be called to perform image restoration processing on the target game image to obtain the restored target game image.

[0139] For example, if the target degradation type is compression degradation and noise degradation, then the recovery algorithm corresponding to compression degradation and the recovery algorithm corresponding to noise degradation can be called to perform image restoration on the target game image to obtain the restored target game image.

[0140] S240 enhances the recovered target game image using a target neural network model.

[0141] The target neural network model is obtained by optimizing shared parameters and target-specific parameters. The shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model. The target-specific parameters are optimized based on the sample data corresponding to the target neural network model.

[0142] In this embodiment of the application, after obtaining the restored target game image, the restored target game image can be enhanced using a target neural network model.

[0143] S250: Obtain the enhanced target game image output by the target network model.

[0144] S260 displays the enhanced target game image.

[0145] The specific implementation methods of steps S240, S250 and S260 can be referred to the specific descriptions of S120~S140, which will not be repeated here.

[0146] The image processing method of this application embodiment, after obtaining the target game image, first performs degradation detection on the target game image, and after obtaining the degradation detection result, restores the target game image according to the degradation detection result, so as to obtain an image that is as close as possible to the game image obtained after server rendering. Then, the restored target game image is enhanced, which can improve the display effect of the enhanced target game image and further improve the user experience quality.

[0147] Furthermore, considering that game images are rendered by different rendering components (such as face and hand rendering components), and that these different rendering components have different basic rendering parameters—for example, the number of basic graphics to be rendered and the attributes of the rendered textures—these components, including specific attributes such as rendering direction and complexity, result in varying rendering difficulties for different regions of the final rendered game image. In this embodiment, an attention mechanism is introduced to perform different levels of enhancement processing on image regions with varying rendering difficulties, thereby improving the efficiency and effectiveness of the enhancement processing. Therefore, in some implementations, please refer to... Figure 10 , Figure 10 The diagram shown is a flowchart of an image processing method according to another embodiment of this application. The target neural network model in this method may include an attention subnetwork and an enhancement subnetwork. This method can be applied to a client (e.g., Figure 1 The method includes: (in the game client)

[0148] S310 receives the target game image.

[0149] S320 uses an attention subnetwork to determine the enhancement level for each image region in the target game image.

[0150] The attention subnetwork is a subnetwork trained based on the attention mechanism. The target game image is input into the attention subnetwork, and the attention subnetwork processes the target game image to obtain the information distribution of features with different importance to the enhancement task in the target game image, that is, to obtain the enhancement degree corresponding to each image region.

[0151] As can be seen from the foregoing, different rendering components have different rendering difficulties. Therefore, we can pre-label the different rendering components in the sample game images with their respective rendering difficulty labels, thereby obtaining the rendering difficulty labels for each sample image and its corresponding image region. Then, we can add an attention mechanism to the target neural network and then add the rendering difficulty labels for each sample image and its corresponding image region to the aforementioned transfer learning process of the target neural network. Finally, we can obtain the attention subnetwork and the enhancement subnetwork.

[0152] S330 enhances the target game image based on the enhancement level of each image region in the target game image and the enhancement sub-network.

[0153] As one implementation method, the enhanced subnetwork can be obtained by optimizing the target neural network model through shared parameters and target-specific parameters. The shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model. The target-specific parameters are optimized based on the sample data corresponding to the target neural network model.

[0154] The enhancement subnetwork can apply different enhancement responses based on the enhancement level of each image region in the target game image. For example, it can apply a larger enhancement response to image regions with higher enhancement levels and a smaller enhancement response to image regions with lower enhancement levels.

[0155] S340: Obtain the enhanced target game image output by the enhanced subnetwork.

[0156] After the target game graphics are enhanced, the client can obtain the enhanced target game image output by the enhanced subnetwork.

[0157] S350 displays the enhanced target game image.

[0158] The image processing method provided in this application, by introducing an attention mechanism, can enhance different image regions with different degrees of enhancement. Under the condition that the processing capability of the game client is certain, it can improve the enhancement processing efficiency and enhancement effect.

[0159] Please see Figure 11 , Figure 11 The diagram shown is a flowchart of an image processing method according to another embodiment of this application. This method can be applied to a server (e.g., Figure 1 (cloud gaming server in China), the method includes:

[0160] S410: Obtain the target image rendering strategy corresponding to the target client.

[0161] The target client can be any game client connected to the cloud gaming server.

[0162] It is understandable that different target clients may use different target image rendering strategies. Therefore, before rendering game data, the target image rendering strategy corresponding to the target client can be obtained first.

[0163] The S420 uses a target image rendering strategy to render game data and generate rendered game images.

[0164] In this embodiment, the target image rendering strategy includes the number of basic graphics to be rendered, rendering texture attributes, or rendering structure for the image to be rendered. After obtaining the target image rendering strategy, parameters such as the number of basic graphics to be rendered, rendering texture attributes, or rendering structure can be adjusted to be consistent with the recorded target image rendering strategy, and the game data can be rendered based on the adjusted parameters to generate the rendered game image.

[0165] S430 sends the rendered game image to the target client so that the target client obtains the target game image corresponding to the rendered game image, and obtains the enhanced target game image corresponding to the target game image through the target neural network model.

[0166] The target neural network model is obtained by optimizing shared parameters and target-specific parameters. The shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model. The target-specific parameters are optimized based on the sample data corresponding to the target neural network model.

[0167] In this embodiment, the cloud gaming server can send the rendered game image to the target game client through the network environment between the server and the target client. In this way, the target game client can receive the target game image corresponding to the rendered game image.

[0168] In some implementations, to save network bandwidth, the cloud gaming server can first compress the rendered game image and then send the compressed game image to the target client. In this case, the target client can first decode the received compressed game image and then obtain the target game image corresponding to the rendered game image.

[0169] After obtaining the target game image corresponding to the rendered game image, the target client can call the deployed target neural network model to enhance the target game image, thereby obtaining the enhanced target game image output by the target neural network model.

[0170] In some implementations, before calling the target neural network model to enhance the target game image, the target client can also perform degradation type detection on the target game image to obtain the corresponding degradation type. Then, it can call the image restoration algorithm corresponding to the degradation detection result to perform image restoration processing on the target game image to obtain the restored target game image. Then, it can call the target neural network model to enhance the restored target game image. In this way, the display effect of the enhanced target game image displayed on the client can be improved, further enhancing the user experience quality.

[0171] S440 receives the target score sent by the target client.

[0172] The target score is obtained by the target client scoring and detecting the target game image.

[0173] After obtaining the target game image, the target client can also perform a rating detection based on human visual aesthetic preferences to obtain the target rating corresponding to the target game image. By performing a rating detection on the target game image, the target rating can be obtained, and then the target rating can be used to determine whether the game image rendered by the cloud gaming server and sent to the target client conforms to human visual aesthetic preferences.

[0174] Once the target client receives the target rating, it can send that rating to the cloud gaming server. Sending only the target rating to the cloud gaming server reduces the amount of data transmitted and saves bandwidth.

[0175] S450 updates the target image rendering strategy based on the target score.

[0176] Since the target score can determine whether the game image rendered by the cloud gaming server conforms to human visual aesthetic preferences after being sent to the target client, the cloud gaming server can update and adjust the target image rendering strategy corresponding to the target client based on the target score.

[0177] Optionally, when the target score is relatively low, such as below a certain score threshold, the cloud gaming server can appropriately increase the rendering and computing resources allocated to the target client (computing resources such as the number of CPUs / GPUs allocated, memory size, etc.). This improves the visual appeal of the image on the server side. Even if the rendered target image degrades upon reaching the target client, resulting in a decrease in visual appeal, the increased rendering resources allocated by the cloud gaming server still improve the target game image to some extent. This allows the target neural network to reduce its boost response, thus conserving processing resources on the target client. Because cloud gaming servers have stronger processing capabilities, increasing the rendering resources allocated to the target client when the target score is relatively low achieves the effect of fully utilizing server computing resources and conserving computing resources on the target client.

[0178] Optionally, considering that the human eye has limited ability to distinguish images when they are displayed on the client, for example, once the visual appeal of a game image reaches a certain level, the human eye can no longer clearly distinguish the differences, the cloud gaming server can appropriately reduce the rendering and computing resources allocated to the target client when the target score is relatively high, such as above a certain score threshold. This can reduce the server load and save server resources while meeting the user's visual appeal requirements for the displayed game images, thereby achieving quality control of the entire cloud gaming system.

[0179] Optionally, the score threshold can be preset based on experience during the game debugging phase.

[0180] The image processing method provided in this embodiment establishes a feedback mechanism between the cloud gaming server and the client. After the rendered game image is sent to the target client, the cloud gaming server can receive the target score for the target game image sent by the target client and update and adjust the target image rendering strategy according to the target score. This achieves the effect of making full use of server computing resources, saving target client computing resources, and reducing server load and saving server resources while meeting the user's visual aesthetic requirements for the displayed game image. It achieves a comprehensive balance between the visual effect of the game image and the rendering complexity of the cloud gaming server.

[0181] Furthermore, since the image rendering strategy has been updated and adjusted, the rendered game images are of different sizes and occupy different bandwidths when sent to different target clients. Therefore, data transmission can be optimized on the server. For example, more bandwidth can be allocated to rendered game images with high bandwidth consumption, while less bandwidth can be allocated to rendered game images with low bandwidth consumption. In this way, compared to using the same bandwidth for transmission, each target client can obtain a better user experience quality without changing the total transmission bandwidth of the server.

[0182] Please see Figure 12 , Figure 12 The diagram shown is a flowchart of an image processing method according to another embodiment of this application. This method can be applied to a server (e.g., Figure 1 The method includes: (cloud gaming server in China)

[0183] S510, when a connection is established with the target client, detects whether there is a historical image rendering strategy corresponding to the target client.

[0184] Historical image rendering strategies can be understood as image rendering strategies that the target client has used in the past.

[0185] Understandably, the target client may not be connecting to the cloud gaming server for the first time. For example, if the target client connected to the cloud gaming server yesterday, played the game, and then logged off, disconnecting from the server, the cloud gaming server can record the image rendering strategy corresponding to the target client. This could involve associating the target client's identifier with its corresponding image rendering strategy and storing this as a historical image rendering strategy. When the target client reconnects to the cloud gaming server, the server can find the historical image rendering strategy based on the target client's identifier and use it as the target image rendering strategy. By associating and storing the target client's identifier with its corresponding image rendering strategy within the cloud gaming server, the server can quickly select the appropriate target image rendering strategy when the target client reconnects, ensuring a high-quality user experience from the outset. Therefore, after establishing a connection with the target client, the cloud gaming server can check for the existence of a historical image rendering strategy corresponding to the target client.

[0186] If the cloud gaming server detects the existence of a historical image rendering strategy, it executes step S521 below; if it detects that no historical image rendering strategy exists, it executes step S522 below.

[0187] As one implementation method, after the target client establishes a connection with the cloud gaming server, the cloud gaming server can first determine whether there is a historical image rendering strategy corresponding to the target client's identifier based on the target client's identifier, which may be a game account, MAC address (Media Access Control Address), or IP address. If a historical image rendering strategy exists, the most recently used historical image rendering strategy is selected as the target image rendering strategy for the target client.

[0188] S521, if a historical image rendering strategy exists, the historical image rendering strategy shall be used as the target image rendering strategy.

[0189] In this embodiment, when a historical image rendering strategy exists, the cloud gaming server uses the historical image rendering strategy as the target image rendering strategy.

[0190] S522, if there is no historical image rendering strategy, select the initial image rendering strategy as the target image rendering strategy.

[0191] The initial image rendering strategy refers to the default image rendering strategy pre-set in the cloud gaming server, which is used when a target client first connects to the cloud gaming server.

[0192] In this embodiment, if there is no historical image rendering strategy, the cloud gaming server will select the initial image rendering strategy as the target image rendering strategy.

[0193] The S530 uses a target image rendering strategy to render game data and generate rendered game images.

[0194] S540 sends the rendered game image to the target client so that the target client obtains the target game image corresponding to the rendered game image, and enhances the target game image through the target neural network model to obtain the enhanced target game image output by the target neural network model.

[0195] The target neural network model is obtained through transfer learning of sample data, which includes a degraded first game image with a degradation type label, a second game image with a subjective rating label, a degraded second game image with a degradation type label, an enhanced second game image, a first natural image with a subjective rating label, a second natural image, and an enhanced second natural image.

[0196] S550 receives the target score sent by the target client, where the target score is obtained by the target client through score detection of the target game image.

[0197] S560 updates the target image rendering strategy based on the target score.

[0198] The specific implementation of steps S530-S560 can be found in the detailed descriptions of S420-S450, and will not be repeated here.

[0199] This application provides an image processing method in which, when a connection is established with a target client, the cloud gaming server detects whether there is a historical image rendering strategy corresponding to the target client, and if there is a historical image rendering strategy, uses the historical image rendering strategy as the target image rendering strategy. This allows the cloud gaming server to quickly select the target image rendering strategy corresponding to the target client when the target game client reconnects to the cloud gaming server, thereby ensuring that the user has a high quality of user experience from the very beginning of cloud gaming.

[0200] Based on the foregoing, the target neural network model for enhancing the target game image, the first neural network model for detecting degradation in the target game image to obtain the predicted degradation type and the probability of each predicted degradation type, and the second scoring neural network model for detecting the aesthetic dimension of the target game image to obtain the corresponding score of the target game image under each aesthetic dimension, and then obtaining the score of the target game image under the predicted degradation type based on the score of the target game image under each aesthetic dimension and the target weight matrix, and obtaining the target score of the target game image based on the score of the target game image under the predicted degradation type and the probability of each predicted degradation type, can be obtained by optimizing the shared parameters and their respective unique parameters. In the optimization process, the sample data used can include the degraded first game image carrying the degradation type label, the second game image carrying the subjective rating label, the degraded second game image carrying the degradation type label, the enhanced second game image, the first natural image carrying the subjective rating label, the second natural image, and the enhanced second natural image. In this scenario, the transfer learning process primarily involves three tasks: the first is degradation type detection, which can be learned using a first initial neural network model; the second is rating detection, which can be learned using a second initial neural network model; and the third is game image enhancement, which can be learned using a target initial neural network model. Since the datasets used for pre-training or joint training of the first, second, and target initial neural network models overlap but are not entirely identical, this embodiment can increase the generalization ability of the neural network models corresponding to each task by sharing parameters and synchronously updating them during training to learn (transfer) information from different data domains. It is understood that, in addition to shared parameters, the three models also have their own unique parameters; that is, the first neural network model includes a first parameter, the second neural network model includes a second parameter, and the target neural network model includes a third parameter.

[0201] In this application embodiment, the first neural network model, the second neural network model, and the target neural network model can share parameters in different ways.

[0202] In one implementation, the first neural network model, the second neural network model, and the target neural network model in this application embodiment can be three separate models, but they share parameters during the training process.

[0203] As another implementation, the first neural network model, the second neural network model, and the target neural network model in the embodiments of this application can also be three sub-models in a multi-task learning model, and the three sub-models share parameters through a network sharing layer.

[0204] The following section primarily uses the first neural network model, the second neural network model, and the target neural network model—three sub-models in a multi-task learning model—to illustrate the training process of these three models through parameter sharing via network sharing layers. Figure 13 As shown, the training process includes steps S601-S604.

[0205] Among them, training processes S601-S604 can be applied to the server (e.g., Figure 1 Cloud gaming servers, or other servers specifically designed for training neural network models, distinct from cloud gaming servers. Figure 1 (Not shown in the image) After the server trains the target neural network model, it can send the target neural network model to the client for deployment, and it can also be applied to the client (e.g., Figure 1 (In the game client), after the client has trained the target neural network model, it can deploy itself. The following explains each step:

[0206] S601, acquire a first sample data set consisting of multiple degraded first game images carrying degradation type labels, a second sample data set consisting of multiple second game images carrying subjective rating labels, multiple degraded second game images carrying degradation type labels, and multiple enhanced second game images, a third sample data set consisting of multiple first natural images carrying subjective rating labels, and a fourth sample data set consisting of multiple second natural images and multiple enhanced second natural images.

[0207] S602, using the first sample data set, the second sample data set, and the third sample data set, transfer learning is performed on the first initial neural network model and the second initial neural network model to obtain an optimized second intermediate neural network model, which includes shared parameters.

[0208] In this embodiment of the application, by utilizing the first sample data set, the second sample data set, and the third sample data set to perform transfer learning on the first initial neural network and the second initial neural network, information between different data domains can be shared / transferred. That is, by using the degradation type detection task to assist the rating detection task, the second intermediate neural network model can obtain good generalization ability even when there are insufficient second game images carrying subjective rating labels.

[0209] In addition, in this embodiment of the application, besides obtaining an optimized second intermediate neural network model, an optimized second intermediate neural network model can also be obtained, and the optimized first intermediate neural network model shares the same shared parameters through the network sharing layer.

[0210] S603 transfers knowledge to the initial target neural network model by sharing parameters, thus obtaining the initial target neural network model after knowledge transfer.

[0211] In this embodiment, after obtaining the shared parameters, knowledge can be transferred to the initial target neural network model using the shared parameters, thereby obtaining the initial target neural network model after knowledge transfer. Similarly, this process is automatically implemented through the network sharing layer.

[0212] Because shared parameters transfer knowledge to the initial target neural network model, the initial target neural network model after knowledge transfer can share / transfer information (feature representation, etc.) learned from different types of data to a certain extent, thus enabling the target neural network model to have better generalization performance.

[0213] S604. Using the second and fourth sample datasets, transfer learning is performed on the initial target neural network model after knowledge transfer and the optimized second intermediate neural network model to obtain the target neural network model.

[0214] In this embodiment, considering that the optimized second intermediate neural network model can already perform good rating detection for game images, the optimized second intermediate neural network model is considered as part of the loss function in the initial target neural network model after knowledge transfer. The rating detection task is used to assist the image enhancement task, so that the final target neural network model can obtain better enhancement effect.

[0215] In addition, in this embodiment, besides obtaining the target neural network model, a first neural network model and a second neural network model can also be obtained. The first neural network model, the second neural network model, and the target neural network model share the same shared parameters through a network sharing layer.

[0216] In the embodiments of this application, such as Figure 14 As shown, step S602 can be achieved through steps S6021-S6023, that is, by using the first sample data set, the second sample data set, and the third sample data set to perform transfer learning on the first initial neural network model and the second initial neural network model to obtain the optimized second intermediate neural network model, including steps S6021-S6023. Each step will be explained below.

[0217] S6021, Using the first sample data set, train the first initial neural network model to obtain the first intermediate neural network model.

[0218] The first intermediate neural network model includes shared parameters and a first parameter.

[0219] The first initial neural network model is a neural network model for degradation type detection. It can be used to classify the degradation type of the first sample data in the first dataset to predict the degradation type corresponding to the first sample data and the probability corresponding to each degradation type. The sum of the probabilities corresponding to each degradation type is 1.

[0220] After predicting the degradation type corresponding to the first sample data and the probability of each degradation type, a loss function can be constructed by combining the degradation type label of the first sample data. As one implementation method, the loss function can be the empirical cross-entropy loss function, specifically expressed as the following formula:

[0221]

[0222] in, This represents the k-th first sample data in the first sample dataset. Let represent the degradation type label corresponding to the k-th first sample data in the first sample dataset, where i represents the i-th degradation type, j represents the number of training samples, W represents the shared parameters of the network's shared layer in the current training phase, and w1 represents the first parameter. This indicates the predicted degradation type corresponding to the first sample data, and the probability corresponding to each degradation type.

[0223] Through the above process, the first initial neural network model can be pre-trained to obtain the first intermediate neural network model.

[0224] S6022, the second initial neural network model is updated using shared parameters, and the updated second initial neural network model is trained using the third sample data set to obtain the second intermediate neural network model.

[0225] The second intermediate neural network model includes the shared parameters after the first update and the second parameters.

[0226] The first neural network model and the second neural network model share parameters through a network sharing layer. Therefore, after obtaining the shared parameters, the second initial neural network model can be updated using the shared parameters, that is, the second initial neural network model uses the parameters of the network sharing layer.

[0227] Based on the foregoing, it is generally easier to obtain first natural images carrying image quality rating labels and first natural images carrying aesthetic dimension rating labels than sample game images carrying human visual aesthetic preference rating labels. For example, first natural images carrying image quality ratings can be obtained directly from the existing natural image quality assessment database TID2013, and first natural images carrying aesthetic dimension rating labels can be obtained directly from the image aesthetic evaluation dataset AVA. Therefore, in some cases, the third sample dataset may include first natural images carrying image quality ratings and first natural images carrying aesthetic dimension rating labels.

[0228] The second initial neural network model is a neural network model used for score detection. Based on the aforementioned third sample dataset, the second initial neural network model can be used to detect the aesthetic dimension of the third sample data in the third dataset to predict the score corresponding to the third sample data under each aesthetic dimension. Then, the scores corresponding to each aesthetic dimension are added together and the average is taken to predict the target score corresponding to the third sample data.

[0229] After predicting the target score corresponding to the third sample data, a loss function can be constructed by combining the subjective rating labels corresponding to the third sample data. As one implementation method, the loss function can be expressed as the following formula:

[0230]

[0231] in, This indicates the prediction of the target score corresponding to the third sample data, specifically for the vector. The vector is obtained by adding all its elements together and taking the mean. This represents the scores corresponding to each degradation type in the third sample data. Specifically, the scores for each aesthetic dimension of the third sample data can be predicted first using the second initial neural network model. Then, by combining the weight matrix provided by the second initial fully connected layer of the neural network, the scores corresponding to each degradation type are obtained. This represents the k-th third sample data in the third sample dataset. w2 represents the subjective rating label corresponding to the kth third sample data in the third sample dataset, specifically the weighted sum of the image quality rating label or the aesthetic dimension rating label, and w2 represents the second parameter.

[0232] S6023, the first intermediate neural network model is updated for the first time using the shared parameters after the first update, and the first intermediate neural network model and the second intermediate neural network model after the first update are jointly trained using the second sample data set to obtain the optimized second intermediate neural network model.

[0233] The optimized second intermediate neural network model includes the shared parameters after the second update and the second parameters after the first update.

[0234] Furthermore, after obtaining the first updated shared parameters, the first intermediate neural network model can be updated using the first updated shared parameters, that is, the first intermediate neural network model uses the first updated shared parameters of the network shared layer.

[0235] Because the size of the third sample dataset is relatively small, the second intermediate neural network model may experience overfitting after pre-training. Considering the need for rating detection of game images, this embodiment employs a multi-task framework, using a degradation type detection task (i.e., the first intermediate neural network model after its initial update) to assist the rating detection task (i.e., the second intermediate neural network model). Joint training and weight fine-tuning are performed on the first intermediate neural network model (including its fully connected layers) and the second intermediate neural network model (including its fully connected layers) on a small-scale second sample dataset. As one implementation method, the loss function in this process can be expressed as the following formula:

[0236]

[0237] in, This represents the k-th unenhanced second sample data in the second sample dataset. This represents the subjective rating label corresponding to the k-th second sample data in the second sample dataset. This represents the rating labels for each aesthetic dimension carried by the k-th second sample data in the second sample dataset. , representing the target score corresponding to the second sample data, specifically the probability of each degradation type corresponding to the second sample data. Scores corresponding to each degradation type in the second sample data The weighted summation, where the probabilities of each degradation type corresponding to the second sample data are... The scores corresponding to each degradation type in the second sample data can be predicted using the first intermediate neural network model. We can first predict the scores for each aesthetic dimension of the second sample data using the second intermediate neural network model, and then combine the weight matrix provided by the fully connected layer of the second intermediate neural network to obtain the scores corresponding to each degradation type. This represents the rating label for the m-th aesthetic dimension corresponding to the second sample data. This represents the predicted score for the m-th aesthetic dimension corresponding to the second sample data. and The hyperparameters are greater than zero for the tradeoffs.

[0238] In this embodiment of the application, step S603, transferring knowledge to the initial target neural network model through shared parameters to obtain the initial target neural network model after knowledge transfer, includes: updating the target initial neural network model using the shared parameters after the second update to obtain the initial target neural network model after knowledge transfer.

[0239] Furthermore, after obtaining the second updated shared parameters, the first intermediate neural network model can be updated using the second updated shared parameters. That is, the initial target neural network model after knowledge transfer uses the second updated shared parameters of the network sharing layer.

[0240] In the embodiments of this application, such as Figure 15 As shown, step S604 can be performed through S6041-S6042, that is, by using the first sample data set, the second sample data set, and the third sample data set to perform transfer learning on the first initial neural network model and the second initial neural network model to obtain the optimized second intermediate neural network model, including S6041-S6042. Each step is explained below.

[0241] S6041, using the fourth sample data set, train the initial target neural network model after knowledge transfer to obtain the target intermediate neural network model.

[0242] The target intermediate neural network model includes the shared parameters after the third update and the third parameter.

[0243] The target initial neural network model is a neural network model for image enhancement. It can be used to perform image enhancement processing on the fourth sample data in the fourth dataset, namely the second natural image, to output the enhancement result, namely the predicted enhanced second natural image.

[0244] After obtaining the predicted enhanced second natural image, the enhanced second natural image from the fourth sample dataset can be used to construct a loss function. As one implementation, the loss function can be expressed as the following formula:

[0245]

[0246] in, The fidelity term, which measures the similarity between the predicted augmented second natural image and the actual augmented second natural image, can be specifically determined based on the predicted augmented second natural image. And the enhanced second natural image We get w3, which represents the third parameter.

[0247] Through the above process, the initial target neural network model after knowledge transfer can be trained to obtain the target intermediate neural network model.

[0248] S6042, update the optimized second intermediate neural network model using the shared parameters updated in the third update, and use the second sample data set to jointly train the updated and optimized second intermediate neural network model and the target intermediate neural network model to obtain the target neural network model.

[0249] The target neural network model includes the shared parameters after the fourth update and the third parameters after the first update.

[0250] Furthermore, after obtaining the shared parameters updated for the third time, the optimized second intermediate neural network model can be updated using the shared parameters updated for the third time. That is, the optimized second intermediate neural network model uses the shared parameters updated for the third time in the network's shared layer.

[0251] Furthermore, the updated and optimized second intermediate neural network model (including the fully connected layers of the second intermediate neural network model) and the target intermediate neural network model are jointly trained on a small-scale second sample dataset. As one implementation method, the loss function in this process can be expressed as the following formula:

[0252]

[0253] in, This represents the k-th unenhanced second sample data in the second sample dataset, i.e., the k-th unenhanced second game image. This represents the enhanced second game image corresponding to the k-th unenhanced second sample data in the second sample dataset. The fidelity term, which measures the similarity between the predicted enhanced second game image and the actual enhanced second game image, can be specifically determined based on the predicted enhanced second game image. And enhanced second game graphics get, The predicted target score is output by the updated and optimized second intermediate neural network model. The weight parameters are greater than zero.

[0254] The desired parameter set is then obtained by minimizing the following loss. ,Right now:

[0255]

[0256] After obtaining the parameter set W and w3, the target neural network model can be obtained based on the parameter set W and w3.

[0257] Unlike traditional methods that use predefined loss functions (such as L2-norm) to train networks, this application considers that the optimized second intermediate neural network model can already score game images well based on human visual aesthetics. In this embodiment, the network part corresponding to the optimized second intermediate neural network model is considered as part of the loss function in the target intermediate neural network model. Using the optimized second intermediate neural network model to assist the target intermediate neural network model can achieve better enhancement results.

[0258] Furthermore, by combining the shared parameter W and the previously obtained updated second parameter, the second neural network model can be obtained. It can be understood that the fully connected layers of the second neural network model are updated during training; therefore, the updated second parameter includes the target weight matrix of the fully connected layers. Additionally, by combining the shared parameter W and the previously obtained first parameter, the first neural network model can be obtained. Thus, the transfer learning process on the first, second, third, and fourth sample datasets is completed, resulting in the first neural network model, the second neural network model, and the target neural network model.

[0259] In some implementations, after obtaining the first, second, third, and fourth sample datasets, for each dataset, the data can be divided into three parts: a first proportion as the training set, a second proportion as the validation set, and a third proportion as the test set, where the sum of the first, second, and third proportions is 1. For example, the first proportion could be 60%, the second proportion could be 20%, and the third proportion could be 20%.

[0260] In this implementation, by performing transfer learning on the first, second, third, and fourth sample data sets, a target neural network model, a first neural network model, and a second neural network model are obtained. This allows knowledge learned from easily obtainable sample data to be transferred to more difficult-to-obtain sample data, thereby simplifying the difficulty of obtaining sample data and reducing the cost of enhancing the target game image received by the game client.

[0261] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0262] See Figure 16 , Figure 16 This is a schematic diagram of an exemplary image processing flow provided in an embodiment of this application; as shown... Figure 16As shown, in the cloud gaming scenario, there are cloud gaming servers and game clients. The game client deploys a multi-task neural network model, which includes a first neural network model, a second neural network model, and a target neural network model. The first neural network model, the second neural network model, and the target neural network model share parameters through a network sharing layer.

[0263] After the game client establishes a connection with the cloud gaming server via the network, the cloud gaming server can obtain the image rendering strategy corresponding to the game client and render the game data according to the image rendering strategy to obtain the rendered game data. Then, the cloud gaming server calls the encoding module to perform image compression encoding operation on the rendered game screen according to the video encoding standard, and sends the encoded data (i.e., compressed image) obtained by compression encoding to the game client via the network.

[0264] The game client can receive encoded data sent by the cloud gaming server over the network, and then call the decoding module to perform decoding operations on the encoded data to restore the game image. At this point, the game client has received the target game image.

[0265] After receiving the target game image, the game client first inputs the target game image into a multi-task neural network model. In the multi-task neural network model, the first neural network model is used to detect the degradation type of the target game image to obtain the predicted degradation type corresponding to the target game image and the probability P corresponding to each predicted degradation type. Then, the game client determines the predicted degradation type with a probability greater than the target probability threshold as the target degradation type, and then calls the image restoration algorithm corresponding to the target degradation type to perform image restoration processing on the target game image to obtain the restored target game image.

[0266] Next, the game client uses the target neural network model in the multi-task neural model to perform image enhancement processing on the restored target game image, obtaining the enhanced target game image output by the target network model. .

[0267] Next, the game client displays the enhanced target game image, allowing users to see game images that better align with human visual aesthetic preferences and improving the user experience.

[0268] Furthermore, the game client can also use a second neural network model within the multi-task neural model to perform score detection on the target game image, thereby obtaining the target score corresponding to the target game image. Specifically, in the score detection process, the second neural network first performs score detection on the target game image in the aesthetic dimension, obtaining the score S corresponding to the target game image in each aesthetic dimension. Then, it processes the scores corresponding to the target game image in each aesthetic dimension through the target weight matrix of the fully connected layer of the target neural network, obtaining the score corresponding to each degradation type of the target game image. Next, the target neural network model scores the target game image for each aesthetic dimension (S) and for each degradation type (S) of the target game image. By performing a weighted summation, the target score corresponding to the target game image is obtained.

[0269] Then, the target score is fed back to the cloud gaming server via the network.

[0270] After receiving the target score, the cloud gaming server can update and adjust the target image rendering strategy for the game client based on whether the target score is higher than the preset score threshold. It then uses the updated target image rendering strategy to render the game data, obtains the rendered game image, and returns to the above steps of calling the encoding module to perform image compression encoding operation on the rendered game screen according to the video encoding standard, and sends the encoded data (i.e., compressed image) obtained by compression encoding to the game client through the network.

[0271] In some implementations, after obtaining the target game image, the game client can decompose the target game image according to the rendering components to obtain individual image regions. Correspondingly, when the game client calls the second neural network model to perform scoring detection on the target game image, it can perform scoring detection on each image region included in the target game image separately to obtain the target score corresponding to each image region. Accordingly, when the game client sends the target score to the cloud gaming server, it sends the target score corresponding to each image region to the cloud gaming server. In this way, the cloud gaming server can accurately determine the target game rendering strategy corresponding to each rendering component based on the target score corresponding to each image region, thereby further improving the display effect of the target game image after it is enhanced on the game client and further improving the user experience quality.

[0272] It should be noted that the above-mentioned specific examples of implementable methods provided in this application can be arbitrarily combined without conflict to form a new image processing method. It should be understood that any new image processing method formed by combining any examples should fall within the protection scope of this application.

[0273] It should also be noted that in some alternative implementations, the execution order of some processes in the embodiments of the present invention may differ from the execution order described in the foregoing specific embodiments. For example, two consecutive processes may actually be executed in parallel, or they may sometimes be executed in reverse order, depending on the functions involved.

[0274] Please see Figure 17 , Figure 17 This illustration shows a block diagram of an image processing apparatus 700 according to an embodiment of this application, which can be applied to a client (e.g., Figure 1 The device 700 (in the game client) includes: an acquisition module 710, an extraction module 720, a target event type determination module 730, and a target risk value calculation module 740.

[0275] Acquisition module 710, target game image receiving module, used to receive target game images;

[0276] The target game image enhancement module 720 is used to enhance the target game image through the target neural network model. The target neural network model is obtained by optimizing shared parameters and target-specific parameters. The shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model. The target-specific parameters are optimized based on the sample data corresponding to the target neural network model.

[0277] The enhanced target game image acquisition module 730 is used to acquire the enhanced target game image output by the target network model.

[0278] Display module 740 is used to display the enhanced target game image.

[0279] In one implementation, the device 700 also includes a degradation detection module and a recovery module.

[0280] The degradation detection module is used to perform degradation detection on the target game image and obtain the degradation detection results.

[0281] The recovery module is used to call the image recovery algorithm corresponding to the degradation detection result, perform image recovery processing on the target game image, and obtain the recovered target game image.

[0282] The target game image enhancement module 720 is also used to enhance the recovered target game image through a target neural network model.

[0283] In one implementation, the degradation detection result includes the predicted degradation type and the probability corresponding to the predicted degradation type. Accordingly, the restoration module is also used to determine the predicted degradation type with a probability greater than the target probability threshold as the target degradation type; and to call the image restoration algorithm corresponding to the target degradation type to perform image restoration processing on the target game image to obtain the restored target game image.

[0284] As one implementation, the target neural network model includes an attention subnetwork and an enhancement subnetwork. Accordingly, the target game image enhancement module 720 is also used to determine the enhancement level corresponding to each image region in the target game image through the attention subnetwork using the attention weight matrix; and to perform enhancement processing on the target game image based on the enhancement level corresponding to each image region in the target game image and the enhancement subnetwork. Accordingly, the enhanced target game image acquisition module 730 is also used to acquire the enhanced target game image output by the enhancement subnetwork.

[0285] In one implementation, the device 700 also includes a scoring detection module and a target scoring transmission module.

[0286] The rating detection module is used to perform rating detection on the target game image and obtain the target rating corresponding to the target game image.

[0287] The target score sending module is used to send the target score to the cloud gaming server, so that the cloud gaming server can determine the target image rendering strategy for the game data based on the target score.

[0288] As one implementation method, the rating detection module is also used to perform aesthetic dimension rating detection on the target game image to obtain the rating of the target game image under each aesthetic dimension; based on the rating of the target game image under each aesthetic dimension and the target weight matrix, the target rating of the target game image is obtained.

[0289] In one implementation, the device 700 further includes: a sample dataset acquisition module, a first migration module, a second migration module, and a third migration module.

[0290] The sample dataset acquisition module is used to acquire a first sample data set consisting of multiple degraded first game images carrying degradation type labels, a second sample data set consisting of multiple second game images carrying subjective rating labels, multiple degraded second game images carrying degradation type labels, and multiple enhanced second game images, a third sample data set consisting of multiple first natural images carrying subjective rating labels, and a fourth sample data set consisting of multiple second natural images and multiple enhanced second natural images.

[0291] The first transfer module is used to perform transfer learning on the first initial neural network model and the second initial neural network model using the first sample data set, the second sample data set and the third sample data set, to obtain an optimized second intermediate neural network model, which includes shared parameters.

[0292] The second transfer module is used to transfer knowledge to the initial target neural network model through shared parameters, thereby obtaining the initial target neural network model after knowledge transfer.

[0293] The third transfer module is used to perform transfer learning on the initial target neural network model after knowledge transfer and the optimized second intermediate neural network model using the second and fourth sample data sets to obtain the target neural network model.

[0294] In one implementation, the device further includes a first training module, a second training module, a third training module, a parameter update module, a fourth training module, and a fifth training module.

[0295] The first training module is used to train the first initial neural network model using the first sample data set to obtain the first intermediate neural network model, which includes shared parameters and first parameters.

[0296] The second training module is used to update the second initial neural network model using shared parameters, and to train the updated second initial neural network model using the third sample data set to obtain the second intermediate neural network model. The second intermediate neural network model includes the shared parameters after the first update and the second parameters.

[0297] The third training module is used to update the first intermediate neural network model using the shared parameters updated in the first update, and to jointly train the first intermediate neural network model and the second intermediate neural network model using the second sample data set to obtain the optimized second intermediate neural network model. The optimized second intermediate neural network model includes the shared parameters updated in the second update and the second parameters updated in the first update.

[0298] The parameter update module is used to update the initial target neural network model using the shared parameters after the second update, so as to obtain the initial target neural network model after knowledge transfer.

[0299] The fourth training module is used to train the initial target neural network model after knowledge transfer using the fourth sample data set to obtain the target intermediate neural network model, which includes the shared parameters after the third update and the third parameters.

[0300] The fifth training module is used to update the optimized second intermediate neural network model using the shared parameters updated in the third update, and to jointly train the updated and optimized second intermediate neural network model and the target intermediate neural network model using the second sample data set to obtain the target neural network model, which includes the shared parameters updated in the fourth update and the third parameters updated in the first update.

[0301] As one implementation, the first natural image carrying a subjective rating label includes at least one of a first natural image carrying an image quality rating, a first natural image carrying an overall image aesthetic rating, or a first natural image carrying a multi-dimensional aesthetic rating label.

[0302] The image processing apparatus provided in this application embodiment can directly enhance and display game images through a target neural network model. Since the image enhancement is direct, the enhanced target game image can improve the user's subjective visual experience of the displayed game screen, making the enhanced target game image more in line with human visual aesthetic preferences. This improves the display effect of the game image and enhances the user experience quality in cloud gaming scenarios. Furthermore, since the target neural network model is obtained by optimizing shared parameters and target-specific parameters, and the shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model, while the target-specific parameters are optimized based on the sample data corresponding to the target neural network model, the target neural network can learn information from different data domains, improve the generalization ability of the target neural network model, and reduce the difficulty of obtaining sample data.

[0303] Please see Figure 18 , Figure 18 This illustration shows a block diagram of an image processing apparatus 800 according to an embodiment of this application, which can be applied to a client (e.g., Figure 1 The device 800 (a game client in the game) includes: a target image rendering strategy acquisition module 810, a rendering module 820, a game image sending module 830, a target score receiving module 840, and a target image rendering strategy updating module 850.

[0304] The target image rendering strategy acquisition module 810 is used to acquire the target image rendering strategy corresponding to the target client.

[0305] The rendering module 820 is used to render game data using a target image rendering strategy to generate rendered game images.

[0306] The game image sending module 830 is used to send the rendered game image to the target client so that the target client can obtain the target game image corresponding to the rendered game image, and obtain the enhanced target game image corresponding to the target game image through the target neural network model. The target neural network model is obtained by optimizing shared parameters and target-specific parameters. The shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model. The target-specific parameters are optimized based on the sample data corresponding to the target neural network model.

[0307] The target score receiving module 840 is used to receive the target score sent by the target client, wherein the target score is obtained by the target client through score detection of the target game image.

[0308] The target image rendering strategy update module 850 is used to update the target image rendering strategy based on the target score.

[0309] In one implementation, the device 800 further includes: a detection module, a first target image rendering strategy determination module, and a second target image rendering strategy determination module.

[0310] The detection module is used to detect whether a historical image rendering strategy corresponding to the target client exists when a connection is established with the target client.

[0311] The first target image rendering strategy determination module is used to use the historical image rendering strategy as the target image rendering strategy if a historical image rendering strategy exists.

[0312] The second target image rendering strategy determination module is used to select the initial image rendering strategy as the target image rendering strategy if no historical image rendering strategy exists.

[0313] In one implementation, the device 800 further includes: a sample dataset acquisition module, a first migration module, a second migration module, and a third migration module.

[0314] The sample dataset acquisition module is used to acquire a first sample data set consisting of multiple degraded first game images carrying degradation type labels, a second sample data set consisting of multiple second game images carrying subjective rating labels, multiple degraded second game images carrying degradation type labels, and multiple enhanced second game images, a third sample data set consisting of multiple first natural images carrying subjective rating labels, and a fourth sample data set consisting of multiple second natural images and multiple enhanced second natural images.

[0315] The first transfer module is used to perform transfer learning on the first initial neural network model and the second initial neural network model using the first sample data set, the second sample data set and the third sample data set, to obtain an optimized second intermediate neural network model, which includes shared parameters.

[0316] The second transfer module is used to transfer knowledge to the initial target neural network model through shared parameters, thereby obtaining the initial target neural network model after knowledge transfer.

[0317] The third transfer module is used to perform transfer learning on the initial target neural network model after knowledge transfer and the optimized second intermediate neural network model using the second and fourth sample data sets to obtain the target neural network model.

[0318] As one implementation, the device also includes...

[0319] The first training module is used to train the first initial neural network model using the first sample data set to obtain the first intermediate neural network model, which includes shared parameters and first parameters.

[0320] The second training module is used to update the second initial neural network model using shared parameters, and to train the updated second initial neural network model using the third sample data set to obtain the second intermediate neural network model. The second intermediate neural network model includes the shared parameters after the first update and the second parameters.

[0321] The third training module is used to update the first intermediate neural network model using the shared parameters updated in the first update, and to jointly train the first intermediate neural network model and the second intermediate neural network model using the second sample data set to obtain the optimized second intermediate neural network model. The optimized second intermediate neural network model includes the shared parameters updated in the second update and the second parameters updated in the first update.

[0322] The parameter update module is used to update the initial target neural network model using the shared parameters after the second update, so as to obtain the initial target neural network model after knowledge transfer.

[0323] The fourth training module is used to train the initial target neural network model after knowledge transfer using the fourth sample data set to obtain the target intermediate neural network model, which includes the shared parameters after the third update and the third parameters.

[0324] The fifth training module is used to update the optimized second intermediate neural network model using the shared parameters updated in the third update, and to jointly train the updated and optimized second intermediate neural network model and the target intermediate neural network model using the second sample data set to obtain the target neural network model, which includes the shared parameters updated in the fourth update and the third parameters updated in the first update.

[0325] As one implementation, the first natural image carrying a subjective rating label includes at least one of a first natural image carrying an image quality rating, a first natural image carrying an overall image aesthetic rating, or a first natural image carrying a multi-dimensional aesthetic rating label.

[0326] The image processing apparatus provided in this application embodiment can directly enhance and display game images through a target neural network model. Since the image enhancement is direct, the enhanced target game image can improve the user's subjective visual experience of the displayed game screen, making the enhanced target game image more in line with human visual aesthetic preferences. This improves the display effect of the game image and enhances the user experience quality in cloud gaming scenarios. Furthermore, since the target neural network model is obtained by optimizing shared parameters and target-specific parameters, and the shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model, while the target-specific parameters are optimized based on the sample data corresponding to the target neural network model, the target neural network can learn information from different data domains, improve the generalization ability of the target neural network model, and reduce the difficulty of obtaining sample data.

[0327] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0328] The following will combine Figure 19 This application describes an electronic device.

[0329] Please see Figure 19 Based on the above image processing method, this application embodiment also provides another electronic device 100 including a processor 102 capable of executing the aforementioned method. The electronic device 100 can be a server or a terminal device, and the terminal device can be a smartphone, tablet computer, computer or portable computer, etc.

[0330] The electronic device 100 also includes a memory 104. The memory 104 stores a program that can execute the contents of the foregoing embodiments, and the processor 102 can execute the program stored in the memory 104.

[0331] The processor 102 may include one or more cores for data processing and message matrix units. The processor 102 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling data stored in the memory 104. Optionally, the processor 102 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 102 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 102 and may be implemented separately using a communication chip.

[0332] The memory 104 may include random access memory (RAM) or read-only memory (ROM). The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described below, etc. The data storage area may also store data acquired by the electronic device 100 during use (e.g., data to be recommended and operating methods).

[0333] The electronic device 100 may also include a network module and a screen. The network module is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals, thereby enabling communication with communication networks or other devices, such as audio playback devices. The network module may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, SIM cards, memory, etc. The network module can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The screen can display interface content and facilitate data interaction.

[0334] In some embodiments, the electronic device 100 may further include a peripheral interface and at least one peripheral device. The processor 102, memory 104, and peripheral interface 106 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral interface via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency component 108, a positioning component 112, a camera 114, an audio component 116, a display screen 118, and a power supply 122.

[0335] Peripheral interface 106 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 102 and memory 104. In some embodiments, processor 102, memory 104 and peripheral interface 106 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 102, memory 104 and peripheral interface 106 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0336] The radio frequency (RF) component 108 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF component 108 communicates with communication networks and other communication devices via electromagnetic signals. The RF component 108 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF component 108 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF component 108 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF component 108 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0337] The positioning component 112 is used to locate the current geographical location of the electronic device to enable navigation or LBS (Location Based Service). The positioning component 112 can be a positioning component based on GPS (Global Positioning System), BeiDou system, or Galileo system.

[0338] Camera 114 is used to capture images or videos. Optionally, camera 114 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device 100, and the rear-facing camera is located on the back of the electronic device 100. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, camera 114 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0339] Audio component 116 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to processor 102 for processing, or input to radio frequency component 108 for voice communication. For stereo acquisition or noise reduction purposes, there may be multiple microphones, each located at a different part of electronic device 100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from processor 102 or radio frequency component 108 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into sound waves that humans can hear, but also into sound waves that humans cannot hear for purposes such as ranging. In some embodiments, audio component 114 may also include a headphone jack.

[0340] Display screen 118 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 118 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 102 for processing. In this case, display screen 118 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 118, which serves as the front panel of electronic device 100; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of electronic device 100 or in a folded design; in still other embodiments, display screen 118 may be a flexible display screen, disposed on a curved or folded surface of electronic device 100. Furthermore, display screen 118 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 118 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0341] Power supply 122 is used to supply power to various components in electronic device 100. Power supply 122 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 122 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0342] This application also provides a computer-readable storage medium. This computer-readable medium stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0343] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media includes non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0344] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various optional implementations above.

[0345] In summary, the image processing method, apparatus, electronic device, storage medium, and computer program product or computer program provided in this application embodiment can directly enhance and display game images on the client side through a target neural network model. Since the image enhancement is direct, the enhanced target game image can improve the user's subjective visual experience of the displayed game screen, making the enhanced target game image more in line with human visual aesthetic preferences, thereby improving the display effect of the game image and enhancing the user experience quality in cloud gaming scenarios. Furthermore, since the target neural network model is obtained by optimizing shared parameters and target-specific parameters, and the shared parameters are jointly optimized based on the sample data corresponding to the target neural network model and the sample data corresponding to the neural network model associated with the target neural network model, and the target-specific parameters are optimized based on the sample data corresponding to the target neural network model, the target neural network can learn information in different data domains, improve the generalization ability of the target neural network model, and at the same time, reduce the difficulty of obtaining sample data.

[0346] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, The method applied to a game client comprises: obtaining a target game image, the target game image being obtained by decoding compressed and encoded data from a cloud game server; performing degradation detection on the target game image to obtain a degradation detection result; calling an image recovery algorithm corresponding to the degradation detection result to perform image recovery processing on the target game image to obtain a recovered target game image; performing enhancement processing on the recovered target game image in a direction conforming to human visual aesthetics by a target neural network model, the target neural network model being obtained by optimizing shared parameters and target specific parameters, the shared parameters being jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to a neural network model associated with the target neural network model, and the target specific parameters being optimized based on sample data corresponding to the target neural network model; obtaining an enhanced target game image output by the target neural network model; displaying the enhanced target game image.

2. The method of claim 1, wherein, The degradation detection result comprises a predicted degradation type and a probability corresponding to the predicted degradation type, and calling an image recovery algorithm corresponding to the degradation detection result to perform image recovery processing on the target game image to obtain a recovered target game image comprises: determining a predicted degradation type with a probability greater than a target probability threshold as a target degradation type; calling an image recovery algorithm corresponding to the target degradation type to perform image recovery processing on the target game image to obtain a recovered target game image.

3. The method of claim 1, wherein, The target neural network model comprises an attention subnetwork and an enhancement subnetwork, and performing enhancement processing on the recovered target game image in a direction conforming to human visual aesthetics by the target neural network model comprises: determining, by the attention subnetwork, an enhancement degree corresponding to each image region in the recovered target game image by using an attention weight matrix; performing enhancement processing on the recovered target game image based on the enhancement degree corresponding to each image region in the recovered target game image and the enhancement subnetwork; The obtaining of the enhanced target game image output by the target neural network model comprises: obtaining an enhanced target game image output by the enhancement subnetwork.

4. The method according to any one of claims 1 to 3, characterized in that, After the obtaining of the target game image, the method further comprises: performing score detection on the target game image to obtain a target score corresponding to the target game image; sending the target score to the cloud game server, so that the cloud game server determines a target image rendering strategy for game data according to the target score.

5. The method of claim 4, wherein, The degradation detection result comprises a predicted degradation type corresponding to the target game image and a probability corresponding to each predicted degradation type, and the performing of score detection on the target game image to obtain a target score corresponding to the target game image comprises: performing score detection on the target game image in an aesthetic dimension to obtain a score corresponding to the target game image in each aesthetic dimension. obtaining a score of the target game image under the predicted degradation type based on the scores of the target game image under each aesthetic dimension and the target weight matrix; obtaining a target score corresponding to the target game image based on the score of the target game image under the predicted degradation type and the probability corresponding to each predicted degradation type.

6. An image processing method characterized by, The method is applied to a cloud game server, and the method comprises the following steps: obtaining a target image rendering strategy corresponding to a target client; rendering game data by using the target image rendering strategy to generate a rendered game image; sending encoded data obtained by compressing and encoding the rendered game image to the target client, so that the target client decodes the encoded data to obtain a target game image corresponding to the rendered game image, and then performs the following processing: performing degradation detection on the target game image to obtain a degradation detection result; calling an image restoration algorithm corresponding to the degradation detection result to perform image restoration processing on the target game image to obtain a restored target game image; performing enhancement processing on the restored target game image in conformity with the direction of human visual aesthetics by using a target neural network model to obtain an enhanced target game image, wherein the target neural network model is obtained by optimizing shared parameters and target unique parameters, the shared parameters are jointly optimized based on sample data corresponding to the target neural network model and sample data corresponding to a neural network model associated with the target neural network model, and the target unique parameters are optimized based on sample data corresponding to the target neural network model; receiving a target score sent by the target client, wherein the target score is obtained by performing score detection on the target game image by the target client; updating the target image rendering strategy according to the target score.

7. The method of claim 6, wherein, The method comprises the following steps: in a case where a connection with the target client is established, detecting whether there is a historical image rendering strategy corresponding to the target client; if the historical image rendering strategy exists, taking the historical image rendering strategy as the target image rendering strategy; if the historical image rendering strategy does not exist, selecting an initial image rendering strategy as the target image rendering strategy.

8. The method of claim 1 or 6, wherein, The target neural network model is obtained by the following steps: obtaining a first sample data set composed of a plurality of degraded first game images carrying degradation type labels, a second sample data set composed of a plurality of second game images carrying subjective score labels, a plurality of degraded second game images carrying degradation type labels and a plurality of enhanced second game images, a third sample data set composed of a plurality of first natural images carrying subjective score labels, and a fourth sample data set composed of a plurality of second natural images and a plurality of enhanced second natural images; The first initial neural network model and the second initial neural network model are subjected to transfer learning by using the first sample data set, the second sample data set and the third sample data set, so as to obtain an optimized second intermediate neural network model, wherein the optimized second intermediate neural network model comprises shared parameters; The knowledge is transferred to an initial target neural network model by using the shared parameters, so as to obtain a knowledge-transferred initial target neural network model; The knowledge-transferred initial target neural network model and the optimized second intermediate neural network model are subjected to transfer learning by using the second sample data set and the fourth sample data set, so as to obtain the target neural network model.

9. The method of claim 1 or 6, wherein, The target neural network model is obtained by the following steps: A first sample data set composed of a plurality of degraded first game images carrying a degradation type label, a second sample data set composed of a plurality of second game images carrying a subjective score label, a plurality of degraded second game images carrying a degradation type label and a plurality of enhanced second game images, a third sample data set composed of a plurality of first natural images carrying a subjective score label, and a fourth sample data set composed of a plurality of second natural images and a plurality of enhanced second natural images are obtained; A first initial neural network model is trained by using the first sample data set, so as to obtain a first intermediate neural network model, wherein the first intermediate neural network model comprises shared parameters and first parameters; A second initial neural network model is updated by using the shared parameters, and the updated second initial neural network model is trained by using the third sample data set, so as to obtain a second intermediate neural network model, wherein the second intermediate neural network model comprises first updated shared parameters and second parameters; The first intermediate neural network model is subjected to first updating by using the first updated shared parameters, and the first updated first intermediate neural network model and the second intermediate neural network model are subjected to joint training by using the second sample data set, so as to obtain an optimized second intermediate neural network model, wherein the optimized second intermediate neural network model comprises second updated shared parameters and first updated second parameters; An initial target neural network model is updated by using the second updated shared parameters, so as to obtain a knowledge-transferred initial target neural network model; The knowledge-transferred initial target neural network model is trained by using the fourth sample data set, so as to obtain a target intermediate neural network model, wherein the target intermediate neural network model comprises third updated shared parameters and third parameters; The third updated shared parameter is used to update the optimized second intermediate neural network model, and the updated optimized second intermediate neural network model and the target intermediate neural network model are jointly trained by using the second sample data set, so as to obtain the target neural network model, wherein the target neural network model comprises fourth updated shared parameters and first updated third parameters.

10. The method of claim 8, wherein, The first natural image carrying a subjective score label comprises at least one of a first natural image carrying an image quality score, a first natural image carrying an image overall aesthetic score, and a first natural image carrying a multi-dimensional aesthetic score label.

11. An image processing apparatus characterized by comprising: The application is applied to a game client, and comprises: A target game image receiving module is configured to obtain a target game image, wherein the target game image is obtained by decoding compressed and encoded data from a cloud game server; A degradation detection module is configured to perform degradation detection on the target game image to obtain a degradation detection result; A recovery module is configured to call an image recovery algorithm corresponding to the degradation detection result, perform image recovery processing on the target game image, and obtain a recovered target game image; A target game image enhancement module is configured to perform enhancement processing on the recovered target game image in a direction conforming to human visual aesthetics by using a target neural network model, wherein the target neural network model is obtained by optimizing shared parameters and target specific parameters, the shared parameters are obtained by jointly optimizing sample data corresponding to the target neural network model and sample data corresponding to a neural network model associated with the target neural network model, and the target specific parameters are obtained by optimizing sample data corresponding to the target neural network model; An enhanced target game image obtaining module is configured to obtain an enhanced target game image output by the target neural network model; A display module is configured to display the enhanced target game image.

12. An image processing apparatus characterized by comprising: The application is applied to a cloud game server, and comprises: A target image rendering strategy obtaining module is configured to obtain a target image rendering strategy corresponding to a target client; A rendering module is configured to perform rendering processing on game data by using the target image rendering strategy to generate a rendered game image. The game image sending module is configured to send encoded data obtained by compressively encoding the rendered game image to the target client, so that the target client decodes the encoded data to obtain a target game image corresponding to the rendered game image, and then performs the following processing: performing degradation detection on the target game image to obtain a degradation detection result; calling an image recovery algorithm corresponding to the degradation detection result to perform image recovery processing on the target game image to obtain a recovered target game image; performing enhancement processing on the recovered target game image in a direction conforming to human visual aesthetics by using a target neural network model to obtain an enhanced target game image, wherein the target neural network model is obtained by optimizing shared parameters and target-specific parameters, the shared parameters are obtained by jointly optimizing sample data corresponding to the target neural network model and sample data corresponding to a neural network model associated with the target neural network model, and the target-specific parameters are obtained by optimizing sample data corresponding to the target neural network model; The target score receiving module is configured to receive a target score sent by the target client, wherein the target score is obtained by performing score detection on the target game image by the target client; The target image rendering strategy updating module is configured to update the target image rendering strategy according to the target score.

13. An electronic device, comprising: Comprise: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the method in any one of claims 1-10.

14. A computer readable storage medium, characterized in that, The computer readable storage medium stores program code, and the program code can be called and executed by the processor to perform the method in any one of claims 1-10.

Images