Cloud game data processing method and system, and readable storage medium

Abstract

The application relates to the technical field of cloud games, and particularly discloses a cloud game data processing method and system and a readable storage medium, which adopt artificial intelligence technology based on deep learning to perform data analysis on first operation instructions and first image data, respectively excavate semantic feature representations of the first operation instructions and the first image data, further realize cross-modal semantic joint of the first operation instructions and the first image data based on a meta-learning mechanism, and generate second image data based on the cross-modal semantic joint. In this way, the operation intention of a player can be more accurately understood, and the operation intention of the player can be dynamically integrated into a generation process of a game picture, so that the response speed of the cloud game and the accuracy of image generation are improved, and a player can have a more smooth and natural game experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud gaming, and more particularly, to a cloud gaming data processing method and system and a readable storage medium. BACKGROUND

[0002] Cloud gaming is an online gaming service that allows users to play games on remote servers over the Internet, rather than running games on local computers or game consoles. Traditional game modes usually require players to download and install games on specific devices, which not only occupies a large amount of local storage space, but also limits the ability of players to switch games between different devices. With the development of cloud computing technology, cloud gaming has emerged as the times require, providing users with a seamless gaming experience.

[0003] In a cloud gaming system, the player's operation instructions and game image data need to be transmitted and processed in real time between the client (such as a mobile phone, computer or game device) and the cloud server. However, in existing cloud gaming data processing technology, due to network delay and server processing capacity limitations, the player's operation instructions may not be timely and accurate to affect the game state, resulting in a decline in game experience. Secondly, the generation and synchronization of game images require efficient data processing algorithms to ensure that high-quality, low-latency image streams are presented on the terminal device. Traditional data processing methods often focus on simple data transmission and rendering, lacking in-depth understanding and analysis of user operation instructions and game picture content, to some extent, limiting the interactivity and intelligent level of cloud gaming.

[0004] Therefore, an optimized cloud gaming data processing method is expected. SUMMARY

[0005] To solve the above technical problems, the present application is proposed. The embodiments of the present application provide a cloud gaming data processing method, system and readable storage medium, which uses artificial intelligence technology based on deep learning to analyze the first operation instruction and the first image data, respectively excavates the semantic feature representation of the first operation instruction and the first image data, and then realizes the cross-modal semantic joint of the first operation instruction and the first image data based on the meta-learning mechanism, and generates the second image data. In this way, the player's operation intention can be more accurately understood and dynamically integrated into the generation process of the game picture, thereby improving the response speed of the cloud game and the accuracy of image generation, providing players with a more smooth and natural gaming experience.

[0006] Accordingly, according to an aspect of the present application, a cloud game data processing method is provided, which comprises: transmitting first image data and first operation instructions to a cloud server by a first terminal device; processing the first image data based on the first operation instructions to obtain second image data at the cloud server; transmitting the second image data to the first terminal device by the cloud server; and displaying a game picture according to the second image data at the first terminal device.

[0007] In the above cloud game data processing method, processing the first image data based on the first operation instructions to obtain second image data at the cloud server comprises: performing semantic coding on the first operation instructions to obtain a first operation instruction semantic coding feature vector; performing image feature extraction on the first image data to obtain a first image semantic feature map; performing feature attention reinforcement on the first image semantic feature map to obtain a reinforced first image semantic feature map; inputting the first operation instruction semantic coding feature vector and the reinforced first image semantic feature map into a MetaNet-based cross-modal joint encoder to obtain an operation instruction semantic guided first image semantic fusion feature map; and generating the second image data based on the operation instruction semantic guided first image semantic fusion feature map.

[0008] In the above cloud game data processing method, performing image feature extraction on the first image data to obtain a first image semantic feature map comprises: inputting the first image data into an image feature extractor based on a cavity convolutional neural network model to obtain the first image semantic feature map.

[0009] In the above cloud game data processing method, performing feature attention reinforcement on the first image semantic feature map to obtain a reinforced first image semantic feature map comprises: inputting the first image semantic feature map into a feature attention selection reinforcement module based on a compression-inhibition structure to obtain the reinforced first image semantic feature map.

[0010] In the cloud game data processing method, inputting the first image semantic feature map into the feature attention selection reinforcement module based on the compression-inhibition structure to obtain the reinforced first image semantic feature map comprises: calculating the global mean of each feature matrix along the channel dimension of the first image semantic feature map to obtain a first image semantic compression information representation feature vector; performing one-dimensional convolution coding on the first image semantic compression information representation feature vector to obtain a first image semantic compression information inter-association representation feature vector; concatenating the first image semantic compression information representation feature vector and the first image semantic compression information inter-association representation feature vector to obtain a first image semantic feature compression information multi-scale representation vector; inputting the first image semantic feature compression information multi-scale representation vector into a compression information feature extraction module containing a multi-layer perception and a SiLU activation function to obtain a first image semantic feature compression information multi-scale association feature vector; performing a normalization operation on the first image semantic feature compression information multi-scale association feature vector using a sigmoid function to obtain a first image semantic feature weight vector; based on the first image semantic feature weight vector, performing feature amplification and inhibition operations on the first image semantic feature map to obtain the reinforced first image semantic feature map.

[0011] In the cloud game data processing method, based on the first image semantic feature weight vector, performing feature amplification and inhibition operations on the first image semantic feature map to obtain the reinforced first image semantic feature map comprises: calculating the Kronecker product of the first image semantic feature weight vector and each feature matrix along the channel dimension of the first image semantic feature map to obtain the reinforced first image semantic feature map.

[0012] In the cloud game data processing method, inputting the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map into the MetaNet-based cross-modal joint encoder to obtain the first image semantic fusion feature map under operation instruction semantic guidance comprises: passing the operation instruction semantic encoding feature vector through a point convolution layer to obtain a first convolution feature vector; passing the first convolution feature vector through a rectified linear unit based on a ReLU function to obtain a first modified convolution feature vector; passing the first modified convolution feature vector through a point convolution layer to obtain a second convolution feature vector; passing the second convolution feature vector through a rectified linear unit based on a Sigmoid function to obtain a second modified convolution feature vector; fusing the second modified convolution feature vector and the reinforced first image semantic feature map to obtain the first image semantic fusion feature map under operation instruction semantic guidance.

[0013] In the cloud game data processing method, the operation instruction semantic guided first image semantic fusion feature map is input into an image generator based on a diffusion model to obtain the second image data.

[0014] According to another aspect of the present application, a cloud game data processing system is provided, comprising: an operation instruction semantic encoding module configured to encode a first operation instruction transmitted by a first terminal device to obtain a first operation instruction semantic encoding feature vector; an image feature extraction module configured to extract image features of first image data transmitted by the first terminal device to obtain a first image semantic feature map; an image feature attention reinforcement module configured to reinforce features of the first image semantic feature map to obtain a reinforced first image semantic feature map; a cross-modal joint encoding module configured to input the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map into a MetaNet-based cross-modal joint encoder to obtain an operation instruction semantic guided first image semantic fusion feature map; and a game picture updating module configured to generate the second image data based on the operation instruction semantic guided first image semantic fusion feature map.

[0015] According to still another aspect of the present application, a readable storage medium is provided, wherein the readable storage medium stores a cloud game data processing program, and the cloud game data processing program is executed by a processor to implement the cloud game data processing method as described above.

[0016] Compared with the prior art, the cloud game data processing method, system and readable storage medium provided by the present application use artificial intelligence technology based on deep learning to analyze the first operation instruction and the first image data, respectively mine the semantic feature representation of the first operation instruction and the first image data, and then realize cross-modal semantic joint of the first operation instruction and the first image data based on a meta-learning mechanism, and generate the second image data. In this way, the operation intention of the player can be more accurately understood and dynamically integrated into the generation process of the game picture, thereby improving the response speed of the cloud game and the accuracy of image generation, and providing a more smooth and natural game experience for the player. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application taken in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of embodiments of the present application and are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and serve to explain the present application, but do not limit the present application. In the drawings, the same reference numerals generally refer to the same components or steps throughout the drawings.

[0018] Figure 1 A flowchart for processing the first image data based on the first operation instruction to obtain second image data in the cloud game data processing method according to the embodiments of the present application.

[0019] Figure 2 An architectural schematic diagram for processing the first image data based on the first operation instruction to obtain second image data in the cloud game data processing method according to the embodiments of the present application.

[0020] Figure 3 A flowchart for inputting the first image semantic feature map into a feature attention selection reinforcement module based on a compression-inhibition structure to obtain the reinforced first image semantic feature map in the cloud game data processing method according to the embodiments of the present application.

[0021] Figure 4 A flowchart for inputting the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map into a cross-modal joint encoder based on MetaNet to obtain a first image semantic fusion feature map under operation instruction semantic guidance in the cloud game data processing method according to the embodiments of the present application.

[0022] Figure 5 A block diagram of the cloud game data processing system according to the embodiments of the present application. DETAILED DESCRIPTION

[0023] Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.

[0024] The present application provides a cloud game data processing method, which comprises: transmitting first image data and a first operation instruction to a cloud server through a first terminal device; processing the first image data based on the first operation instruction to obtain second image data at the cloud server; transmitting the second image data to the first terminal device by the cloud server; and displaying a game picture according to the second image data at the first terminal device.

[0025] That is, the first image data generated by the player during the game and the operation instruction are transmitted to the cloud server in real time through the first terminal device (such as a smart phone, a tablet computer or a television box, etc.). After receiving the data on the cloud server side, the first image data is processed according to the first operation instruction sent by the player to simulate the real-time changes of the game world under the operation of the player, and the updated game picture, that is, the second image data, is generated. Subsequently, the cloud server quickly returns the generated second image data to the first terminal device, and the first terminal device displays the updated game picture on the screen after receiving the second image data, so that the player can see the operation and result in the game in real time. This closed-loop process not only greatly improves the interactivity and immersion of the game, but also makes cloud gaming a new type of entertainment that can be flexibly run on various devices.

[0026] More specifically, at the cloud server, the first image data is processed based on the first operation instruction to obtain the second image data. Traditional data processing methods often focus on simple data transmission and rendering, lack of in-depth understanding and analysis of user operation instructions and game picture content, and to some extent limit the interactivity and intelligent level of cloud gaming.

[0027] To solve the above technical problems, the technical concept of the present application is to use artificial intelligence technology based on deep learning to analyze the first operation instruction and the first image data, respectively excavate the semantic feature representation of the first operation instruction and the first image data, and then realize the cross-modal semantic joint of the first operation instruction and the first image data based on the meta-learning mechanism, and generate the second image data. In this way, the operation intention of the player can be more accurately understood and dynamically integrated into the generation process of the game picture, thereby improving the response speed of cloud gaming and the accuracy of image generation, and providing a more smooth and natural game experience for the player.

[0028] Figure 1 A flow chart for processing the first image data based on the first operation instruction to obtain the second image data in the cloud gaming data processing method according to the embodiment of the present application. Figure 2 An architecture schematic diagram for processing the first image data based on the first operation instruction to obtain the second image data in the cloud gaming data processing method according to the embodiment of the present application. As Figure 1 and Figure 2As shown, according to the cloud game data processing method of the embodiment of the present application, the method comprises the steps of: S110, performing semantic coding on the first operation instruction to obtain a first operation instruction semantic coding feature vector; S120, performing image feature extraction on the first image data to obtain a first image semantic feature map; S130, performing feature attention reinforcement on the first image semantic feature map to obtain a reinforced first image semantic feature map; S140, inputting the first operation instruction semantic coding feature vector and the reinforced first image semantic feature map into a MetaNet-based cross-modal joint encoder to obtain a first image semantic fusion feature map under operation instruction semantic guidance; and S150, generating the second image data based on the first image semantic fusion feature map under operation instruction semantic guidance.

[0029] In the above cloud game data processing method, the step S110, the first operation instruction is coded to obtain a first operation instruction semantic coding feature vector. It can be understood that, considering that the first operation instruction may contain various complex player actions, such as character movement, attack, use of props, etc., directly processing the original operation instruction (such as key input, mouse movement, etc.) may not be able to fully capture the intention or complex logic behind it. Therefore, in order to effectively understand the player's intention, in the technical solution of the present application, the first operation instruction is further coded to convert it into a higher level feature vector representation form with clear semantic meaning, so that the meaning of the first operation instruction can be more accurately expressed, and a response more in line with the player's expectations can be made in the game environment. In the embodiment of the present application, a pre-trained deep neural network model such as Transformer or BERT can be used to code the first operation instruction to learn the context information and potential semantic structure of the operation instruction, thereby generating a first operation instruction semantic coding feature vector with rich semantic information.

[0030] In the cloud game data processing method, the step S120 is to perform image feature extraction on the first image data to obtain a first image semantic feature map. In a specific example of the present application, the processing manner of performing image feature extraction on the first image data to obtain a first image semantic feature map is to input the first image data into an image feature extractor based on a dilated convolutional neural network model to obtain the first image semantic feature map. It should be understood that, considering that a game picture usually contains a large amount of complex detail information, an efficient feature extraction method is needed to ensure the realism and smoothness of the picture. Therefore, in the technical solution of the present application, a dilated convolutional neural network (DCNN) model with excellent performance in the field of image processing is used to perform deep feature extraction on the first image data. The dilated convolution can expand the receptive field at a lower computational cost, thereby capturing long-range dependencies and context information in the image while maintaining a high spatial resolution. In traditional image processing algorithms, a pooling layer is often set to indirectly increase the receptive field, but a large amount of image information is lost in this process. The dilated convolution increases the receptive field without pooling operation by introducing a hole (i.e. zero padding), thereby reducing information loss and retaining more image details to provide more accurate input for generating high-quality game pictures.

[0031] In the cloud game data processing method, the step S130 is to perform feature attention reinforcement on the first image semantic feature map to obtain a reinforced first image semantic feature map. In a specific example of the present application, the processing manner of performing feature attention reinforcement on the first image semantic feature map to obtain a reinforced first image semantic feature map is to input the first image semantic feature map into a feature attention selection reinforcement module based on a compression-inhibition structure to obtain the reinforced first image semantic feature map. It should be understood that, considering that different elements in a game picture have different importance for player operation, for example, the player character, the enemy, the prop, etc. can be the key attention focus, while the background or other non-key elements are relatively secondary. Therefore, in order to more accurately focus on and respond to player operation, in the technical solution of the present application, a feature attention selection reinforcement module based on a compression-inhibition structure is introduced to perform attention reinforcement on the first image semantic feature map, to highlight the key information related to player operation and suppress irrelevant or secondary features, thereby improving the pertinence and efficiency of image processing. Specifically, the feature attention selection reinforcement module generates attention weights based on the correlation between each feature channel by performing correlated feature learning in the channel dimension of the first image semantic feature map, and adaptively redistributes the original image features, which helps to more accurately capture the core information related to player operation and improve the accuracy of subsequent processing.

[0032] Figure 3 This is a flowchart illustrating the process of inputting the first image semantic feature map into a feature attention selection and enhancement module based on a compression-inhibition structure to obtain the enhanced first image semantic feature map in a cloud gaming data processing method according to an embodiment of this application. Figure 3 As shown, step S130 includes: S131, calculating the global mean of each feature matrix along the channel dimension of the first image semantic feature map to obtain the first image semantic compressed information representation feature vector; S132, performing one-dimensional convolutional encoding on the first image semantic compressed information representation feature vector to obtain the first image semantic compressed information inter-correlation representation feature vector; S133, concatenating the first image semantic compressed information representation feature vector and the first image semantic compressed information inter-correlation representation feature vector to obtain the first image semantic feature compressed information multi-scale representation vector; S134, inputting the first image semantic feature compressed information multi-scale representation vector into a compressed information feature extraction module containing a multilayer perceptron and a SiLU activation function to obtain the first image semantic feature compressed information multi-scale correlation feature vector; S135, using the Sigmoid function to normalize the first image semantic feature compressed information multi-scale correlation feature vector to obtain the first image semantic feature weight vector; S136, based on the first image semantic feature weight vector, performing feature amplification and suppression operations on the first image semantic feature map to obtain the enhanced first image semantic feature map.

[0033] More specifically, step S136 includes: calculating the Kronecker product of the first image semantic feature weight vector and each feature matrix along the channel dimension of the first image semantic feature map to obtain the enhanced first image semantic feature map.

[0034] That is, step S130 includes: performing attention selection on the first image semantic feature map using the following feature enhancement formula to obtain the enhanced first image semantic feature map, wherein the feature enhancement formula is:

[0035]

[0036] V s =concat(V s1 Conv 1d (V s1 ))

[0037]

[0038] in, It is the xth semantic feature map of the first image o The coordinates in the channel are (i o jo ) the eigenvalues of the first image semantic feature map, H and W are the height and width of the first image semantic feature map, respectively, is the xth eigenvalue of the first image semantic compressed information representation feature vector, o s1 is the first image semantic compressed information representation feature vector, Conv 1d (·) represents one-dimensional convolution encoding, concat(·, ·) represents concatenation, V s is the first image semantic feature compressed information multi-scale representation vector, MLP(·) represents a multi-layer perceptron, sigmoid represents a sigmoid function, and F is the first image semantic feature map, represents the calculation of the vector and the Kronecker product of each feature matrix along the channel dimension of the feature map, and F' is the reinforced first image semantic feature map.

[0039] In the cloud game data processing method described above, the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map are input into a MetaNet-based cross-modal joint encoder to obtain a first image semantic fusion feature map guided by operation instruction semantics. It should be understood that the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map respectively represent the intention of the player and the current state of the game picture, but since they belong to different modalities, direct fusion may exist semantic gap. Therefore, in the technical solution of the present application, a cross-modal joint encoder with a meta-learning mechanism is used to jointly encode the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map, so as to establish an effective semantic association between the operation instruction semantic feature and the image semantic feature. Specifically, the MetaNet model can learn the transferable knowledge between different modalities, and by dynamically adjusting its internal parameters, it can adapt to the joint representation of different operation instructions and image features, and can convert the first operation instruction semantic encoding feature vector into a form that is comparable and joint on the semantic level with image data, and embed and fuse it into the reinforced first image semantic feature map, thereby realizing cross-modal semantic fusion.

[0040] Figure 4 A flowchart of the cloud game data processing method according to the embodiment of the present application, in which the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map are input into a MetaNet-based cross-modal joint encoder to obtain a first image semantic fusion feature map guided by operation instruction semantics. As Figure 4 ​As shown, the step S140 includes: S141, passing the operation instruction semantic encoding feature vector through a point convolution layer to obtain a first convolution feature vector; S142, passing the first convolution feature vector through a rectified linear unit based on a ReLU function to obtain a first rectified convolution feature vector; S143, passing the first rectified convolution feature vector through a point convolution layer to obtain a second convolution feature vector; S144, passing the second convolution feature vector through a rectified linear unit based on a Sigmoid function to obtain a second rectified convolution feature vector; and S145, fusing the second rectified convolution feature vector and the reinforced first image semantic feature map to obtain the first image semantic fusion feature map guided by the operation instruction semantics.

[0041] In the cloud game data processing method described above, the step S150 generates the second image data based on the first image semantic fusion feature map guided by the operation instruction semantics. In a specific example of the present application, the implementation of generating the second image data based on the first image semantic fusion feature map guided by the operation instruction semantics is to input the first image semantic fusion feature map guided by the operation instruction semantics into a diffusion model-based image generator to obtain the second image data. It should be understood that, in order to ensure that the generated image not only conforms to the operation intention of the player, but also maintains the coherence and authenticity of the game picture, a diffusion model-based image generator is selected in the technical solution of the present application. The diffusion model has shown significant advantages in the field of game picture generation due to its powerful generation capability and precise control over image details. Specifically, the diffusion model can gradually recover a clear image from random noise through simulating the diffusion and denoising process in the image generation process, and after multiple iterations and optimization, it can generate second image data that conforms to the operation intention of the player and has high quality.

[0042] In particular, in the above scheme, the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map respectively express the semantic encoding features of the first operation instruction and the image semantic features of the first image data based on the feature attention selective weighting reinforcement, so that after the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map are input into the MetaNet-based cross-modal joint encoder, the first image semantic fusion feature map guided by the operation instruction semantics will have feature aggregation generation image semantic domain regression representation imbalance due to the difference between the image semantic feature distribution constraint of the feature matrix dimension and the semantic encoding feature distribution constraint of the channel dimension, and after the feature regression representation aggregation is performed through the clustering operation, the image semantic generation probability density distribution regression convergence logic of the intra-class features and the out-of-class features will be inconsistent.

[0043] Therefore, in a preferred embodiment of the present application, after the clustering operation based on the distance between feature values of the first image semantic fusion feature map under the operation instruction semantic guide, further optimization of the intra-class features and the out-of-class features is performed, including the following steps: determining the number of feature values in the clustering set obtained by the clustering operation of the first image semantic fusion feature map under the operation instruction semantic guide, i.e. the number of intra-class features, and subtracting the number of intra-class features from the total number of feature values of the first image semantic fusion feature map under the operation instruction semantic guide to obtain the number of out-of-class features; dividing the total number of feature values by the number of intra-class features and the number of out-of-class features respectively to obtain a class importance value and an out-of-class proportion value, and calculating the reciprocal of the class importance value to obtain a class constraint value; calculating the power function of each feature value of the first image semantic fusion feature map under the operation instruction semantic guide with the class importance value as the exponent, and adding the exponential value with the natural function as the base and the class constraint value as the exponent, and then multiplying by the out-of-class proportion value to obtain the first image semantic fusion modulation feature map under the operation instruction semantic guide; point multiplying the first image semantic fusion feature map under the operation instruction semantic guide by the class importance value to obtain the first image semantic fusion ontology feature map under the operation instruction semantic guide; calculating the weighted sum of the first image semantic fusion modulation feature map under the operation instruction semantic guide and the first image semantic fusion ontology feature map under the operation instruction semantic guide with a weight hyperparameter to obtain the first image semantic fusion feature map under the operation instruction semantic guide after optimization.

[0044] That is, the first image semantic fusion feature map under the operation instruction semantic guide is optimized based on the feature distribution of intra-class features and out-of-class features by the following optimization formula to obtain the first image semantic fusion feature map under the operation instruction semantic guide after optimization, wherein the optimization formula is:

[0045]

[0046] Wherein F represents the first image semantic fusion feature map under the operation instruction semantic guide, n represents the total number of feature values of the first image semantic fusion feature map under the operation instruction semantic guide, k represents the number of intra-class features, n-k represents the number of out-of-class features, represents the out-of-class proportion value, represents the class importance value, represents the class constraint value, and β represents the weight hyperparameter, represents the power function value of each feature value of the first image semantic fusion feature map under the operation instruction semantic guide with the class importance value as the exponent, represents point multiplication, and F' represents the first image semantic fusion feature map under the operation instruction semantic guide after optimization.

[0047] Here, in the technical solution of the present application, while the first image semantic fusion feature map is subjected to clustering operation under the guidance of the operation instruction semantics, the clustering importance measure of the feature values of the first image semantic fusion feature map under the guidance of the operation instruction semantics is combined with the regression constraint measure of the clustering operation to further respond to the modulation of the out-of-class scaling factor, and the clustering scaling of the feature ontology representation of the first image semantic fusion feature map under the guidance of the operation instruction semantics is used to establish the feature purification simplicity and effectiveness of the clustering operation for the entire feature set of the first image semantic fusion feature map under the guidance of the operation instruction semantics, and to suppress the logical inconsistency of the image semantic generation probability density distribution regression convergence caused by the clustering operation, so as to improve the image semantic generation regression iteration effect of the first image semantic fusion feature map under the guidance of the operation instruction semantics input into the image generator based on the diffusion model, that is, to improve the speed of generation training and the image quality of the obtained second image data.

[0048] In summary, the cloud game data processing method according to the embodiments of the present application is illustrated, which uses artificial intelligence technology based on deep learning to analyze the first operation instruction and the first image data, respectively excavates the semantic feature representation of the first operation instruction and the first image data, and then realizes the cross-modal semantic joint of the first operation instruction and the first image data based on the meta-learning mechanism, and generates the second image data. In this way, the operation intention of the player can be more accurately understood and dynamically integrated into the generation process of the game picture, thereby improving the response speed of the cloud game and the accuracy of image generation, and providing the player with a more smooth and natural game experience.

[0049] Figure 5 The block diagram of the cloud game data processing system according to the embodiments of the present application is shown. As shown in Figure 5 The cloud game data processing system 100 according to the embodiments of the present application includes: an operation instruction semantic encoding module 110, configured to perform semantic encoding on the first operation instruction transmitted by the first terminal device to obtain a first operation instruction semantic encoding feature vector; an image feature extraction module 120, configured to perform image feature extraction on the first image data transmitted by the first terminal device to obtain a first image semantic feature map; an image feature attention reinforcement module 130, configured to perform feature attention reinforcement on the first image semantic feature map to obtain a reinforced first image semantic feature map; a cross-modal joint encoding module 140, configured to input the first operation instruction semantic encoding feature vector and the reinforced first image semantic feature map into a MetaNet-based cross-modal joint encoder to obtain a first image semantic fusion feature map under the guidance of the operation instruction semantics; and a game picture updating module 150, configured to generate the second image data based on the first image semantic fusion feature map under the guidance of the operation instruction semantics.

[0050] Here, those skilled in the art can understand that the specific operations of each step in the cloud game data processing system described above have been described in detail above with reference to the description of the cloud game data processing method of Figures 1 to 4 , and thus repetitive description thereof will be omitted.

[0051] Exemplary computer program product and computer readable storage medium

[0052] In addition to the methods and systems described above, embodiments of the present application can also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps of the cloud game data processing method according to various embodiments of the present application described in the above exemplary method part of the specification.

[0053] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0054] In addition, embodiments of the present application can also be a computer readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform steps of the cloud game data processing method according to various embodiments of the present application described in the above exemplary method part of the specification.

[0055] The computer readable storage medium can take any combination of one or more readable storage media. The readable storage medium can be a readable signal storage medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0056] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application.

Claims

1. A cloud gaming data processing method, characterized in that, include: The first image data and the first operation command are transmitted to the cloud server through the first terminal device; On the cloud server, based on the first operation instruction, the first image data is processed to obtain the second image data; The cloud server transmits the second image data to the first terminal device; On the first terminal device, the game screen is displayed based on the second image data; Specifically, on the cloud server, based on the first operation instruction, the first image data is processed to obtain second image data, including: The first operation instruction is semantically encoded to obtain the semantic encoding feature vector of the first operation instruction; Image feature extraction is performed on the first image data to obtain a first image semantic feature map; Feature attention enhancement is performed on the first image semantic feature map to obtain an enhanced first image semantic feature map; The first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map are input into a MetaNet-based cross-modal joint encoder to obtain a first image semantic fusion feature map guided by operation instruction semantics. Based on the semantic fusion feature map of the first image guided by the operation instruction, the second image data is generated. The method of enhancing the first image semantic feature map with feature attention to obtain an enhanced first image semantic feature map includes: inputting the first image semantic feature map into a feature attention selection enhancement module based on a compression-inhibition structure to obtain the enhanced first image semantic feature map, wherein the feature attention selection enhancement module learns the associated features of the first image semantic feature map in the channel dimension, generates attention weights based on the correlation between each feature channel, and adaptively redistributes the original image features. The process involves several steps, including: First, after performing clustering operations based on the distance between feature values ​​on the first image semantic fusion feature map guided by the operation command, further optimization of intra-class and out-of-class features is performed. This includes: determining the number of feature values ​​within the clusters obtained by the clustering operation on the first image semantic fusion feature map guided by the operation command, i.e., the number of intra-class features; and subtracting the number of intra-class features from the total number of feature values ​​in the first image semantic fusion feature map guided by the operation command to obtain the number of out-of-class features; dividing the total number of feature values ​​by the number of intra-class features and the number of out-of-class features respectively to obtain the class importance value and the out-of-class proportion value, and calculating the reciprocal of the class importance value to obtain the class constraint value; and calculating the operation command semantic guidance... Each feature value of the first image semantic fusion feature map is a power function of the class importance value, and is added to an exponent value with the class constraint value as the base of the natural function, and then multiplied by the out-of-class ratio value to obtain the first image semantic fusion modulation feature map guided by the operation instruction semantics; the first image semantic fusion feature map guided by the operation instruction semantics is multiplied by the class importance value to obtain the first image semantic fusion ontology feature map guided by the operation instruction semantics; the weighted sum of the first image semantic fusion modulation feature map guided by the operation instruction semantics and the first image semantic fusion ontology feature map guided by the operation instruction semantics is calculated using weight hyperparameters to obtain the optimized first image semantic fusion feature map guided by the operation instruction semantics.

2. The cloud gaming data processing method according to claim 1, characterized in that, Image feature extraction is performed on the first image data to obtain a first image semantic feature map, including: The first image data is input into an image feature extractor based on a dilated convolutional neural network model to obtain the semantic feature map of the first image.

3. The cloud gaming data processing method according to claim 2, characterized in that, The first image semantic feature map is input into a feature attention selection enhancement module based on a compression-suppression structure to obtain the enhanced first image semantic feature map, including: Calculate the global mean of each feature matrix along the channel dimension of the semantic feature map of the first image to obtain the semantic compression information representation feature vector of the first image; One-dimensional convolutional encoding is performed on the feature vector representing the semantic compression information of the first image to obtain the feature vector representing the correlation between the semantic compression information of the first image. The first image semantic compression information representation feature vector and the first image semantic compression information inter-association representation feature vector are concatenated to obtain the first image semantic feature compression information multi-scale representation vector. The first image semantic feature compressed information multi-scale representation vector is input into the compressed information feature extraction module containing a multilayer perceptron and a SiLU activation function to obtain the first image semantic feature compressed information multi-scale associated feature vector; The Sigmoid function is used to normalize the multi-scale associated feature vector of the first image semantic feature compression information to obtain the first image semantic feature weight vector. Based on the first image semantic feature weight vector, feature amplification and suppression operations are performed on the first image semantic feature map to obtain the enhanced first image semantic feature map.

4. The cloud gaming data processing method according to claim 3, characterized in that, Based on the first image semantic feature weight vector, feature amplification and suppression operations are performed on the first image semantic feature map to obtain the enhanced first image semantic feature map, including: The enhanced first image semantic feature map is obtained by calculating the Kronecker product of the first image semantic feature weight vector and each feature matrix along the channel dimension of the first image semantic feature map.

5. The cloud gaming data processing method according to claim 4, characterized in that, The semantically encoded feature vector of the first operation instruction and the enhanced semantic feature map of the first image are input into a cross-modal joint encoder based on MetaNet to obtain a semantically fused feature map of the first image under the semantic guidance of the operation instruction, including: The semantically encoded feature vector of the operation instruction is passed through a point convolutional layer to obtain the first convolutional feature vector; The first convolutional feature vector is passed through a modified linear unit based on the ReLU function to obtain a first modified convolutional feature vector; The first modified convolutional feature vector is passed through a point convolutional layer to obtain the second convolutional feature vector; The second convolutional feature vector is passed through a rectified linear unit based on the Sigmoid function to obtain a second rectified convolutional feature vector; The second modified convolutional feature vector is fused with the enhanced first image semantic feature map to obtain the first image semantic fusion feature map guided by the operation instruction semantics.

6. The cloud gaming data processing method according to claim 5, characterized in that, Based on the semantic fusion feature map of the first image guided by the operation instruction, the second image data is generated, including: The semantic fusion feature map of the first image guided by the operation instruction is input into the diffusion model-based image generator to obtain the second image data.

7. A cloud gaming data processing system, characterized in that, include: An operation instruction semantic encoding module is used to semantically encode the first operation instruction transmitted by the first terminal device to obtain the first operation instruction semantic encoding feature vector. The image feature extraction module is used to extract image features from the first image data transmitted by the first terminal device to obtain a first image semantic feature map; An image feature attention enhancement module is used to perform feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map; A cross-modal joint encoding module is used to input the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a MetaNet-based cross-modal joint encoder to obtain a first image semantic fusion feature map guided by operation instruction semantics. The game screen update module is used to generate second image data based on the semantic fusion feature map of the first image under the semantic guidance of the operation instructions. The process of performing feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map includes: The first image semantic feature map is input into the feature attention selection enhancement module based on the compression-inhibition structure to obtain the enhanced first image semantic feature map. The feature attention selection enhancement module learns the correlation features of the first image semantic feature map in the channel dimension, generates attention weights based on the correlation between each feature channel, and adaptively redistributes the original image features. The process involves several steps, including: First, after performing clustering operations based on the distance between feature values ​​on the first image semantic fusion feature map guided by the operation command, further optimization of intra-class and out-of-class features is performed. This includes: determining the number of feature values ​​within the clusters obtained by the clustering operation on the first image semantic fusion feature map guided by the operation command, i.e., the number of intra-class features; and subtracting the number of intra-class features from the total number of feature values ​​in the first image semantic fusion feature map guided by the operation command to obtain the number of out-of-class features; dividing the total number of feature values ​​by the number of intra-class features and the number of out-of-class features respectively to obtain the class importance value and the out-of-class proportion value, and calculating the reciprocal of the class importance value to obtain the class constraint value; and calculating the operation command semantic guidance... Each feature value of the first image semantic fusion feature map is a power function of the class importance value, and is added to an exponent value with the class constraint value as the base of the natural function, and then multiplied by the out-of-class ratio value to obtain the first image semantic fusion modulation feature map guided by the operation instruction semantics; the first image semantic fusion feature map guided by the operation instruction semantics is multiplied by the class importance value to obtain the first image semantic fusion ontology feature map guided by the operation instruction semantics; the weighted sum of the first image semantic fusion modulation feature map guided by the operation instruction semantics and the first image semantic fusion ontology feature map guided by the operation instruction semantics is calculated using weight hyperparameters to obtain the optimized first image semantic fusion feature map guided by the operation instruction semantics.

8. A readable storage medium, characterized in that, The readable storage medium stores a cloud gaming data processing program, which, when executed by a processor, implements the cloud gaming data processing method as described in any one of claims 1 to 6.

Images