Cloud game data processing method and system and readable storage medium

Through deep learning technology, semantic analysis and cross-modal joint of operation instructions and image data in cloud games has been solved, and the problem of timeliness of operation instructions and image generation accuracy in cloud games has been improved, and the game experience and intelligence level has been improved.

CN118965254AActive Publication Date: 2024-11-15WUHAN ZHUOXUN INTERACTIVE INFORMATION TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410954270.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-15
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

Due to network delay and server processing capabilities, existing cloud game data processing technology cannot affect the game status in a timely and accurate manner, the game experience is reduced, and there is a lack of in-depth understanding and analysis of user operation instructions and game screen content. , limiting the interactiveness and intelligence level of cloud games.

Method used

Using artificial intelligence technology based on deep learning to analyze the semantic features of operation instructions and image data, the cross-modal semantic alliance is realized through the meta-learning mechanism, more accurate game picture data is generated, dynamically integrated into the game process, and the response speed and image generation are improved. accuracy.

Benefits of technology

It achieves a smoother and natural gaming experience, improves the response speed of cloud games and the accuracy of image generation, and enhances the interactiveness and intelligence level of the game.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118965254A_ABST
    Figure CN118965254A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of cloud games, and particularly discloses a cloud game data processing method and system and a readable storage medium, and the method comprises the steps: carrying out the data analysis of a first operation instruction and first image data through employing an artificial intelligence technology based on deep learning; semantic feature representations of the first operation instruction and the first image data are mined respectively, and then cross-modal semantic combination of the first operation instruction and the first image data is realized based on a meta-learning mechanism, so that second image data is generated. Therefore, the operation intention of the player can be more accurately understood and dynamically fused into the game picture generation process, so that the response speed of the cloud game and the image generation accuracy are improved, and smoother and more natural game experience is provided for the player.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of cloud gaming technology, and more specifically, to a cloud gaming data processing method, system, and readable storage medium. Background Art

[0002] Cloud gaming is an online gaming service that allows users to play games on a remote server over the Internet, rather than running the game on a local computer or game console. Traditional gaming modes usually require players to download and install games on specific devices, which not only takes up a lot of local storage space, but also limits the player's ability to switch games between different devices. With the development of cloud computing technology, cloud gaming has emerged, providing users with a seamless gaming experience.

[0003] In a cloud gaming system, the player's operating instructions and game image data need to be transmitted and processed in real time between the client (such as a mobile phone, computer or gaming device) and the cloud server. However, in existing cloud gaming data processing technologies, due to network latency and server processing power limitations, the player's operating instructions may not be able to affect the game status in a timely and accurate manner, resulting in a decline in gaming experience. Secondly, the generation and synchronization of game images require efficient data processing algorithms to ensure high-quality, low-latency image streams on terminal devices. Traditional data processing methods often focus on simple data transmission and rendering, lacking in-depth understanding and analysis of user operating instructions and game screen content, which limits the interactivity and intelligence of cloud gaming to a certain extent.

[0004] Therefore, an optimized cloud gaming data processing method is expected. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a cloud game data processing method, system and readable storage medium, which use artificial intelligence technology based on deep learning to perform data analysis on the first operation instruction and the first image data, respectively mine the semantic feature representations of the first operation instruction and the first image data, and then realize the cross-modal semantic union of the first operation instruction and the first image data based on the meta-learning mechanism, and generate the second image data. In this way, the player's operating intention can be understood more accurately and dynamically integrated into the generation process of the game screen, thereby improving the response speed of the cloud game and the accuracy of image generation, providing players with a smoother and more natural gaming experience.

[0006] Accordingly, according to one aspect of the present application, a cloud game data processing method is provided, which includes: transmitting first image data and a first operation instruction to a cloud server via a first terminal device; on the cloud server, based on the first operation instruction, processing the first image data to obtain second image data; the cloud server transmits the second image data to the first terminal device; on the first terminal device, displaying a game screen based on the second image data.

[0007] In the above-mentioned cloud game data processing method, on the cloud server, based on the first operation instruction, the first image data is processed to obtain the second image data, including: semantically encoding the first operation instruction to obtain a first operation instruction semantic encoding feature vector; performing image feature extraction on the first image data to obtain a first image semantic feature map; performing feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map; inputting the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a MetaNet-based cross-modal joint encoder to obtain a first image semantic fusion feature map under the semantic guidance of the operation instruction; and generating the second image data based on the first image semantic fusion feature map under the semantic guidance of the operation instruction.

[0008] In the above-mentioned cloud game data processing method, image features are extracted from the first image data to obtain a first image semantic feature map, including: inputting the first image data into an image feature extractor based on a void convolutional neural network model to obtain the first image semantic feature map.

[0009] In the above-mentioned cloud game data processing method, feature attention enhancement is performed on the first image semantic feature map to obtain an enhanced first image semantic feature map, including: inputting the first image semantic feature map into a feature attention selection enhancement module based on a compression-suppression structure to obtain the enhanced first image semantic feature map.

[0010] In the above-mentioned cloud game data processing method, the first image semantic feature map is input into a feature attention selection and enhancement module based on a compression-suppression structure to obtain the enhanced first image semantic feature map, including: calculating the global mean of each feature matrix of the first image semantic feature map along the channel dimension to obtain a first image semantic compression information representation feature vector; performing one-dimensional convolution encoding on the first image semantic compression information representation feature vector to obtain a first image semantic compression information association representation feature vector; cascading the first image semantic compression information representation feature vector and the first image semantic compression information association representation feature vector to obtain a first image semantic feature compression information multi-scale representation vector; inputting the first image semantic feature compression information multi-scale representation vector into a compression information feature extraction module including a multi-layer perceptron and a SiLU activation function to obtain a first image semantic feature compression information multi-scale association feature vector; using a Sigmo id function to normalize the first image semantic feature compression information multi-scale association feature vector to obtain a first image semantic feature weight vector; based on the first image semantic feature weight vector, performing feature amplification and suppression operations on the first image semantic feature map to obtain the enhanced first image semantic feature map.

[0011] In the above-mentioned cloud game data processing method, based on the first image semantic feature weight vector, feature amplification and suppression operations are performed on the first image semantic feature map to obtain the enhanced first image semantic feature map, including: calculating the Kronecker product of the first image semantic feature weight vector and each feature matrix of the first image semantic feature map along the channel dimension to obtain the enhanced first image semantic feature map.

[0012] In the above-mentioned cloud game data processing method, the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map are input into a cross-modal joint encoder based on MetaNet to obtain a first image semantic fusion feature map under the guidance of operation instruction semantics, including: passing the operation instruction semantic encoding feature vector through a point convolution layer to obtain a first convolution feature vector; passing the first convolution feature vector through a rectified linear unit based on a ReLU function to obtain a first corrected convolution feature vector; passing the first corrected convolution feature vector through a point convolution layer to obtain a second convolution feature vector; passing the second convolution feature vector through a rectified linear unit based on a Sigmoid function to obtain a second corrected convolution feature vector; fusing the second corrected convolution feature vector with the enhanced first image semantic feature map to obtain the first image semantic fusion feature map under the guidance of the operation instruction semantics.

[0013] In the above-mentioned cloud game data processing method, the second image data is generated based on the first image semantic fusion feature map under the guidance of the operation instruction semantics, including: inputting the first image semantic fusion feature map under the guidance of the operation instruction semantics into an image generator based on a diffusion model to obtain the second image data.

[0014] According to another aspect of the present application, a cloud gaming data processing system is provided, which includes: an operation instruction semantic encoding module, used to semantically encode a first operation instruction transmitted by a first terminal device to obtain a first operation instruction semantic encoding feature vector; an image feature extraction module, used to extract image features of first image data transmitted by the first terminal device to obtain a first image semantic feature map; an image feature attention enhancement module, used to enhance feature attention of the first image semantic feature map to obtain an enhanced first image semantic feature map; a cross-modal joint encoding module, used to input the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a cross-modal joint encoder based on MetaNet to obtain a first image semantic fusion feature map under the guidance of operation instruction semantics; a game screen update module, used to generate the second image data based on the first image semantic fusion feature map under the guidance of the operation instruction semantics.

[0015] According to another aspect of the present application, a readable storage medium is provided, characterized in that a cloud game data processing program is stored on the readable storage medium, and when the cloud game data processing program is executed by a processor, the cloud game data processing method as described above is implemented.

[0016] Compared with the prior art, the cloud game data processing method, system and readable storage medium provided by the present application use artificial intelligence technology based on deep learning to perform data analysis on the first operation instruction and the first image data, respectively mine the semantic feature representation of the first operation instruction and the first image data, and then realize the cross-modal semantic union of the first operation instruction and the first image data based on the meta-learning mechanism, and generate the second image data. In this way, the player's operation intention can be understood more accurately and dynamically integrated into the generation process of the game screen, thereby improving the response speed of the cloud game and the accuracy of image generation, providing players with a smoother and more natural game experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 This is a flowchart of processing the first image data based on the first operation instruction to obtain the second image data in the cloud game data processing method according to an embodiment of the present application.

[0019] Figure 2 This is a schematic diagram of an architecture for processing the first image data based on the first operation instruction to obtain the second image data in a cloud gaming data processing method according to an embodiment of the present application.

[0020] Figure 3 A flowchart of inputting the first image semantic feature map into a feature attention selection enhancement module based on a compression-suppression structure in a cloud game data processing method according to an embodiment of the present application to obtain the enhanced first image semantic feature map.

[0021] Figure 4 The present invention provides a flowchart of inputting the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a MetaNet-based cross-modal joint encoder in the cloud game data processing method according to an embodiment of the present application to obtain the first image semantic fusion feature map under the guidance of the operation instruction semantics.

[0022] Figure 5 4 is a block diagram of a cloud gaming data processing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0024] The present application provides a cloud game data processing method, which includes: transmitting first image data and a first operation instruction to a cloud server via a first terminal device; on the cloud server, based on the first operation instruction, processing the first image data to obtain second image data; the cloud server transmits the second image data to the first terminal device; on the first terminal device, displaying a game screen based on the second image data.

[0025] That is, the first image data and operation instructions generated by the player during the game are transmitted to the cloud server in real time through the first terminal device (such as a smart phone, tablet computer or TV box, etc.). After receiving the data on the cloud server side, the first image data is immediately processed according to the first operation instruction sent by the player to simulate the real-time changes of the game world under the player's operation, and an updated game screen, i.e., the second image data, is generated. Subsequently, the cloud server quickly transmits the generated second image data back to the first terminal device. After the first terminal device receives the second image data, it immediately displays the updated game screen on the screen, so that the player can see his operations and results in the game in real time. This closed-loop process not only greatly enhances the interactivity and immersion of the game, but also makes cloud gaming a new form of entertainment that can be flexibly run on various devices.

[0026] More specifically, on the cloud server, based on the first operation instruction, the first image data is processed to obtain the second image data. Traditional data processing methods often focus on simple data transmission and rendering, and lack in-depth understanding and analysis of user operation instructions and game screen content, which to a certain extent limits the interactivity and intelligence level of cloud games.

[0027] In response to the above technical problems, the technical concept of this application is to use artificial intelligence technology based on deep learning to perform data analysis on the first operation instruction and the first image data, respectively mine the semantic feature representations of the first operation instruction and the first image data, and then realize the cross-modal semantic union of the first operation instruction and the first image data based on the meta-learning mechanism, and generate the second image data. In this way, the player's operation intention can be understood more accurately and dynamically integrated into the generation process of the game screen, thereby improving the response speed of cloud games and the accuracy of image generation, providing players with a smoother and more natural gaming experience.

[0028] Figure 1 This is a flowchart of processing the first image data based on the first operation instruction to obtain the second image data in the cloud game data processing method according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of an architecture for processing the first image data based on the first operation instruction to obtain the second image data in a cloud game data processing method according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the cloud game data processing method of the embodiment of the present application, the steps include: S110, semantically encoding the first operation instruction to obtain a first operation instruction semantic encoding feature vector; S120, performing image feature extraction on the first image data to obtain a first image semantic feature map; S130, performing feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map; S140, inputting the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a MetaNet-based cross-modal joint encoder to obtain a first image semantic fusion feature map under the guidance of the operation instruction semantics; S150, generating the second image data based on the first image semantic fusion feature map under the guidance of the operation instruction semantics.

[0029] In the above-mentioned cloud game data processing method, the step S110 semantically encodes the first operation instruction to obtain the semantic encoding feature vector of the first operation instruction. It should be understood that, considering that the first operation instruction may contain various complex player actions, such as character movement, attack, use of props, etc., directly processing the original operation instruction (such as key input, mouse movement, etc.) may not be able to fully capture the intention or complex logic behind it. Therefore, in order to effectively understand the player's intention, in the technical solution of the present application, the first operation instruction is further semantically encoded to convert it into a higher-level feature vector representation with clear semantic meaning, so that the meaning of the first operation instruction can be more accurately expressed, and then a response that is more in line with the player's expectations can be made in the game environment. In an embodiment of the present application, a pre-trained deep neural network model such as Transformer or BERT can be used to encode the first operation instruction to learn the context information and potential semantic structure of the operation instruction, thereby generating a semantic encoding feature vector of the first operation instruction with rich semantic information.

[0030] In the above-mentioned cloud game data processing method, the step S120 performs image feature extraction on the first image data to obtain a first image semantic feature map. In a specific example of the present application, the processing method of performing image feature extraction on the first image data to obtain the first image semantic feature map is to input the first image data into an image feature extractor based on a hole convolutional neural network model to obtain the first image semantic feature map. It should be understood that considering that the game screen usually contains a large amount of complex detail information, an efficient feature extraction method is required to ensure the realism and smoothness of the screen. Therefore, in the technical solution of the present application, a hole convolutional neural network (Dilated Convolutional Neural Network, DCNN) model that performs well in the field of image processing is used to perform deep feature extraction on the first image data. Among them, the hole convolution can expand the receptive field at a lower computational cost, thereby capturing the long-range dependency and contextual information in the image while maintaining a high spatial resolution. In traditional image processing algorithms, a pooling layer is often set to indirectly increase the receptive field, but a large amount of image information is lost in this process. The atrous convolution introduces holes (i.e., zero padding) to increase the receptive field without performing pooling operations, thereby reducing information loss, retaining more image details, and providing more accurate input for generating high-quality game graphics.

[0031] In the above-mentioned cloud game data processing method, the step S130 performs feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map. In a specific example of the present application, the processing method of performing feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map is to input the first image semantic feature map into a feature attention selection enhancement module based on a compression-suppression structure to obtain the enhanced first image semantic feature map. It should be understood that, considering that in the game screen, different elements have different response importance to player operations, for example, player characters, enemies, props, etc. may be the key focus of attention, while the background or other non-critical elements are relatively minor. Therefore, in order to pay more attention to and respond to player operations more accurately, in the technical solution of the present application, a feature attention selection enhancement module based on a compression-suppression structure is introduced to enhance the attention of the first image semantic feature map to highlight the key information related to the player's operation, while suppressing irrelevant or minor features, thereby improving the pertinence and efficiency of image processing. Specifically, the feature attention selection and enhancement module learns the associated features of the channel dimension of the first image semantic feature map, generates attention weights based on the correlation between each feature channel, and adaptively redistributes the original image features, which helps to more accurately capture the core information related to the player's operation and improve the accuracy of subsequent processing.

[0032] Figure 3 A flowchart of inputting the first image semantic feature map into a feature attention selection enhancement module based on a compression-suppression structure to obtain the enhanced first image semantic feature map in a cloud game data processing method according to an embodiment of the present application. Figure 3 As shown, the step S130 includes: S131, calculating the global mean of each feature matrix along the channel dimension of the first image semantic feature map to obtain a first image semantic compression information representation feature vector; S132, performing one-dimensional convolution encoding on the first image semantic compression information representation feature vector to obtain a first image semantic compression information association representation feature vector; S133, cascading the first image semantic compression information representation feature vector and the first image semantic compression information association representation feature vector to obtain a first image semantic feature compression information multi-scale representation vector; S134, inputting the first image semantic feature compression information multi-scale representation vector into a compression information feature extraction module including a multi-layer perceptron and a SiLU activation function to obtain a first image semantic feature compression information multi-scale association feature vector; S135, using a Sigmoid function to perform a normalization operation on the first image semantic feature compression information multi-scale association feature vector to obtain a first image semantic feature weight vector; S136, based on the first image semantic feature weight vector, performing feature amplification and suppression operations on the first image semantic feature map to obtain the enhanced first image semantic feature map.

[0033] More specifically, the step S136 includes: calculating the Kronecker product of the first image semantic feature weight vector and each feature matrix of the first image semantic feature map along the channel dimension to obtain the enhanced first image semantic feature map.

[0034] That is, the step S130 includes: performing attention selection on the first image semantic feature map using the following feature enhancement formula to obtain the enhanced first image semantic feature map, wherein the feature enhancement formula is:

[0035]

[0036] V s =concat(V s1 ,Conv 1d (V s1 ))

[0037]

[0038] in, is the first image semantic feature map x o The coordinates in the channel are (i o , jo ), H and W are the height and width of the first image semantic feature map, respectively. is the xth feature vector representing the semantic compression information of the first image o The characteristic value, V s1 is the feature vector representing the semantic compression information of the first image, Conv 1d (·) represents one-dimensional convolutional coding, concat(·,·) represents cascade, V s is the multi-scale representation vector of the compressed information of the semantic feature of the first image, MLP(·) represents a multi-layer perceptron, sigmoid represents a sigmoid function, F is the semantic feature map of the first image, represents the Kronecker product of the calculation vector and each feature matrix of the feature map along the channel dimension, and F' is the enhanced first image semantic feature map.

[0039] In the above-mentioned cloud game data processing method, in step S140, the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map are input into a cross-modal joint encoder based on MetaNet to obtain the first image semantic fusion feature map under the guidance of the operation instruction semantics. It should be understood that the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map represent the player's intention and the current state of the game screen respectively, but because the two belong to different modes, direct fusion may have a semantic gap. To this end, in the technical solution of the present application, a cross-modal joint encoder with a meta-learning mechanism is used to jointly encode the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map to establish an effective semantic association between the operation instruction semantic features and the image semantic features. Specifically, the MetaNet model can learn transferable knowledge between different modalities, and will adapt to the joint representation of different operation instructions and image features by dynamically adjusting its internal parameters, and can convert the first operation instruction semantic encoding feature vector into a form comparable and united with the image data at the semantic level, and embed and fuse it into the enhanced first image semantic feature map, thereby realizing cross-modal semantic fusion.

[0040] Figure 4 The present invention is a flow chart of inputting the first operation instruction semantic encoding feature vector and the first enhanced image semantic feature map into a cross-modal joint encoder based on MetaNet to obtain the first image semantic fusion feature map under the guidance of the operation instruction semantics in the cloud game data processing method according to an embodiment of the present application. Figure 4As shown, the step S140 includes: S141, passing the operation instruction semantic encoding feature vector through a point convolution layer to obtain a first convolution feature vector; S142, passing the first convolution feature vector through a rectified linear unit based on a ReLU function to obtain a first corrected convolution feature vector; S143, passing the first corrected convolution feature vector through a point convolution layer to obtain a second convolution feature vector; S144, passing the second convolution feature vector through a rectified linear unit based on a Sigmoid function to obtain a second corrected convolution feature vector; S145, fusing the second corrected convolution feature vector with the enhanced first image semantic feature map to obtain the first image semantic fusion feature map under the guidance of the operation instruction semantics.

[0041] In the above-mentioned cloud game data processing method, the step S150 generates the second image data based on the semantic fusion feature map of the first image under the guidance of the semantics of the operation instruction. In a specific example of the present application, the implementation method of generating the second image data based on the semantic fusion feature map of the first image under the guidance of the semantics of the operation instruction is to input the semantic fusion feature map of the first image under the guidance of the semantics of the operation instruction into an image generator based on a diffusion model to obtain the second image data. It should be understood that in order to ensure that the generated image not only meets the player's operation intention but also maintains the coherence and authenticity of the game screen, an image generator based on a diffusion model is selected in the technical solution of the present application. The diffusion model has shown significant advantages in the field of game screen generation with its powerful generation ability and precise control of image details. Specifically, the diffusion model can gradually restore a clear image from random noise by simulating the diffusion and denoising process in the image generation process, and after multiple rounds of iterations and optimization, it can generate high-quality second image data that meets the player's operation intention.

[0042] In particular, in the above scheme, the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map respectively express the semantic encoding features of the first operation instruction and the image semantic features of the first image data based on feature attention selective weighted enhancement. In this way, after the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map are input into the MetaNet-based cross-modal joint encoder, the first image semantic fusion feature map obtained under the guidance of the operation instruction semantics will have an imbalance in the feature aggregation generation of image semantic domain regression representation due to the difference in the image semantic feature distribution constraints of the feature matrix dimension and the semantic encoding feature distribution constraints of the channel dimension. After the feature regression representation is aggregated through clustering operation, the regression convergence logic of the image semantic generation probability density distribution of in-class features and out-of-class features will be inconsistent.

[0043] Therefore, in a preferred embodiment of the present application, after performing a clustering operation based on the distance between eigenvalues ​​on the first image semantic fusion feature map under the guidance of the semantics of the operation instruction, further optimization of in-class features and out-of-class features is performed, including the following steps: determining the number of eigenvalues ​​in the clustering set obtained by the clustering operation of the first image semantic fusion feature map under the guidance of the semantics of the operation instruction, that is, the number of in-class features, and subtracting the number of in-class features from the total number of eigenvalues ​​of the first image semantic fusion feature map under the guidance of the semantics of the operation instruction to obtain the number of out-of-class features; dividing the total number of eigenvalues ​​by the number of in-class features and the number of out-of-class features respectively to obtain a class importance value and an out-of-class proportion value, and calculating the inverse of the class importance value to obtain a class constraint value; calculating the operation Each eigenvalue of the first image semantic fusion feature map under the guidance of operation instruction semantics is a power function with the class importance value as the exponent, and is added to the exponential value with the natural function as the base and the class constraint value as the exponent, and then multiplied by the out-of-class proportion value to obtain the first image semantic fusion modulation feature map under the guidance of operation instruction semantics; the first image semantic fusion feature map under the guidance of operation instruction semantics is dot-multiplied with the class importance value to obtain the first image semantic fusion ontology feature map under the guidance of operation instruction semantics; the weighted sum of the first image semantic fusion modulation feature map under the guidance of operation instruction semantics and the first image semantic fusion ontology feature map under the guidance of operation instruction semantics is calculated with the weight hyperparameter to obtain the optimized first image semantic fusion feature map under the guidance of operation instruction semantics.

[0044] That is, the feature distribution of the first image semantic fusion feature map under the guidance of the operation instruction semantics is optimized based on the in-class features and out-of-class features to obtain the optimized first image semantic fusion feature map under the guidance of the operation instruction semantics, wherein the optimization formula is:

[0045]

[0046] Wherein, F represents the semantic fusion feature map of the first image under the guidance of the semantics of the operation instruction, n represents the total number of feature values ​​of the semantic fusion feature map of the first image under the guidance of the semantics of the operation instruction, k represents the number of in-class features, nk represents the number of out-of-class features, represents the out-of-class proportion value, represents the class importance value, represents the class constraint value, β represents the weight hyperparameter, It represents the power function value of calculating each eigenvalue of the first image semantic fusion feature map under the guidance of the semantics of the operation instruction with the class importance value as the exponent, ⊙ represents the dot product, and F' represents the optimized semantic fusion feature map of the first image under the guidance of the semantics of the operation instruction.

[0047] Here, in the technical solution of the present application, while performing a clustering operation on the first image semantic fusion feature map under the guidance of the operation instruction semantics, the clustering importance measure of the feature value of the first image semantic fusion feature map under the guidance of the operation instruction semantics and the regression constraint measure of the clustering operation are combined to further perform responsive modulation with the out-of-class scale factor, and based on the clustering scaling of the feature ontology representation of the first image semantic fusion feature map under the guidance of the operation instruction semantics, the simplicity and effectiveness of the feature purification of the overall feature set of the first image semantic fusion feature map under the guidance of the operation instruction semantics is established by the clustering operation, and the inconsistency of the regression convergence logic of the image semantic generation probability density distribution caused by the clustering operation is suppressed, so as to improve the image semantic generation regression iteration effect of the image generator based on the diffusion model input of the feature set of the first image semantic fusion feature map under the guidance of the operation instruction semantics, that is, to improve the speed of generation training and the image quality of the obtained second image data.

[0048] In summary, the cloud game data processing method according to the embodiment of the present application is explained, which uses artificial intelligence technology based on deep learning to perform data analysis on the first operation instruction and the first image data, respectively mines the semantic feature representation of the first operation instruction and the first image data, and then realizes the cross-modal semantic union of the first operation instruction and the first image data based on the meta-learning mechanism, and generates the second image data. In this way, the player's operation intention can be understood more accurately and dynamically integrated into the generation process of the game screen, thereby improving the response speed of the cloud game and the accuracy of image generation, providing players with a smoother and more natural game experience.

[0049] Figure 5 is a block diagram of a cloud gaming data processing system according to an embodiment of the present application. Figure 5 As shown, according to the cloud gaming data processing system 100 of the embodiment of the present application, it includes: an operation instruction semantic encoding module 110, which is used to semantically encode the first operation instruction transmitted by the first terminal device to obtain a first operation instruction semantic encoding feature vector; an image feature extraction module 120, which is used to extract image features of the first image data transmitted by the first terminal device to obtain a first image semantic feature map; an image feature attention enhancement module 130, which is used to enhance the feature attention of the first image semantic feature map to obtain an enhanced first image semantic feature map; a cross-modal joint encoding module 140, which is used to input the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a cross-modal joint encoder based on MetaNet to obtain a first image semantic fusion feature map under the guidance of the operation instruction semantics; a game screen update module 150, which is used to generate the second image data based on the first image semantic fusion feature map under the guidance of the operation instruction semantics.

[0050] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned cloud game data processing system have been referred to above. Figures 1 to 4 The description of the cloud gaming data processing method has been introduced in detail, and therefore, its repeated description will be omitted.

[0051] Exemplary computer program products and computer-readable storage media

[0052] In addition to the above-mentioned methods and systems, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the cloud gaming data processing method according to various embodiments of the present application described in the above-mentioned exemplary method section of this specification.

[0053] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0054] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the cloud gaming data processing method according to various embodiments of the present application described in the above exemplary method section of this specification.

[0055] The computer readable storage medium can adopt any combination of one or more readable storage media. The readable storage medium can be a readable signal storage medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0056] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A cloud game data processing method, characterized in that: include: Transmitting the first image data and the first operation instruction to the cloud server through the first terminal device; On the cloud server, based on the first operation instruction, processing the first image data to obtain second image data; The cloud server transmits the second image data to the first terminal device; On the first terminal device, displaying a game screen according to the second image data; Wherein, in the cloud server, based on the first operation instruction, processing the first image data to obtain second image data includes: Performing semantic encoding on the first operation instruction to obtain a first operation instruction semantic encoding feature vector; Performing image feature extraction on the first image data to obtain a first image semantic feature map; Performing feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map; Inputting the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a cross-modal joint encoder based on MetaNet to obtain a first image semantic fusion feature map guided by the operation instruction semantics; The second image data is generated based on the semantic fusion feature map of the first image under the guidance of the semantics of the operation instruction.

2. The cloud game data processing method according to claim 1, characterized in that: Extracting image features from the first image data to obtain a first image semantic feature map includes: The first image data is input into an image feature extractor based on a hole convolutional neural network model to obtain the first image semantic feature map.

3. The cloud game data processing method according to claim 2, characterized in that: Performing feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map includes: The first image semantic feature map is input into a feature attention selection enhancement module based on a compression-suppression structure to obtain the enhanced first image semantic feature map.

4. The cloud game data processing method according to claim 3, characterized in that: Inputting the first image semantic feature map into a feature attention selection enhancement module based on a compression-suppression structure to obtain the enhanced first image semantic feature map, comprising: Calculating the global mean of each feature matrix along the channel dimension of the first image semantic feature map to obtain a first image semantic compression information representation feature vector; Performing one-dimensional convolution encoding on the first image semantic compression information representation feature vector to obtain a first image semantic compression information association representation feature vector; Cascading the first image semantic compression information representation feature vector and the first image semantic compression information association representation feature vector to obtain a first image semantic feature compression information multi-scale representation vector; Inputting the first image semantic feature compression information multi-scale representation vector into a compression information feature extraction module including a multi-layer perceptron and a SiLU activation function to obtain a first image semantic feature compression information multi-scale correlation feature vector; Using a Sigmoid function to normalize the multi-scale associated feature vector of the first image semantic feature compression information to obtain a first image semantic feature weight vector; Based on the first image semantic feature weight vector, feature amplification and suppression operations are performed on the first image semantic feature map to obtain the enhanced first image semantic feature map.

5. The cloud game data processing method according to claim 4, characterized in that: Based on the first image semantic feature weight vector, performing feature amplification and suppression operations on the first image semantic feature map to obtain the enhanced first image semantic feature map, including: The Kronecker product of the first image semantic feature weight vector and each feature matrix of the first image semantic feature map along the channel dimension is calculated to obtain the enhanced first image semantic feature map.

6. The cloud game data processing method according to claim 5, characterized in that: Inputting the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a cross-modal joint encoder based on MetaNet to obtain a first image semantic fusion feature map guided by the operation instruction semantics, including: Passing the operation instruction semantic encoding feature vector through a point convolution layer to obtain a first convolution feature vector; Passing the first convolution feature vector through a rectified linear unit based on a ReLU function to obtain a first rectified convolution feature vector; Passing the first modified convolutional feature vector through a point convolution layer to obtain a second convolutional feature vector; Passing the second convolution feature vector through a rectified linear unit based on a Sigmoid function to obtain a second rectified convolution feature vector; The second corrected convolution feature vector is fused with the enhanced first image semantic feature map to obtain the first image semantic fusion feature map under the guidance of the operation instruction semantics.

7. The cloud game data processing method according to claim 6, characterized in that: Generating the second image data based on the semantic fusion feature map of the first image under the guidance of the semantics of the operation instruction includes: The semantic fusion feature map of the first image guided by the semantics of the operation instruction is input into an image generator based on a diffusion model to obtain the second image data.

8. A cloud game data processing system, characterized in that: include: An operation instruction semantic coding module, used for semantically coding a first operation instruction transmitted by a first terminal device to obtain a semantic coding feature vector of the first operation instruction; An image feature extraction module, used to extract image features from the first image data transmitted by the first terminal device to obtain a first image semantic feature map; An image feature attention enhancement module, used for performing feature attention enhancement on the first image semantic feature map to obtain an enhanced first image semantic feature map; A cross-modal joint encoding module, used for inputting the first operation instruction semantic encoding feature vector and the enhanced first image semantic feature map into a cross-modal joint encoder based on MetaNet to obtain a first image semantic fusion feature map under the guidance of operation instruction semantics; The game screen update module is used to generate the second image data based on the first image semantic fusion feature map under the guidance of the operation instruction semantics.

9. A readable storage medium, characterized in that: The readable storage medium stores a cloud gaming data processing program, and when the cloud gaming data processing program is executed by a processor, the cloud gaming data processing method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Cloud game realizing method and device, cloud server, terminal and medium

    CN110368686A

  • Translation method and device based on multi-modal machine learning, equipment and storage medium

    CN111597830A

  • Cloud game control method and device, equipment and storage medium

    CN114146405A

  • Image semantic segmentation method based on dynamic convolution attention

    CN116129115A

  • Cloud game operation control method and device, computer equipment and storage medium

    CN116764200A