Cloud game virtual human dialogue method and system based on multi-modal model

Abstract

The invention discloses a cloud game virtual human dialogue method and system based on a multi-modal model. The method comprises the following steps: constructing a multi-modal sample data set; acquiring embedded vectors corresponding to all the data by using a multi-modal encoder, updating the data set based on the embedded vectors and preset markers, and training the large language model to obtain a game response reasoning model; collecting real-time multi-modal data, collecting a corresponding real-time knowledge base, and obtaining a real-time task prompt set by utilizing the pre-trained SyncTOD model, the real-time multi-modal data and the real-time knowledge base; calculating the similarity between a historical task prompt set and a real-time task prompt set of the historical task response data by utilizing a similarity method, and selecting first mu samples with relatively high similarity from the historical task response data; and optimizing the output of the game response reasoning model based on the real-time task prompt set and the mu samples, and returning the optimized response to the user. According to the invention, the output consistency and the multi-modal interaction effect of the cloud game system can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud gaming, and in particular relates to a cloud gaming virtual human dialogue method and system based on a multimodal model. Background Art

[0002] With the rapid development of cloud computing technology, cloud gaming, as a streaming interactive system that can run online and avoid running on the edge, is becoming more and more popular among users. The characteristic of cloud gaming is that it processes complex calculations and rendering through cloud servers and transmits the results to user devices in the form of streams, thereby reducing the computing power requirements of edge devices.

[0003] However, existing cloud gaming systems often have the following problems in rendering and multimodal output when interacting with users: 1. Rendering is inconsistent with multimodal output. For example, information in different modes such as images, sounds, and text cannot be coordinated, affecting the user's gaming experience; 2. Existing systems cannot efficiently process image, sound, and text input from users, affecting the quality of multimodal output; 3. Cloud gaming rendering is separated from multimodal output, making it difficult to achieve efficient real-time interaction. Summary of the invention

[0004] In view of the above problems, the present invention proposes a cloud game virtual human dialogue method and system based on a multimodal model, which not only improves the multimodal interaction effect of cloud games, but also realizes the efficient deployment and operation of the system, providing strong support for the further development of cloud games. In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0005] A cloud game virtual human dialogue system based on a multimodal model includes the following steps:

[0006] S1, collects multimodal data when cloud games execute tasks and constructs a multimodal sample dataset;

[0007] S2, using a multimodal encoder to obtain an embedding vector corresponding to all data in the multimodal sample dataset, updating the multimodal sample dataset based on the embedding vector and the preset marker, and using the updated multimodal sample dataset to train the large language model to obtain a game response inference model;

[0008] S3, collects the real-time multimodal data involved in the current game task, collects the real-time knowledge base corresponding to the game task, and uses the pre-trained SyncTOD model, real-time multimodal data and real-time knowledge base to obtain the real-time task prompt set;

[0009] S4, using a similarity calculation method to calculate the similarity between the historical task prompt set and the real-time task prompt set of the historical task response data, and selecting the first μ samples with higher similarity from the historical task response data;

[0010] S5, based on the real-time task prompt set obtained in step S3 and the samples selected in step S4, the response output by the game response inference model obtained in step S2 is optimized, and the optimized response is transmitted back to the user.

[0011] The step S2 comprises the following steps:

[0012] S2.1, using the high-resolution input module (GLHR) of the MammothModa model to process the image data in the multimodal sample dataset to obtain the image fusion vector;

[0013] S2.2, constructing a modal projector, wherein the modal projector includes a first linear layer, an activation function layer, a first Dropout layer, a second linear layer, and a second Dropout layer connected in sequence;

[0014] S2.3, inputting the image fusion vector obtained in step S2.1 and the sound data in the multimodal sample data set into the multimodal encoder for encoding, inputting the encoded vector into the modal projector to obtain the corresponding embedding vector, and obtaining the corresponding embedding vector based on the text data in the multimodal sample data set;

[0015] S2.4, updating the multimodal sample data set obtained in step S1 based on the preset marker and the embedding vector obtained in step S2.3;

[0016] S2.5, using the updated multimodal sample data set obtained in step S2.4 to train the large language model to obtain a game response inference model.

[0017] The method for obtaining the image fusion vector is as follows:

[0018] a. Adjust the image size of the image data in the multimodal sample dataset so that the size of each frame image is an integer multiple of 336;

[0019] b, divide each frame image into 336*336 image blocks;

[0020] c. Downsample each frame of the image adjusted in step a to obtain a 336*336 global image, and concatenate the image vector of the global image with the image block vector of the corresponding frame obtained in step b to obtain an image fusion vector.

[0021] The step S3 comprises the following steps:

[0022] S3.1, obtaining real-time multimodal data during the current game task dialogue, and processing the real-time multimodal data according to the methods of step S1 and step S2 to obtain a real-time multimodal data set;

[0023] S3.2, updating the real-time multimodal dataset according to the method of steps S2.1-S2.4, and collecting the real-time knowledge base corresponding to the current game task based on the real-time multimodal dataset;

[0024] S3.3, using the entity prediction model, the closing prediction model, and the response prediction model included in the SyncTOD model to obtain the real-time task prompt set corresponding to the real-time multimodal dataset.

[0025] The step S4 comprises the following steps:

[0026] S4.1, concatenating the real-time multimodal data obtained in step S3 with the corresponding real-time knowledge base to obtain an operation representation vector;

[0027] S4.2, obtaining the operation representation vector of each sample in the historical task response data D according to the method of step S4.1, and selecting the top k samples with high similarity to the operation representation vector obtained in step S4.1 by using the maximum inner product search algorithm;

[0028] S4.3, input the first k samples obtained in step S4.2 into the SyncTOD model to obtain the historical task prompt set, calculate the similarity between the historical task prompt set and the real-time task prompt set, select the first μ samples with higher similarity, and μ <k。

[0029] The calculation formula of the similarity is:

[0030]

[0031] In the formula, H m represents the historical task prompt set of the mth sample, dc m represents the closing prediction of the mth sample, et m represents the entity type of the mth sample, Represents the historical task prompt set H m and real-time task reminder set The similarity between them, 1 is the indicator function, and J is the Jaccard similarity.

[0032] A cloud game virtual human dialogue system based on a multimodal model, comprising:

[0033] Data collection module: used to build a multimodal sample dataset based on the multimodal data of players using cloud games;

[0034] Model building module: used to obtain the corresponding embedding vector based on the multimodal sample data set obtained by the data collection module and the preset multimodal encoder, update the multimodal sample data set based on the embedding vector and the preset marker, and use the updated multimodal sample data set to train the large language model to obtain the game response inference model;

[0035] Real-time task information collection module: used to collect real-time multimodal data under the current game task, collect the real-time knowledge base corresponding to the game task, and use the pre-trained SyncTOD model, real-time multimodal data and real-time knowledge base to obtain the real-time task prompt set;

[0036] Sample selection module: used to obtain the historical task prompt set corresponding to the historical task response data based on the pre-trained SyncTOD model, calculate the similarity between the historical task prompt set and the real-time task prompt set using the similarity calculation method, and select the first μ samples with higher similarity from the historical task response data;

[0037] Response optimization module: used to optimize the response based on the samples selected by the sample selection module, the real-time task prompt set obtained by the real-time task information collection module and the game response reasoning model of the model construction module, and send the optimized response back to the user.

[0038] Beneficial effects of the present invention:

[0039] 1. Combining the MammothModa multimodal large model with the S3 system framework, and using the contextual learning and prompt mechanism SyncTOD, we have achieved high-quality unified processing of multimodal information such as images, voices, and texts, improved the output consistency and multimodal interaction effect of the cloud gaming system, solved the problem of inconsistent multimodal output in the existing system, and ultimately improved the user's immersive experience;

[0040] 2. The SyncTOD small auxiliary model is introduced to provide prompts. The prompt mechanism selects the best context prompts, which improves the system's ability to respond in different situations and improves the output quality and coherence of multimodal output;

[0041] 3. It is mainly used in the field of cloud game NPC and virtual humans. Through edge deployment, it can utilize the hardware support of cloud L20 and other graphics cards to achieve real-time image, sound, text and other interactions, ensuring the efficient operation of the virtual human dialogue system, realizing the business implementation of the virtual human dialogue system, and meeting the business needs of cloud game virtual humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A sample diagram of a multimodal sample dataset;

[0044] Figure 2 It is a structural schematic diagram of the modal projector;

[0045] Figure 3 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Multilayer Perceptron (MLP): is a feedforward neural network and one of the basic forms of neural networks. It consists of an input layer, one or more hidden layers, and an output layer, each of which consists of many neurons (also called nodes or units).

[0048] S3: From the paper S3: A SIMPLE STRONG SAMPLE-EFFECTIVE MULTIMODAL DIALOG SYSTEM, it is an efficient multimodal dialogue system that focuses on achieving powerful dialogue generation capabilities through a simple architecture. It can process and generate dialogue content based on multiple modalities such as text, images, and voice.

[0049] MammothModa: From the paper MammothModa:MULTI-MODAL LARGE LANGUAGE MODEL is a large multimodal language model that can understand and generate across multiple modalities. It performs well in processing and generating complex multimodal data.

[0050] SyncTOD: From the paper Synergizing In-context Learning with Hints for End-to-end Task-oriented Dialog Systems, it is a task-oriented dialogue system that optimizes the dialogue generation process by combining context learning and hint mechanisms. It can use hints and context information in end-to-end dialogue generation to improve the accuracy and coherence of the dialogue.

[0051] A cloud game virtual human dialogue method based on a multimodal model, such as Figure 3 As shown, the following steps are included:

[0052] S1, collects multimodal data when running cloud games and constructs a multimodal sample dataset in JSON format, including the following steps:

[0053] S1.1, collects multimodal data of users’ conversations after logging into the cloud game;

[0054] The multimodal data includes at least two of image data, sound data, and text data. In cloud games, image data generally includes scene maps, player perspectives, NPC (Non-Player Character) images, etc. in the game, sound data generally includes background music in the game, NPC dialogue audio, player voice input, etc., and text data generally includes text entered by the player, game plot dialogue, system prompts, etc. The above data can be collected through tools such as log4j, and the collected data can be stored in the database.

[0055] S1.2, preprocessing the multimodal data obtained in step S1.1 to construct a multimodal sample data set;

[0056] Specifically, data cleaning is first performed to remove noise and abnormal data such as incomplete audio clips, blurred images, invalid text input, etc. Then, sample construction is performed to synchronize image, sound, and text data according to timestamps to construct a multimodal sample dataset.

[0057] S1.3, converting the multimodal sample dataset obtained in step S1.2 into JSON format data;

[0058] Specifically, each data in the multimodal sample dataset is represented as a JSON object, which contains a "role" (used to indicate whether the information is from a user or a robot), a "type" (used to indicate whether the information contains an image, audio, or text), and a message content (when the data is an image or audio, the message content displays its file path; when the data is text, the message content directly displays the text data). Our chat layout allows media content to be inserted at any time in the conversation (before or after text content), and it may be multiple times (multiple images or audio). Figure 1 As shown, a data sample corresponding to a conversation in the converted multimodal sample data set is shown, and the data sample contains a total of 4 messages.

[0059] S2, using a multimodal encoder to obtain an embedding vector corresponding to all data in the multimodal sample dataset, updating the multimodal sample dataset based on the embedding vector and the preset marker, and using the updated multimodal sample dataset to train the large language model to obtain a game response inference model, including the following steps:

[0060] S2.1, using the high-resolution input module (GLHR) of the MammothModa model to process the image data in the multimodal sample dataset to obtain the image fusion vector;

[0061] This application combines MammothModa's High-Resolution Input Module (GLHR) to dynamically split the input image into manageable small blocks to achieve flexible fine-grained visual understanding. Specifically, it includes the following steps:

[0062] a. Adjust the image size of the image data in the multimodal sample dataset so that the size of each frame image is an integer multiple of 336;

[0063] b, divide each frame image into 336*336 image blocks;

[0064] For example, given an image of (h,w), where h is the height of the image and w is the width of the image. First resize the image and pad it to (p h ×336,p w ×336), where

[0065] c. Downsample each frame of the image adjusted in step a to obtain a 336*336 global image, and concatenate the image vector of the global image with the image block vector of the corresponding frame obtained in step b to obtain an image fusion vector;

[0066] For each image, a global view is obtained by directly resizing it to a multiple of 336*336, and then multiple local views are created by dynamically splitting it into 336*336 blocks. This approach ensures that both global information and detailed local information can be preserved and used for further processing.

[0067] S2.2, constructing a modal projector, wherein the modal projector includes a first linear layer, an activation function layer, a first Dropout layer, a second linear layer, and a second Dropout layer connected in sequence;

[0068] like Figure 2 As shown in the figure, the modal projector is an MLP module consisting of two linear layers, a ReLU activation function and two Dropouts. The modal projector can be used to obtain the corresponding embedding vector (Token Embedding). The modal projector is able to convert the input data into a Token Embedding of only 4 tokens. For example, if the dimension of the Token Embedding of the large language model is 32, the output projection of the modal projector can be set to produce a 128-dimensional Embedding, which is then divided into 4 separate Token Embeddings. It is assumed that a small number of output modal tokens are sufficient for basic visual understanding. In addition, using only 4 tokens can significantly reduce the sequence length passed to the Transformer of the large language model.

[0069] S2.3, inputting the image fusion vector obtained in step S2.1 and the sound data in the multimodal sample data set into the multimodal encoder for encoding, inputting the encoded vector into the modal projector to obtain the corresponding embedding vector, and obtaining the corresponding embedding vector based on the text data in the multimodal sample data set;

[0070] For the image fusion vector, this application chooses CLIP as the encoder. Image encoders such as CLIP usually aggregate the outputs of processed modal objects through pooling to produce a single final embedding vector. But for our task, we use separate Embedding vectors respectively. For the audio data in the multimodal sample dataset, the ImageBind multimodal encoder can be used for processing, which can process various modalities including audio, images, and videos. This application uses a simple MLP module to map the Embeddings corresponding to images and audio directly to the TokenEmbedding of LLM. By adjusting the Embeddings of various modal objects (such as images or audio), they can be ensured to be compatible with LLM.

[0071] S2.4, updating the multimodal sample data set obtained in step S1 based on the preset marker and the embedding vector obtained in step S2.3;

[0072] The preset markers include modal markers for indicating the start and end of different modal objects in the data, object markers for indicating different modal objects such as audio and images, subject markers for indicating the speaker of the conversation, and message markers for indicating the start and end of each message in the cloud game conversation.

[0073] This embodiment uses the modal tags [M] and [ / M] to mark the start and end of different modal objects, respectively. The object tag [audio] is used to represent the audio object, and the object tag [img] is used to represent the image object. When the data is text, the text object is not updated. The subject tag [user] is used to represent the user, and the subject tag [bot] is used to represent the robot. [RS] is used to represent the start of each message in the conversation, and [ / RS] is used to represent the end of each message in the conversation. Based on the above rules, each sample in the dataset is processed into a string containing these special characters, as shown below:

[0074] [RS][user][M][img][img][img][img][ / M][ / RS]

[0075] [RS][user][M][audio][audio][audio][audio][ / M][ / RS]

[0076] [RS][user]Where is the red dot located on the map? [ / RS]

[0077] [RS][bot]Please follow me,and I will take you there.[ / RS]

[0078] S2.5, using the updated multimodal sample data set obtained in step S2.4 to train the large language model to obtain a game response inference model;

[0079] To strike a balance between task-specific performance and model stability, the LLM model header and embedding layers can be unfrozen during specific training. When certain layers of the model are unfrozen, the weights of these layers can be updated during training, allowing the model to better adapt to the data distribution of a specific task. For example, when performing a generative task, unfreezing the model header can help the model output more accurate results, while unfreezing the embedding layer can make the model's input feature representation closer to the requirements of the target task. However, if the entire model is unfrozen, especially those layers that have been pre-trained on a large scale, it may cause the model to overfit on a specific task or lose some general language understanding capabilities. Therefore, unfreezing only certain layers (such as the model header and embedding layer) can fine-tune the model to adapt to a specific task without significantly changing the core architecture of the model. Doing so can retain the extensive knowledge gained by the model during pre-training while fine-tuning the task-related parts. Therefore, by unfreezing only some layers, it is possible to improve the model's task performance while avoiding over-adjustment and maintaining the stability and broad adaptability of the model. In addition, during the multi-task pre-training stage, various data types including bilingual subtitles, interleaved text-image pairs, OCR, etc. can be used to enhance the fine-grained recognition and OCR capabilities of the model while reducing hallucinations.

[0080] In the supervised fine-tuning phase, the focus is on training the model to understand user intent and extract relevant information from images to provide accurate responses. The training data is the same in content and format, including multi-turn visual dialogues, general VQA, external knowledge (such as Wikipedia), etc. We adopt an image cropping strategy to increase the effective resolution and ensure detailed capture. All model parameters are open for training, and the layer-by-layer learning rate is decayed to minimize changes to the pre-trained parameters.

[0081] In specific applications, a machine with 8 L20 graphics cards can be used to achieve integrated training and pushing, which is used for both fine-tuning of multimodal models and inference processes. Fine-tuning by real-time acquisition of inference data ensures the efficiency and adaptability of the system, improves the training and inference efficiency of the system, and reduces the consumption of hardware resources. Specifically, one is used for real-time acquisition and preprocessing of inference data: this graphics card is responsible for collecting data generated during the inference process and preprocessing it, such as data cleaning, labeling or format conversion, to prepare for the subsequent fine-tuning process. One is responsible for rendering the screen of cloud games: this graphics card continues to undertake the real-time rendering task of the game screen to ensure that the user's gaming experience is not disturbed, and at the same time transmits the rendering data synchronously with other multimodal data as part of the inference data. One is responsible for the conversion of text to sound: this graphics card handles the TTS tasks required during the game process, converts text into voice output, and the data it generates will also be used as input to participate in real-time fine-tuning. Five for reasoning and fine-tuning of multimodal models: These five graphics cards process the reasoning tasks of multimodal models in parallel, including the interaction of image, sound and text data. At the same time, they are also responsible for real-time fine-tuning of the collected reasoning data and updating model parameters to adapt to the latest data distribution and user needs. This setting allows the system to continuously improve the accuracy and responsiveness of the model while maintaining the reasoning ability. It not only improves the efficiency of the system, but also enhances the flexibility and adaptability of the model, which is particularly suitable for application scenarios such as cloud games that have extremely high requirements for real-time and interactivity.

[0082] S3, collects the real-time multimodal data involved in the current game task, collects the real-time knowledge base corresponding to the game task, and uses the pre-trained SyncTOD model, real-time multimodal data and real-time knowledge base to obtain the real-time task prompt set, including the following steps:

[0083] S3.1, obtaining real-time multimodal data during the current game task dialogue, and processing the real-time multimodal data according to the methods of step S1 and step S2 to obtain a real-time multimodal data set;

[0084] This part mainly obtains the user's multimodal input (image, audio and text) in real time and converts it into a real-time multimodal dataset in the JSON file format described in step 1.3, so as to facilitate subsequent input into the model for reasoning.

[0085] S3.2, updating the real-time multimodal dataset according to the method of steps S2.1-S2.4, and collecting the real-time knowledge base corresponding to the current game task based on the real-time multimodal dataset;

[0086] S3.3, using the entity prediction model, the closing prediction model, and the response prediction model included in the SyncTOD model to obtain a real-time task prompt set corresponding to the real-time multimodal data set;

[0087] The entity prediction model, the closing prediction model, and the response prediction model are all pre-trained based on historical task response data. The historical task response data contains multiple dialogues, represented by D. c i represents the i-th dialogue, K i represents the knowledge base of the game task related to the ith dialogue, y i It represents the historical response of the cloud gaming platform to the i-th dialogue, and n represents the number of dialogues in the historical task response data, that is, the number of samples.

[0088] In the contextual learning setting, Retrieval-augmented Generation (RAG) is a popular approach. RAG retrieves the most similar samples to the current dialogue c from the historical task response data D and uses them to generate predicted responses. By retrieving historical dialogue samples similar to the current dialogue context, the system can learn from the effective dialogue strategies in these samples to improve the accuracy and naturalness of responses. For example, when a player asks how to complete a task, the system can retrieve dialogue records of similar tasks and generate corresponding answers.

[0089] SyncTOD is a system that combines LLM contextual learning with prompts, which combines retrieval-augmented generation (RAG) technology to better align with NPC utterances in historical task response data D. The SyncTOD auxiliary model plays a key role in the system, which can generate a series of task-related prompts based on the player's dialogue history and game task status. These prompts include but are not limited to the expected entity type (such as the item the player is looking for or the type of NPC that needs to be interacted with), the expected response length (adjusted according to the depth and complexity of the dialogue), and the timing of the end of the dialogue (to ensure the fluency of the dialogue and the satisfaction of the player).

[0090] SyncTOD uses three types of prompts: entity type, response length, and dialogue end. These guide LLM to generate expected responses, taking into account player expectations, game context adaptability, and dialogue coherence, ensuring that the generated responses are both natural and consistent with the game's narrative structure.

[0091] Entity Type (ET): An entity is a key element in an NPC response. For example, in the NPC response "How does Lovell Lodge sound?", the hotel name "Lovell Lodge" is a key entity. Given a conversation context and knowledge base, SyncTOD predicts the list of entity types in the system response. To predict entity types, SyncTOD learns an entity prediction model P(et|c,K) on historical task response data D, where et represents the entity type.

[0092] Entity prediction model: Based on historical task response data D and through training on large-scale conversation data, the system can identify common entity types in different situations and predict entities that may appear in the current conversation. For example, in tourism conversations, common entity types may include hotels, attractions, and transportation.

[0093] Conversation Closure (DC): The style of conversation closing varies depending on the game scenario, and each different scenario has a different way to close the conversation. ChatGPT generates similar, lengthy, and open-ended responses to the user's closing remarks. In order to guide LLM to successfully close the conversation, SyncTOD uses the closing prediction dc of a given dialog box as a prompt to guide LLM to successfully close the conversation, that is, when the closing prediction dc is true, add the prompt: "The conversation must be closed"; conversely, when the closing prediction dc is false, add the prompt: "The conversation cannot be closed". For training conversations, define dc = True if and only if y i It is the last sentence in the conversation.

[0094] Closing prediction model: By analyzing the ending patterns of a large number of conversations, the system is able to predict when the conversation should end in the current context. For example, after the player completes a task or obtains the required information, the system will generate a prompt to end the conversation to ensure that the conversation closes naturally.

[0095] Response size (RS): For the historical task response data D, the response size rs of each sample is equal to the response y i According to the above principles, the response data set corresponding to the historical task response data D is obtained. Among them, rs i Indicates the response size of the ith conversation, SyncTOD in the response dataset The response prediction model P(rs|c,K) is obtained by learning so that the number of words in the response does not exceed rs. Based on the real-time multimodal dataset and real-time knowledge base Get real-time task data Use SyncTOD to predict ET, RS and DC respectively, and get the real-time task prompt set Indicates the entity type obtained based on real-time task data. Indicates the response size obtained based on real-time task data. Represents the shutdown prediction based on real-time task data.

[0096] Response prediction model: By analyzing the length and complexity of the conversation, the system can predict the appropriate response length for the current situation. For example, in a simple question-and-answer scenario, the response can be short; while in complex task guidance, the response needs to be more detailed and comprehensive.

[0097] S4, using a similarity calculation method to calculate the similarity between the historical task prompt set and the real-time task prompt set of the historical task response data, and selecting μ samples with the highest similarity from the historical task response data, including the following steps:

[0098] S4.1, concatenating the real-time multimodal data obtained in step S3 with the corresponding real-time knowledge base to obtain an operation representation vector;

[0099] S4.2, obtaining the operation representation vector of each sample in the historical task response data D according to the method of step S4.1, and selecting the top k samples that are most similar to the operation representation vector obtained in step S4.1 by using the maximum inner product search algorithm;

[0100] S4.3, input the first k samples obtained in step S4.2 into the SyncTOD model to obtain the historical task prompt set, calculate the similarity between the historical task prompt set and the real-time task prompt set, select the μ samples with the highest similarity, and μ <k;

[0101] In this embodiment, μ=2. The similarity calculation formula is:

[0102]

[0103] In the formula, H m represents the historical task prompt set of the mth sample, dc m represents the closing prediction of the mth sample, et m represents the entity type of the mth sample, Represents the historical task prompt set H m and real-time task reminder set The similarity between them, 1 is the indicator function, and J is the Jaccard similarity.

[0104] This application first uses the maximum inner product to search for semantically closer The samples are sorted and then the similarity is calculated using the task prompt set obtained by the SyncTOD model. The two with the highest similarity scores are selected as the final samples. Through the above method, SyncTOD can effectively train NPC responses that are consistent with the generated dialogue style.

[0105] S5, optimizing the response based on the real-time task prompt set obtained in step S3, the game response inference model obtained in step S2, and the sample selected in step S4, and transmitting the optimized response back to the user;

[0106] The real-time multimodal data, real-time knowledge base, real-time task prompt set and the samples selected in step S4 are input into the game response reasoning model to optimize the final response, and natural language prompts are used to guide the LLM to generate a more expected response. The selected examples are provided to the LLM in the form of natural language prompts as part of the context. This information helps the LLM generate responses that align with the style of the training data. After receiving these natural language prompts and the selected examples, the LLM generates a conversational response that meets the prompt requirements. Then, the conversation text corresponding to the generated response is converted into natural speech using text-to-speech technology (such as Tacotron 2 or WaveNet), and the speech parameters (such as speaking rate, pitch, timbre) are configured so that the speech output conforms to the personality settings of the virtual person.

[0107] This application can process user feedback information (such as voice input, text input, operation instructions) in real time, and use SyncTOD's prompt mechanism and context learning function to dynamically adjust the virtual person's dialogue strategy to improve the intelligence and adaptability of interaction. According to user feedback and changes in game scenes, the virtual person's expression, action and dialogue content are dynamically adjusted to achieve a synchronization mechanism between multimodal output and game scenes, ensuring that the virtual person's actions, expressions and dialogues can be reflected in the game environment in real time.

[0108] The embodiment of the present application also proposes a cloud game virtual human dialogue system based on a multimodal model, including:

[0109] Data collection module: used for multimodal data when users use cloud games, and constructs a multimodal sample data set in JSON format;

[0110] Model building module: used to obtain the corresponding embedding vector based on the multimodal sample data set obtained by the data collection module and the preset multimodal encoder, update the multimodal sample data set based on the embedding vector and the preset marker, and use the updated multimodal sample data set to train the large language model to obtain the game response inference model;

[0111] Real-time task information collection module: used to collect real-time multimodal data under the current game task, collect the real-time knowledge base corresponding to the game task, and use the pre-trained SyncTOD model, real-time multimodal data and real-time knowledge base to obtain the real-time task prompt set;

[0112] Sample selection module: used to obtain the historical task prompt set based on the pre-trained SyncTOD model, calculate the similarity between the historical task prompt set and the real-time task prompt set using the similarity calculation method, and select μ samples with the highest similarity from the historical task response data;

[0113] Response optimization module: used to optimize the response based on the samples selected by the sample selection module, the real-time task prompt set obtained by the real-time task information collection module and the game response reasoning model of the model construction module, and send the optimized response back to the user.

[0114] An embodiment of the present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned cloud game virtual human dialogue method based on the multimodal model are performed.

[0115] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned cloud game virtual human dialogue method based on a multimodal model are executed. Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the embodiment of the above-mentioned cloud game virtual human dialogue method based on a multimodal model can be executed.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud game virtual human dialogue method based on a multimodal model, characterized in that: The steps include: S1, collects multimodal data when cloud games execute tasks and constructs a multimodal sample dataset; S2, using a multimodal encoder to obtain an embedding vector corresponding to all data in the multimodal sample dataset, updating the multimodal sample dataset based on the embedding vector and the preset marker, and using the updated multimodal sample dataset to train the large language model to obtain a game response inference model; S3, collects the real-time multimodal data involved in the current game task, collects the real-time knowledge base corresponding to the game task, and uses the pre-trained SyncTOD model, real-time multimodal data and real-time knowledge base to obtain the real-time task prompt set; S4, using a similarity calculation method to calculate the similarity between the historical task prompt set and the real-time task prompt set of the historical task response data, and selecting the first μ samples with higher similarity from the historical task response data; S5, based on the real-time task prompt set obtained in step S3 and the samples selected in step S4, the response output by the game response inference model obtained in step S2 is optimized, and the optimized response is transmitted back to the user.

2. The cloud game virtual human dialogue method based on a multimodal model according to claim 1, characterized in that: The step S2 comprises the following steps: S2.1, using the high-resolution input module (GLHR) of the MammothModa model to process the image data in the multimodal sample dataset to obtain the image fusion vector; S2.2, constructing a modal projector, wherein the modal projector includes a first linear layer, an activation function layer, a first Dropout layer, a second linear layer, and a second Dropout layer connected in sequence; S2.3, inputting the image fusion vector obtained in step S2.1 and the sound data in the multimodal sample data set into the multimodal encoder for encoding, inputting the encoded vector into the modal projector to obtain the corresponding embedding vector, and obtaining the corresponding embedding vector based on the text data in the multimodal sample data set; S2.4, updating the multimodal sample data set obtained in step S1 based on the preset marker and the embedding vector obtained in step S2.3; S2.5, using the updated multimodal sample data set obtained in step S2.4 to train the large language model to obtain a game response inference model.

3. The cloud game virtual human dialogue method based on a multimodal model according to claim 2 is characterized in that: The method for obtaining the image fusion vector is as follows: a. Adjust the image size of the image data in the multimodal sample dataset so that the size of each frame image is an integer multiple of 336; b, divide each frame image into 336*336 image blocks; c. Downsample each frame of the image adjusted in step a to obtain a 336*336 global image, and concatenate the image vector of the global image with the image block vector of the corresponding frame obtained in step b to obtain an image fusion vector.

4. The cloud game virtual human dialogue method based on a multimodal model according to claim 1, characterized in that: The step S3 comprises the following steps: S3.1, obtaining real-time multimodal data during the current game task dialogue, and processing the real-time multimodal data according to the methods of step S1 and step S2 to obtain a real-time multimodal data set; S3.2, updating the real-time multimodal dataset according to the method of steps S2.1-S2.4, and collecting the real-time knowledge base corresponding to the current game task based on the real-time multimodal dataset; S3.3, using the entity prediction model, the closing prediction model, and the response prediction model included in the SyncTOD model to obtain the real-time task prompt set corresponding to the real-time multimodal dataset.

5. The cloud game virtual human dialogue method based on a multimodal model according to claim 1, characterized in that: The step S4 comprises the following steps: S4.1, concatenating the real-time multimodal data obtained in step S3 with the corresponding real-time knowledge base to obtain an operation representation vector; S4.2, obtaining the operation representation vector of each sample in the historical task response data D according to the method of step S4.1, and selecting the top k samples with high similarity to the operation representation vector obtained in step S4.1 by using the maximum inner product search algorithm; S4.3, input the first k samples obtained in step S4.2 into the SyncTOD model to obtain the historical task prompt set, calculate the similarity between the historical task prompt set and the real-time task prompt set, and select the first μ samples with higher similarity, where μ<k.

6. The cloud game virtual human dialogue method based on a multimodal model according to claim 5, characterized in that: The calculation formula of the similarity is: In the formula, H m represents the historical task prompt set of the mth sample, dc m represents the closing prediction of the mth sample, et m represents the entity type of the mth sample, Represents the historical task prompt set H m and real-time task reminder set The similarity between them, 1 is the indicator function, and J is the Jaccard similarity.

7. A cloud game virtual human dialogue system based on a multimodal model, characterized in that: include: Data collection module: used to build a multimodal sample dataset based on the multimodal data of players using cloud games; Model building module: used to obtain the corresponding embedding vector based on the multimodal sample data set obtained by the data collection module and the preset multimodal encoder, update the multimodal sample data set based on the embedding vector and the preset marker, and use the updated multimodal sample data set to train the large language model to obtain the game response inference model; Real-time task information collection module: used to collect real-time multimodal data under the current game task, collect the real-time knowledge base corresponding to the game task, and use the pre-trained SyncTOD model, real-time multimodal data and real-time knowledge base to obtain the real-time task prompt set; Sample selection module: used to obtain the historical task prompt set corresponding to the historical task response data based on the pre-trained SyncTOD model, calculate the similarity between the historical task prompt set and the real-time task prompt set using the similarity calculation method, and select the first μ samples with higher similarity from the historical task response data; Response optimization module: used to optimize the response based on the samples selected by the sample selection module, the real-time task prompt set obtained by the real-time task information collection module and the game response reasoning model of the model construction module, and send the optimized response back to the user.

Images