Multi-mode collaborative board game entertainment method and system based on artificial intelligence

Abstract

The invention provides a multi-mode collaborative board game entertainment method and system based on artificial intelligence, and relates to the technical field of artificial intelligence. The network server is innovated in a mode of AI voice recognition, an AI host model and a multi-modal generation model, game automation is achieved, and multi-modal feedback of characters, voice and dynamic scenes is provided; wherein the AI host can automatically determine the current progress condition of a game and generate an optional plot branch text in combination with a player instruction, the collaborative architecture multi-modal AI model generates multi-modal data matched with the plot branch text according to the plot branch text, and the multi-modal data has consistency and accuracy, enriches the output form of the board game entertainment system, and improves the performance of the board game entertainment system. And flexible and changeable multi-mode game guidance is provided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based multimodal collaborative tabletop game entertainment method and system. Background Art

[0002] Tabletop games, also known as board games, are games played on a table or other flat surface. These games typically have a specific theme, rules, and objectives, requiring players to use strategy, skill, memory, logical reasoning, or simple luck to win.

[0003] There are many different types of board games, including but not limited to strategy games, abstract games, customization games, and family games. Some of these games primarily use text or language to construct the in-game environment, plot, adventure, and player interaction, such as role-playing games like script-killing and online role-playing games. These games often require one or more hosts to interact face-to-face with multiple players.

[0004] The host's responsibilities in a game include, but are not limited to, introducing the rules, setting the story, guiding the plot, directing player actions, recording progress, judging test results, livening up the game atmosphere, and controlling the pace of the game. The host's familiarity with the script, his or her verbal and literary skills, and his or her experience interacting with players directly determine the overall gaming experience.

[0005] Currently, human hosts occupy the vast majority of tabletop game occasions. However, human hosts often face the following problems: 1. Players have a high demand for playing different role-playing games, but it takes a long time for the host to become familiar with the script; 2. Each tabletop game lasts a long time, and the host's high intensity makes it easy for fatigue to lead to mistakes; 3. The host's performance in each game is unstable, and different types of role-playing require hosts with different styles, resulting in a large upper and lower limit range for each game experience.

[0006] In addition, the sense of immersion in this type of text-based board game relies on the players' brains to construct it, which makes it difficult for novice players and players with weak imagination to get started, reducing the fun of the game. The ability to generate game scenes in real time is an urgent need for this type of game. Although there are existing AI-based board game systems, they still have fixed paradigms, generate single information, and cannot meet multimodal scene requirements.

[0007] Therefore, it is necessary to provide a multimodal collaborative table game entertainment method and system based on artificial intelligence to solve the technical problems of existing AI-based table game systems, such as fixed paradigms, single generated information, and inability to meet multimodal scenario requirements. Summary of the Invention

[0008] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a multimodal collaborative table game entertainment method and system based on artificial intelligence. It innovates in the mode of "AI voice recognition + AI host model + multimodal generation model". The AI host can automatically determine the current progress of the game and generate optional plot branch texts based on player instructions. The collaborative architecture multimodal AI model generates multimodal data matching the plot branch text. These multimodal data are not only consistent and accurate, but also enrich the output form of the table game entertainment system and provide flexible and changeable multimodal game guidance.

[0009] To achieve the above objectives, this application proposes an artificial intelligence-based multimodal collaborative board game entertainment system, comprising:

[0010] An input device, used to receive information data input by a player and communicate with a network server; wherein the information data includes voice information data;

[0011] An output device, configured to provide corresponding multimodal data to the player and communicate with the network server; wherein the multimodal data is used to advance the gaming process of the board game;

[0012] The network server is used to generate corresponding multimodal data based on the information data input by the player, including:

[0013] The AI speech recognition model is used to perform AI speech recognition on the voice information data, obtain the corresponding player voice text data, remove invalid information data, and convert it into the corresponding player command;

[0014] The AI host model is used to automatically determine the current progress of the game, generate optional plot branch text, and generate corresponding plot text output based on player instructions;

[0015] The collaborative architecture multimodal AI model is used to generate several single-modal data according to the plot text output through the multimodal generation model, and coordinate and combine them through the coordination controller to obtain alternative multimodal data and output the results through the scoring model. The scores greater than the set threshold are output as multimodal data.

[0016] As a further solution, the collaborative architecture multimodal AI model processes the plot text output through an input layer, which includes a natural speech parsing module, an intent recognition module, and a feature encoder; wherein,

[0017] The natural speech analysis module is used to parse the plot branch text to obtain text analysis content; the intention recognition module is used to perform intention recognition on the text analysis content to obtain text intention content; the feature encoder maps the text analysis content and text intention content to a unified feature space shared by different modalities, converts them into structured content data for use by the multimodal generation model.

[0018] As a further solution, the multimodal generation model includes a video generation module, a picture generation module, an audio generation module and a text generation module, and generates video modal data, picture modal data, audio modal data and text modal data accordingly.

[0019] As a further solution, the multimodal generation model uses hierarchical contrastive learning to perform cross-modal alignment fine-tuning and introduces a game state memory mechanism; hierarchical contrastive learning: different modalities are divided into two or more levels in a unified feature space, and modal alignment is performed on the corresponding levels respectively; game state memory mechanism: storage / prompt means are used to memorize the current game state, and the current game state is aligned in a unified feature space.

[0020] As a further solution, when using the high-level semantics / low-level details partitioning:

[0021] High-level semantic alignment is to align high-level semantic information of different modalities in the global feature space;

[0022] Low-level detail alignment is to align the low-level detail information of different modalities in the local feature space.

[0023] As a further solution, the coordination controller includes a timing synchronization module, which includes a data stream timestamp system, a time synchronization system and a dynamic buffer control system; wherein,

[0024] The data stream timestamp system is used to establish a unified global time coordinate system for all data streams, stamp each key frame / audio clip with a unified timestamp, and assign different modal data to a comparable reference time axis;

[0025] The time synchronization system is used to use a synchronization algorithm to analyze the time embedding of video / audio features in real time, calculate the spatiotemporal alignment weights through a temporal convolutional network model, calculate the deviation amount and generate compensation instructions;

[0026] The dynamic buffer control system is used to dynamically adjust the original buffer and synchronization queue according to the compensation instruction and the real-time synchronization state to elastically release the streaming data.

[0027] As a further solution, the coordination controller includes a consistency checker, which is composed of several consistency detection modules, including a semantic consistency detection module, a physical consistency detection module, a style consistency detection module and an emotional consistency detection module.

[0028] As a further solution, the exception handling module includes multiple sub-processing modules; wherein the sub-processing modules include:

[0029] The modal desynchronization sub-processing module is used to perform timestamp deviation detection, and the processing method is dynamic frame skipping / insertion;

[0030] Content conflict sub-processing module, used to execute consistency checker alarm, and the processing method is partial image or voice regeneration;

[0031] The resource limit sub-processing module is used to perform GPU memory monitoring and automatically downgrade the build quality;

[0032] The generation failure sub-processing module is used to perform empty output detection. The processing method is to call the backup material library or directly output the text and restart the generation pipeline;

[0033] The logical deadlock sub-processing module is used to perform heartbeat packet timeout detection. The processing method is to roll back the state and restart the asynchronous thread.

[0034] As a further solution, the scoring model scores the output results of the candidate multimodal data, retains the candidate multimodal data whose scores are not greater than the set threshold, and feeds them back to the collaborative architecture multimodal AI model to regenerate the candidate multimodal data;

[0035] Among them, if the set score is not reached within the set number of cycles and time, the player will be given the result with the highest score generated in the output process, or a degradation strategy will be adopted to output low-modal data.

[0036] On the other hand, the present invention further provides an artificial intelligence-based multimodal collaborative table game entertainment method, which uses an artificial intelligence-based multimodal collaborative table game entertainment system as described in any of the above items, comprising the following steps:

[0037] Step 1: Receive player input data through the input device and send it to the network server;

[0038] Step 2: Perform AI voice recognition on the voice information data through the AI voice recognition model to obtain the corresponding player voice text data and remove invalid information data, and convert it into the corresponding player command;

[0039] Step 3: The AI host model automatically determines the current progress of the game, generates optional plot branch text, and generates corresponding plot text output based on player instructions;

[0040] Step 4: Generate several unimodal data according to the plot text output through the multimodal generation model;

[0041] Step 5: The single modal data are coordinated and combined through the coordination controller to obtain the candidate multimodal data;

[0042] Step 6: Score the output results through the scoring model, and output the results with scores greater than the set threshold as multimodal data;

[0043] Step 7: The output device receives the multimodal data and feeds back the corresponding multimodal data to the player through a corresponding hardware device; wherein the multimodal data is used to advance the game process of the board game entertainment;

[0044] Step 8: Repeat steps 1 to 7 until the game process of the board game entertainment ends.

[0045] Compared with related technologies, the present invention can provide an artificial intelligence-based multimodal collaborative board game entertainment method and system with the following advantages:

[0046] This invention, based on AI speech recognition, an AI host model, and a multimodal generation model, establishes an interactive entertainment system that automates games and provides multimodal feedback using text, voice, and dynamic scenes. This system addresses the time cost of the host learning scripts and improves the host's fluctuating performance. Multimodal feedback enhances the player's sense of immersion and gaming experience. A coordinated controller is provided to efficiently control the synergistic effects of the multimodal model, creating a scoring model feedback mechanism that improves the quality of gaming experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A schematic diagram of the structure of a multi-modal collaborative table game entertainment system based on artificial intelligence provided by the present invention;

[0050] Figure 2A schematic diagram of the game information flow provided by the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of the collaborative architecture multimodal AI model provided by the present invention;

[0052] Figure 4 A schematic diagram of the coordination controller structure provided by the present invention;

[0053] Figure 5 A schematic diagram of the steps of a multimodal collaborative table game entertainment method based on artificial intelligence provided by the present invention.

[0054] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0056] See also Figure 1 , the embodiment of the present application provides a multimodal collaborative table game entertainment system based on artificial intelligence, including:

[0057] An input device, configured to receive information data input by a player and communicate with a network server; wherein the information data includes voice information data and preset option text information data;

[0058] An output device, configured to provide corresponding multimodal data to the player and communicate with the network server; wherein the multimodal data is used to advance the gaming process of the board game;

[0059] The network server is used to generate corresponding multimodal data based on the information data input by the player, including:

[0060] The AI speech recognition model is used to perform AI speech recognition on the voice information data, obtain the corresponding player voice text data, remove invalid information data, and convert it into the corresponding player command;

[0061] The AI host model is used to automatically determine the current progress of the game, generate optional plot branch text, and generate corresponding plot text output based on player instructions;

[0062] The collaborative architecture multimodal AI model is used to generate several single-modal data according to the plot text output through the multimodal generation model, and coordinate and combine them through the coordination controller to obtain alternative multimodal data and output the results through the scoring model. The scores greater than the set threshold are output as multimodal data.

[0063] It should be noted that traditional AI board game systems often only have a single AI host function and can only provide simple and fixed game guidance based on the player's voice. For example, patent document CN112675526A discloses an AI judge system, its game hosting method and game system. This patent collects player behavior data through visual and text perception modules and uses a basic rule library to provide game guidance. However, its technical solution focuses on the recognition of player behavior under a static rule library. Its system framework limits its application scope to a small range, making it difficult to adapt to the different plots and rules of different game scripts, and it has not formed a collaborative cross-modal interaction system.

[0064] This system innovates with the model of "AI speech recognition + AI hosting model + multimodal generation model", specifically:

[0065] In this embodiment, the player's voice information data is transmitted to the network through the input device and enters the AI voice recognition model; the AI voice recognition model converts the player's voice information into text information, and intelligently removes invalid information (such as communication or chatting between players) and converts it into player instructions to be input into the AI host model.

[0066] The AI host can automatically determine the current progress of the game and generate optional plot branch text based on player instructions. The collaborative architecture multimodal AI model generates matching multimodal data based on the plot branch text. These multimodal data are not only consistent and accurate, but also enrich the output form of the board game entertainment system and provide flexible and diverse multimodal game guidance.

[0067] This text is output to the multimodal generation model (such as the video generation module, image generation module, audio generation module, and text generation module) under the collaborative architecture (collaborative architecture multimodal AI model), which generates text options, voice options, scene images, scene videos, etc. to advance the game progress, and is coordinated and combined together through the collaborative framework. The scoring model scores the output results of the multimodal collaboration. If the score is greater than the set value, the result is output. The output results of one or more combinations are provided to the player through devices such as speakers and displays.

[0068] The above process is repeated to complete a round of game. The game information flow is as follows: Figure 2As shown, the game flow does not require players to be physically present in the same space, making online play possible via a mobile app. Player input is recorded as audio by default in the information flow. Alternatively, the AI host can input text messages by touching the screen. Using touch to input text bypasses the AI voice recognition model, improving information flow efficiency and providing players with a more engaging gameplay experience.

[0069] Note: Input devices can be mobile phones, computers, microphones, sensors, etc., and output devices can be monitors, speakers, etc., and display devices such as AR glasses and VR glasses can be used to provide players with a better gaming experience.

[0070] Next, we will introduce the technical implementation solutions and technical details of each part:

[0071] The AI speech recognition model converts the player's voice information into text information. It can automatically determine the source of the voice through voiceprint recognition and microphone number input, improve the model's analysis of different character plots, and increase the efficiency of interaction between players and the system.

[0072] The AI speech recognition model includes a streamlined text information model, which is used to distinguish which conversations need to be provided to the AI host model and which conversations are communication or chatting between players, to avoid incorrect input information affecting the game experience.

[0073] One feasible approach is to build a System Tuning structure based on a large language model, aggregate player conversation information, and connect the context when necessary to extract player intent and key information.

[0074] The AI host is based on the Large Language Model (LLM) and is optimized for this application scenario, using methods including but not limited to transfer learning models or dynamic rule constraints.

[0075] Through transfer learning, the AI host model is fine-tuned from a general language model to a state suitable for role-playing games; through dynamic rule constraints, the AI host model can quickly adapt to the rules and plot changes of different game scripts.

[0076] The AI host can use multiple language models that have undergone transfer learning.

[0077] In transfer learning, for different types of role-playing games, the fine-tuning objects can be based on different language models, or the same language model can be fine-tuned differently. Feasible fine-tuning solutions include but are not limited to: PromptTuning design, building a script hosting script and structure database, and vertical field fine-tuning of Embedding.

[0078] After fine-tuning, the effectiveness of each model is scored. The scoring criteria can be based on weighted performance in dimensions such as accuracy, response latency, and player feedback, and a scoring database for different migration models for different game type labels can be established.

[0079] Therefore, the AI host model includes a language model selection module. It selects the best transfer language model based on the game type information, background text, and character information of the game selected by the player, giving the player the best gaming experience. Figure 1 shown.

[0080] The AI host's responsibilities include, but are not limited to, introducing game rules, setting up the story, guiding the plot, directing player actions, recording progress, generating plot branches, judging test results, interacting with players, livening up the game atmosphere, and controlling the game's pacing. Because the game is completely controlled and advanced by the AI model, the upper and lower limits of performance are more easily controlled.

[0081] The AI host can provide players with selectable options by directly using single numbers or letters as selection options to improve the efficiency of interaction with players, and can effectively solve the pain point of novice players not knowing how to advance the game. Of course, it is also acceptable for players to enter instructions other than the options provided by the AI host.

[0082] The collaborative architecture multimodal AI model of this system is the core component of this system, and its structure is as follows Figure 3 As shown, the input is the text information provided by the AI host, and the output is one or more multimodal results;

[0083] Text information is extracted through text parsing and intent recognition at the input layer.

[0084] The text data is mapped to a unified feature space shared by different modalities, and the text information is converted into structured data for easy use by various generation modules.

[0085] Then, in the multimodal generation model, one or more models are selected for content generation.

[0086] Subsequently, the generated content is passed to the coordination controller, which needs to ensure that the output of different modal contents is synchronized.

[0087] For example, when generating a battle scene, the text description should be moved to different positions, the pictures should be switched, the audio should be combined with the battle sound effects, and the video should be coherent.

[0088] A consistency checker is built in the coordination controller to evaluate the multimodal consistency. If the verification result is abnormal, targeted processing is required.

[0089] Finally, the output layer outputs the integrated modal results to the scoring model to score the results, and the scoring model decides whether to regenerate from a certain step or pass it to the player.

[0090] The input layer includes a natural speech analysis module, an intent recognition module, and a feature encoder; the multimodal generation model includes a video generation module, an image generation module, an audio generation module, and a text generation module, and generates video modal data, image modal data, audio modal data, and text modal data accordingly.

[0091] Among them, the natural speech analysis module is used to parse the plot branch text to obtain text analysis content; the intention recognition module is used to perform intent recognition on the text analysis content to obtain text intention content; the feature encoder maps the text analysis content and text intention content to a unified feature space shared by different modalities, converts it into structured content data and provides it for use by the multimodal generation model.

[0092] Patent document CN114067410A discloses an AI-based tabletop game interaction system. The patent proposes a multimodal interactive platform, the core of which is to collect player information during tabletop games to enhance the player experience. However, the patent focuses solely on player information collection and image recognition, and does not address the use of transfer learning to fine-tune models or dynamic rules to better replace game hosts, bypassing the time cost of hosts learning and familiarizing themselves with tabletop game scripts, and improving players' consistent gameplay experience. Furthermore, the fixed interactive device limits possible cross-scenario interaction modes, such as online app modes.

[0093] To ensure semantic consistency, the multimodal generative model uses hierarchical contrastive learning for cross-modal alignment fine-tuning and introduces game state memory mechanisms, such as external storage for memory and prompt engineering, for alignment within the game. Optionally, a cross-modal shared feature space can be used to ensure input consistency within the multimodal model. During AI model training for the multimodal generative model, fine-tuning embeddings can be used for vertical domain optimization, while hierarchical contrastive learning can also be used to train the multimodal model in tandem to improve model output.

[0094] Hierarchical contrastive learning divides different modalities into two or more layers of semantics in the feature space, such as high-level semantics and low-level details. High-level semantic alignment is to align the high-level semantic information of different modalities (such as "character occupation") in the global feature space. Low-level detail alignment is to align the low-level detail information of different modalities (such as "character expression") in the local feature space. A hierarchical loss function is used to simultaneously optimize high-level features and low-level details. Here is a possible total loss function formula:

[0095] L=λ1×L high +λ2×Llow

[0096] Among them, L high and L low Denote the losses of the high-level and low-level layers, respectively; λ1 and λ2 are weight coefficients. This weighted approach improves model optimization efficiency and provides players with more accurate multimodal interaction results.

[0097] like Figure 4 As shown in the figure, the coordination controller of the collaborative framework consists of a timing synchronization module, a consistency checker, and an exception handling module, which are used to ensure the consistency of the output multimodal results in time, semantics, and style. In addition, the sub-modules in this structure can be used as optional modules and skipped when the consistency of the generated results is expected to be good.

[0098] The coordination controller includes a timing synchronization module, which includes a data stream timestamp system, a time synchronization system and a dynamic buffer control system; wherein,

[0099] The data stream timestamp system is used to establish a unified global time coordinate system for all data streams. Through multi-modal high-precision clock synchronization, a unified timestamp is applied to each key frame / audio clip, giving data of different modalities a comparable reference time axis.

[0100] The time synchronization system is used to use a synchronization algorithm to analyze the time embedding of video / audio features in real time, calculate the spatiotemporal alignment weights through a temporal convolutional network model, calculate the deviation amount and generate compensation instructions;

[0101] The dynamic buffer control system is used to dynamically adjust the original buffer and synchronization queue according to the compensation instruction and the real-time synchronization state to elastically release the streaming data.

[0102] Within a certain range, the system dynamically adjusts the video frame rate to, for example, 24-60 fps and uses a PhaseNet predictor for frame generation. It also adapts to flexible audio sampling rates, such as 44.1-192 kHz, and uses SOLA-FS for time stretching and delay compensation. Furthermore, it also sets a synchronization tolerance of ±10ms. These functions work together to form a closed loop of "monitoring-decision-compensation," ultimately outputting synchronized streams with a time deviation of less than 20ms. This architecture effectively reduces computational load while maintaining accuracy, making it more suitable for real-time scenarios.

[0103] The coordination controller includes a consistency checker, which is composed of several consistency detection modules, including a semantic consistency detection module, a physical consistency detection module, a style consistency detection module and an emotional consistency detection module; wherein,

[0104] Semantic consistency detection module: uses cross-modal contrastive learning such as CLIP to improve the model and verify the matching of image and sound effects;

[0105] Physical consistency detection module: Uses a physical engine simulation verification such as the OpenCV+Librosa code solution to verify the synchronization of sound and picture;

[0106] Style consistency detection module: Use style transfer feature space comparison, such as the GroupViT model, to verify the matching degree of the cartoon style before and after;

[0107] Emotion consistency detection module: Use multimodal emotion encoders such as visual emotion CNN + audio emotion RNN to verify the coordination between the scene and the soundtrack.

[0108] The coordination controller generates a test. If the consistency is incorrect, the problem is addressed in the exception handling module. If the consistency is correct, the test further checks for other anomalies. The coordination controller can also assist in the training and evaluation of multimodal AI models, achieving efficient utilization. Processing by the coordination controller ensures that the output results of the multimodal model are highly consistent.

[0109] The exception handling module of the coordination controller is used to quickly detect and handle possible exceptions. Different exceptions require different detection methods and classification processing methods. Therefore, the exception detection module and classification processing module are combined for description, including but not limited to the following modules:

[0110] The modal desynchronization sub-processing module is used to perform timestamp deviation detection, and the processing method is dynamic frame skipping / insertion;

[0111] Content conflict sub-processing module, used to execute consistency checker alarm, and the processing method is partial image or voice regeneration;

[0112] The resource limit sub-processing module is used to perform GPU memory monitoring and automatically downgrade the build quality;

[0113] The generation failure sub-processing module is used to perform empty output detection. The processing method is to call the backup material library or directly output the text and restart the generation pipeline;

[0114] The logical deadlock sub-processing module is used to perform heartbeat packet timeout detection. The processing method is to roll back the state and restart the asynchronous thread.

[0115] The recovery confirmation module gradually restores the output after detecting that the abnormal situation has been resolved, such as asynchronously loading high-quality resources into the backup buffer. Through the exception handling module, the output results obtained by the player are optimized.

[0116] The scoring model of the collaborative framework can realize a self-feedback mechanism. The model is based on the Reward Model and is based on a comprehensive scoring of multiple aspects such as plot style relevance, previous player feedback, and content compliance.

[0117] The workflow is as follows: after inputting candidate multimodal data that has been verified for consistency, the model selects one or more modalities from the multimodal input and scores the generated results. If the score is greater than a set value, the result is directly output to the player.

[0118] On the contrary, if the score does not reach the set value, the result will be retained and fed back to the collaborative framework multimodal AI model to regenerate the adjusted multimodal result and perform the scoring process again.

[0119] If the set score is not reached within the set number of cycles and time, the player will be given the result with the highest score generated during the output process, or a demotion strategy will be adopted to output it in text form to avoid the player waiting for too long.

[0120] This structure can provide players with a higher-quality gaming experience and enable the model to generate output that better meets player expectations and safety standards.

[0121] This system may include an account system or an NFC card system. After saving the progress in the middle of the game, the player's account or NFC card can be used to continue the game at a later time.

[0122] like Figure 5 As shown, this embodiment further provides an artificial intelligence-based multimodal collaborative table game entertainment method, which uses an artificial intelligence-based multimodal collaborative table game entertainment system as described in any of the above items, including the following steps:

[0123] Step 1: Receive player input data through the input device and send it to the network server;

[0124] Step 2: Perform AI voice recognition on the voice information data through the AI voice recognition model to obtain the corresponding player voice text data and remove invalid information data, and convert it into the corresponding player command;

[0125] Step 3: The AI host model automatically determines the current progress of the game, generates optional plot branch text, and generates corresponding plot text output based on player instructions;

[0126] Step 4: Generate several unimodal data according to the plot text output through the multimodal generation model;

[0127] Step 5: The single modal data are coordinated and combined through the coordination controller to obtain the candidate multimodal data;

[0128] Step 6: Score the output results through the scoring model, and output the results with scores greater than the set threshold as multimodal data;

[0129] Step 7: The output device receives the multimodal data and feeds back the corresponding multimodal data to the player through a corresponding hardware device; wherein the multimodal data is used to advance the game process of the board game entertainment;

[0130] Step 8: Repeat steps 1 to 7 until the game process of the board game entertainment ends.

[0131] In summary, the present invention, with AI speech recognition + AI hosting model + multimodal generation model as the core, has established an interactive entertainment system that realizes game automation and provides multimodal feedback of text + voice + dynamic scenes. It solves the time cost of the host learning the script, improves the problem of the host's performance fluctuation, uses multimodal feedback to enhance the player's sense of game substitution and playing experience, and provides a coordination controller to efficiently control the synergistic effect of the multimodal model. The scoring model feedback mechanism is formed to improve the player's playing quality.

[0132] The above are only some embodiments of the present application and are not intended to limit the patent scope of the present application. All equivalent structural transformations made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multimodal collaborative table game entertainment system based on artificial intelligence, characterized by: include: An input device, configured to receive information data input by a player and communicate with a network server; wherein the information data includes voice information data and preset option text information data; An output device, configured to provide corresponding multimodal data to the player and communicate with the network server; wherein the multimodal data is used to advance the gaming process of the board game; The network server is used to generate corresponding multimodal data based on the information data input by the player, including: The AI speech recognition model is used to perform AI speech recognition on the voice information data, obtain the corresponding player voice text data, remove invalid information data, and convert it into the corresponding player command; The AI host model is used to automatically determine the current progress of the game, generate optional plot branch text, and generate corresponding plot text output based on player instructions; The collaborative architecture multimodal AI model is used to generate several single-modal data according to the plot text output through the multimodal generation model, and coordinate and combine them through the coordination controller to obtain alternative multimodal data and output the results through the scoring model. The scores greater than the set threshold are output as multimodal data.

2. The multimodal collaborative board game entertainment method based on artificial intelligence according to claim 1, characterized in that: The collaborative architecture multimodal AI model processes the plot text output through an input layer, which includes a natural speech analysis module, an intent recognition module, and a feature encoder; wherein, The natural speech analysis module is used to parse the plot branch text to obtain text analysis content; the intention recognition module is used to perform intention recognition on the text analysis content to obtain text intention content; the feature encoder maps the text analysis content and text intention content to a unified feature space shared by different modalities, converts them into structured content data for use by the multimodal generation model.

3. The multimodal collaborative board game entertainment method based on artificial intelligence according to claim 1, characterized in that: The multimodal generation model includes a video generation module, a picture generation module, an audio generation module and a text generation module, and generates video modal data, picture modal data, audio modal data and text modal data accordingly.

4. The multimodal collaborative board game entertainment method based on artificial intelligence according to claim 2, characterized in that: The multimodal generation model uses hierarchical contrastive learning to perform cross-modal alignment fine-tuning and introduces a game state memory mechanism; hierarchical contrastive learning: different modalities are divided into two or more levels in a unified feature space, and modal alignment is performed on the corresponding levels respectively; game state memory mechanism: storage / prompt means are used to memorize the current game state, and the current game state is aligned in the unified feature space.

5. The multimodal collaborative board game entertainment method based on artificial intelligence according to claim 4, characterized in that: When using the high-level semantics / low-level details partition: High-level semantic alignment is to align high-level semantic information of different modalities in the global feature space; Low-level detail alignment is to align the low-level detail information of different modalities in the local feature space.

6. The multimodal collaborative board game entertainment method based on artificial intelligence according to claim 1, characterized in that: The coordination controller includes a timing synchronization module, which includes a data stream timestamp system, a time synchronization system and a dynamic buffer control system; wherein, The data stream timestamp system is used to establish a unified global time coordinate system for all data streams, stamp each key frame / audio clip with a unified timestamp, and assign different modal data to a comparable reference time axis; The time synchronization system is used to use a synchronization algorithm to analyze the time embedding of video / audio features in real time, calculate the spatiotemporal alignment weights through a temporal convolutional network model, calculate the deviation amount and generate compensation instructions; The dynamic buffer control system is used to dynamically adjust the original buffer and synchronization queue according to the compensation instruction and the real-time synchronization state to elastically release the streaming data.

7. The multimodal collaborative board game entertainment method based on artificial intelligence according to claim 1, characterized in that: The coordination controller includes a consistency checker, which is composed of several consistency detection modules, including a semantic consistency detection module, a physical consistency detection module, a style consistency detection module and an emotional consistency detection module.

8. The multimodal collaborative board game entertainment method based on artificial intelligence according to claim 1, characterized in that: The exception handling module includes multiple sub-processing modules; wherein the sub-processing modules include: The modal desynchronization sub-module is used to perform timestamp deviation detection, and the processing method is dynamic frame skipping / insertion; Content conflict sub-processing module, used to execute consistency checker alarm, and the processing method is partial image or voice regeneration; The resource limit sub-processing module is used to perform GPU memory monitoring and automatically downgrade the build quality; The generation failure sub-processing module is used to perform empty output detection. The processing method is to call the backup material library or directly output the text and restart the generation pipeline; The logical deadlock sub-processing module is used to perform heartbeat packet timeout detection. The processing method is to roll back the state and restart the asynchronous thread.

9. The multimodal collaborative board game entertainment method based on artificial intelligence according to claim 1, characterized in that: The scoring model scores the output results of the candidate multimodal data, retains the candidate multimodal data whose score results are not greater than the set threshold, and feeds them back to the collaborative architecture multimodal AI model to regenerate the candidate multimodal data; Among them, if the set score is not reached within the set number of cycles and time, the player will be given the result with the highest score generated in the output process, or a degradation strategy will be adopted to output low-modal data.

10. An artificial intelligence-based multimodal collaborative tabletop game entertainment method, using an artificial intelligence-based multimodal collaborative tabletop game entertainment system according to any one of claims 1 to 9, characterized in that: The following steps are involved: Step 1: Receive player input data through the input device and send it to the network server; Step 2: Perform AI voice recognition on the voice information data through the AI voice recognition model to obtain the corresponding player voice text data and remove invalid information data, and convert it into the corresponding player command; Step 3: The AI host model automatically determines the current progress of the game, generates optional plot branch text, and generates corresponding plot text output based on player instructions; Step 4: Generate several unimodal data according to the plot text output through the multimodal generation model; Step 5: The single modal data are coordinated and combined through the coordination controller to obtain the candidate multimodal data; Step 6: Score the output results through the scoring model, and output the results with scores greater than the set threshold as multimodal data; Step 7: The output device receives the multimodal data and feeds back the corresponding multimodal data to the player through a corresponding hardware device; wherein the multimodal data is used to advance the game process of the board game entertainment; Step 8: Repeat steps 1 to 7 until the game process of the board game entertainment ends.

Images