A method and system for generating a virtual world based on AI

Abstract

The application discloses a method and system for generating a virtual world based on AI, and relates to the field of virtual world construction. The method comprises the following steps: receiving perfect information of a current frame of a virtual world input by a user at a current moment, wherein the perfect information comprises at least one of the following information: theme type, role type, interaction preference type, style type and control signal; adding the perfect information to the current frame of the virtual world by using an AI large model to obtain a next frame of the virtual world; and outputting the next frame of the virtual world. The application can generate a personalized virtual world.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual world construction, and in particular to a method and system for generating a virtual world based on AI. BACKGROUND

[0002] With the continuous iteration and evolution of artificial intelligence (AI), virtual reality (VR) and game engine technology, immersive virtual worlds have increasingly become the core research direction and key application scenario in the field of digital technology. The current construction of virtual worlds still mainly relies on artificial modeling, and mainstream commercial game engines such as Unreal Engine and Unity are used as the core technical support. In this mode, the development team needs to complete the scene building and character design in the virtual world in a templated manner in advance. On this basis, the virtual space that the user finally enters is essentially a digital environment limited by the preset framework.

[0003] Although this mode can meet the basic three-dimensional immersive experience needs, its characteristics of excessive reliance on artificial modeling and fixed template generation of scenes not only restrict the user's exploration range to the preset environment and make them unable to break free from passive experience, but also make it difficult for users to generate personalized virtual scenes and character images according to their own interests and preferences. This predefined experience logic restricts the depth of immersive interaction, resulting in poor overall user experience. SUMMARY

[0004] The purpose of the present application is to provide a method and system for generating a virtual world based on AI, which can generate personalized virtual worlds.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a method for generating a virtual world based on AI, comprising:

[0007] receiving perfect information of a current frame of a virtual world input by a user at a current time, the perfect information including at least one of the following information: theme type, character type, interaction preference type, style type and control signal;

[0008] adding the perfect information to the current frame of the virtual world through an AI large model to obtain a next frame of the virtual world;

[0009] outputting the next frame of the virtual world.

[0010] The present disclosure can dynamically improve the virtual world by combining the user's intention, generate a personalized virtual world, effectively improve the matching degree of the virtual world and the user's demand, and enhance the user's participation and interaction experience in the virtual world by receiving the user's input and generating the next frame of the virtual world in real time with the help of the AI large model.

[0011] In one embodiment, the adding of the improvement information to the current frame of the virtual world to obtain the next frame of the virtual world comprises:

[0012] Obtaining the current semantic feature corresponding to the improvement information;

[0013] Obtaining the historical semantic feature of the virtual world of the preset number of frames before the current frame;

[0014] Obtaining the context representation according to the historical semantic feature and the current semantic feature;

[0015] Obtaining the specification vector corresponding to the type label included in the improvement information according to the context representation and the improvement information;

[0016] Obtaining the condition vector for generative world modeling according to the specification vector and the context representation;

[0017] Obtaining the visual context according to the key frame and the preset number of frames before the current frame;

[0018] Obtaining the current latent variable of the virtual world of the current frame;

[0019] Predicting the latent variable at the next time according to the current latent variable, the visual context and the condition vector;

[0020] Obtaining the virtual world of the next frame according to the latent variable at the next time.

[0021] By fusing the historical semantic feature and the current semantic feature to construct the context representation, and predicting the latent variable at the next time by combining multi-dimensional information (visual context, etc.) to generate the next frame of the virtual world, it can be ensured that the generation of the virtual world not only continues the historical logic, but also responds to the current input, significantly improves the coherence, rationality and accuracy of the evolution of the virtual world, and makes the dynamic changes of the virtual world more consistent with the natural law and user expectations.

[0022] In one embodiment, the obtaining of the specification vector corresponding to the type label included in the improvement information according to the context representation and the improvement information comprises:

[0023] Obtaining the type label included in the improvement information according to the context representation;

[0024] Ontology mapping the type label to obtain the specification label word corresponding to each type label;

[0025] Obtain the specification vector based on all specification tags.

[0026] Converting type labels into standardized label terms and obtaining specification vectors through ontology mapping can unify and standardize the type information input by users, reduce semantic ambiguity, and enable large AI models to more accurately understand the specific connotations of each category of information. This improves the accuracy of controlling type features during the generation of the virtual world and ensures the consistency between the generated content and the type specified by the user.

[0027] In one embodiment, obtaining the virtual world of the next frame based on the latent variables of the next moment includes:

[0028] Obtain the scene of the next frame based on the latent variables of the next moment;

[0029] Check if there are any standardized tags corresponding to a role in the standardized tag list;

[0030] If so, create a virtual character based on the specification vector, the specification tags corresponding to the character, and the condition vector;

[0031] The virtual world of the next frame is obtained based on the virtual character and the scene of the next frame.

[0032] First, a scene is generated based on latent variables. Then, characters are created specifically according to the relevant normative tags of the characters and integrated with the scene to generate a virtual world. This enables the collaborative construction of scenes and characters, ensuring the adaptability of virtual characters and scenes in terms of style and logic, avoiding the disconnect between characters and environment, and enhancing the integrity and immersion of the virtual world.

[0033] In one embodiment, receiving the virtual world completion information of the current frame input by the user at the current moment includes:

[0034] Receive complete information about the virtual world in the current frame of the current time frame, in different modalities, from the user's input at the current moment;

[0035] The acquisition of the current semantic features corresponding to the complete information includes:

[0036] By using interpolation functions or resampling functions, the complete information of the current frame of different modalities is mapped to the same time axis to obtain unified time axis information corresponding to each modality;

[0037] The unified time axis information corresponding to each mode is standardized to obtain the standard information corresponding to each mode;

[0038] Map the standard information corresponding to each modality to the same dimension to obtain the dimensional information corresponding to each modality;

[0039] Add type identifiers and time location codes to the dimensional information corresponding to each modality to obtain complete information corresponding to each modality;

[0040] The complete information corresponding to each modality is spliced to obtain a unified input vector;

[0041] The current semantic feature corresponding to the unified input vector is obtained.

[0042] Through the steps of unifying the time axis of multi-modal input, standardization processing and dimension mapping, different modal user inputs such as text, voice and gesture can be effectively compatible, breaking the limitation of single input mode, improving the flexibility and convenience of user interaction with the virtual world, and ensuring that various input information can be efficiently and accurately analyzed and utilized by the AI large model.

[0043] In one embodiment, the method further comprises:

[0044] According to the unified input vector, the specification vector and the condition vector, a semantic response of the virtual character at the current time is obtained;

[0045] After outputting the next frame of the virtual world, the method further comprises:

[0046] Outputting the semantic response in the next frame of the virtual world.

[0047] Outputting the semantic response of the virtual character while generating the next frame of the virtual world makes the character in the virtual world not only exist statically, but also make semantic feedback in line with the context based on user input, enhancing the intelligence and interactivity of the character, and allowing users to obtain a more natural and real communication experience in the virtual world, improving the sense of immersion.

[0048] In one embodiment, the method further comprises:

[0049] Obtaining a current global state machine of the currently output virtual world;

[0050] Obtaining a migration token according to the user identity information;

[0051] Obtaining a target global state of the target virtual world to be migrated at the current time;

[0052] Obtaining character attribute information of the character according to the current global state machine;

[0053] Obtaining a global state of the target virtual world at the next time according to the character attribute information and the target global state;

[0054] Outputting the global state of the target virtual world at the next time.

[0055] Supporting the migration of the role from the current virtual world to the target virtual world and keeping the adaptation of the role attribute to the state of the target world, the role continuity across virtual worlds is realized, the experience break of the user when switching between different virtual worlds is avoided, the expansibility and user stickiness of the virtual world system are improved, and the needs of the user for coherent interaction in multiple virtual worlds are met.

[0056] In an embodiment, the method further comprises:

[0057] According to the context representation, obtaining the probability of the perfect information belonging to each category label in each type;

[0058] Obtaining the maximum value in all category label probabilities;

[0059] Detecting whether the maximum value is less than a preset threshold;

[0060] If less, stop using AI to generate the virtual world.

[0061] By detecting the category probability of the perfect information and stopping generation when it is lower than the threshold, the generation of low-quality and unexpected virtual world content by AI when it cannot accurately understand the user's intention can be avoided, the reliability and quality of virtual world generation are effectively guaranteed, the impact of invalid generation on user experience is reduced, and the trust of the user in the system is enhanced.

[0062] In a second aspect, the present application provides a system for generating a virtual world based on AI, comprising:

[0063] A receiving module is configured to receive perfect information of a current frame of a virtual world input by a user at a current time, wherein the perfect information comprises at least one of the following information: theme type, role type, interaction preference type, style type and control signal.

[0064] A processing module is configured to add the perfect information to the current frame of the virtual world by an AI large model to obtain a next frame of the virtual world.

[0065] An output module is configured to output the next frame of the virtual world.

[0066] In the aspect of adding the perfect information to the current frame of the virtual world to obtain the next frame of the virtual world, the processing module is specifically configured to:

[0067] Obtain a current semantic feature corresponding to the perfect information;

[0068] Obtain a historical semantic feature of a preset number of frames of the virtual world before the current frame;

[0069] Obtain a context representation according to the historical semantic feature and the current semantic feature;

[0070] According to the context representation and the perfect information, a specification vector corresponding to a type label included in the perfect information is obtained;

[0071] According to the specification vector and the context representation, a condition vector used for generative world modeling is obtained;

[0072] According to a preset number of frames before the key frame and the current frame, a visual context is obtained;

[0073] A current latent variable of the virtual world of the current frame is obtained;

[0074] According to the current latent variable, the visual context and the condition vector, a latent variable of a next time is predicted;

[0075] According to the latent variable of the next time, a virtual world of a next frame is obtained. BRIEF DESCRIPTION OF DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0077] Figure 1 is a flow of a method for generating a virtual world based on AI according to an exemplary embodiment Figure 1 ;

[0078] Figure 2 is a flow of a method for generating a virtual world based on AI according to an exemplary embodiment Figure 2 ;

[0079] Figure 3 is a flow of a method for generating a virtual world based on AI according to an exemplary embodiment Figure 3 ;

[0080] Figure 4 is a functional module schematic diagram of a system for generating a virtual world based on AI provided by another embodiment of the present application. DETAILED DESCRIPTION

[0081] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0082] The above objects, features and advantages of the present application will become more apparent from the following detailed description of the application, taken in conjunction with the accompanying drawings and specific embodiments.

[0083] Figure 1 is a flowchart of a method for generating a virtual world based on AI according to an exemplary embodiment, as shown in Figure 1 The method includes the following steps S101-S103:

[0084] In step S101, the perfect information of the current frame of the virtual world input by the user at the current moment is received, and the perfect information includes at least one of the following information: theme type, character type, interaction preference type, style type and control signal.

[0085] The theme type is the core theme of the virtual world expected by the user (such as space exploration, ancient market, future city), which determines the core style and functional positioning of the scene; the character type is the character attribute (such as mechanical artificial NPC, magic player avatar, cute pet partner) that the user wants to add or adjust, which includes character appearance, ability, interaction logic, etc.; the interaction preference type is the specific demand of the user for the association logic and dynamic interaction rule between elements in the virtual scene, which is essentially to make the scene elements interact or exist in the expected way, rather than in the way of operating the device. For example, when the virtual scene is a natural scene, the interaction preference type can be that when the wind blows, the leaves will sway, and the fallen leaves on the ground will flutter, or when the vine is touched, the vine will automatically expand to both sides; when the virtual scene is a science fiction scene, the interaction preference type can be that when the laboratory button is pressed, the laboratory light will change color, and the equipment will emit a running sound effect, or when the robot approaches the wall, the wall will automatically pop up a passage; when the virtual scene is a social scene, the interaction preference type can be that when the virtual colleague speaks, a real-time subtitle is displayed next to the avatar, or when the banquet table is clicked, a menu introduction panel is automatically popped up. In short, the interaction preference type focuses on how the scene elements are linked or responded, which is the core demand of the user for the reality and convenience of the virtual world. The style type is the visual style of the scene or the character (such as cyberpunk, low-polygon cartoon, hyper-realism), which determines the rendering details, color matching, material performance, etc.; the control signal is the immediate operation instruction of the user (such as moving to the left building, making the character raise the right hand, and enlarging the sculpture in the scene), which belongs to the short-term and specific interaction demand.

[0086] The virtual world of the current frame refers to the complete state of the virtual world at a certain moment (similar to a static snapshot of a video frame), which includes the scene layout at this time (such as forest, laboratory, virtual conference room), character / object position, existing interaction logic (such as door can be opened, button can be triggered) and all elements, which is the basic scene that the user wants to optimize.

[0087] The virtual world of the current frame can be a first frame. When it is a first frame, the disclosure does not focus on how the first frame is generated, because the disclosure is a process performed on the generated first frame. For example, the first frame can be an initial picture of a virtual world that a user wants to generate and input to an AI model. Then the AI model generates a virtual world based on the picture. The disclosure will add other content to the virtual world based on user needs. The added content can include theme type, character type, interaction preference type, and style type.

[0088] The virtual world of the current frame can be a virtual world of a certain frame. The disclosure will add other content to the virtual world displayed in the frame based on user needs. The added content can also include theme type, character type, interaction preference type, and style type.

[0089] In step S102, the perfect information is added to the virtual world of the current frame by the AI large model to obtain the virtual world of the next frame.

[0090] The AI will analyze the core demand of the user's perfect information and adapt it to the existing elements of the virtual world of the current frame. For example, the user's perfect information is to add a stuffed cat to the existing sofa in the living room. The AI large model will adapt to the current frame of the virtual world (i.e. the living room scene displayed at the moment, including the position, style of the sofa, and whether there is space, etc.). Specifically, the AI will first check the sofa state of the current living room, such as a three-seater sofa with a light gray color in a Nordic style, and the right seat is empty (without other items). Then, according to the characteristics of the stuffed cat (medium size, white and brown interlaced fur, and a common curled posture), a matching model is generated, which is suitable for the sofa seat (not too large to occupy the entire sofa), the fur color forms a soft contrast with the light gray sofa (not conspicuous), and the posture is a natural curl on the sofa right side (consistent with the habit of a cat in reality). At the same time, the AI will handle the appearance, and the stuffed cat will not suddenly appear on the sofa, but will simulate a dynamic process of slowly walking from the corner of the sofa and then curling down, avoiding harsh flashing. Finally, the generated next frame of the virtual world is a scene of a Nordic style living room with a light gray sofa on the right side of which there is a naturally curled stuffed cat, which not only meets the user's demand to add a cat, but also makes the new element fully integrated with the existing living room environment, making it look like the cat was supposed to be there.

[0091] In step S103, the virtual world of the next frame is output.

[0092] In the present disclosure, by receiving the user input according to his own needs to perfect the perfect information of the virtual world of the current frame, the AI large model is used to integrate the user's needs into the current frame of the virtual world and output the next frame, which solves the pain points of the traditional virtual world relying on the preset of the development team and the user can only passively experience, gives the user the right to guide the evolution of the virtual world, supports the user to adjust the core elements of the virtual world in real time according to his own needs, breaks the limitation of the fixed scene, makes the virtual world generation more in line with the user's personalized needs, and effectively improves the dynamic nature of the virtual world and the user's immersion.

[0093] In one embodiment, the step S102 above, the perfect information is added to the virtual world of the current frame by the AI large model, and the virtual world of the next frame is obtained, including the following sub-steps A1-A9:

[0094] A1, obtain the current semantic feature corresponding to the perfect information .

[0095] In one embodiment, the perfect information of the virtual world of the current frame in different modalities input by the user at the current time can be received, for example: voice input, keyboard input and mouse input.

[0096] When voice input, the voice signal is collected by the microphone , which is used for semantic understanding and dialogue interaction;

[0097] When keyboard input, the key code stream is collected , including key press / release events and time stamps;

[0098] When mouse input, two-dimensional coordinates and click events are collected , wherein, and represent the position of the mouse in the screen at time t, represents whether a click event occurs at time t, when , it means that there is a click event, and when , it means no click event.

[0099] The multi-modal input stream can be collected through the hardware interface, and a high-precision time stamp is added.

[0100] When receiving the perfect information of the virtual world of the current frame in different modalities input by the user at the current time, at this time, the current semantic feature corresponding to the perfect information is obtained , including the following sub-steps A11-A16:

[0101] A11, map the perfect information of the current frame in different modalities to the same time axis by an interpolation function or a resampling function, and obtain the unified time axis information corresponding to each modality.

[0102] Because different modalities have different sampling frequencies (e.g., 16 kHz for speech, and trigger-based for keyboard / mouse events), in order to ensure subsequent unified processing, this step adopts a timestamp alignment and interpolation mechanism:

[0103] Modality input sampling:

[0104] ;

[0105] wherein, denotes the sampling sequence of modality , and denotes the sampling value of modality at time . Here, the time may be the current sampling time point, and at this time can also be referred to as the original signal collected at the current sampling time point, is the sampling time set corresponding to modality .

[0106] Time alignment:

[0107]

[0108] wherein, denotes an interpolation / resampling function that aligns inputs of different modalities to the same time axis; denotes the sampling value of modality at time . The corresponding discrete sampling value is Mathematically, it can be considered that is a continuous-time function, and the sampling time refers to the set of time points at which the sensor / sampling device records data. Due to the limitation of sampling frequency, the device can only record values at certain discrete time points (rather than continuously). is always a subset of t , i.e., the discretization of the sampling time. When interpolation / resampling is performed, it is trying to restore or approximate the continuous-time signal from the discrete sampling values and map it to the same time axis.

[0109] Interpolation refers to estimating the values of unsampled points between sampling points through certain functions (linear, polynomial, spline, deep learning methods, etc.). For example, to map the sampling frequency of the mouse from 60 Hz to 90 Hz, the system will interpolate to fill in the points between the two known points.

[0110] Resampling can change the sampling frequency of a signal to a target frequency, for example, by upsampling (increasing the number of samples) or downsampling (reducing the number of samples) to match the target frequency.

[0111] For example, the sampling frequency of a voice input of 16 kHz must be downsampled when mapped to 90 Hz.

[0112] When the original sampling time points of a signal have already been aligned with the same time axis (for example, keyboard events are discrete triggers consistent with uniform clock events themselves), they can be used directly without additional interpolation or resampling.

[0113] This step maps inputs of different frequencies to a uniform time axis through interpolation / resampling.

[0114] A12. Standardize the uniform time axis information corresponding to each modality to obtain standard information corresponding to each modality.

[0115] This step normalizes numerical values and maps voice to semantic embedding.

[0116] A13. Map the standard information corresponding to each modality to the same dimension to obtain dimension information corresponding to each modality.

[0117] A14. Add type identification and time position coding to the dimension information corresponding to each modality to obtain complete information corresponding to each modality.

[0118] A15. Concatenate the complete information corresponding to each modality to obtain a uniform input vector.

[0119] This step uses the function to concatenate all modalities into a uniform input vector .

[0120] After completing sampling and time alignment, map each modality input to the same feature space to form a uniform input vector:

[0121] The uniform input vector is represented as: ;

[0122] wherein represents the uniform input vector at sampling time t, and the uniform input vector dimension is fixed at d; is a feature concatenation and standardization function.

[0123] The specific implementation of the function includes five steps (Step0-Step4) to ensure that all modality inputs can be mapped to a uniform input vector space with a fixed dimension d after completing time alignment and interpolation / resampling.

[0124] Step 0, assign target sub-dimension for each modality , and satisfy , , , or , denotes the target sub-dimension assigned for speech input, denotes the target sub-dimension assigned for keyboard input, denotes the target sub-dimension assigned for mouse input. Define the existence mask for modality input at each time instant t , denotes , , denotes the mask corresponding to speech input, denotes the mask corresponding to keyboard input, denotes the mask corresponding to mouse input, if the modality is missing, set the zero vector and keep the mask as 0.

[0125] Step 1, standardize the signal of each modality.

[0126] Speech signal s(t): (1) Extract the original speech feature matrix using Mel spectrum or MFCC features; (2) Perform mean-variance normalization: ; (3) Map to semantic vector using speech encoder (CNN / Transformer), here, can also be called the standard information corresponding to speech input; where, denotes the original speech feature matrix extracted by Mel spectrum or MFCC features, denotes the mean of the original speech feature matrix in the time dimension, and σ denotes the standard deviation of the original speech feature matrix in the time dimension, used for scale normalization; ε denotes a small constant to prevent the denominator from being zero, generally taking the value of 10 -6 ; denotes the normalized speech feature matrix, used as input for the subsequent speech encoder (CNN / Transformer); R denotes the set of real numbers; denotes the semantic feature dimension (usually 128, 256, or 512, depending on the model configuration). Therefore, means that the speech input at time t is encoded into a dimensional semantic vector, which is used for subsequent alignment and fusion with other modalities.

[0127] Keyboard input k(t): using One-hot encoding + timestamp normalization; get Here, Also known as the standard information corresponding to the keyboard input; wherein R represents a set of real numbers; Keyboard input feature dimension (usually 128, 256 or 512, according to model configuration) is represented by Dk, so, Means that the keyboard input at time t is encoded into a semantic vector of dimension Dk, Used for subsequent alignment and fusion with other modalities.

[0128] Mouse input : (1) screen coordinate normalization: ; (2) click event is represented by binary or count; (3) concatenated into Here, Also known as the standard information corresponding to the mouse input. Wherein, And represent the position of the mouse in the screen at time t, represents the width of the screen (unit: pixel), and H represents the height of the screen (unit: pixel), used for coordinate normalization; wherein R represents a set of real numbers; Mouse input feature dimension (usually 128, 256 or 512, according to model configuration) is represented by Dm, so, Means that the mouse input at time t is encoded into a semantic vector of dimension Dm, Used for subsequent alignment and fusion with other modalities.

[0129] Step 2, dimension self-adaptation.

[0130] Map the standardized signals of each modality to the same dimension, using the normalization method (linear mapping is used for normalization), the expression is:

[0131] ;

[0132] Wherein, represents , or , represents the dimension information corresponding to the voice input, represents the dimension information corresponding to the keyboard input, represents the dimension information corresponding to the mouse input; represents , or , wherein, represents the self-adaptation mapping matrix corresponding to the voice input, denotes the self-adaption mapping matrix corresponding to the keyboard input, denotes the self-adaption mapping matrix corresponding to the mouse input; denotes , or , denotes the normalized speech input, denotes the normalized keyboard input, denotes the normalized mouse input; denotes , or , wherein, denotes the bias term of the speech input in the linear transformation denotes the bias term of the keyboard input in the linear transformation, denotes the bias term of the mouse input in the linear transformation, and if a certain modality is missing, a zero vector is output and a mask is recorded.

[0133] Step 3, modality identification and time coding.

[0134] The modality embedding vector is added to the dimension information corresponding to each modality (the modality embedding vector represents the identification of the modality, which is used to distinguish data from different sources in a unified feature space, so that the model can learn the feature differences of different modalities), and the time position coding pe(t) (which represents the position information of the input in the time sequence, which is used to maintain the continuity of the time sequence, so that the model can capture the context relationship when processing dynamic signals (such as speech)) is added: ;

[0135] wherein, denotes , or , denotes the complete information corresponding to the speech input, denotes the complete information corresponding to the keyboard input, denotes the complete information corresponding to the mouse input, denotes , or , denotes the speech modality embedding vector, denotes the keyboard modality embedding vector, denotes the mouse modality embedding vector; denotes the linear projection result of the time position coding vector in the dimension , which is used to provide the time sequence position information.

[0136] This step enhances the robustness of the model to modality distinction and time sequence alignment.

[0137] Step 4: splicing and fusion.

[0138] The final unified input vector representation is as follows: ;

[0139] in, It represents a unified input vector, which uses a masking mechanism to ensure that missing modes are automatically set to zero while maintaining a fixed dimension d.

[0140] A16. Obtain the current semantic features corresponding to the unified input vector. .

[0141] A2. Obtain the historical semantic features of the virtual world for a preset number of frames prior to the current frame.

[0142] A3. Obtain contextual representation based on historical and current semantic features. .

[0143] This step will obtain a unified input vector. The information is transformed into standardized specification information (world theme t, character type r, interaction preference e, style s, etc.), and its semantic embedding and conditional fusion vectors are given as direct inputs for subsequent generative world modeling.

[0144] By combining multimodal semantic encoding with temporal aggregation, a stable context representation is obtained. The multimodal encoding representation is as follows: Here, It is not an abstract function, but a multimodal encoder that takes a uniform input vector Converted into semantic features The specific implementation can be divided into the following steps:

[0145] Step 1: Linear mapping and normalization.

[0146] Will The initial embedding is obtained through a single linear mapping: ;

[0147] in, This represents the weight matrix of the linear mapping layer, used to... Projected onto the hidden layer space This represents the bias vector, used for translation correction after mapping. This indicates the initial embedding of the first layer.

[0148] LayerNorm or BatchNorm are used to ensure numerical stability.

[0149] Step 2: Multi-layer encoding (CNN / Transformer optional).

[0150] If the trained Transformer encoder is adopted:

[0151] ;

[0152] The above formula is the standard Transformer encoder layer. Among them, represents a multi-head attention module for capturing feature dependency relationships in the time dimension or between modalities; represents a feedforward network layer for further extracting high-level semantic features; "+" represents a residual connection that can preserve input feature information in the deep structure of the network, thereby enhancing training stability and gradient propagation capability, represents the initial embedding of the layer, represents the initial embedding of the -1 layer.

[0153] Step 3 Time aggregation.

[0154] Align and pool the features of a window to obtain a stable context representation:

[0155] ;

[0156] Among them, represents the context representation after time window aggregation, indicating that the feature at this point contains the continuity information of the time series, g is a positive integer, represents the time step alignment of the feature sequence in the time window, ensuring that different modalities or different frame features correspond on the same time scale.

[0157] In this step, the window alignment and pooling implementation steps include:

[0158] 1. Window segmentation:

[0159] At the current time t, take the semantic feature sequence of the last g frames:

[0160] ;

[0161] Among them, represents the current semantic feature corresponding to the unified input vector, represents the historical semantic features of the virtual world of a predetermined number of frames before the current frame;

[0162] 2. Time alignment:

[0163] If the sampling frequencies of each modality are different, first align The features in the input are mapped to a unified reference time axis through interpolation / resampling to ensure consistent input length.

[0164] 3. Feature pooling:

[0165] Apply aggregation functions to the feature sequences within the window:

[0166] Mean Pooling: ;

[0167] or,

[0168] Max Pooling: ;

[0169] 4. Stabilization and Output:

[0170] Perform on the aggregated vector Normalization (such as LayerNorm) reduces numerical jitter and yields a final, stable context representation. .

[0171] Step 4: Output the context representation .

[0172] A4. Based on the context and complete information, obtain the specification vector corresponding to the type label included in the complete information. .

[0173] In one embodiment, obtaining the specification vector corresponding to the type label included in the completion information based on the context representation and completion information includes the following sub-steps:

[0174] A41. Based on the context, retrieve the type tags included in the complete information.

[0175] Context representation Multi-head classification is performed using a classifier. The formula for multi-head classification is expressed as follows:

[0176]

[0177] The aggregated context representation Perform multi-head classification to obtain... , , and , This represents the classification probability of each topic tag within a theme type. This represents the classification probability of each role label within the role type. This represents the classification probability of each preference label within the interaction preference type. a classification probability of each style label in a style type, a classification weight matrix corresponding to a theme type, a bias vector corresponding to a theme type, a classification weight matrix corresponding to a role type, a bias vector corresponding to a role type, a classification weight matrix corresponding to an interaction preference type, a bias vector corresponding to an interaction preference type, a classification weight matrix corresponding to a style type, a bias vector corresponding to a style type. is a linear classifier that projects the context features into a "theme classification space"; then softmax turns it into a probability distribution, and the rest of the formulas are the same.

[0178] The probability of each type label in each type included in the refined information is predicted through multi-head classification.

[0179] For example, the obtained after multi-head classification in the above steps is a classification probability of each role in a role type, which can also be said to be a probability distribution of the role type obtained through multi-head classification.

[0180] Suppose there are three role labels in the role type: warrior, archer, and mage. The obtained after multi-head classification prediction is [0.15, 0.75, 0.10], that is, the probability of the role type in the refined information being a warrior is 0.15, the probability being an archer is 0.75, and the probability being a mage is 0.10.

[0181] Since the classifier first outputs a probability distribution, not a direct type label, the can be obtained to get the predicted type label. Continue with the above example: since , the type label corresponding to r is an archer. Similarly, .

[0182] In one embodiment, the method further comprises:

[0183] According to the context representation, the probability of the refined information belonging to each category label in each type is obtained; the maximum value in all category label probabilities is obtained; it is detected whether the maximum value is less than a preset threshold; if it is less than, the generation of the virtual world using AI is stopped.

[0184] Specifically, for each type (topic type t, role type r, preference type e, style type s), each type label corresponds to a probability, and the probabilities of all type labels of each type form a probability distribution The maximum probability is selected from all probability distributions as a point estimate result. If the point estimate result is less than a preset threshold, the prediction result is not directly used for virtual world generation, but a safe / conservative backup solution is adopted to ensure the stability and robustness of the entire process, for example, the virtual world is generated in the original templated manner.

[0185] A42, the type label is mapped to the ontology to obtain the corresponding standard label word of each type label.

[0186] The type label obtained by the multi-head classification is mapped to a standardized ontology word table: , thereby obtaining the standard label word .

[0187] The type label predicted above is only an "internal number" and needs to be mapped to a standardized ontology word table. The ontology word table is a "standard set" defined in advance, for example:

[0188] Topic ontology word table: {history, science fiction, fairy tale, city};

[0189] Role ontology word table: {warrior, princess, dinosaur, robot};

[0190] Preference ontology word table: {cooperation, competition, exploration};

[0191] Style ontology word table: {Disney, Cyberpunk, Realistic};

[0192] The above indicates that the predicted type label t is mapped to the standard label word in the "topic ontology word table". For example, assuming that the classifier prediction result is: topic label = "Elsa", then "Elsa" can be mapped to "fairy tale" through the topic ontology word table.

[0193] A43, obtain a specification vector according to all standard label words .

[0194] The type label obtained by the ontology word table mapping in the above step is a discrete symbol, and the model cannot be directly used for calculation. In order to enable the model to further perform feature fusion, reasoning and generation, it is necessary to convert these discrete labels into continuous vector representation.

[0195] By embedding matrix mapping, the label "princess" can become a 128-dimensional vector, and the label "dinosaur" becomes another 128-dimensional vector. These vectors can capture semantic relationships (for example, the vectors of "princess" and "prince" may be closer).

[0196] Specifically, the following steps are included:

[0197] Step 1: Build an embedding dictionary.

[0198] Maintain an embedding matrix for each type (topic type, role type, interaction preference type, style type), and each row in the embedding matrix represents an embedding vector of a type label.

[0199] Embedding matrix It is preset as a random value or a pre-trained vector at system initialization, and is used as a trainable parameter to participate in optimization in the subsequent training process, so as to realize the continuous representation of label semantics.

[0200] ;

[0201] Where, |T| represents the number of topic label categories in the topic type, |R| represents the number of role label categories in the role type, |E| represents the number of interaction preference label categories in the interaction preference type, and |S| represents the number of style label categories in the style type, is the embedding dimension.

[0202] Step 2: Label index lookup.

[0203] For each discrete label mapped by the ontology word table (such as: ), find its index position idx in the corresponding preset type dictionary, and then take out the corresponding embedding vector from the embedding matrix:

[0204] ; Where, represents the corresponding embedding vector, represents the corresponding embedding vector, represents the corresponding embedding vector, represents the corresponding embedding vector.

[0205] Suppose there is a preset role (Role) dictionary: R={warrior, wizard, archer, princess, dinosaur};

[0206] "warrior" -> index position idx=0;

[0207] "wizard" -> index position idx=1;

[0208] "archer" -> index position idx = 2;

[0209] "princess" -> index position idx = 3;

[0210] "dinosaur" -> index position idx = 4;

[0211] When the ontology word table mapping output = princess;

[0212] The system will find the index position of the princess in the preset role dictionary idx = 3;

[0213] Then go to the 3rd row of the embedding matrix Take out the corresponding embedding vector The extracted embedding vector is the embedding vector corresponding to the princess.

[0214] Step3 splice to form a specification vector;

[0215] Splice all embedding vectors: ;

[0216] Step4 engineering implementation details.

[0217] Random initialization can also be used. Pre-trained word vectors (such as word2vec, BERTembedding) can be used to assign similar initial values to semantically similar type labels.

[0218] These embedding matrices are updateable during model training and can be self-adapted and optimized so that labels with similar semantics are also close in vector space.

[0219] The spliced can be connected to LayerNorm or Dropout to improve generalization ability.

[0220] A5, according to the specification vector and the context representation, obtain a conditional vector for generative world modeling.

[0221] Fuse the specification vector and the context representation to form a conditional vector for generative world modeling. When fusing, linear fusion can be used, wherein represents the specification vector, represents the context representation, and and are spliced and multiplied by , and are trainable parameters (weight matrix and bias term, not preset dead, which will be learned during training).

[0222] A6, Obtain visual context from key frame and pre-set number of frames before current frame .

[0223] key frame with history frames , providing temporal reference. The system automatically determines key frame Ic from user input events, which can be provided by the user, system timing sampling or interactive event triggered generation; at the same time, the system maintains a time sliding cache for recording frames n recent frames to form a history frame set . Key frame and history frame together serve as input for generative model, providing temporal context and visual anchor information, ensuring the continuity and stability of the generated sequence in terms of temporal and spatial structure.

[0224] The visual context is obtained by the following formula: ;

[0225] Where, is the visual context, also known as visual latent variable, representing the temporal context of the scene, Enc() is a 3D causal convolutional encoder or a Transformer encoder.

[0226] A7, Obtain the current latent variable of the virtual world of the current frame .

[0227] A8, Predict the latent variable of the next time according to the current latent variable, visual context and condition vector .

[0228] In the latent variable space, use the pre-trained diffusion-Transformer model to predict the latent variable of the future frame, realizing the continuous dynamic generation of the world.

[0229] Diffusion prediction iteration:

[0230] ;

[0231] Where, represents the current latent variable of the virtual world of the current frame, represents the visual context, represents the condition vector, represents the latent variable of the next time, represents the pre-trained diffusion-Transformer model, represents the Gaussian noise term, which is used to maintain the randomness and diversity of the diffusion denoising process.

[0232] In the initialization phase, the current latent variable can be mapped by the input scene frame through the encoder to obtain the latent variable representation, or randomly sampled from the Gaussian distribution, and in the iteration phase, the prediction result of the previous time is recursively obtained. Through the above mechanism, the system forms a continuous time sequence representation in the latent space, so that the generated virtual world has dynamic consistency.

[0233] The first frame can be regarded as a "hidden snapshot of the current world", which is obtained from the scene input; and each subsequent frame is calculated by the model according to the previous frame.

[0234] A9, according to the latent variable of the next time to obtain the virtual world of the next frame.

[0235] In one embodiment, the latent variable of the next time to obtain the virtual world of the next frame, including the following sub-steps A91-A94:

[0236] A91, according to the latent variable of the next time to obtain the next frame.

[0237] The decoding function is represented as: ;

[0238] wherein, represents a 3D causal decoder; represents the generated next frame, which can also be referred to as the frame corresponding to the next time; represents the latent variable of the next time.

[0239] A92, detecting whether there is a role corresponding to the specification label word in the specification label word.

[0240] If not, the next frame obtained above is the virtual world of the next frame.

[0241] A93, if there is, a virtual role is created according to the specification vector , the role corresponding to the specification label word and the condition vector.

[0242] A94, according to the virtual role and the next frame to obtain the virtual world of the next frame.

[0243] If there is a role corresponding to the specification label word in the specification label word, a virtual role needs to be created.

[0244] In the generated world, a role consistent with the user's setting is created according to the role label and the style label in the specification vector, and the role generation formula is: ; ​

[0245] in, For generated virtual characters; Generate functions for the character and bind the skeleton and semantic interface.

[0246] The output avatar has a semantic interface, which is used to implement language / command interaction.

[0247] In one embodiment, the method further includes: obtaining the semantic response of the character at the current moment based on the unified input vector, specification vector, and condition vector; at this time, after outputting the virtual world of the next frame, the method further includes: outputting the semantic response in the virtual world of the next frame.

[0248] The virtual avatar needs to support language / dialogue interaction, so a semantic interface is bound to it. The semantic interface mapping function is represented as follows: ;

[0249] in, This represents the semantic response (text / voice / expression) of a virtual character at time t. This represents a semantic decoding function, combined with a unified input vector. Specification embedding With condition vector The semantic parsing function can be implemented by a trained semantic decoding model or by calling an existing natural language processing model. This interface ensures that the character can generate personalized language during interaction.

[0250] This step is used for semantic interface registration, that is, through... Map the input to a semantic response .

[0251] In one embodiment, the method further includes: obtaining the current global state machine of the currently output virtual world; obtaining a migration token based on user identity information; obtaining the target global state of the target virtual world to be migrated at the current moment; obtaining the role attribute information of the virtual character based on the current global state machine; obtaining the global state of the target virtual world at the next moment based on the role attribute information and the target global state; and outputting the global state of the target virtual world at the next moment.

[0252] This step enables cross-world migration, allowing users' virtual characters to seamlessly migrate between multiple virtual worlds. This mechanism ensures the scalability of "interoperability between multiple virtual worlds" and is the core of realizing metaverse-level cross-world social interaction.

[0253] At the start of the migration, the system generates a migration token. To ensure the uniqueness of migrations, migration tokens can be implemented using the following formula: ;in, represents a migration token; represents a user identity; represents a virtual role attribute snapshot.

[0254] In order to ensure the security of migration, the migration token can also be implemented by the following formula: ; wherein, represents a migration token; represents a user identity; Sig represents a digital signature, which is used to prevent forgery.

[0255] When the virtual role enters the target virtual world, the system updates the global state machine of the target virtual world by the following formula: ; represents the global state of the target virtual world j at time t; represents the global state of the target virtual world j at the next time .

[0256] The present disclosure also provides a virtual world map mapping function, which aims to establish a unified "virtual world map" for the virtual worlds generated by multiple users, and is implemented through a graph structure and a vector index, and is mainly used for: registration and unique identification of multiple virtual worlds; relationship modeling between virtual worlds (migratable, social, similarity); retrieval and recommendation of virtual world entrances; two-dimensional / three-dimensional visual display of the virtual world map, facilitating user selection of cross-virtual world access.

[0257] In implementing this step, the inputs are: virtual world metadata: virtual world theme, style, rules, load, delay, and other information; user portrait and access intention: semantic labels and conditional vectors from the above steps; inter-virtual world relationship data: friend relationship, similarity, interaction record.

[0258] First, the registration and unique identification of multiple virtual worlds, this step assigns a globally unique to each generated virtual world at the time of registration, which is implemented by the following formula:

[0259] ;

[0260] wherein, represents a virtual world node; represents the semantic specification of the virtual world (theme, style, rule summary); represents the creator identity; represents the generation timestamp; represents a hash function, such as SHA-256, used to ensure global uniqueness.

[0261] The system can call the registry service to calculate and stored into the distributed database as the unique entry for subsequent retrieval and cross-virtual-world access.

[0262] Secondly, the relationship modeling between virtual worlds, including: all virtual worlds are mapped as a graph structure: where V represents the set of virtual worlds, each represents a virtual world; E represents the set of edges, each represents the relationship between two virtual worlds, such as transferable, friends, similar themes, etc.

[0263] Node attribute function: ;

[0264] Edge attribute function: ;

[0265] where theme represents the content theme or main semantic category of the virtual world, such as "prehistoric era", "Frozen", etc., represents the visual and performance style of the virtual world (such as realistic, cartoon, two-dimensional, steam punk, etc.), which is the rendering attribute of the world itself, not the interaction relationship with other worlds, so it belongs to the node attribute; represents the environmental constraints and interaction logic within the virtual world, such as physical rules, social mechanisms, task systems, etc., which are only effective within the world, so they also belong to the node attribute; load represents the system resource occupation rate or online user number of the virtual world, used for dynamic scheduling; latency represents the average network latency for accessing the virtual world; similarity represents the semantic similarity between two virtual worlds, represents the strength of social relationship; protocol represents the interactive communication protocol or transfer rule between virtual worlds, which defines the compatible way of information format, interactive interface, permission verification, etc. when the character or data is migrated from one virtual world to another.

[0266] For users to recommend accessible virtual worlds, including: the system generates a world semantic vector for each virtual world, and the user's intention vector is denoted as , the similarity between the user and the virtual world is calculated by the cosine similarity:

[0267] ;

[0268] where, represents the similarity between the user and the virtual world v, ranging from [0, 1]; represents the intention vector of the user ; a semantic vector representing the virtual world v.

[0269] The system takes the top K similar virtual worlds as recommended virtual worlds that the user can enter, and the user can select the recommended virtual worlds in the map.

[0270] Map layout and visualization steps:

[0271] The system uses graph embedding or force-directed algorithms to embed virtual world nodes in a two-dimensional or three-dimensional coordinate system to generate an interactive map.

[0272] ;

[0273] wherein, represents a virtual world node layout energy function, which is used to control the spatial distribution and visualization effect of the virtual world in a two-dimensional or three-dimensional coordinate system, represents the horizontal coordinate of the virtual world node, represents the vertical coordinate of the virtual world node; represents the weight (similarity / social strength) between virtual world nodes; represents a regularization coefficient.

[0274] The present disclosure also provides rendering and state synchronization functions, which are used to ensure that the rendering screens of all clients are consistent and the state machines are synchronized in a multi-person concurrent virtual world. Rendering provides users with a smooth and immersive visual experience; synchronization ensures that the global state machine remains consistent across different clients, avoiding "different screens and state conflicts".

[0275] When performing rendering and state synchronization, the following steps are included:

[0276] S1: Server global state update.

[0277] The server updates the world global state according to the operation set of all users

[0278] ;

[0279] wherein, represents the virtual world global state at time t, including scene objects, character attributes, interaction relationships, etc. represents the virtual world global state at the previous time represents the operation set (Action Set) of all users at time t, including interaction behaviors, movements, task triggers, etc.; F( ) represents a state update function, which is used to update the virtual world global state at the previous time and the operation set ​​​Compute global state change at time t.

[0280] S2: Incremental computation and broadcast.

[0281] To reduce bandwidth, the server only computes the increment from the previous state and broadcasts:

[0282] ;

[0283] S3: Client prediction and rollback.

[0284] When the client does not receive the server update, it uses a prediction function to generate a temporary state:

[0285] ;

[0286] where, denotes the predicted state; denotes the unified input vector; denotes the prediction function.

[0287] S4: Conflict detection and consistency guarantee.

[0288] In the case of high concurrency, if different user operations conflict, the system determines the operation order through the timestamp / vector clock.

[0289] ;

[0290] where, , and denote two different user or client submitted operation events, denote the logical clock of operation , used to record the time order of event occurrence; PID(a i ) denotes the unique identification of the process or client to which operation a i belongs; denote the logical clock of operation , used to record the time order of event occurrence; PID(a j ) denotes the unique identification of the process or client to which operation a j belongs, and ≺ denotes the precedence relationship.

[0291] S5: Consistency condition.

[0292] Finally, all clients should converge to a consistent state:

[0293] .

[0294] where, denotes the universal quantifier, indicating for all, Any two clients, Clients at time t Held world state, Clients at time t Held world state.

[0295] S6: Render layer execution.

[0296] The synchronized state is input to the rendering engine to generate a unified picture:

[0297] Using frame interpolation and buffering mechanisms to ensure smooth pictures;

[0298] Ensure that all users see consistent virtual world changes at the same time.

[0299] With the development of artificial intelligence, virtual reality and game engines, immersive virtual worlds have gradually become an important research and application direction. Current virtual world construction mainly relies on artificial modeling and commercial game engines (such as Unity, Unreal Engine). In this mode, scenes and characters are often designed and rendered in advance by developers, and users can only enter the preset world for limited interaction. Although this solution can provide basic three-dimensional immersive experience, it has the following shortcomings: high development cost, low expansion efficiency, insufficient personalization and single interaction mode.

[0300] The existing relatively close implementation solutions mainly have three categories:

[0301] Metaverse platforms (such as Roblox, Meta Horizon Worlds).

[0302] This type of platform allows users to enter three-dimensional scenes through virtual avatars and socialize, but its scenes are still manually constructed by developers or users through editors, lacking AI automatic generation capabilities, and the world creation threshold for users is high.

[0303] Virtual meeting and social platforms (such as VRChat, AltspaceVR).

[0304] Users can communicate through voice or activities in preset virtual spaces, but the interaction form is limited to action commands and voice calls, making it difficult to achieve natural semantic understanding, and lacking cross-world interaction mechanisms.

[0305] AI-driven virtual character applications.

[0306] Some systems introduce large language models to achieve dialogue or companionship with virtual characters, but mostly stay within two-dimensional interfaces or single scenes, unable to generate complete virtual worlds, and lacking immersive interaction and social functions for multiple users.

[0307] In summary, although the prior art provides virtual world experience and social functions to some extent, it still has the following defects:

[0308] 1. Lack of personalization.

[0309] Existing systems mainly rely on manual modeling and fixed templates to generate scenes, and users can only act in pre-set environments.

[0310] Users cannot quickly generate unique virtual worlds and character images according to their interests.

[0311] Virtual worlds are highly homogenized, and users lack long-term immersion and motivation to create.

[0312] The present disclosure allows users to generate exclusive worlds and virtual images according to their interests, such as generating a T-Rex in a Jurassic scene or an Elsa world with an Ice and Snow theme, greatly improving immersion and creativity.

[0313] 2. Single interaction mode.

[0314] The interaction mode mainly relies on fixed action libraries and voice calls, without combining AI semantic understanding and multi-modal perception.

[0315] User interactions are limited to simple chatting, lacking real and natural communication experience.

[0316] Virtual worlds lack depth in social atmosphere, making it difficult to form a high-stickiness community.

[0317] The present disclosure can provide multiple interaction modes, including voice, keyboard, and mouse.

[0318] 3. Lack of cross-world interaction.

[0319] Virtual worlds in existing platforms are independent of each other, without establishing a cross-world migration and interconnection mechanism.

[0320] User characters cannot enter other virtual worlds, limiting social interactions to a single scene.

[0321] User interactions are fragmented, and social networks cannot be expanded.

[0322] The present disclosure allows different users' worlds to interconnect through a world map mechanism, and characters can freely migrate, such as a T-Rex entering Elsa's world for interaction, enabling true cross-scene socialization.

[0323] 4. Insufficient character social system.

[0324] Character attributes are mostly static, social activities are pre-set by humans, and there is a lack of dynamic generation mechanism.

[0325] Users lack goals and feedback in long-term use, and immersion gradually decreases.

[0326] User activity is insufficient, and the platform is difficult to maintain long-term operation.

[0327] Therefore, the purpose of the present disclosure is to propose an AI large model-based virtual world generation and interaction method and device, which can automatically generate personalized virtual worlds, support role migration and interaction across worlds, and combine role growth systems and intelligent social systems, thereby significantly improving the immersion, scalability, and user stickiness of virtual worlds, and the AI large model automatically generates the world, reducing the cost of artificial modeling; The world map mechanism ensures the infinite expansion of the virtual world and the consistency of multi-user concurrent interaction.

[0328] Based on the same inventive concept, the embodiments of the present application also provide an AI-based virtual world generation system for implementing the AI-based virtual world generation method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more AI-based virtual world generation system embodiments provided below can refer to the limitations of the AI-based virtual world generation method described above, which will not be repeated here.

[0329] In one exemplary embodiment, as shown in Figure 4 An AI-based virtual world generation system is provided, which includes:

[0330] A receiving module is configured to receive perfect information of a virtual world of a current frame input by a user at a current time, wherein the perfect information includes at least one of the following information: theme type, role type, interaction preference type, style type, and control signal.

[0331] A processing module is configured to add the perfect information to the virtual world of the current frame by using an AI large model to obtain a virtual world of a next frame.

[0332] An output module is configured to output the virtual world of the next frame.

[0333] In the aspect of adding the perfect information to the virtual world of the current frame to obtain the virtual world of the next frame, the processing module is specifically configured to:

[0334] Obtain a current semantic feature corresponding to the perfect information.

[0335] Obtain a historical semantic feature of a preset number of frames of the virtual world before the current frame.

[0336] Obtain a context representation according to the historical semantic feature and the current semantic feature.

[0337] Obtain a specification vector corresponding to a type label included in the perfect information according to the context representation and the perfect information.

[0338] obtaining a condition vector for generative world modeling according to the specification vector and the context representation;

[0339] obtaining a visual context according to a preset number of frames before the key frame and the current frame;

[0340] obtaining a current latent variable of the virtual world of the current frame;

[0341] predicting a latent variable of a next time according to the current latent variable, the visual context and the condition vector;

[0342] obtaining a virtual world of a next frame according to the latent variable of the next time.

[0343] In an embodiment, in the aspect of obtaining the specification vector corresponding to the type label included in the refinement information according to the context representation and the refinement information, the processing module is specifically configured to:

[0344] obtaining a type label included in the refinement information according to the context representation;

[0345] performing ontology mapping on the type label to obtain a specification label word corresponding to each type label;

[0346] obtaining the specification vector according to all the specification label words.

[0347] In an embodiment, in the aspect of obtaining the virtual world of the next frame according to the latent variable of the next time, the processing module is specifically configured to:

[0348] obtaining a scene of a next frame according to the latent variable of the next time;

[0349] detecting whether there is a specification label word corresponding to a role in the specification label word;

[0350] if there is, creating a role according to the specification vector, the specification label word corresponding to the role and the condition vector;

[0351] obtaining the virtual world of the next frame according to the role and the scene of the next frame.

[0352] In an embodiment, in the aspect of receiving the refinement information of the virtual world of the current frame input by the user at the current time, the receiving module is specifically configured to:

[0353] receiving the refinement information of the virtual world of the current frame in different modalities input by the user at the current time;

[0354] In the aspect of obtaining the current semantic feature corresponding to the refinement information, the processing module is specifically configured to:

[0355] The perfect information of the current frame of different modalities is mapped to the same time axis by an interpolation function or a resampling function, to obtain unified time axis information corresponding to each modality.

[0356] The unified time axis information corresponding to each modality is standardized to obtain standard information corresponding to each modality.

[0357] The standard information corresponding to each modality is mapped to the same dimension to obtain dimension information corresponding to each modality.

[0358] The type identifier and the time position code are added in the dimension information corresponding to each modality to obtain complete information corresponding to each modality.

[0359] The complete information corresponding to each modality is spliced to obtain a unified input vector.

[0360] The current semantic feature corresponding to the unified input vector is obtained.

[0361] In an embodiment, the processing module is further configured to:

[0362] According to the unified input vector, the specification vector and the condition vector, a semantic response of the role at the current time is obtained.

[0363] The output module is further configured to, after outputting the next frame of the virtual world, output the semantic response in the next frame of the virtual world.

[0364] In an embodiment, the processing module is further configured to:

[0365] A current global state machine of the current output virtual world is obtained.

[0366] A migration token is obtained according to the user identity information.

[0367] A target global state of the target virtual world to be migrated at the current time is obtained.

[0368] Role attribute information of the role is obtained according to the current global state machine.

[0369] The global state of the target virtual world at the next time is obtained according to the role attribute information and the target global state.

[0370] The global state of the target virtual world at the next time is output.

[0371] In an embodiment, the processing module is further configured to:

[0372] According to the context representation, the probability that the perfect information belongs to each category in each type is obtained.

[0373] The maximum value in each category probability in each type is obtained.

[0374] detecting whether a category with a maximum value less than a preset threshold value appears;

[0375] If it appears, stop using AI to generate a virtual world.

[0376] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0377] The principles and implementation modes of the present application are described by using specific examples herein, and the above embodiments are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation modes and application ranges will be changed according to the idea of the present application. In conclusion, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A method for generating a virtual world based on AI, the method comprising: The method comprises: receiving perfect information of a current frame of a virtual world input by a user at a current time, the perfect information comprising at least one of the following: theme type, character type, interaction preference type, style type and control signal; adding the perfect information to the current frame of the virtual world by an AI large model to obtain a next frame of the virtual world; outputting the next frame of the virtual world. The adding of the perfect information to the current frame of the virtual world to obtain the next frame of the virtual world comprises: obtaining a current semantic feature corresponding to the perfect information; obtaining a historical semantic feature of a preset number of frames of the virtual world before the current frame; obtaining a context representation according to the historical semantic feature and the current semantic feature; obtaining a specification vector corresponding to a type label included in the perfect information according to the context representation and the perfect information; obtaining a conditional vector for generative world modeling according to the specification vector and the context representation; obtaining a visual context representing a scene according to a key frame and a preset number of frames before the current frame; obtaining a current latent variable of the virtual world of the current frame; predicting a latent variable at a next time according to the current latent variable, the visual context and the conditional vector; the current latent variable is obtained by mapping a scene frame by an encoder; obtaining the next frame of the virtual world according to the latent variable at the next time.

2. The method of claim 1, wherein, The obtaining of the specification vector corresponding to the type label included in the perfect information according to the context representation and the perfect information comprises: obtaining the type label included in the perfect information according to the context representation; performing ontology mapping on the type label to obtain a specification label word corresponding to each type label; obtaining the specification vector according to all the specification label words.

3. The method of claim 2, wherein, The obtaining of the next frame of the virtual world according to the latent variable at the next time comprises: obtaining the next frame according to the latent variable at the next time; detecting whether there is a specification label word corresponding to a character in the specification label words; if yes, creating a virtual character according to the specification vector, the specification label word corresponding to the character and the conditional vector; obtaining the next frame of the virtual world according to the virtual character and the next frame.

4. The method of claim 3, wherein, The receiving of the perfect information of the current frame of the virtual world input by the user at the current time comprises: receiving the perfect information of the current frame of the virtual world in different modalities input by the user at the current time. The obtaining of the current semantic feature corresponding to the perfect information comprises: mapping the perfect information of the current frame in different modalities to a same time axis by an interpolation function or a resampling function to obtain unified time axis information corresponding to each modality; performing standardization processing on the unified time axis information corresponding to each modality to obtain standard information corresponding to each modality; mapping the standard information corresponding to each modality to a same dimension to obtain dimension information corresponding to each modality; adding a type identifier and a time position code in the dimension information corresponding to each modality to obtain complete information corresponding to each modality; splicing the complete information corresponding to each modality to obtain a unified input vector; obtaining a current semantic feature corresponding to the unified input vector.

5. The method of claim 4, wherein, The method further comprises: obtaining a semantic response of a virtual character at the current time according to the unified input vector, the specification vector and the conditional vector. After the outputting of the next frame of the virtual world, the method further comprises: outputting the semantic response in the next frame of the virtual world.

6. The method of claim 5, wherein, The method further comprises: obtaining a current global state machine of a current output virtual world; obtaining a migration token according to user identity information; obtaining a target global state of a target virtual world to be migrated at a current time; obtaining role attribute information of a role according to the current global state machine; obtaining a global state of the target virtual world at a next time according to the role attribute information and the target global state; outputting the global state of the target virtual world at the next time.

7. The method of claim 6, wherein, The method further comprises: obtaining probabilities of perfect information belonging to various category labels in various types according to the context representation; obtaining a maximum value in all category label probabilities; detecting whether the maximum value is less than a preset threshold value; if less than, stopping using AI to generate a virtual world. 8.A system for generating a virtual world based on AI, the system comprising: The system comprises: a receiving module configured to receive perfect information of a current frame of a virtual world input by a user at a current time, the perfect information comprising at least one of the following information: theme type, role type, interaction preference type, style type, and control signal; a processing module configured to add the perfect information to the current frame of the virtual world by using an AI large model to obtain a next frame of the virtual world; an output module configured to output the next frame of the virtual world. In the aspect of adding the perfect information to the current frame of the virtual world to obtain the next frame of the virtual world, the processing module is specifically configured to: obtain a current semantic feature corresponding to the perfect information; obtain historical semantic features of a preset number of frames of the virtual world before the current frame; obtain a context representation according to the historical semantic features and the current semantic feature; obtain a specification vector corresponding to a type label included in the perfect information according to the context representation and the perfect information; obtain a condition vector for generative world modeling according to the specification vector and the context representation; obtain a visual context representing a scene according to a key frame and a preset number of frames before the current frame; obtain a current latent variable of the virtual world of the current frame; predict a next-time latent variable according to the current latent variable, the visual context, and the condition vector; the current latent variable is obtained by mapping a scene frame by an encoder; obtain the next frame of the virtual world according to the next-time latent variable.

Images