A virtual reality system and method for providing a virtual reality experience to a user

Abstract

The application provides a kind of based on 3D virtual space's realistic embodied AI interaction system and its implementation method, belong to artificial intelligence and virtual reality technical field, the system includes: open object visual space positioning system, for identifying the environmental information under current visual angle / panoramic field of view by visual language model;Hierarchical memory and narrative memory generation system, for establishing hierarchical memory architecture, store the memory information corresponding to conversation;Director engine and context assembly system, for assembling structured context information, task reasoning is carried out by director engine;Natural language driven multi-track generative animation system, for arranging time sequence to multiple track instructions by large language model, and compiling into target instruction;In front-end analysis and execution the target instruction, generate the animation sequence of the role.The AI interaction system of the application has visual perception, autonomous space movement, emotional state expression and continuous behavior control ability, and improves the sense of reality.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and virtual reality technology, and in particular to a 3D virtual space-based embody AI interactive system and its implementation method. Background Technology

[0002] Currently, with the rapid development of large language models and multimodal technologies, AI-powered interactive products are undergoing a paradigm shift from "text-based dialogue" to "realistic companionship." Dialogue systems such as Character.ai and Replika have already achieved highly human-like emotional interaction, while 3D products like NYXverse and Starry Night Song are attempting to extend dialogue capabilities into virtual spaces, providing users with a more immersive social experience.

[0003] Existing AI character interaction systems based on 3D virtual space still have fundamental limitations in terms of embodyation and spatial intelligence, specifically manifested in the following four core technical bottlenecks: (I) Lack of Spatial Perception and Visual Pathfinding Capabilities: Existing AI characters generally lack open visual pathfinding capabilities in 3D virtual environments. Mainstream products adopt an interaction scheme of "points of interest + track switching," where characters can only teleport between preset points or slide along fixed tracks, unable to "see the target and walk there autonomously" like humans—that is, lacking the end-to-end capability of directly mapping visual input into continuous navigation actions. Although the field of computer vision has accumulated a large amount of positioning technology, target detection solutions such as the YOLO series can only identify objects of predefined categories and cannot handle arbitrary targets in open scenes; while visual language models have zero-sample recognition capabilities, their accuracy is low when directly outputting 3D coordinates, with systematic errors at the tens of centimeters or even meters, making it difficult to support accurate pathfinding. This capability gap prevents AI characters from truly "understanding" spatial layouts, let alone autonomously moving to any visible location in their field of vision according to user instructions.

[0004] (II) Low Degree of Freedom and High Production Costs in Animation Control: In terms of character animation, existing digital human systems generally adopt a single-channel mode of "one sentence triggers one pre-made animation". Most NPCs in mainstream animation systems can only play a limited number of preset actions. The construction of the action library relies on artists manually keying each frame or cleaning motion capture data. The production cycle of a single complex action is measured in hours. Even with a multi-track hybrid approach, there is still the rigidity of "frame-by-frame switching"—there is a lack of natural semantic transitions between actions, and it is impossible to generate continuous, diverse, and semantically consistent action sequences based on real-time instructions. In other words, existing systems lack the ability to generate "free and continuous actions" in real time, which severely limits the expressiveness and realism of AI characters.

[0005] (III) Bottlenecks in Planning and Decomposition of Complex Tasks: At the behavioral decision-making level, traditional AI dialogue systems adopt a reactive "question-and-answer" architecture, which cannot automatically decompose complex goals such as "help me organize the books on the table onto the bookshelf" into multi-step sub-tasks such as "locating the table → walking to the table → grabbing the book → locating the bookshelf → walking to the bookshelf → placing the book" and execute them sequentially. This lack of long-term task planning capabilities causes AI characters to either not respond or exhibit broken behavioral logic when faced with multi-step instructions. Although the field of embodied intelligence has proposed frameworks such as hierarchical planning and end-to-end VLA models, these solutions are currently mainly aimed at physical robots, and their lightweight and real-time deployment on virtual characters is not yet mature.

[0006] (iv) System Latency and Pipeline Efficiency Issues: In real-time interactive scenarios, traditional architectures employ a "serial perception-inference-execution" process, resulting in severe response latency. In a typical scenario, each time the AI ​​moves to a new location, it must sequentially complete: scene rendering and capture (~0.5 seconds) → visual language model analysis (~6 seconds) → large language model decision (~5 seconds) → action command generation (~3 seconds), totaling approximately 14 seconds of "standing still" waiting time. This frame-by-frame processing mode not only disrupts the immersive experience of the interaction but also highlights the lack of overall planning and pipeline scheduling capabilities for multi-step actions in existing systems. Although recent research has proposed efficiency optimization methods such as adaptive token pruning and streaming action generation, integrating them into a unified immersive interactive system remains a technological gap.

[0007] In summary, existing AI character interaction products have significant technical shortcomings in dimensions such as spatial perception, action generation, task planning, and real-time response, making it difficult to support truly realistic and empathetic interaction. Therefore, there is an urgent need for a novel realistic and empathetic AI interaction system architecture to at least partially solve the problems existing in the aforementioned technologies. Summary of the Invention

[0008] This invention provides a 3D virtual space-based embody AI interaction system and its implementation method, which addresses the shortcomings of existing AI character interaction products in lacking proactive spatial perception and autonomous behavioral decision-making capabilities, enabling AI characters to possess visual perception, spatial movement, emotional expression, and continuous behavioral control capabilities.

[0009] In a first aspect, the present invention provides a 3D virtual space-based embody AI interaction system, comprising: An open object visual spatial positioning system is used to acquire environmental images of a character in the current viewpoint / panoramic field of view when receiving task information input by the user, and to identify environmental information in the current viewpoint / panoramic field of view through a visual language model; A hierarchical memory and narrative memory generation system is used to establish a hierarchical memory architecture at the task level, conversation level, and cross-conversation level. After the conversation ends, the fragmented sequences in the conversation are converted into narrative memory text through a large language model, and the narrative memory text is converted into memory information and stored in the hierarchical memory architecture. The director engine and context assembly system are used to assemble the task information, environment information, memory information, location information and state information corresponding to the character into structured context information, and input the structured context information into the director engine, perform task reasoning through the director engine and output the character behavior planning result; wherein, the character behavior planning result includes at least one behavior unit; A natural language-driven multi-track generative animation system is used to define multiple track instructions corresponding to the behavior unit through a domain-specific language; to perform time-series arrangement of the multiple track instructions corresponding to each behavior unit through a large language model, and compile them into target instructions corresponding to the behavior unit; and to parse and execute the target instructions at the front end to generate the animation sequence of the character.

[0010] According to the present invention, a 3D virtual space-based embody AI interaction system is provided, wherein the open object visual space positioning system comprises: The target view environment scanning module is used to capture a single environment image from the character's current viewpoint, divide the single environment image into a grid, perform coarse localization of each object in the current viewpoint environment using a visual language model, crop out the target area based on the coarse localization of each object, and generate environmental information under the current viewpoint using the visual language model; wherein, the environmental information under the current viewpoint includes a scene description and a list of objects under the current viewpoint. A panoramic view environment scanning module is used to acquire environmental images from multiple directions within a panoramic view; stitch the environmental images from multiple directions into a panoramic image, and divide the panoramic image into a grid; coarsely locate target objects in the panoramic view environment using the visual language model; determine the sub-image where the target object is located based on the coarse location of the target object; and generate environmental information under the panoramic view using the visual language model; wherein, the environmental information under the panoramic view includes a scene description under the panoramic view and the position information of the target object; the target object is related to the target point in the task information; The coordinate transformation and automatic pathfinding module is used to transform the position information of the target object into a three-dimensional world coordinate system, perform path planning through ray casting and navigation mesh sampling, and drive the character to move towards the target point according to the path planning result.

[0011] According to the present invention, a 3D virtual space-based embody AI interaction system is provided, wherein the hierarchical memory and narrative memory generation system comprises: A hierarchical memory architecture is used to establish a hierarchical memory architecture of task level, session level, and cross-session level. The task level stores multi-turn structured context information of the director engine during task inference. The session level stores session-level fragment sequences, using fragments from each task within the entire session as the basic unit. When the number of fragments reaches a preset fragment number threshold, compression is automatically triggered, and the compressed session-level fragment sequence is written to the cross-session level, which then initiates a new session record. The cross-session level stores the memory information of each session, forming a cross-session vector database. The narrative memory generation module is used to format the fragmented sequence in the session into a labeled text list in chronological order after the session ends; input the text list into a large language model, output the narrative memory text, and vectorize and store the narrative memory text in the cross-session vector database; wherein, the cross-session vector data supports the director engine's semantic retrieval and recent memory retrieval, and provides memory information injection for the director engine's task reasoning process.

[0012] The present invention provides a 3D virtual space-based embody AI interaction system. The types of fragments include: The specific action descriptions performed by the character, the character's voice content, the concise summary of the director engine during the task reasoning process, the original task text input by the user, the task execution result information, and the scene description generated by the target perspective environment scanning module or the panoramic view environment scanning module. The memory information includes the following fields: Narrative text in the third person; The text embedding vector corresponding to the narrative text; Importance scores assessed using a large language model; Unix timestamps generated from memory information; The memory source type includes user-initiated sessions, user-initiated autonomous actions without user participation, and mixed sessions generated by user intervention during the execution of user-initiated autonomous actions. The dominant sentiment label when information is written to memory; Tag list.

[0013] According to the present invention, a 3D virtual space-based embody AI interaction system is provided, wherein the director engine and context assembly system include: The context assembly module is used to assemble the task information, environment information, memory information, location information and state information corresponding to the character into structured context information before each call to the director engine; The director engine is used to perform task reasoning based on the structured context information and output character behavior planning results according to planning principles under different scenarios. These planning principles include: if the current viewpoint information is sufficient to complete the task, or if environmental information under a panoramic view is required but has not yet been acquired, a single behavior unit is output; when the complete information required to complete the task is known, multiple behavior units are planned and output based on the continuous actions required to complete the task; for behavior units that require a panoramic view environment to determine, task reasoning is re-performed after the panoramic view environment scanning module completes its work; each behavior unit includes a configuration description corresponding to multiple trajectory commands of the character.

[0014] According to the present invention, a 3D virtual space-based embody AI interaction system is provided, wherein the natural language-driven multi-track generative animation system includes: The multi-track instruction definition module is used to define multiple track instructions corresponding to each behavior unit through a domain-specific language. The multiple track instructions include animation track instructions for controlling skeletal animation, expression track instructions for controlling character facial expressions, movement track instructions for controlling character spatial movement, subtitle track instructions for displaying subtitle text, and special track instructions carrying special instructions. The target instruction generation module is used to perform time-series arrangement of multiple track instructions corresponding to each behavior unit through a large language model, determine the start time and duration of each track instruction, and output a key frame array corresponding to the multiple track instructions; group the key frame array according to track instructions, sort the key frames according to the start time within each track instruction, generate string fragments corresponding to each track instruction, and concatenate the string fragments corresponding to each track instruction to generate the target instruction corresponding to the behavior unit; The animation generation module is used to parse the target instructions at the front end, create a coroutine queue in ascending order of the start time of each track instruction, trigger the operation of the corresponding track instruction when the corresponding time point is reached, and continuously drive the skeletal animation by superimposing the breathing channel, micro-swaying channel, head shaking channel and blinking channel to generate the animation sequence of the character.

[0015] According to the present invention, a 3D virtual space-based embody AI interaction system is provided, the system further comprising: An autonomous behavior-driven system is used to maintain multiple independent intrinsic demand channels. During the standby state of the role, the autonomous demand value is accumulated and calculated. When the autonomous demand value reaches the demand threshold, an autonomous task is planned in advance. When the triggering condition of the autonomous task is met, the autonomous task of the role is triggered. Among them, the priority of user input tasks is higher than the priority of autonomous tasks. When user input tasks are detected, the autonomous task is paused.

[0016] According to the present invention, a realistic embody AI interaction system based on 3D virtual space is provided. When the character moves towards the target point, the prefetching mode is injected into the target instruction through the special track instruction. After the front end receives the target instruction, the environmental information of the target point under the panoramic view is identified in advance through the panoramic view environment scanning module, and the movement track instruction in the target instruction is executed synchronously. When the character reaches the target point, if the environmental information of the target point in the panoramic view has been obtained, the next round of animation sequence of the character is immediately generated through the director engine, the context assembly system, and the natural language-driven multi-track generative animation system; wherein, the orientation of the character when it reaches the target point is corrected according to the absolute yaw angle of the virtual camera in the prefetch mode; the multi-round director engine reasoning process within the same task shares memory information.

[0017] According to the present invention, a 3D virtual space-based embody AI interaction system is provided, wherein during the reasoning process, the visual language model, the large language model, and the director engine continuously run the breathing channel, micro-swaying channel, head shaking channel, and blinking channel to drive skeletal animation. In the character behavior planning result, the facial expression state of the facial expression track instruction in the previous behavior unit is passed to the facial expression track instruction in the next behavior unit through a preset weight; If the character is executing a movement track command, the walking animation will continue to play until the target point is reached.

[0018] Secondly, the present invention also provides a method for implementing a 3D virtual space-based embody AI interaction system, comprising: When receiving task information input by the user, the system acquires environmental images of the character in the current viewpoint / panoramic view and identifies environmental information in the current viewpoint / panoramic view through a visual language model. After the task information session ends, the fragmented sequence in the session is converted into narrative memory text through a large language model, and the narrative memory text is converted into memory information and stored in a hierarchical memory architecture. The task information, environment information, memory information, location information, and state information corresponding to the character are assembled into structured context information, and the structured context information is input into the director engine. The director engine performs task reasoning and outputs the character behavior planning result; wherein, the character behavior planning result includes at least one behavior unit. The large language model is used to time-order multiple track instructions corresponding to each behavior unit and compile them into target instructions corresponding to the behavior unit; the target instructions are parsed and executed at the front end to generate the animation sequence of the character.

[0019] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the implementation method of the 3D virtual space-based embody AI interaction system as described above.

[0020] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the implementation method of the 3D virtual space-based embody AI interactive system as described above.

[0021] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the implementation method of the 3D virtual space-based embody AI interactive system as described above.

[0022] The beneficial effects of the technical solutions provided by some embodiments of the present invention include at least the following: 1) This invention provides a 3D virtual space-based embody AI interaction system and its implementation method. By using an open object visual space positioning system to identify environmental information under the current viewpoint / panoramic field of view, it overcomes the limitations of visual pathfinding at fixed points, enabling the AI ​​character to identify and autonomously travel to any visible target point within its field of view. A hierarchical memory and narrative memory generation system stores the memory information of the session. A director engine and context assembly system assemble the task information, environmental information, memory information, location information, and state information corresponding to the character into structured context information, achieving coherent reasoning within the task, experience transfer between tasks, and cross-session knowledge accumulation, which is beneficial for improving... The accuracy of task reasoning is improved; the director engine performs task reasoning and outputs character behavior planning results, realizing multi-step task decomposition and executable behavior planning for complex tasks. A natural language-driven multi-track generative animation system is used to sequentially arrange multiple track instructions corresponding to each behavior unit, driving multiple track instructions in parallel to generate the character's animation sequence. This achieves parallel animation generation of five track instructions with semantic understanding and sequential arrangement separated, enabling free and continuous action control. The AI ​​interaction system of this invention enables AI characters to possess visual perception, spatial movement, emotional expression, and continuous behavior control capabilities, enhancing the user's sense of realism and immersion.

[0023] 2) This invention proposes an open visual pathfinding complete technology chain of "two-stage mesh fine localization based on visual language model + ray projection + navigation mesh adaptive pathfinding", which does not rely on pre-trained fixed object categories and can perform accurate 3D localization and navigation of any visible object, improving the processing efficiency of multi-object scenes. 3) This invention proposes a hierarchical memory architecture of task-level-conversation-cross-conversation level, which can realize the continuous memory of AI characters across tasks and conversations; when the conversation ends, the fragmented sequence is automatically compressed into narrative memory information for permanent storage, and it supports the retrieval of historical memory at the semantic level and on-demand active query, so that AI can refer to past events, get rid of the limitations of pure conversation history stacking, and achieve true cross-conversation memory continuity.

[0024] 4) This invention proposes a five-dimensional structured context assembly scheme to provide a complete decision-making basis for the director engine; the reasoning memory inheritance mechanism enables multiple rounds of director engine calls for the same task to maintain logical coherence and avoid redundant analysis.

[0025] 5) This invention proposes a four-layer separation architecture based on a director engine for semantic planning, a large language model for temporal arrangement, a deterministic compiler for target instruction generation, and multi-track parallel execution. Semantic understanding and temporal arrangement each perform their respective functions. It supports independent and parallel control of five-track instructions, with each track instruction using an independent start timestamp to achieve asynchronous instruction-level arrangement. This breaks through the limitations of traditional single-threaded block-level synchronous temporal expression and can generate arbitrarily complex behavioral sequences through code combination. In addition, a four-channel overlay system (breathing / slight swaying / head shaking / blinking) runs independently outside the target instruction execution layer to achieve a natural sense of life for the character at all times.

[0026] 6) This invention proposes a prefetching mechanism that utilizes the physical movement time of AI characters to synchronously execute visual analysis based on a visual language model, which can reduce user perception latency.

[0027] 7) This invention proposes a multi-channel demand-driven autonomous behavior system, which defines multiple independent demand channels. When the demand value reaches a preset condition, an autonomous task is triggered and the task target is submitted, avoiding the delay of LLM call at the moment of triggering. In addition, by prioritizing user input tasks, a natural switching between "responding to the player when the player is in focus and living autonomously when the player is out of focus" is achieved, giving the AI ​​character a true sense of autonomous life. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram illustrating an application scenario of the 3D virtual space-based embody AI interaction system provided by the present invention. Figure 2 This is a schematic diagram of the structure of the 3D virtual space-based embody AI interaction system provided by the present invention; Figure 3 This is a schematic diagram of coarse localization of an environment image overlaid with an 8×8 grid in a certain scene, provided by the present invention. Figure 4 This invention provides a method for... Figure 3 A detailed diagram of the precise positioning of the target area after magnification and overlay of a 6×6 grid; Figure 5 This is a schematic diagram of panoramic image stitching provided by the present invention; Figure 6 This is a schematic diagram of coarse positioning using a 12×6 grid for panoramic images provided by the present invention; Figure 7This is a flowchart illustrating the implementation method of the 3D virtual space-based embody AI interaction system provided by the present invention. Figure 8 This is a schematic diagram showing the positions of the target object and the character displayed in the Unity game engine front-end during the preparation stage of a certain scenario, as provided by this invention. Figure 9 This is a schematic diagram showing the successful positioning of a panoramic scan target object in the Unity game engine front-end display provided by the present invention; Figure 10 This is a schematic diagram of the user input task completion displayed on the front end of the Unity game engine provided by the present invention; Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] First, the main technical terms involved in this invention will be explained: Large Language Models (LLMs) are a class of artificial intelligence models trained on deep neural networks (Transformer architecture). They typically have billions to hundreds of billions of parameters and are capable of understanding and generating natural language text, performing tasks such as reasoning, dialogue, and code generation. This invention can use large language models such as Gemini-3.0-flash-preview as inference decision engines.

[0032] Visual Language Model (VLM): A multimodal model that integrates a visual encoder with an LLM, capable of simultaneously processing image and text inputs and outputting descriptions of image content, question answers, or image-based inference results. This invention can use VLMs such as qwen3-vl-plus and qwen-vl-flash for scene scanning and object recognition.

[0033] Navigation Mesh (NavMesh): A pre-baked triangular mesh data structure in game engines (such as Unity) that describes the area where characters can walk in the scene and supports automatic path planning and obstacle avoidance.

[0034] NavMeshAgent: A path planning and character movement control component provided by the Unity engine, attached to the character object. Calling SetDestination() automatically calculates the shortest path on the NavMesh, driving the character to move along the path and avoid obstacles.

[0035] Domain-Specific Language (DSL): A dedicated description language designed for specific application scenarios. The Domain-Specific Language V2 DSL defined in this invention is a dedicated protocol format for transmitting multi-track behavior instructions from the Python inference backend to the Unity game engine frontend. It includes five tracks of instructions: animation, morph, move, text, and special, which differs from general-purpose programming languages.

[0036] Normalized coordinates: A coordinate system that maps image pixel coordinates to the interval [0,1], independent of resolution. In this invention, the object positions output by VLM are all represented by image normalized coordinates, (0,0) corresponds to the upper left corner of the image, (1,1) corresponds to the lower right corner, and the Y-axis points downwards.

[0037] Raycasting is a technique that emits a virtual ray from a point in space along a specified direction and detects its collision points with objects in the scene. This invention uses the Camera.ViewportPointToRay() function in the Unity game engine frontend to generate a camera ray from normalized coordinates, and Physics.Raycast() to detect the ray's intersection with the ground, thus achieving the conversion from 2D image coordinates to 3D world coordinates.

[0038] Text embedding: A dense vector representation of text mapped to a high-dimensional vector space by an embedding model. Texts with similar semantics are closer together in the vector space. This invention uses models such as text-embedding-v3 to vectorize memorized text, supporting semantic similarity retrieval.

[0039] RGB Image: A standard color image format that represents color information using three channels: red (R), green (G), and blue (B). It is the basic input for computer vision systems to acquire visual information. In this invention, the virtual camera at the front end of the Unity game engine captures RGB screenshots from the AI ​​character's current perspective and sends them via TCP to the Python inference backend as input for VLM visual analysis.

[0040] Panoramic Capture (PANO): In this invention, it specifically refers to the technical process of taking one RGB image each in the four directions (forward, right, left, and backward) sequentially using a Unity virtual camera, and then stitching them together in a 2×2 layout ("front left top, right right top, left left bottom, back right bottom") to create a panoramic image. In the V2 DSL defined in this invention, PANO is a special instruction among special track instructions, used to trigger the AI ​​to scan the 360-degree field of view around the current position to obtain object information in any direction.

[0041] Director Engine: The reasoning and decision-making engine of this invention is the central decision-making unit of the entire system. It is responsible for performing task reasoning based on a large language model, outputting structured role behavior planning results, and guiding the compilation and execution of all subsequent actions.

[0042] Behavior Unit (beat): The basic behavior planning unit output by the director engine of this invention. Each beat represents a complete action segment with a clear "stage intention," including configuration descriptions for five tracks: animation, morph, move, text, and special instructions, as well as coordinate context and missing information declarations. Multiple beats constitute the beats array, which is the result of character behavior planning, planned and output by the director engine all at once. It can be understood as "a series of small theaters arranged by the director."

[0043] Session: A continuous interactive recording unit in the memory system of this invention, encompassing all fragmented memory sequences accumulated from the start of the session (when the user initiates input or the AI ​​character's autonomous behavior triggers it) to the end of the session (approximately 70 seconds of timeout with no new events and no active tasks). After the session ends, the memory system automatically compresses the fragmented sequences into a narrative long-term memory and writes it into permanent storage.

[0044] like Figure 1 The diagram illustrates an application scenario of this invention. The proposed 3D virtual space-based embody AI interaction system comprises two parts: a Python inference backend and a Unity game engine frontend. The Python inference backend is responsible for running the core AI-based inference logic, while the Unity game engine frontend is responsible for environmental image acquisition, animation rendering, and command execution. Through the interaction between the Python inference backend and the Unity game engine frontend, the system achieves multiple core functionalities, including visual positioning, memory storage, task inference, and command generation.

[0045] Please see Figure 2 , Figure 2One of the structural schematic diagrams of a 3D virtual space-based embody AI interaction system provided as an embodiment of the present invention includes: An open object visual space positioning system 110 is used to acquire environmental images of a character in the current viewpoint / panoramic field of view when receiving task information input by a user, and to identify environmental information in the current viewpoint / panoramic field of view through a visual language model; The hierarchical memory and narrative memory generation system 120 is used to establish a hierarchical memory architecture at the task level, conversation level, and cross-conversation level. After the conversation ends, the fragmented sequence in the conversation is converted into narrative memory text through a large language model, and the narrative memory text is converted into memory information and stored in the hierarchical memory architecture. The director engine and context assembly system 130 are used to assemble the task information, environment information, memory information, location information and state information corresponding to the character into structured context information, and input the structured context information into the director engine, perform task reasoning through the director engine and output the character behavior planning result; wherein, the character behavior planning result includes at least one behavior unit. A natural language-driven multi-track generative animation system 140 is used to define multiple track instructions corresponding to a behavior unit through a domain-specific language; to perform time-series arrangement of the multiple track instructions corresponding to each behavior unit through a large language model, and compile them into target instructions corresponding to each behavior unit; and to parse and execute each target instruction sequentially at the front end to generate the animation sequence of the character.

[0046] This invention identifies environmental information under the current viewpoint / panoramic field of view through an open object visual spatial positioning system, breaking through the limitations of visual pathfinding at fixed points. The AI ​​character can identify and autonomously move to any visible target point in the field of view. It stores the memory information of the session through a hierarchical memory and narrative memory generation system, and assembles the task information, environmental information, memory information, position information and state information corresponding to the character into structured context information through a director engine and context assembly system. This enables coherent reasoning within the task, experience transfer between tasks, and knowledge accumulation across sessions, which helps to improve the accuracy of task reasoning. The director engine performs task reasoning and outputs the character behavior planning results, realizing multi-step task decomposition and executable behavior planning for complex tasks. Through a natural language-driven multi-track generative animation system, it arranges multiple track instructions corresponding to each behavior unit in a temporal sequence, drives multiple track instructions in parallel, and generates the character's animation sequence. This achieves the parallel animation generation of five track instructions with semantic understanding and temporal arrangement separated, realizing free and continuous action control. The AI ​​interaction system of this invention enables the AI ​​character to have visual perception, spatial movement, emotional expression and continuous behavior control capabilities, enhancing the user's sense of realism and immersion.

[0047] Open FMNav (Open Object Vision Navigation) is a relatively new and challenging topic in the field of computer vision in recent years. Current state-of-the-art technologies achieve a success rate of only 55%, with path length-weighted success rates even lower at 24%. This invention addresses the problem of precise spatial positioning of any visible object within a 3D virtual scene by combining visual pathfinding with the 3D virtual environment.

[0048] Based on the field of view coverage, this invention provides two environmental scanning paths: a scanning path based on a single environmental image (suitable for unidirectional scene analysis from the current viewpoint) and a scanning path based on a panoramic image (PANO path, suitable for target search in a 360-degree panoramic view). The positioning results of both scanning paths are normalized coordinates of the target object in the image. After ray casting and NavMesh pathfinding in the Unity game engine front-end, these coordinates are converted into 3D world coordinates to drive character pathfinding. The target object is an object related to the user's input task; for example, if the user's input task is "find the red cube outside the door," then the target object is the "red cube."

[0049] In some possible embodiments, the open object visual spatial positioning system 110 includes: The target view environment scanning module 110-1 is used to capture a single environment image from the character's current viewpoint, divide the single environment image into a grid, perform coarse localization of each object in the current viewpoint environment through a visual language model, crop out the target area based on the coarse localization of each object, and generate environmental information under the current viewpoint through a visual language model; wherein, the environmental information under the current viewpoint includes a scene description and a list of objects under the current viewpoint. The panoramic view environment scanning module 110-2 is used to acquire environmental images from multiple directions within the panoramic view; it stitches these images together to form a panoramic image, divides the panoramic image into a grid, performs coarse localization of target objects in the panoramic view environment using a visual language model, determines the sub-image containing the target object based on the coarse localization, and generates environmental information within the panoramic view using the visual language model. This environmental information includes a scene description within the panoramic view and the positional information of the target objects; the target objects are related to the target points in the task information. The coordinate transformation and automatic pathfinding module 110-3 is used to transform the position information of the target object into a three-dimensional world coordinate system, perform path planning through ray casting and navigation mesh sampling, and drive the character to move towards the target point according to the path planning result.

[0050] Specifically, the target view environment scanning module 110-1 executes a scanning path based on a single environment image. The target view environment scanning module 110-1 specifically performs the following steps: S110-1-1, Image Acquisition: Take RGB screenshots from the current perspective of the AI ​​character using the Unity game engine front-end as environment images. The recommended resolution is 854×480 or higher. Save the images as PNG format and send them to the Python inference back-end via TCP. S110-1-2, First Stage Coarse Positioning: (1) Call _draw_grid(image_path, cols=8, rows=8) to overlay a grid onto the original image. For example... Figure 3 The image shown is a coarse positioning diagram of an environment image overlaid with an 8×8 grid in a certain scene. The grid lines are yellow (255,255,0), 2px wide, with vertical line spacing equal to the image width / 8 and horizontal line spacing equal to the image height / 8. Each grid cell is labeled with a number (column letters AH, row numbers 1-8, such as A1, D4). The font size is set to max(14, min(image width, image height) / (max(number of columns, number of rows)×3)). The numbering uses a black rectangular background with yellow text.

[0051] (2) Convert the gridded image into a base64 string and construct an API request to send to the visual language model. The visual language model can be qwen3-vl-plus, with temperature parameter temperature=0.1 and maximum token count max_tokens=500. The message structure of the API request carries the scene analysis rules and output format requirements, as well as the base64 string and the task description string of the user input task.

[0052] (3) VLM returns coarse location results in JSON format. The coarse location results need to be filtered to remove entries with empty grid fields, keeping only valid objects.

[0053] For example, the coarse location result in JSON format is as follows: {"scene": "Living room, with a red sofa and TV in front", "objects": [{"name": "Red sofa", "grid": "D4", "type": "solid"}, {"name": "Doorway", "grid": "C2", "type": "opening"}]} The "scene" field is used for initial scene description, "objects" is a list of objects, "name" is the name of the object, the "type" field distinguishes the object type, "solid" is a solid object (sofa, cabinet, etc.), "opening" is a hollow object (door, window, etc.), and "grid" defines the grid position of the object.

[0054] S110-1-3, Second Stage Precision Positioning: For each valid object returned in step S110-1-2, perform the following operations independently using a ThreadPoolExecutor (with up to 4 threads running in parallel): (1) Calculate the pixel coordinates of the target area to be cropped: Extract the column index col (A->0, B->1, ..., H->7) and row index row (starting from 0) from the grid number.

[0055] The size of a single pixel cell can be set as cell_w = image width / 8, cell_h = image height / 8; For solid objects (such as sofas and cabinets), expand one grid to the top, bottom, left, and right of the target grid to obtain the pixel coordinates (x1, y1) and (x2, y2) of the two corner points of the target area: x1 = max(0, (col-1)×cell_w), x2 = min(image width, (col+2)×cell_w) y1=max(0, (row-1)×cell_h), y2=min(image height, (row+2)×cell_h) For opening type objects (doors / windows and other openwork objects), the target grid and the two rows below it (covering the walkable points at the bottom of the door frame) are cropped to obtain the pixel coordinates (x1, y1) and (x2, y2) of the two corner points of the target area: x1 = max(0, (col-1)×cell_w), x2 = min(image width, (col+2)×cell_w) y1 = row × cell_h, y2 = min(image height, (row + 3) × cell_h) All coordinates are restricted to the image boundary.

[0056] (2) Crop out the target area and enlarge its smallest side to 512px: crop = Image.crop((x1, y1, x2, y2)) In this context, Image.crop is the cropping operation, crop is the target area to be cropped, and crop.width and crop.height are the width and height of the target area, respectively.

[0057] If min(crop.width, crop.height) < 512, calculate the scaling factor scale = 512 / min(crop.width, crop.height), and use LANCZOS interpolation to enlarge the image to (int(crop.width×scale), int(crop.height×scale)). Here, min is the minimum function, and int is the rounding function. (3) Overlay a 6×6 grid (column AF, row 1-6) onto the enlarged target area, and call VLM to perform precise localization of the target object from the current viewpoint: For solid type: Construct a hint for VLM to ask which grid cell the target object is located in within a 6x6 grid. The return format can be {"grid": "C3", "reason": "The center of the sofa is located in column C, row 3"}, where "reason" is the reason. If the target object is not within the target area, return NONE.

[0058] For the "opening" type: Build a cue word for the VLM, specifically requiring the location of a walkable point at the bottom center of the door / window, not the center of the door frame. If the VLM returns NONE, mark the object as discarded=True and skip subsequent steps.

[0059] like Figure 4 The image shows the pair. Figure 3 A detailed diagram of the precise positioning of the target area after magnification and overlay of a 6×6 grid.

[0060] (4) Coordinate verification (solid type only, maximum of 2 outer retry rounds, maximum of 2 inner iterations per round): a. Convert the fine_grid obtained from the precise localization into normalized coordinates within the magnified target region: fine_norm_x = (fine-position grid column index + 0.5) / 6 fine_norm_y = (fine-positioning grid row index + 0.5) / 6 Where fine_norm_x and fine_norm_y are the x and y coordinates of the normalized coordinates within the magnified target region, respectively.

[0061] b. Draw a red cross mark in the enlarged target area, with the horizontal line from (px-20, py) to (px+20, py) and the vertical line from (px, py-20) to (px, py+20), with a line width of 3px; also draw a 5px solid red circle in the center to obtain the annotation diagram.

[0062] c. Send the annotated drawing to VLM and ask whether the red crosses in the annotated drawing accurately mark the location of the object.

[0063] d. VLM return format: {"correct": true / false, "reason": "...", "grid": "Suggested correction for the grid (only has a value when correct=false)"}, where "correct" indicates whether it is correct or not.

[0064] e. If correct=false and a different grid value is provided than the current one, update fine_grid, recalculate fine_norm_x and fine_norm_y, move the red cross, and send the verification again (inner iteration, up to 2 times).

[0065] f. The opening type skips the verification step and directly uses the precise positioning result.

[0066] (5) Reverse coordinate calculation to the normalized coordinates of the original environment image: x = (x1 + fine_norm_x × (x2 - x1)) / total image width, rounded to 4 decimal places. y = (y1 + fine_norm_y × (y2 - y1)) / total image height, rounded to 4 decimal places. S110-1-4. Return environmental information from the current viewpoint (after filtering out _discarded objects): For example, the return result of VLM can be: {"scene_description": "There is a red sofa in front of the living room", "objects": [{"name": "Red sofa", "x": 0.5312, "y": 0.4583, "type": "solid", "direction": "front"}]} The environmental information from the current perspective includes the scene description "scene_description" and the location information of the object list "objects", including the name of the target object "name", coordinates (x, y), type "type", and direction "direction".

[0067] Understandably, for a single environmental image, its direction is fixed as "front".

[0068] The panoramic view environment scanning module 110-2 executes a scanning path based on panoramic images. It is used when the AI ​​character needs to search for target objects in a 360-degree field of view. It achieves all-round coverage by capturing environmental images in four directions and stitching them together in a 2×2 panoramic view. This is similar to the two-stage positioning algorithm used in the target view environment scanning module 110-1.

[0069] Specifically, the panoramic view environment scanning module 110-2 performs the following steps: S110-2-1, Unity game engine front-end triggers shooting from four directions: When the Unity game engine frontend executes a PANO instruction in a special instruction track within a target instruction (including prefetch mode): (1) If it is in prefetch mode (metadata.prefetch=true), a virtual camera is generated at the target point position of the current target command. The initial yaw angle of the virtual camera is passed in by the prefetchYaw parameter in the target command.

[0070] (2) Rotate the virtual camera to the four absolute yaw angles in sequence and capture environmental images, with the shooting order corresponding to the layout of the panoramic image: Front (initial yaw angle + 0°) → Top left quadrant of the panoramic view; Right (initial yaw angle + 90°) → Top right quadrant of the panoramic view; Left (initial yaw angle -90°) → Lower left quadrant of the panoramic view; Back (initial yaw angle + 180°) → Lower right quadrant of the panoramic view.

[0071] The resolution of a single environmental image in each direction is 910×512 pixels.

[0072] (3) Pack the four environmental images and the initial yaw angle (prefetchYaw) into a TCP message (MSG_PREFETCH message type) and send it back to the Python inference backend.

[0073] S110-2-2, Panoramic Image Stitching: like Figure 5 The image shown is a schematic diagram of panoramic image stitching. The Python inference backend uses vision_panorama_tool to stitch four environmental images into a 2×2 panoramic image according to the following layout: Top left quadrant: Forward view (yaw angle = prefetchYaw + 0°) Top right quadrant: Rightward image (yaw angle = prefetchYaw + 90°) Lower left quadrant: Leftward image (yaw angle = prefetchYaw - 90° / + 270°) Lower right quadrant: Backward image (yaw angle = prefetchYaw + 180°) Panoramic image total size: 1820×1024 by default.

[0074] S110-2-3, Panoramic Coarse Positioning (12×6 grid): like Figure 6 The diagram shows a coarse localization diagram of a 12×6 grid in a panoramic image. A 12-column × 6-row grid is superimposed on the panoramic image (each sub-image region corresponds to 3 columns × 3 rows). The VLM is called to identify the column and row regions where the target object (specified by the focus parameter) is located and determine its orientation (front / right / back / left).

[0075] For example, the VLM return format can be: {"direction": "right", "coarse_grid": "G2", "description": "The red cube is located in the middle right"}, where "coarse_grid" is the network where the target object (red cube) is located, and "description" is the location description information of the target object.

[0076] S110-2-4, Sub-map fine positioning (6×6 grid): Determine the pixel range of the sub-image within the panorama based on its orientation: front (top left): x range [0, panoramic width / 2), y range [0, panoramic height / 2). right (top right): x range [panoramic width / 2, panoramic width), y range [0, panoramic height / 2). left (bottom left): x range [0, panorama width / 2), y range [panorama height / 2, panorama height) back (bottom right): x range [panoramic width / 2, panoramic width), y range [panoramic height / 2, panoramic height] Overlay a 6×6 grid onto the subgraph and call VLM to return the normalized coordinates (sub_x, sub_y) of the target object within the subgraph.

[0077] S110-2-4, Coordinate Transformation (1) Use (sub_x, sub_y) as the viewport coordinates of the camera in this direction.

[0078] (2) The Python inference backend sends the location information back to Unity: including direction, sub_x, sub_y, and prefetchYaw.

[0079] (3) The Unity game engine front end calculates the direction offset angle based on the direction parameter and transmits it to the coordinate transformation and automatic pathfinding module 110-3 for unified processing.

[0080] The coordinate transformation and automatic pathfinding module 110-3 receives the normalized coordinates (x, y) and direction parameters returned by the Python inference backend, converts the 2D image coordinates into 3D world coordinates, and drives the character's pathfinding. Specifically, the coordinate transformation and automatic pathfinding module 110-3 executes the following steps: S110-3-1, Camera aspect ratio adaptation: Temporarily set the main camera's aspect property to the aspect ratio during image acquisition (910 / 512=1.775) to ensure that the ray direction corresponds exactly to the image acquisition and avoid ray offset due to different actual rendering resolution.

[0081] S110-3-2, Constructing the viewport coordinates (viewportPoint): viewportPoint = new Vector3(x, 1.0f - y, 0f) Because the viewport coordinates of the Unity game engine front end are upward on the Y-axis (0=bottom, 1=top), while the image coordinates are downward on the Y-axis (0=top), the Y-axis needs to be flipped to correspond to the correct screen position.

[0082] S110-3-3, Prefetch Yaw Orientation Correction (Dedicated to Panoramic View Environment Scanning and Prefetch Mode): If the target command carries a prefetchYaw parameter (non-zero), it means that the coordinates come from the image captured by the virtual camera during the prefetching phase, rather than the actual camera orientation in the current frame. The direction offset angle is calculated based on the direction parameter (front: +0°, right: +90°, back: +180°, left: -90°), and the main camera is temporarily rotated to an absolute yaw angle = prefetchYaw + direction offset angle; immediately after raycasting is complete, the main camera is restored to its actual orientation.

[0083] This mechanism ensures that even if NavMesh pathfinding changes the character's facing direction upon arrival, raycasting is still based on the correct viewpoint in prefetch mode, and the coordinate mapping result is completely consistent with the scene in prefetch mode.

[0084] S110-3-4, Emit a ray and obtain the 3D hit point: ray = _mainCamera.ViewportPointToRay(viewportPoint) Physics.Raycast(ray, out hit, Mathf.Infinity, _raycastLayerMask) Here, `ray` is a ray emitted from the viewport coordinates of the main camera; `ViewportPointToRay` is a built-in method of the Unity game engine used to convert viewport coordinates into a ray; `Physics.Raycast` is a built-in method of the Unity game engine used to perform physical detection along the ray and return information about the first object hit; `out hit` represents output parameters, including the position of the hit point, normal, collider, etc.; `Mathf.Infinity` represents the ray length; and `_raycastLayerMask` represents a layer mask that only includes rays from the ground and walkable surfaces, excluding interference from transparent objects, UI, etc.

[0085] S110-3-5, Ground Projection: groundProjected = new Vector3(hit.point.x, transform.position.y,hit.point.z) Here, groundProjected represents the newly generated point, and (hit.point.x, hit.point.y, hit.point.z) are the three-dimensional coordinates of the hit point.

[0086] In this embodiment, the hit point Y coordinate hit.point.y is replaced with the AI ​​character's current Y coordinate transform.position.y to obtain a newly generated point, eliminating the influence of terrain undulations on the target point height and ensuring that the navigation mesh query is performed at the correct height layer.

[0087] S110-3-6, NavMesh sampling alignment: NavMesh.SamplePosition(groundProjected, out navHit, 10f,NavMesh.AllAreas) NavMesh.SamplePosition is used to search for the nearest valid position of the navigation grid around a specified point and return that position; out navHit represents the output parameter, which contains information about the found valid points; the search radius is 10f (meters); and NavMesh.AllAreas means searching all navigation grid areas.

[0088] In this embodiment, the nearest valid NavHit.position of NavMesh is found within a 10-meter search radius, with groundProjected as the center, and the case where the ray hit point falls outside the edge of NavMesh or on the edge of the slope is handled.

[0089] S110-3-7, Path Length Calculation and Speed ​​Adaptation: NavMesh.CalculatePath(transform.position, navHit.position,NavMesh.AllAreas,path) NavMesh.CalculatePath is used to traverse the array of corner points (path.corners) on the path, and accumulate the Euclidean distance between adjacent corner points to obtain the total path length pathLength. transform.position represents the starting point, i.e. the current position of the AI ​​character, navHit.position represents the ending point, i.e. the valid point obtained by NavMesh sampling in step S110-3-6, and path is the output parameter that stores the calculated path information.

[0090] The required speed (requiredSpeed) is calculated based on the duration parameter of the target command. requiredSpeed ​​= duration>0 ? pathLength / duration : 3.5f (default speed 3.5m / s) Set the AI ​​character's movement speed NavMeshAgent.speed = requiredSpeed ​​to ensure that the character reaches the target point precisely within duration seconds, maintaining consistency with the timing of the target command.

[0091] S110-3-8, Start Pathfinding: _navMeshAgent.SetDestination(navHit.position) Among them, _navMeshAgent is the navigation component for the AI ​​character, and SetDestination is used to set the target point, automatically calculate the path and start moving. In this way, the optimal path can be automatically calculated and the character can be driven to move. When encountering obstacles, it will automatically detour.

[0092] S110-3-9. Orientation Correction After Reaching the Target Point (FaceTarget): The 3D hit point's ground projection coordinates are cached as `_pendingFacingTarget`, and a facing correction task `_hasPendingFacingTarget=true` is set to wait for execution. When the arrival detection function `HasArrived()` detects that `navMeshAgent.remainingDistance <= stoppingDistance + 0.05f`, the character facing correction coroutine `FaceTargetCoroutine` is started: It smoothly rotates the character's orientation to face the target point within approximately 1 second using Slerp spherical interpolation, eliminating the facing deviation caused by the NavMesh's path and ensuring the character faces the user's expected direction upon arrival. Here, `navMeshAgent.remainingDistance` represents the remaining distance from the character's current position to the path's end point, and `stoppingDistance` is the stopping distance parameter of the NavMeshAgent component.

[0093] The above-mentioned hierarchical memory and narrative memory generation system 120 is used to solve the problem of memory continuity of AI characters across tasks and conversations. Through a three-layer memory architecture and automatic conversion from fragmented experiences to narrative memories, the AI ​​character can remember past events and refer to them when making reasoning decisions, thus achieving long-term memory capabilities.

[0094] In some possible embodiments, the hierarchical memory and narrative memory generation system 120 includes: The hierarchical memory architecture 120-1 is used to establish a hierarchical memory architecture of task level, session level, and cross-session level. Among them, the task level stores the multi-round structured context information of the director engine during the task inference process; the session level stores the session-level fragment sequence with fragments in each task of the whole session as the basic unit. When the number of fragments reaches the preset fragment number threshold, compression is automatically triggered, and the compressed session-level fragment sequence is written to the cross-session level, and the session level opens a new session record; the cross-session level stores the memory information of each session, forming a cross-session vector database. The narrative memory generation module 120-2 is used to format the fragmented sequence in the session into a labeled text list in chronological order after the session ends; input the text list into a large language model, output the narrative memory text, and vectorize and store the narrative memory text in a cross-session vector database.

[0095] Specifically, the hierarchical memory architecture 120-1 adopts a three-layer architecture of short-term memory, medium-term memory, and long-term memory. Among them, short-term memory is a task-level information record, storing the multi-turn context of the current task reasoning process, including the list of reasoning processes output by the director engine in each turn. It is discarded after the task ends and is not retained across tasks.

[0096] Mid-term memory is a session-level information record, with fragments as the basic unit, which is continuously accumulated throughout the entire session lifecycle and supports cross-task context transfer.

[0097] In some possible embodiments, the types of fragments include: The specific action description performed by the character (ai_action), the character's voice content (ai_speech), the concise summary of the director engine during the task reasoning process (ai_thinking), the original task text input by the user (user_input), the task execution result information (ask_result), and the scene description generated by the target perspective environment scanning module or the panoramic vision environment scanning module (vision). In addition, there is no upper limit to the number of fragments accumulated in the session-level medium memory. When the number of fragments reaches 200, compression is automatically triggered: the current fragments are summarized and written into long-term memory, a new session is started to continue recording, and the context is not lost throughout the process.

[0098] Long-term memory is a cross-session-level information record that persistently stores the memory information corresponding to each session in a cross-session vector database.

[0099] For example, a JSON file (py / data / memories.json) can be used as the storage medium to store the memory information corresponding to each session.

[0100] In some possible embodiments, the memory information includes the following fields: The narrative text in third-person format; the text embedding vector (which can be generated by text-embedding-v3, default 1024 dimensions) corresponding to the narrative text; the importance score (integer value 1-10) evaluated by a large language model; and the Unix timestamp generated from the memory information. Memory source type ("user" = user-initiated conversation; "autonomous" = AI autonomous behavior without user participation; "dialogue" = mixed conversation generated by user intervention during AI autonomous behavior execution). The dominant sentiment label (extracted from the highest-valued sentiment field of the sentiment state machine, such as "curiosity" or "comfort") when writing memory information; A list of tags.

[0101] Furthermore, this cross-session vector database supports vector retrieval based on semantic similarity, without relying on keyword matching.

[0102] The narrative memory generation module 120-2 is used to automatically convert the fragment list into a single narrative memory after a session ends (e.g., after a timeout of approximately 70 seconds with no new tasks, or a clear signal of session end). Specifically, the narrative memory generation module 120-2 performs the following steps: S120-2-1, Fragmentation Formatting: Format the fragment sequence in the session into a text list of labeled types in chronological order.

[0103] For example, suppose the name of the digital human (AI character) in the virtual world is Klee, the type_labels mapping can be as follows: ai_action → "action", ai_speech → "Klee said", ai_thinking → "Klee thought of". user_input → "User says", task_result → "Result", vision → "What you see" Based on the above mapping, an example of formatting a fragment sequence into a labeled text list is as follows: [Action] Klee walks towards the doorway. "[Klee says] Klee, go outside and see what's there!" "[I see] an open view ahead, and a red cube to the right." "[User says] What is that red square?" S120-2-2, LLM-based narrative memory text generation: Input a formatted text list into an LLM (e.g., Gemini-3.0-flash-preview), and design prompts. The output should be narrative memory text, retaining four core types of information: what the user said, what the AI ​​character saw, what the AI ​​character thought, and what the AI ​​character did. Visual details (scene description, object location) must be preserved. Pure visual question-and-answer contamination should be filtered out, and sentiment markers should be extracted. For example, the output format of the narrative memory text could be: {"narrative": "third-person narrative text", "importance": 7, "tags": ["exploration", "discovery of objects"]}.

[0104] In some embodiments, fragment splicing can be used as a fallback when the acquisition of a simplified summary during the reasoning process fails, to ensure that the memory is not lost.

[0105] S120-2-3, Vectorized Storage: The DashScope text-embedding-v3 API is called to convert the third-person narrative text into a high-dimensional vector, and write it along with importance, timestamp, and tags into a cross-session vector database (memories.json).

[0106] Furthermore, cross-session vector data supports semantic retrieval and recent memory retrieval for the director engine, and provides a memory information injection mechanism for the director engine's task reasoning process.

[0107] Specifically, semantic retrieval (search_memory) involves, given a query text, first calling the text-embedding-v3 model to generate a query vector, and then calculating a comprehensive score for each memory piece in the cross-session vector data. For example, it calculates the cosine similarity *sim* between the query vector and each memory piece, the normalized importance value of that memory piece (norm_im = importance / 10), and the timeliness score (timestamp = exp(-D / 30), where the timeliness score decays over a 30-day period, with a higher score for smaller numbers of days ago (D). Finally, it calculates the comprehensive score based on this information. score = sim×0.5 + norm_im×0.3 + timestamp× 0.2 Sort by overall score in descending order, and return the top 3 memory records by default.

[0108] In some embodiments, the director engine can actively query cross-session vector data through the memory_search tool during inference, which returns a maximum of Top-5 results.

[0109] In some embodiments, recent memory retrieval returns the three most recent memories in descending order of timeliness score, ensuring temporal continuity and complementing semantic retrieval.

[0110] In some embodiments, the memory injection mechanism (context_assembler execution) refers to automatically injecting three parts of memory into the inference context before each call to the director engine: ① Semantic relevance: Perform semantic retrieval on the user's current input and inject the top-3 relevant historical memories (automatic injection, no tool calls required); ②Recent Sessions: Inject the 3 most recent sessions to ensure temporal continuity; ③ Current session fragment sequence (current_session_fragments): Injects all accumulated fragments within the current session, supporting cross-task context passing.

[0111] In some embodiments, the director engine can also proactively retrieve relevant history (up to Top-5 entries) on demand using the memory_search tool during task reasoning. This combination of three-way injection and on-demand tool supplementation enables the director engine to reference past events, breaking free from the limitations of pure session history stacking and achieving true cross-session memory continuity.

[0112] In some embodiments, the present invention employs a delayed fragmentation mechanism in the hierarchical memory and narrative memory generation system 120: that is, in order to ensure the temporal accuracy of the fragment sequence, when the AI ​​character's autonomous behavior is being executed and user input is detected at the same time, the hierarchical memory and narrative memory generation system 120 temporarily stores the user input fragments in the _pending_user_fragments buffer, and then flushes them to the session fragment sequence after the current task is completed, so as to avoid the user fragments being incorrectly inserted before the AI ​​character's action description due to temporal preemption.

[0113] The Director Engine and Context Assembly System 130 address the problem of structured assembly of multi-source information in AI reasoning, and how the Director Engine completes semantic-level character behavior planning based on complete structured context information. The Director Engine is the central decision-making unit of the entire system, responsible for performing one or more rounds of task reasoning based on structured context information, outputting a sequence of structured behavioral units (beats array), and guiding the compilation and execution of all subsequent actions.

[0114] In some possible embodiments, the director engine and context assembly system 130 include: The context assembly module 130-1 is used to assemble the task information, environment information, memory information, position information and state information corresponding to the character into structured context information before each call to the director engine; Director Engine 130-2 is used to perform task reasoning based on structured context information and output character behavior planning results according to planning principles in different scenarios. Specifically, the ContextAssembler module 130-1 is used to assemble task information, environment information, memory information, location information, and state information into context information in a unified JSON format before each call to the director engine.

[0115] The task information (task field) includes the user's original input text (original_input), the current inference round (round, starting from 1 and incrementing after each PANO scan), and the trigger reason for this round (status, deterministically concatenated by agent_loop, no LLM generation required). The trigger reason status includes the following: New task → "New task"; PANO scan completed → "Nth round PANO scan completed, new environmental information obtained, please continue processing the task based on the new information"; Beat verification failed → "Nth round beat verification failed, please regenerate". The memory information (memory field) includes: performing semantic retrieval on user input and injecting the Top-3 relevant historical memories (semantic_relevant); injecting the 3 most recent historical memories in descending order of timestamp (recent_sessions) to ensure temporal continuity; a list of all inference rounds within the current task (reasoning_memory, task-level short-term memory, inherited across PANO rounds); and all accumulated fragment sequences within the current session (current_session_fragments).

[0116] The environment information (environment field) is the result of the VLM environment scan, including scene text description (scene_description) and information on located objects (objects, including name, normalized coordinates, and orientation). The environment information for the first round of task inference by the director engine comes from a single environment image scan, while subsequent rounds come from the PANO scan analysis results.

[0117] The position information (positions field) includes: the current world coordinates of the AI ​​character (ai.world, [x, y, z]) and the current world coordinates of the user character (player.world, [x, y, z]). The current world coordinates of the user character are passed in with each user input message from the Unity game engine frontend. When the autonomous task is executed, the Python inference backend actively sends a request message to the Unity game engine frontend to obtain the real-time position coordinates.

[0118] The state information (state field) includes the AI ​​character’s current emotional state string (such as comfort / curiosity / stress, etc.), provided by the EmotionStateMachine.

[0119] Director Engine 130-2 uses Gemini-3.0-flash-preview as its inference model. It receives structured context information in JSON format assembled by the context assembly module, performs character behavior planning through task inference, and outputs structured character behavior planning results (beats).

[0120] The output of the director engine is strictly limited by prompts (closed vocabulary constraints) to prevent unauthorized creation of command types that are not supported by the system.

[0121] In some embodiments, the closed vocabulary constraints involved in the director engine's prompts include: Animation vocabulary (corresponding to animation track instructions): 15 fixed animation names (walking / idle / wave right hand / raise hand to celebrate / hug / hands on hips / pat head / finger forward / make a V sign / girl raises hand / take a photo V / cat pose / shrug / run / jump); Facial Expression List (corresponding to facial expression track instructions): 6 facial expressions (neutral / happy / angry / sad / playful / surprised); Move word list (corresponding to move orbit command): 3 types (ray movement / world movement / rotation).

[0122] In some embodiments, before officially outputting the character behavior planning results, the director engine can use the following tools to supplement the missing information in the current context as needed: Memory search: When the pre-injected memory is insufficient to support the decision, actively query semantically relevant historical memory (returning a maximum of Top-5 results). Position query (position_query): Queries the current world coordinates of the AI ​​character and the user (only used when there is no valid data in the position information); Emotion query: Query the detailed numerical value of the current emotional state (only used when the state information is insufficient to support a decision).

[0123] The tool call results are directly injected into the structured context information of this round of reasoning, and the director engine completes the character behavior planning based on the supplementary information.

[0124] In some embodiments, the output of the director engine includes: The reasoning process of this round (reasoning string) is stored in the task-level reasoning process list (reasoning_memory) for reference in subsequent rounds; The natural language description of the AI ​​character's behavior (reaction string) is stored in the conversation fragment sequence (type=ai_thinking). The result of the role behavior planning (beat array) is verified and then proceeds to the subsequent compilation process.

[0125] Each beat (behavioral unit) has a tracks field that describes what it does (such as "walking", "happy", "ray movement") or track instructions in natural language, and a context field that provides coordinate data. Beats do not contain timestamps and do not care about timing details; their timing arrangement is handled by the compilation process.

[0126] In some embodiments, the director engine outputs structured data, such as JSON format. If the JSON format output by the director engine is malformed (e.g., mismatched brackets, missing fields), a repair prompt is sent to the director engine to regenerate the data using the temperature parameter temperature=0.3, ensuring that subsequent deterministic processes obtain valid JSON format data.

[0127] In the director engine, the planning principles for generating single or multiple action units in different scenarios include: if the current view information is sufficient to complete the task, or if environmental information under the panoramic view is required but has not yet been acquired, output a single action unit (single beat); when the complete information required to complete the task is known, plan multiple action units (multiple beats) based on the continuous actions required to complete the task and output them; for action units that require the environment under the panoramic view to be determined, re-perform task reasoning after the panoramic view environment scanning module has completed the task; the action unit includes the configuration descriptions corresponding to the character's multiple track commands.

[0128] Specifically, the director follows these rules to decide whether to output a single action unit or multiple action units: Single action unit scenario: The current view information is sufficient to complete the task, or a PANO scan is required but the environmental information under the panoramic view has not yet been acquired (output a single action unit with PANO, trigger prefetch mode, wait for the environmental scan results in prefetch mode, and then plan subsequent actions). Multiple Behavior Unit Scenarios: When the complete information required to complete the task is known, multiple behavior units are pre-planned based on the continuous actions required to complete the task; for behavior units that rely on PANO scanning to determine their coordinates, the missing information declaration field declares type="vision", which is then truncated by the beat verification function (BeatScanner) and the beat is not committed to the Unity game engine frontend.

[0129] In other words, the director engine does not include actions that it cannot confirm can be executed in the beats array, to avoid false promises that cause errors in the Unity game engine front-end execution; after the PANO scan is completed, the character task planning is re-executed based on the real environment information.

[0130] In some embodiments, reasoning memory is shared across multiple rounds of director engine calls within the same task (e.g., first round PANO prefetching, second round replanning) through a reasoning memory inheritance mechanism, which includes: The reasoning process (reasoning string) output from each round of task reasoning is appended to the reasoning process list (reasoning_memories); During the next round of assembling structured context information, this list of reasoning processes is injected as a field (memory.reasoning_memory) to store memory information; The director engine can directly view the reasoning process of past rounds, avoiding repeated analysis or contradictory decisions in multiple rounds of planning.

[0131] The Natural Language-Driven Multitrack Generative Animation System 140 realizes real-time generation from natural language semantic description to five-track parallel animation sequence. The core design separates semantic understanding and temporal arrangement into different processing layers.

[0132] In some possible embodiments, the natural language-driven multi-track generative animation system 140 includes: The multi-track instruction definition module 140-1 is used to define multiple track instructions corresponding to each behavior unit through a domain-specific language. The multiple track instructions include animation track instructions that control skeletal animation, expression track instructions that control character facial expressions, movement track instructions that control character spatial movement, subtitle track instructions that display subtitle text, and special track instructions that carry special instructions. The target instruction generation module 140-2 is used to perform time-series orchestration of multiple track instructions corresponding to each behavior unit based on the large language model, determine the start time and duration of each track instruction, and output key frame arrays corresponding to multiple track instructions; group the key frame arrays according to track instructions, sort the key frames in each track instruction by start time, generate string fragments corresponding to each track instruction, and concatenate the string fragments corresponding to each track instruction to generate the target instruction corresponding to the behavior unit; The animation generation module 140-3 is used to parse the target instructions on the front end, create a coroutine queue in ascending order of the start time of each track instruction, trigger the operation of the corresponding track instruction when the corresponding time point is reached, and continuously drive the skeletal animation by superimposing the breathing channel, micro-swaying channel, head shaking channel and blinking channel to generate the character's animation sequence.

[0133] Specifically, this invention defines the V2 DSL format based on a domain-specific language. V2 DSL is the core instruction protocol for communication between the Python inference backend and the Unity game engine frontend. The complete format is as follows: v2; B{ instruction 1 | instruction 2 | instruction 3 | ...} "v2;" is the version identifier, and "B{...}" is a block, with track instructions separated by "|". Each action unit corresponds to multiple track instructions, and the same track can contain multiple instructions (each with its own independent start time), which are also separated by "|". The format of a five-track instruction is as follows: (1) Animation track instructions (anim, skeletal animation layer): Format: anim:{start},{animation code},{duration},fade={fade-in time},layer={layer number}, Here, `start` represents the start time, and `duration` represents the duration; animation code (anim_code) is divided into two categories: full-body animation and part mask animation. Full-body animation (F prefix, layer=0 base layer): F0 Idle, F1 Walking, F2 Waving right hand, F3 Raising hand in celebration, F4 Hugging, F5 Hands on hips, F6 Touching head, F7 Pointing forward, F8 Making a V sign, F9 Girl raising hand, F10 Taking a photo with a V sign, F11 Cat pose, F12 Shrugging, F13 Running, F14 Jumping; Partial Mask Animation (Partial Code + Animation Number, layer>=1 overlay): The system defines seven AvatarMask types—HD (Head), LA / LH (Left Arm, mask covers all bones of the left arm and left hand, arm and hand are controlled in tandem), RA / RH (Right Arm, mask covers all bones of the right arm and right hand, arm and hand are controlled in tandem), LHO (Left Hand Only, further limited to the hand bones based on the left arm mask, used for scenes requiring independent hand control, currently not enabled), RHO (Right Hand Only, similarly, currently not enabled), LL (Left Leg), and RL (Right Leg). Example: RH2 indicates that the right arm performs the F2 waving motion; the mask only affects the right arm and right hand bones, without interfering with other parts.

[0134] The layer parameter determines the layer number of the Animator. There is no upper limit to the number of layers: layer=0 plays the full-body running animation, layer=1 overlays the right arm waving (RA / RH mask), and layer=2 overlays the head action (HD mask). Each layer is blended independently according to its own mask range and does not interfere with each other.

[0135] The `fade` parameter controls the fade-in duration (in seconds) of the animation. Smooth transitions are achieved using Layers[layer].Play(clip, fadeDuration) of Animator (a third-party animation library for Unity that replaces the built-in Animator and supports direct code-based control of animation clip playback, blending, and layer management).

[0136] Here are two examples of animation track instructions: Whole body: anim:0.0,F1,5.0,fade=0.3,layer=0; Overlay of parts: anim:0.5,RH2,2.0,fade=0.2,layer=1; (2) Expression track instructions (morph, expression blending layer): Format: morph:{start},{emoticon code},{duration},blend=smooth The emoticon codes include K0 neutral, K1 happy, K2 angry, K3 sad, K4 playful, and K5 surprised. The emoticon codes support mixed weights (the sum of each weight does not exceed 1.0). For example, a mixed weight emoticon track instruction is morph:0.0,K1=0.7+K4=0.3,5.0,blend=smooth.

[0137] Blend represents the transition method of the expression. When blend=smooth, it means a smooth transition. At this time, the Unity game engine front end uses time interpolation to make the weight of the blend shape component (BlendShape) transition smoothly.

[0138] Here is an example of an expression track command: morph:0.0,K1,5.0,blend=smooth (3) Move command (space movement layer): Format: move:{start},{move command},{duration},ease=smooth, [direction={direction}], [prefetchYaw={angle}] The movement command types include Raycast (MR), World Move (MW), and Rotate in Place (RT). `prefetchYaw` records the initial yaw angle of the camera during prefetching, used for coordinate reconstruction in the Unity game engine front-end; this is optional. `ease` represents the velocity curve during movement; `ease=smooth` indicates the use of a smooth easing curve.

[0139] An example of a movement trajectory command under the ray movement type is: move:0.0,MR(0.603,0.707),5.0,ease=smooth,direction=front, where the coordinates in parentheses are the viewport normalized coordinates (x,y), and direction specifies the camera orientation (front / right / back / left) when the image is captured.

[0140] An example of a movement track command under the World Movement type is: move:0.0,MW(1.2,0.1,3.4),5.0,ease=smooth, where the coordinates in parentheses are the Unity world coordinates (x,y,z), which can be directly set as the NavMeshAgent target.

[0141] An example of a movement track command under the stationary rotation type is: move:0.0,RT(90),1.0, where the value in parentheses is the rotation angle, a positive value = right turn, and a negative value = left turn.

[0142] (4) Subtitle track instructions (text, subtitle text layer): Format: text:{start},"{text content}",{duration} Double quotation marks in the text must be escaped as \".

[0143] Here is an example of a subtitle track instruction: text:0.5,"Klee goes outside to see what's there!",4.0 (5) Special track instructions (special, special instruction layer), including blocking instructions and non-blocking instructions: ① Two examples of blocking instructions (where the Unity game engine's TimelinePlayer pauses the timeline during execution and resumes only after the Python inference backend responds): Panoramic shooting (PANO): special:5.0,PANO, target object name,0 Single image (IMG): special:5.0,IMG,Target description,0 ② Several examples of non-blocking instructions (executed immediately, without pausing the timeline) are as follows: Right arm IK: special:0.5,IK_R,front,0.6,0.4,3.0 Its parameters include direction, viewport normalized x-coordinate, viewport normalized y-coordinate, and duration (seconds).

[0144] Turn off IK: special:0.5,IK_OFF,2.0 Micro-motion parameters: special:0.0,IDLE,0.8 (sets the tension value) The target instruction generation module 140-2 generates target instructions based on the V2 DSL format and five track instruction formats defined by the multi-track instruction definition module 140-1.

[0145] Specifically, the timing of multiple track instructions corresponding to each action unit is first performed using the timing orchestration component (InstructionGenerator) of the large language model (Gemini-3.0-flash-preview). This timing orchestration includes: It receives a single action unit and, referencing common pattern templates in the prompt words, autonomously determines the start time and duration of five track instructions, for example: The movement and walking animations start simultaneously (both start at 0.0 and have the same duration). The text subtitles will start after a delay of 0.3-0.5 seconds (and will be displayed after the animation starts), and will continue until 0.5 seconds before the movement ends; The facial expression is set at the start of the task and lasts for the entire duration of the action unit; Special instructions such as PANO are scheduled at the end of the movement (start = movement duration).

[0146] After completing the timing arrangement above, output the keyframe array. The following is an example of a keyframe array: [{"track": "anim", "start": 0.0, "duration": 5.0, "anim_code": "F1","layer": 0}, {"track": "move", "start": 0.0, "duration": 5.0, "type": "ray movement","target": [0.603, 0.707], "direction": "front"}, {"track": "text", "start": 0.5, "duration": 4.0, "content": "Klee goes outside to check!"}, {"track": "special", "start": 5.0, "duration": 0, "special_type": "PANO", "focus": "object outside the door"}] The "track" field defines the track command type, including move / anim / morph / text / special. The "target" field defines the target point of the current movement track command.

[0147] The post-processing step forcibly injects the orientation and coordinates of the target point (beat.context.move_target) of the current action unit into the move keyframe, ensuring that the information aligns with the director engine's intent. Figure 1 It can infer coordinates independently without relying on LLM.

[0148] Finally, the keyframe array is deterministically compiled using a deterministic compiler. This compiler groups the keyframe array by track instruction, sorts each track by start time, generates corresponding DSL string fragments, and concatenates them into the complete string "v2; B{...}", thus obtaining the target instruction. This step is a purely deterministic computation, does not call LLM, and its time consumption is negligible.

[0149] The following is a complete example of a target instruction: v2; B{ move:0.0,MR(0.603,0.707),5.0,ease=smooth,direction=front |anim:0.0,F1,5.0,fade=0.3,layer=0 | morph:0.0,K1,5.0,blend=smooth | text:0.5,"Klee goes outside to check!",4.0 | special:5.0,PANO,Object outside the door,0} The target instruction directs the AI ​​character named Klee to go outside and look around, and provides five track commands: move, anim, morph, text, and special.

[0150] The animation generation module 140-3 is used to parse the target instructions generated by the target instruction generation module 140-2 and drive the AI ​​character to perform tasks according to the target instructions, thereby generating continuous animations.

[0151] Specifically, in the Unity game engine front-end, the TimelinePlayer control parses and executes the target instructions: (1) Verify the "v2;" prefix and extract the content of block B{...}.

[0152] (2) Separate each instruction string by "|" and parse the track type, start time and parameters.

[0153] (3) Create a queue of coroutines in ascending order of their start times, and trigger the corresponding operation when the corresponding time point is reached: Animation Track Instructions (anim): Based on the layer parameter, call Animator (a third-party Unity animation library that replaces the built-in Animator and supports direct code-based control of animation clip playback, blending, and layer management) to achieve a smooth fade-in for the corresponding layer; if the animation code contains a part prefix (HD / LA / LH / RA / RH / LHO / RHO / LL / RL), first setMask for that layer (an Animator layer method that applies an AvatarMask to a specified layer) to the corresponding AvatarMask (a built-in resource of the Unity game engine that defines which bone parts are affected by skeletal animation), restricting the animation to only affect the specified bone branches; layer=0 base layer mask=null, covering the entire skeleton.

[0154] Expression Track Command (morph): Sets the BlendShape weight; when blend=smooth, it uses coroutine time interpolation for a smooth transition.

[0155] Ray movement trajectory command (move(MR)): Projects a viewport-normalized coordinate (x, 1-y) ray onto the navigation mesh and calls NavMeshAgent.SetDestination to start pathfinding.

[0156] World Move Orbit Command (move(MW)): Directly calls NavMeshAgent.SetDestination(newVector3(x,y,z)) to start pathfinding.

[0157] Subtitle Track Command (text): Displays subtitle text, hides it after the duration ends.

[0158] Blocking orbit command (PANO / IMG): Pauses the timeline coroutine (yield return null loop wait), triggers the taking of a picture and sends the obtained environmental image to the Python inference backend. The timeline is resumed after receiving the "continue" command from the Python inference backend.

[0159] IK_R / IK_L orbital commands: Project viewport coordinate rays into 3D world coordinates, set IK target points, activate IK controllers, and dynamically modify the animation mask of the corresponding arm bones (excluding IK-controlled bones to avoid conflicts with bone animation).

[0160] In some practical applications, the animation generation module 140-3 overlays a four-channel overlay system (IdleMicroMotion) during animation generation. This four-channel overlay system continuously drives the skeleton through four independent channels using different algorithms, allowing the character to maintain a natural sense of life while performing the main animation. These four independent channels include: (1) Breathing channel: Sine wave, frequency 0.2-0.4Hz, driving the spine to slightly translate along the Y-axis (amplitude 0.5-2cm), the amplitude of which is linearly scaled by the micro-motion parameter (tension).

[0161] (2) Micro-oscillation channel: Two sine waves are superimposed (frequency 0.13Hz and 0.17Hz) to drive the spine to rotate slightly along the Z-axis, achieving a natural drifting sensation.

[0162] (3) Head movement channel: Perlin noise function drives the neck to rotate along the Y-axis at a frequency of about 0.1 Hz and an amplitude of about 3-8 degrees, to achieve irregular natural head micro-movements.

[0163] (4) Blinking channel: Triggered at random intervals (3-7 seconds), driving the BlendShape eyelid closure weight to rise from 0 to 1 and then fall back to 0, lasting 0.15 seconds per cycle.

[0164] The tension parameter is injected (range 0.0-1.0) from the IDLE (standby state) command of the target instruction; it automatically increases breathing amplitude when moving and decreases when stationary. In some possible embodiments, as the character moves toward the target point, a prefetch mode is injected into the target instruction through a special trajectory command. After receiving the target instruction, the front end uses the panoramic view environment scanning module to pre-identify the environmental information of the target point in the panoramic view and synchronously executes the movement trajectory command in the target instruction. When the character reaches the target point, if the environmental information of the target point in the panoramic view has been obtained, the next round of animation sequence of the character is immediately generated through the director engine, context assembly system and natural language driven multi-track generative animation system; among them, the orientation of the character when it reaches the target point is corrected according to the absolute yaw angle of the virtual camera in the prefetch mode; the multi-round director engine reasoning process within the same task shares memory information.

[0165] Specifically, in prefetch mode, the physical movement time for an AI character to move from its current position to the target point is typically 3-10 seconds. During this movement, VLM (Visual Modeling) analysis is performed simultaneously, completely masking any AI processing delays. The character can then immediately execute the next action upon reaching the target point, without waiting. This functionality is achieved through the following steps: (1) When the director engine plans a behavior unit, it injects the PANO instruction into the field of its special track instruction; (2) When the Python inference backend sends the target instruction for this action unit, it carries metadata related to the prefetch mode: {"prefetch": true, "prefetch_target": "target name"}, where the "prefetch" field defines whether the prefetch mode is enabled. The timing orchestration component (InstructionGenerator) injects the initial yaw angle (prefetchYaw) parameter into the target instruction's movement trajectory instruction.

[0166] (3) After receiving the target instruction, the Unity game engine front end immediately (about 0 seconds, before the movement has started) places a virtual camera at the target point of the movement track instruction and rotates the virtual camera from the initial yaw angle to the absolute yaw angle. Then, it takes panoramic images from four directions (about 0.5-1 seconds) and sends the images back to the Python inference back end. At the same time, it starts executing the movement track instruction (the character starts to move).

[0167] (4) When the Python inference backend receives the panoramic image (about 0.5-1 seconds), it starts VLM analysis in the thread pool (about 3-6 seconds), during which the character continues to move.

[0168] (5) When the character reaches the target point (about 5-10 seconds), the VLM analysis is usually completed, and the Python inference backend immediately generates the next round of target instructions based on the analysis results and sends them.

[0169] In some embodiments, navigation network path planning does not guarantee that the character's orientation upon arrival is consistent with the orientation during prefetching (the orientation may change due to detouring around obstacles), requiring orientation correction. By recording the initial yaw angle of the virtual camera during prefetching in the target instruction, the Unity game engine front-end temporarily rotates the camera to the corresponding absolute yaw angle before performing raycasting, ensuring that the 3D world position corresponding to the normalized coordinates is completely consistent with the viewpoint during prefetching. After raycasting is completed, the camera is restored to its actual orientation.

[0170] In some possible embodiments, during the inference process, the visual language model, the large language model, and the director engine continuously run the breathing channel, micro-swaying channel, head swaying channel, and blinking channel to drive the skeletal animation. In the character behavior planning results, the facial expression state of the facial expression track instruction in the previous behavior unit is passed to the facial expression track instruction in the next behavior unit through preset weights; If the character is executing a movement track command, the walking animation will continue to play until the target point is reached.

[0171] Specifically, the AI ​​role interaction system of the present invention uses AI components in environmental scanning, task inference, and timing orchestration. When the AI ​​role interaction system of the present invention starts up, all AI components are preheated in parallel, triggering the establishment of HTTP connections and the corresponding prompt word caching. Visual language model warm-up: In the open object visual space positioning system of the present invention, both the target view environment scanning module and the panoramic view environment scanning module use a visual language model for environment analysis. A 32×32 white PNG placeholder image is sent to it, max_tokens=1, triggering the API connection, and the corresponding prompt words are cached on the HTTP end.

[0172] Director Engine Warm-up: This invention uses a director engine based on a large language model for task reasoning, and sends a minimum placeholder context JSON string, max_tokens=10, to the large language model to pre-cache the prompt words of the director engine.

[0173] Preheating of the temporal orchestration component: This invention performs temporal orchestration through the temporal orchestration component of the large language model, sends a minimal placeholder beat JSON to the large language model, max_tokens=10, and generates a prompt word cache with the preheating instruction.

[0174] The warm-up of the above AI components is typically completed within 10-15 seconds. After warm-up, all AI components reuse the cached HTTP connections and prompt word caches when called, which can improve the response time of the first user request by about 30-50%.

[0175] In some embodiments, during AI inference (in the WAITING_PREFETCH or WAITING_COMPLETE state), the visual sense of waiting is eliminated through the following mechanism: (1) The four-channel overlay system runs continuously in the front end of the Unity game engine and is executed independently of the target instruction. By overlaying the breathing / slight swaying / head shaking / blinking channels, the character always maintains a natural sense of life and has no static or stiff feeling.

[0176] (2) The facial expression state of the previous behavior unit is maintained to the next behavior unit through the BlendShape weights, without any sudden change.

[0177] (3) If the character is executing a movement track command, the walking animation (F1) will continue to play until the target is reached, thus improving the pause problem.

[0178] In some embodiments, the Python inference backend uses the asyncio event loop: (1) The Python inference backend starts the TCP server by calling the asynchronous programming function asyncio.start_server. Each Unity game engine frontend connection corresponds to an independent asynchronous processing coroutine.

[0179] (2) All AI component API calls are executed in the default thread pool via loop.run_in_executor(None, func), without blocking the asyncio event loop.

[0180] (3) All behavioral units simultaneously initiate AI calls to generate target instructions in parallel across multiple behavioral units. The total time is approximately equal to the generation time of the slowest behavioral unit.

[0181] (4) Session timeout polling: The main loop asyncio.sleep(1) performs periodic checks without interfering with any processing flow.

[0182] This invention, through the above asynchronous non-blocking architecture, can accelerate inference and execution speeds and reduce waiting time.

[0183] In some possible embodiments, the AI ​​character interaction system of the present invention further includes: The autonomous behavior driving system 150 is used to maintain multiple independent intrinsic demand channels. During the role's standby state, it accumulates and calculates autonomous demand values. When the autonomous demand value reaches the demand threshold, it plans autonomous tasks in advance. When the triggering conditions of the autonomous task are met, it triggers the role's autonomous task. Among them, the priority of user input tasks is higher than the priority of autonomous tasks. When user input tasks are detected, autonomous tasks are paused.

[0184] Specifically, the autonomous behavior driving system 150 includes an intrinsic demand system (NeedsSystem) that maintains two independent demand channels: hunger (hunger, growth_rate = 1.0 / second) and boredom (boredom, growth_rate = 0.8 / second), each accumulating demand values ​​independently. The main loop calls update() once per second, checking whether the corresponding autonomous demand value exceeds a threshold (e.g., 600 units) when the system is idle. When the demand value reaches 50% of the threshold (e.g., 300 units), an AI component is called in a background thread to generate a more specific task description in advance (e.g., "go to the kitchen to find the red cube" instead of "find something to eat"), reducing the initial response latency of autonomous behavior.

[0185] The conditions for triggering this autonomous behavior must be met simultaneously: (1) the system is in IDLE (standby) state; (2) the cooldown time (30 seconds) has elapsed; and (3) there are currently no autonomous tasks being fulfilled. When the autonomous behavior is triggered, the autonomous demand value is immediately reset to zero, and a flag is set to prevent repeated triggering. The state is reset after the task is completed.

[0186] After the autonomous task is triggered, a single image scan command (special command "v2;B{ special:0,IMG,scan,0}") is sent to the Unity game engine frontend to request the capture of an image of the current scene environment. After Unity returns the environment image, the entire AI character interaction process is completed using the task description as "virtual user input": VLM environment scan → director engine inference → beat verification function verification → five-track instruction generation → deterministic compilation → target instruction sent to the Unity game engine frontend for execution.

[0187] During the inference process, the director engine marks the task source (autonomous task) and the demand type (hunger / boredom) in the structured context information. The director engine can choose to record only the memory structure and not perform any actions (suitable for everyday scenes with nothing worth paying attention to), or choose to perform actions (suitable for discovering objects or environmental changes worth interacting with).

[0188] In some embodiments, when the AI ​​character interaction system receives user input, it immediately pauses all autonomous tasks and resumes them upon completion of the user task. This ensures that system resources are not contested by autonomous tasks during user interaction and prevents the sudden insertion of autonomous behavior during user instruction execution.

[0189] In some embodiments, when the autonomous demand value reaches a 50% threshold, a background thread asynchronously invokes the AI ​​component to generate a more specific task description by combining the current environmental context information (VLM scan results, memory fragments, location information), and pre-generates role behavior planning results. When autonomous behavior is triggered, the pre-generated role behavior planning results are used directly, avoiding the delay in AI component invocation at the moment of triggering. If the planning fails or is not completed, the default task description is used as a fallback.

[0190] The above is a description of the main functions of the 3D virtual space-based embody AI interaction system proposed in this invention. Compared with the prior art, this invention has the following technical effects: 1. The open object visual spatial positioning system of the present invention is the first to create a complete open visual pathfinding technology chain of "VLM two-stage mesh fine positioning + Unity ray casting + NavMesh adaptive pathfinding". It does not rely on pre-trained fixed object categories and can perform accurate 3D positioning and navigation of any visible object. The target-view environment scanning module achieves progressive positioning based on a single environment image through a four-step process: "8×8 coarse positioning grid → 3×3 neighborhood clipping → 6×6 fine positioning grid → coordinate verification." Parallel fine positioning (up to 4 threads) improves the efficiency of multi-object scene processing. For opening-type objects (doors / windows), a dedicated offset clipping method is used to ensure that the positioning point falls on walkable ground. The panoramic view environment scanning module achieves target positioning in any direction of the 360-degree field of view through "Unity four-direction shooting + 2×2 stitching + 12×6 panoramic image coarse positioning + 6×6 sub-image fine positioning." The Unity game engine front-end raycasting and NavMesh pathfinding use mechanisms such as camera aspect ratio adaptation, Y-axis flip correction, absolute yaw angle correction, ground projection, navigation grid sampling alignment, path length adaptive speed calculation, and orientation correction after arrival to accurately convert the normalized coordinates of 2D images into 3D navigation targets and drive the character to automatically find its way.

[0191] 2. The hierarchical memory and narrative memory generation system of this invention proposes a three-layer memory architecture: short-term memory (task-level reasoning context) → medium-term memory (six types of fragmented sequences at the conversation level, with no upper limit on accumulation) → long-term memory (cross-conversation JSON vector database). This architecture enables AI characters to continuously remember information across tasks and conversations. At the end of a conversation, fragmented sequences are automatically compressed into third-person narrative memories, filtering out pure visual question-and-answer contamination and extracting sentiment markers. In case of summary failure, fragment splicing is used as a fallback to ensure that memories are not lost. A text embedding vectorization + comprehensive scoring retrieval algorithm is used to achieve semantic-level historical memory retrieval. During task reasoning, the cross-conversation vector database supports memory information injection and on-demand active querying, enabling AI components to reference past events, breaking free from the limitations of pure conversation history stacking, and achieving true cross-conversation memory continuity.

[0192] 3. The director engine and context assembly system of this invention propose a five-dimensional structured context assembly scheme and a central planning architecture with closed vocabulary constraints for the director engine. Before each director engine call, context assembly automatically completes three-way memory injection, VLM environment result fusion, and position information filling, providing the director engine with complete decision-making basis. The director engine limits the output range of animation, expression, and movement with a closed vocabulary and supports multiple on-demand tool calls. The dual mechanism of "automatic injection + active supplementation" ensures the integrity of decision information. The output result repair mechanism of the director engine ensures that abnormal output does not block subsequent processes. The reasoning memory inheritance mechanism ensures that multiple rounds of director calls for the same task maintain logical coherence and avoids repetitive analysis.

[0193] 4. The natural language-driven five-track generative animation system of this invention proposes a four-layer separation architecture: director engine task reasoning (semantic planning) → timing arrangement → deterministic compiler (target instruction generation) → parallel execution (TimelinePlayer). Semantic understanding and timing arrangement each perform their respective functions. The V2 DSL protocol of this invention supports independent and parallel control of five tracks: anim / morph / move / text / special. Each track achieves instruction-level asynchronous arrangement with an independent start timestamp, breaking through the limitations of traditional single-threaded block-level synchronous timing expression. The closed parameterized definition of animation code (F series) and expression code (K series) allows the timing arrangement component to generate arbitrarily complex behavior sequences through code combination. The four-channel overlay system (breathing / slight swaying / head shaking / blinking) runs independently outside the target instruction execution layer, realizing the natural life feeling of the character at all times.

[0194] 5. This invention also proposes a prefetching mode mechanism that utilizes the physical movement time (3-10 seconds) of an AI character to synchronously execute VLM visual analysis. Combined with orientation recording, virtual camera prefetching, asyncio parallel architecture, and system parallel warm-up technology, the perceived latency for users is reduced from approximately 14 seconds (approximately 6 seconds of VLM analysis + approximately 5 seconds of inference + approximately 3 seconds of instruction generation, serially superimposed) to nearly 0 seconds (completed in parallel during movement, executed upon arrival). The continuous micro-motion and continuous playback of the four-channel overlay system eliminate the visual waiting sensation.

[0195] 6. The autonomous behavior-driven system of this invention defines multiple independent demand channels (hunger, boredom, etc.) through multi-channel demand-driven autonomous behavior. Each channel's autonomous demand value grows at an independent rate, triggering an autonomous task when it reaches a preset threshold. When the demand value reaches the 50% threshold, the AI ​​component is invoked to pre-generate calculations, directly using the pre-generated plan upon triggering, avoiding delays in AI component invocation at the moment of triggering. Autonomous tasks and user tasks share the same director-scanner-instruction generation-compiler pipeline, requiring no additional development. All demand growth is paused when the player inputs information and resumes after the task is completed, achieving a natural switch of "responding to the player when in focus and autonomously living when out of focus," giving the AI ​​character a true sense of autonomous life.

[0196] This invention defines a V2 DSL format based on a domain-specific language. Through comparative analysis with similar solutions, the necessity of the V2 DSL format design in this invention is analyzed: 1) Comparison with multi-track synchronization: Compared to the method where each action block simultaneously specifies the animation frames and durations of all tracks, resulting in complete timing synchronization between different tracks, V2 DSL assigns an independent start timestamp to each instruction. Tracks are arranged completely freely on the timeline, supporting arbitrary "before / after, overlap, and interval" timing relationships. This better alleviates fine-grained timing differences between tracks, such as "movement starts first while the subtitle is delayed by 0.3 seconds." It upgrades from "block-level synchronization" to "instruction-level asynchronicity," expanding the complexity of actions that can be described from single synchronous actions to multi-instruction concurrent scenarios.

[0197] 2) Comparison with Single-Thread Motion Control Systems: Traditional digital human animation systems generally adopt a single-threaded mode, where one action triggers a pre-made animation. Animation, facial expressions, and voice captions are forcibly coupled within the same time unit and cannot be independently arranged. V2 DSL completely decouples the five tracks: full-body skeletal animation (anim), facial expression blendshape (morph), spatial movement (move), caption text (text), and special instructions (special). Each track has an independent time sequence that can be arbitrarily combined and superimposed. This means that a sequence of actions can simultaneously occur: walking animation + happy expression + speaking caption + moving towards a target + PANO trigger, without any manual keyframes. A complex action that would require animators to manually keyframe for hours to complete in a single-threaded system can be completed in about 3 seconds by V2 DSL through LLM generation and deterministic compilation.

[0198] 3) Comparison with LLM's direct animation coordinate generation scheme: Some schemes attempt to have LLM directly output animation instructions containing precise timestamps and skeletal transformation data. However, LLM's generation of timing values ​​has low accuracy and poor consistency, easily leading to track conflicts or timing misalignments. V2 DSL avoids this problem through a design that separates semantic understanding from timing arrangement: the director LLM only outputs natural language-level semantic descriptions ("walking," "happy," "ray movement"), the InstructionGenerator LLM maps the semantics to keyframes with timestamps, and the DeterministicCompiler finally compiles the keyframes into precise DSL strings. LLM is responsible for "what to do," while the deterministic algorithm is responsible for "when to do it," each performing its own function, eliminating the accuracy risks associated with LLM's direct generation of timing values.

[0199] Please see Figure 7 , Figure 7 A flowchart illustrating a method for implementing a 3D virtual space-based embody AI interaction system, provided as an embodiment of the present invention, includes: S1. Upon receiving task information input by the user, acquire the environmental image of the character in the current viewpoint / panoramic view, and identify the environmental information in the current viewpoint / panoramic view through a visual language model; S2. After the task information session ends, the fragmented sequence in the session is converted into narrative memory text through the large language model, and the narrative memory text is converted into memory information and stored in the hierarchical memory architecture. S3. Assemble the task information, environment information, memory information, location information and state information corresponding to the character into structured context information, and input the structured context information into the director engine. Perform task reasoning through the director engine and output the character behavior planning result; wherein, the character behavior planning result includes at least one behavior unit. S4. The multiple track instructions corresponding to each behavior unit are time-series arranged using a large language model and compiled into target instructions corresponding to the behavior unit; the target instructions are parsed and executed at the front end to generate the character's animation sequence.

[0200] The implementation method of the 3D virtual space-based embody AI interaction system described above is based on the 3D virtual space-based embody AI interaction system described above, and the brief descriptions can be referred to accordingly.

[0201] It is understandable that the above steps are not strictly performed in the order described by the numbers, but should be understood as a whole.

[0202] The following example, using the user inputting the task "Find the red cube outside the door," fully describes the entire process from receiving the instruction to task completion: Preparation phase: System warm-up and standby Four AI components are warmed up in parallel: a visual language model for environment scanning, a visual language model for panoramic image stitching, a director engine based on a large language model, and a large language model for temporal orchestration. Each component sends a minimum placeholder input (32×32 white PNG or a minimum JSON string, with max_tokens limited to 1-10), triggering the establishment of an HTTP connection and the corresponding prompt word's server-side cache. The Unity game engine frontend completes the TCP connection, disabling the Nagle algorithm (TCP_NODELAY) to reduce latency in small data packet transmission. The NeedsSystem starts timing, with the hunger and boredom demand channels' autonomous demand values ​​accumulating independently from 0 at their respective set growth rates (growth_rate).

[0203] like Figure 8 The image shown is a schematic diagram of the interface displaying the target object and character positions in the Unity game engine front-end during the preparation phase of a certain scene.

[0204] Phase 1: User Input Tasks and Image Acquisition When a user-input task is detected, the Unity game engine frontend captures an RGB image, packages the raw task text input by the user, and the environment image, and sends it to the Python inference backend via TCP. Upon receiving the image, the Python inference backend stores the environment image in a temporary directory, extracts the AI ​​character's world coordinates and the user's world coordinates from the message, and then calls the function to create the user task.

[0205] Phase 2: The internal demand system is paused. When user input is detected, pause all autonomous demand value growth. All current autonomous demand values ​​are frozen and will resume after the task is completed.

[0206] Phase 3: VLM Environment Scan The VLM algorithm analyzes the RGB image and performs a two-stage mesh localization: In the first stage, the image is divided into an 8×8 grid, and VLM returns the coarse grid coordinates of the target object. If necessary, the second stage crops a 3×3 neighborhood sub-image and further divides it into a 6×6 fine grid, with LLM returning the precise grid coordinates. The final output includes a scene description and a list of objects.

[0207] Phase 4: Context Assembly ContextAssembler assembles complete structured context information: task information (original task text original_input, round number round = 1, current round trigger reason status ="new task"), memory information (semantic relevant memory, recent session memory, task-level short-term memory reasoning_memory=[], current session fragments=[]), environment information (scene description scene_description, object list objects), location information (world coordinates of AI character and user), and state information (emotion tag emotion).

[0208] Phase 5: Director Engine's First Round of Reasoning The director engine calls the LLM (Logical Learning Model), inputting complete structured context information. The LLM returns JSON-formatted output: a list of reasoning processes, a natural language description of the AI ​​character's behavior (reaction), and a sequence of behavioral units (beats). In this embodiment, the director engine outputs one beat: beat_id=1, the tracks field contains the track instruction type (move / anim / morph / text / special), the context field contains the target point (move_target), the missing information declaration missing_info=[], and the task completion flag task_end=false. The reasoning is written to the reasoning memory, and the reaction is written to the fragmented memory.

[0209] Phase 6: Verification and Prefetch Inference The BeatScanner function detects "PANO" in the "special" field of the beats and automatically triggers prefetch mode. It detects a conflict between "task_end" and the prefetch mode and automatically corrects "task_end" to false. The validation passes.

[0210] Phase 7: Timing Orchestration to Generate Keyframe Array The timing orchestration component calls LLM, inputting the beat structure and DSL format specification. LLM autonomously determines the start time and duration of each track. It returns 5 keyframes: move, anim, morph, text, and special.

[0211] Phase 8: Deterministic compiler generates target instructions The deterministic compiler compiles keyframes into V2 DSL strings as target instructions. The compilation process is purely deterministic, with no LLM calls.

[0212] Phase 9: Sending target commands and state switching The target instruction and prefetch mode information metadata (beat_id, prefetch=true, task_end=false, prefetch_focus) are sent to the Unity game engine frontend via TCP. The state machine transitions from the processing state (PROCESSING) to the waiting prefetch state (WAITING_PREFETCH).

[0213] Phase 10: Unity Game Engine Front-End Execution of Target Instructions and Synchronous Prefetching TimelinePlayer parses the target instruction, detects that metadata.prefetch=true, immediately places a virtual camera at the move target point, rotates it to the current AI character's orientation, and captures environmental images from four directions, sending them back to the Python inference backend via TCP. Simultaneously, TimelinePlayer begins executing the target instruction: the character plays animations and expressions, and NavMeshAgent begins driving the character's movement.

[0214] Phase 11: Parallel processing of panoramic images using Python inference backend The panoramic image is analyzed in a thread pool: the first stage is coarse localization (dividing the image into a 12×6 grid, calling VLM to analyze the four directions, and returning the direction of the target object and the coarse grid coordinates); the second stage is fine localization (cropping the sub-image, dividing it into a 6×6 grid, and VLM returning the precise grid coordinates); coordinate correction; and finally, scene descriptions in the four directions and the precise coordinates and direction of the target object are output. During this process, the character continuously moves, and the VLM analysis is completed during the movement, resulting in low perceived latency for the user.

[0215] like Figure 9 The image shown is a diagram illustrating the successful localization of the target object during panoramic scanning, displayed on the front end of the Unity game engine.

[0216] Phase 12: Supplementing Fragmented Memories and Replanning Tasks The panoramic image analysis results are added to the `new_info` field of the first round of fragmented memory: `scene`, `objects`. This triggers the task replanning process: discarding any remaining unsent beats, changing the `task_id`, incrementing `round` by 1, and re-calling the director engine for character task planning.

[0217] Phase 13: Director Engine Second Round of Reasoning The ContextAssembler assembles a new context: task.round=2, task.status="Round 1 PANO scan completed, new environmental information obtained, please continue processing the task based on the new information", reasoning_memory contains the first round of reasoning, session_fragments contains the first round of fragments (including new_info), the environmental information is updated to the panoramic image analysis results, and the location information is updated to the latest coordinates. The director engine calls LLM, and LLM returns JSON: reasoning, reaction, beats. In this round, the director engine outputs 3 beats. The reasoning is written to the reasoning memory, and the reaction is written to the fragment memory.

[0218] Phase 14: Pre-scanning and position detection BeatScanner performs a full pre-scan of 3 beats. It detects beats whose move_target is in world coordinates and approximately equal to the current player / AI coordinates (error ≤ 0.3m), automatically marking them as "needs_position" (to be filled in after BLOCK_COMPLETE returns the latest coordinates). Other beats are marked as "ready".

[0219] Phase 15: Parallel generation of target instructions The timing orchestration component is invoked in parallel to generate keyframes for the three beats. The deterministic compiler compiles these keyframe arrays into target instructions. The beat of needs_position generates the target instruction template (coordinate placeholder "__POS__").

[0220] Phase 16: Send the first beat Send the target instruction and prefetch mode information metadata for the first beat. The state machine then transitions to the WAITING_COMPLETE state.

[0221] Phase 17: The Unity game engine frontend executes the first beat and returns a behavior block execution complete message (BLOCK_COMPLETE). That is, the TimelinePlayer executes the first beat: playing animations, displaying emoticons, showing subtitles, and executing special instructions. After execution, the Unity game engine frontend sends a BLOCK_COMPLETE message (type="block_complete", beat_id, task_end=false, ai_pos, player_pos).

[0222] Phase 18: Populate the needs_position template and send the next beat. Upon receiving the BLOCK_COMPLETE message, Python extracts the latest coordinates, replaces the "__POS__" placeholder in the target instruction template with a string, and generates the complete target instruction. It then sends the target instruction and metadata for the next beat. The state machine remains in the WAITING_COMPLETE state.

[0223] Phase 19: Unity game engine front-end executes remaining beats TimelinePlayer automatically retrieves beats from the queue for execution. Each beat sends a BLOCK_COMPLETE message upon completion. After the last beat completes, if metadata.task_end=true is detected, a TASK_COMPLETE message is sent.

[0224] Phase 20: Task Closure and Required System Restoration The Python inference backend receives a TASK_COMPLETE message, and the state machine switches to standby (IDLE) state, indicating an increase in internal drive demand and system recovery requirements. The session manager refreshes the last session time and starts a 30-second timeout countdown.

[0225] like Figure 10 The image shown is a diagram illustrating the completion of a user input task displayed on the front end of the Unity game engine.

[0226] Phase 21: Session Timeout and Writing of Memory Information If no new user input occurs within 30 seconds, the session manager triggers a session timeout. It then invokes the LLM (Local Management Library), taking `reasoning_memory` and `session_fragments` as input, to generate a third-person narrative memory text. The memory information, consisting of the narrative memory text, embedding, importance, and emotion, is written to the cross-session vector database `memories.json`, and the session ends.

[0227] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a 3D virtual space-based embody AI interaction system provided in the above-described method embodiments.

[0228] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0229] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute a 3D virtual space-based embody AI interaction system provided by the above-described method embodiments.

[0230] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a 3D virtual space-based embody AI interactive system provided by the methods described above.

[0231] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0232] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0233] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D virtual space-based embody AI interaction system, characterized in that, include: An open object visual spatial positioning system is used to acquire environmental images of a character in the current viewpoint / panoramic field of view when receiving task information input by the user, and to identify environmental information in the current viewpoint / panoramic field of view through a visual language model; A hierarchical memory and narrative memory generation system is used to establish a hierarchical memory architecture at the task level, session level, and cross-session level. After the session ends, the fragmented sequence in the session is converted into narrative memory text using a large language model, and the narrative memory text is converted into memory information and stored in the hierarchical memory architecture. The director engine and context assembly system are used to assemble the task information, environment information, memory information, location information and state information corresponding to the character into structured context information, and input the structured context information into the director engine, perform task reasoning through the director engine and output the character behavior planning result; wherein, the character behavior planning result includes at least one behavior unit; A natural language-driven multi-track generative animation system is used to define multiple track instructions corresponding to the behavior unit through a domain-specific language; to perform time-series arrangement of the multiple track instructions corresponding to each behavior unit through a large language model, and compile them into target instructions corresponding to the behavior unit; and to parse and execute the target instructions at the front end to generate the animation sequence of the character.

2. The 3D virtual space-based embody AI interaction system according to claim 1, characterized in that, The open object visual spatial positioning system includes: The target view environment scanning module is used to capture a single environment image from the character's current viewpoint, divide the single environment image into a grid, perform coarse localization of each object in the current viewpoint environment using a visual language model, crop out the target area based on the coarse localization of each object, and generate environmental information under the current viewpoint using the visual language model; wherein, the environmental information under the current viewpoint includes a scene description and a list of objects under the current viewpoint. A panoramic view environment scanning module is used to acquire environmental images from multiple directions within a panoramic view; stitch the environmental images from multiple directions into a panoramic image, and divide the panoramic image into a grid; coarsely locate target objects in the panoramic view environment using the visual language model; determine the sub-image where the target object is located based on the coarse location of the target object; and generate environmental information under the panoramic view using the visual language model; wherein, the environmental information under the panoramic view includes a scene description under the panoramic view and the position information of the target object; the target object is related to the target point in the task information; The coordinate transformation and automatic pathfinding module is used to transform the position information of the target object into a three-dimensional world coordinate system, perform path planning through ray casting and navigation mesh sampling, and drive the character to move towards the target point according to the path planning result.

3. The 3D virtual space-based embody AI interaction system according to claim 1, characterized in that, The hierarchical memory and narrative memory generation system includes: A hierarchical memory architecture is used to establish a hierarchical memory architecture of task level, session level, and cross-session level. The task level stores multi-turn structured context information of the director engine during task inference. The session level stores session-level fragment sequences, using fragments from each task within the entire session as the basic unit. When the number of fragments reaches a preset fragment number threshold, compression is automatically triggered, and the compressed session-level fragment sequence is written to the cross-session level, which then initiates a new session record. The cross-session level stores the memory information of each session, forming a cross-session vector database. The narrative memory generation module is used to format the fragmented sequence in the session into a labeled text list in chronological order after the session ends; input the text list into a large language model, output the narrative memory text, and vectorize and store the narrative memory text in the cross-session vector database; wherein, the cross-session vector data supports the director engine's semantic retrieval and recent memory retrieval, and provides memory information injection for the director engine's task reasoning process.

4. The 3D virtual space-based embody AI interaction system according to claim 3, characterized in that, The types of fragments include: The specific action descriptions performed by the character, the character's voice content, the concise summary of the director engine during the task reasoning process, the original task text input by the user, the task execution result information, and the scene description generated by the target perspective environment scanning module or the panoramic view environment scanning module. The memory information includes the following fields: Narrative text in the third person format; The text embedding vector corresponding to the narrative text; Importance scores assessed using a large language model; Unix timestamps generated from memory information; The memory source type includes user-initiated sessions, user-initiated autonomous actions without user participation, and mixed sessions generated by user intervention during the execution of user-initiated autonomous actions. The dominant sentiment label when information is written to memory; Tag list.

5. The 3D virtual space-based embody AI interaction system according to claim 1, characterized in that, The director engine and context assembly system include: The context assembly module is used to assemble the task information, environment information, memory information, location information and state information corresponding to the character into structured context information before each call to the director engine; The director engine is used to perform task reasoning based on the structured context information and output character behavior planning results according to planning principles under different scenarios. These planning principles include: if the current viewpoint information is sufficient to complete the task, or if environmental information under a panoramic view is required but has not yet been acquired, a single behavior unit is output; when the complete information required to complete the task is known, multiple behavior units are planned and output based on the continuous actions required to complete the task; for behavior units that require a panoramic view environment to determine, task reasoning is re-performed after the panoramic view environment scanning module completes its work; each behavior unit includes a configuration description corresponding to multiple trajectory commands of the character.

6. The 3D virtual space-based embody AI interaction system according to claim 5, characterized in that, The natural language-driven multi-track generative animation system includes: The multi-track instruction definition module is used to define multiple track instructions corresponding to each behavior unit through a domain-specific language. The multiple track instructions include animation track instructions for controlling skeletal animation, expression track instructions for controlling character facial expressions, movement track instructions for controlling character spatial movement, subtitle track instructions for displaying subtitle text, and special track instructions carrying special instructions. The target instruction generation module is used to perform time-series arrangement of multiple track instructions corresponding to each behavior unit through a large language model, determine the start time and duration of each track instruction, and output a key frame array corresponding to the multiple track instructions; group the key frame array according to track instructions, sort the key frames according to the start time within each track instruction, generate string fragments corresponding to each track instruction, and concatenate the string fragments corresponding to each track instruction to generate the target instruction corresponding to the behavior unit; The animation generation module is used to parse the target instructions at the front end, create a coroutine queue in ascending order of the start time of each track instruction, trigger the operation of the corresponding track instruction when the corresponding time point is reached, and continuously drive the skeletal animation by superimposing the breathing channel, micro-swaying channel, head shaking channel and blinking channel to generate the animation sequence of the character.

7. The 3D virtual space-based embody AI interaction system according to claim 6, characterized in that, As the character moves toward the target point, it injects the prefetch mode into the target command through the special track command. After receiving the target command, the front end generates a virtual camera at the target point in advance in the prefetch mode. The panoramic view environment scanning module identifies the environmental information of the target point in the panoramic view in advance and executes the movement track command in the target command synchronously. When the character reaches the target point, if the environmental information of the target point in the panoramic view has been obtained, the next round of animation sequence of the character is immediately generated through the director engine, the context assembly system, and the natural language-driven multi-track generative animation system; wherein, the orientation of the character when it reaches the target point is corrected according to the absolute yaw angle of the virtual camera in the prefetch mode; the multi-round director engine reasoning process within the same task shares memory information.

8. The 3D virtual space-based embody AI interaction system according to claim 6, characterized in that, During the reasoning process, the visual language model, the large language model, and the director engine continuously run the breathing channel, micro-swaying channel, head swaying channel, and blinking channel in the front end to drive the skeletal animation. In the character behavior planning result, the facial expression state of the facial expression track instruction in the previous behavior unit is passed to the facial expression track instruction in the next behavior unit through a preset weight; If the character is executing a movement track command, the walking animation will continue to play until the target point is reached.

9. The 3D virtual space-based embody AI interaction system according to claim 1, characterized in that, The system also includes: An autonomous behavior-driven system is used to maintain multiple independent intrinsic demand channels. During the standby state of the role, the autonomous demand value is accumulated and calculated. When the autonomous demand value reaches the demand threshold, an autonomous task is planned in advance. When the triggering condition of the autonomous task is met, the autonomous task of the role is triggered. Among them, the priority of user input tasks is higher than the priority of autonomous tasks. When user input tasks are detected, the autonomous task is paused.

10. A method for implementing a 3D virtual space-based embody AI interaction system, characterized in that, The method includes: When receiving task information input by the user, the system acquires environmental images of the character in the current viewpoint / panoramic view and identifies environmental information in the current viewpoint / panoramic view through a visual language model. After the task information session ends, the fragmented sequence in the session is converted into narrative memory text through a large language model, and the narrative memory text is converted into memory information and stored in a hierarchical memory architecture. The task information, environment information, memory information, location information, and state information corresponding to the character are assembled into structured context information, and the structured context information is input into the director engine. The director engine performs task reasoning and outputs the character behavior planning result; wherein, the character behavior planning result includes at least one behavior unit. The large language model is used to time-order multiple track instructions corresponding to each behavior unit and compile them into target instructions corresponding to the behavior unit; the target instructions are parsed and executed at the front end to generate the animation sequence of the character.

Images