Method and system for generating three-dimensional video driven by intelligent agent

Abstract

The invention discloses a method, a system and a computer program product for generating a three-dimensional video. The method comprises the following steps of: receiving text input which comprises content describing a dynamic scene; a three-dimensional video generation process is interactively supervised through three intelligent agents, and the intelligent agents autonomously decide and execute actions conforming to physical rules in a three-dimensional environment through a physical perception intelligent agent module on the basis of a physical engine and a motion planning algorithm; meanwhile, action coherence is ensured through a time sequence optimization module; and the consistency of the generated content and the input description is ensured through the semantic verification module. Automatic generation of a dynamic scene is realized through intelligent agent simulation, high-quality three-dimensional video content can be interactively created, and the method can be widely applied to the fields of virtual reality, game development, movie and television special effects and the like. In addition, the invention further provides a corresponding generation device, a computer storage medium and an electronic equipment implementation scheme.

Description

Technical Field

[0001] The present invention relates to video processing, and in particular, the present invention relates to a method, system and computer program product for agent-driven three-dimensional video generation. Background Art

[0002] Automatic video content generation technology is reshaping the paradigm of digital content production. Its core value is reflected in three dimensions: first, it industrializes the video production process through algorithms, significantly reducing content production costs in news media, online education, digital marketing and other fields; second, the intelligent recommendation system based on user portraits can generate personalized video content for thousands of people, greatly improving user stickiness; finally, in emergency response scenarios, this technology can achieve a minute-level closed loop from information collection to video output, ensuring the temporal and spatial validity of information.

[0003] The current mainstream technology route has obvious dimensional defects: although the generation paradigm based on NLP to 2D images is mature, it is limited by the inherent limitations of two-dimensional representation, which are manifested as follows: (1) the lack of depth perception leads to distortion of scene spatial relationships, and the error in object distance calculation exceeds the reasonable threshold; (2) single-view rendering is difficult to support multi-angle observation, and the scene complexity is limited; (3) the loss of geometric structure information causes the blurring of surface details, especially in key links such as occlusion handling and backside reconstruction.

[0004] To break through the dimensional barrier, a new generation of technologies has begun to introduce three-dimensional representation systems: through three-dimensional representation methods such as point clouds and neural radiance fields (NeRF), digital twin scenes with complete geometric properties (depth, normals, materials) and physical properties (lighting, motion trajectory) are established. This 3D-first generation strategy can achieve: (1) millimeter-level spatial positioning; (2) continuous rendering that supports any perspective; and (3) physics-based dynamic simulation. Summary of the Invention

[0005] The following description includes exemplary methods, systems, and computer program products that embody the present invention. However, it should be understood that, in one or more aspects, the described invention can be practiced without these specific details. In other cases, well-known structures and techniques are not shown in detail to avoid obscuring the present invention.

[0006] According to one aspect of the present invention, a method for generating three-dimensional videos driven by an intelligent agent is disclosed, which includes: receiving a text describing dynamic scene content and a motion control vector for the object; and generating a dynamic three-dimensional video about the dynamic scene content based on the content of the text.

[0007] According to another aspect of the present invention, a computer system is disclosed, comprising: a memory; and at least one processor operatively coupled to the memory and configured to execute the method described above.

[0008] According to yet another aspect of the present invention, a computer program product is disclosed. The computer program product includes program instructions, and the program instructions can be executed by a computing device to enable the computing device to perform the method described above.

[0009] This innovative agent-driven approach to dynamic 3D scene generation, proposed in this paper, integrates three types of agents: scene understanding, motion planning, and physical simulation, enabling autonomous decision-making and supporting real-time interactive feedback of results. This method innovatively supports semantic-level control of 3D motion from natural language, significantly outperforming existing approaches in terms of scene realism. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The invention itself, as well as the mode of use, objects, features and advantages of its preferred embodiments, may be better understood by reading the following detailed description of illustrative embodiments with reference to the accompanying drawings in which:

[0011] Figure 1 A structural block diagram of a system for generating three-dimensional video driven by an intelligent agent according to an embodiment of the present invention is shown; DETAILED DESCRIPTION

[0012] The following description includes exemplary methods, systems, and storage media that embody the present invention. However, it should be understood that, in one or more aspects, the described invention can be practiced without these specific details. In other cases, well-known protocols, structures, and techniques are not shown in detail in order to avoid obscuring the present invention.

[0013] Embodiments of the present invention are described below with reference to the accompanying drawings. In the following description, many specific details are set forth in order to more fully understand the present invention. However, it will be apparent to those skilled in the art that the present invention may be implemented without some of these specific details. Furthermore, it should be understood that the present invention is not limited to the specific embodiments described. On the contrary, any combination of the following features and elements may be considered to implement the present invention, regardless of whether they relate to different embodiments. Therefore, the following aspects, features, embodiments, and advantages are provided for illustrative purposes only and should not be considered as elements or limitations of the appended claims unless expressly set forth in the claims.

[0014] As mentioned earlier, traditional video generation techniques based on 2D images have significant limitations, prompting a shift in research towards a 3D data-driven video generation paradigm. Breakthroughs have been made in this area, particularly in the area of text-to-4D generation, which can directly convert text input containing object descriptions and motion features into 3D dynamic scenes with spatiotemporal continuity.

[0015] While existing technologies can successfully create impressive 3D visual effects, they still face three key challenges in the generation process: First, ensuring physical perception of dynamic scenes is difficult, especially when dealing with complex object interactions, often resulting in motion trajectories that violate physical laws. Second, the stability of long-term sequence generation is insufficient, with significant quality degradation as video length increases. Finally, the ability to provide fine-grained control over text descriptions is limited, making it difficult to precisely fine-tune the timing and spatial relationships of specific actions. These technical bottlenecks severely restrict the usability of generated content in real-world applications.

[0016] The present invention proposes an agent-driven 3D video generation method and system, aiming to break through the bottleneck of existing technologies through a multimodal agent collaborative framework. The system innovatively constructs three core modules: first, the physical perception agent module verifies the physical perception of the scene in real time to ensure that all movements comply with rigid body dynamics and collision detection rules; second, the temporal stability agent module adopts a memory-enhanced network architecture to effectively maintain the visual consistency of long-term series generation; finally, the semantic control agent module realizes fine-grained mapping of text to 3D actions, supporting precise control of motion trajectories, interaction relationships and spatial layouts. The breakthrough of this system is that it transforms the offline rendering process of traditional 3D generation technology into a real-time decision-making process of an agent. During the generation process, the agent actively intervenes to supervise the perspective and time, so that more rounds of iterative training are performed at perspectives and time points with poor results, thereby improving the generation effect.

[0017] Figure 1 FIG. 1 shows a block diagram of a system 100 for generating 3D video driven by an intelligent agent according to an embodiment of the present invention. Figure 1As shown, system 100 includes a static 3D video generation module 110, a 3D video generation module 120, a physical perception agent module 130, a temporal stability agent module 140, and a semantic control agent module 150. Static 3D video generation module 110 is configured to receive a text 101 describing a dynamic scene and, based on the content of text 101, generate a static 3D video 103 of the dynamic scene. 3D video generation module 120 is configured to receive a physical perception supervision vector 131, a temporal stability supervision vector 141, a semantic control supervision vector 151 for an object, and the static 3D video 103. Based on the physical perception supervision vector 131, the temporal stability supervision vector 141, and the semantic control supervision vector 151, for each image in the static 3D video 103, the object in the image is caused to move in accordance with physical perception, temporal stability, and semantic consistency, thereby generating a 3D video 105.

[0018] In terms of physical perception assurance, the system has designed a physical perception agent module based on deep reinforcement learning. This module uses an improved version of the PPO algorithm framework. Its state space contains 128-dimensional physical parameters such as mass, velocity, angular velocity, etc. of all rigid bodies in the scene, and the action space defines the adjustment vector of the rigid body motion. The reward function comprehensively considers the three dimensions of energy conservation, momentum conservation, and collision penalty. Through the GPU-accelerated continuous collision detection algorithm, it can complete the physical verification of complex scenes within 2 milliseconds. The intelligent agent also integrates a physics-based inverse dynamics solver. When a motion that does not conform to the laws of physics is detected, it can quickly calculate a reasonable joint angle correction solution through the Jacobian matrix transpose method.

[0019] In terms of maintaining temporal stability, the system has built a temporal stability agent module with a memory enhancement mechanism. The core of this module is a bidirectional recurrent neural network containing 512 LSTM units, which is combined with a multi-head attention mechanism to capture long-distance dependencies. The network input is an RGB-D sequence of the past 128 frames. After extracting spatiotemporal features through three-dimensional convolution, an optical flow-based feature matching algorithm is used to establish inter-frame correspondences. The specially designed temporal consistency loss function contains three components: pixel-level L1 loss, perceptual VGG loss, and adversarial loss, which effectively suppresses the flickering and mutation problems common in traditional methods. The module also implements a dynamic key frame selection algorithm that can adaptively adjust the reference weights of historical frames according to the degree of scene change.

[0020] In terms of semantic control, the system has developed a semantic control agent module based on a hierarchical Transformer architecture. This module uses a three-stage processing pipeline: first, natural language instructions are converted into 768-dimensional semantic vectors using a BERT-style text encoder; then, semantic information is injected into the motion prediction network through a conditional normalization layer; and finally, a differentiable renderer is used to translate control parameters into actual changes in the 3D scene. This module supports fine-grained motion editing capabilities, including trajectory planning based on Bezier curves, contact point control based on physical constraints, and scene layout adjustment based on bounding boxes. Users can precisely control complex actions such as "the character moves around obstacles at a speed of 1.5 meters per second" through natural language instructions.

[0021] In terms of system integration, the three agent modules communicate via a high-throughput distributed message bus. The physical perception agent publishes physical verification results at a 500Hz frequency, the timing stability agent maintains a double-buffered frame history, and the semantic control agent receives user commands via a gRPC interface. The system employs an innovative timestamp-based optimistic concurrency control algorithm, resolving inter-agent decision conflicts through 64-bit version numbers. The entire architecture is deployed on the Kubernetes container platform, supporting dynamic load balancing and failover to ensure that all computational tasks complete within the strict 16-millisecond deadline.

[0022] In terms of performance, the system achieves several breakthrough metrics while maintaining 4K resolution rendering quality: interactive latency is kept below 50 milliseconds, a real-time generation rate of 60 frames per second is supported, physics simulation accuracy reaches the level of commercial physics engines, and a PSNR (Time Sequence Noise Ratio) indicator exceeds 35dB. Compared to existing technologies, this system transforms the traditional offline rendering process into a real-time intelligent decision-making process. While maintaining cinematic quality, it significantly improves the interactivity and controllability of 3D video generation, providing a new technical solution for fields such as virtual production, game development, and industrial simulation.

[0023] In one embodiment, an embodiment of the present invention also discloses a computer system, comprising: one or more processing units; a memory, wherein the memory stores computer program instructions, and the computer program instructions can be executed by the one or more processing units to enable the one or more processing units to perform the steps of the method described above.

[0024] In one embodiment, an embodiment of the present invention further discloses a computer program product, comprising a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and the program instructions can be executed by one or more processing units to enable the one or more processing units to perform the steps of the method described above.

[0025] The present invention may be a system, method, and / or computer program product. The computer program product includes a computer-readable storage medium. The computer-readable storage medium carries computer-readable program instructions for causing a processor to implement various aspects of the present invention. The method of the present invention may be executed on a standalone computer system, a distributed computing system, or a cloud platform.

[0026] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination of the foregoing.

[0027] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0028] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages and conventional procedural programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network or a wide area network, or may be connected to an external computer.

[0029] The flow chart and block diagram in the accompanying drawings have shown the possible architecture, function and operation of the system, method and computer-readable storage medium according to multiple embodiments of the present invention.In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and the part for this module, program segment or instruction comprises one or more executable instructions for realizing the logical function of regulation.In some as replacement implementations, the function marked in the box also can occur in a sequence different from that marked in the accompanying drawings.For example, two continuous boxes can actually be performed substantially in parallel, and they also can be performed in reverse order sometimes, and this depends on the function involved.

[0030] The description of the present invention is presented for the purpose of illustration and description and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The embodiments are selected and described in order to best explain the principles of the invention, practical applications, and to enable others of ordinary skill in the art to understand the various embodiments of the invention with various modifications that are suitable for the specific purposes contemplated. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing on the market, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a three-dimensional video, comprising: Receive a text containing content describing a dynamic scene; generating a static three-dimensional video of the dynamic scene content according to the content of the text; as well as According to the physical perception, temporal stability, and semantic consistency, for each picture in the static three-dimensional video, the object in the picture is made to move according to the physical perception, temporal stability, and semantic consistency, thereby obtaining a dynamic three-dimensional video.

2. An agent-driven 3D video generation method, characterized in that: The following steps are involved: Constructing a multimodal agent collaboration framework, the framework including a physical perception agent module, a temporal stability agent module, and a semantic control agent module; Verifying the physical plausibility of the generated scene in real time through the physical perception agent module, including constraining motion based on rigid body dynamics and collision detection rules; Maintaining visual consistency across frames by using a memory-augmented network architecture through the temporally stable agent module; The semantic control agent module parses input text instructions to achieve fine-grained mapping of text to three-dimensional actions, including the regulation of motion trajectories, interaction relationships, and spatial layout; During the generation process, real-time decision-making is performed through the multimodal intelligent agent collaborative framework to dynamically adjust the iterative training rounds of the rendering perspective and time nodes.

3. The method according to claim 2, characterized in that The real-time verification of the physical perception agent module includes: Establish rigid body motion constraint equations to calculate the relationship between acceleration, velocity, and displacement of objects in three-dimensional space; use bounding box collision detection algorithm to perform real-time verification of the interaction of objects in the scene.

4. The method according to claim 2, characterized in that The memory-enhanced network architecture of the temporally stable agent module includes: Long short-term memory (LSTM) unit, used to store the motion state features of historical frames; The attention mechanism module is used to calculate the feature association weights of the current frame and the historical frames.

5. The method according to claim 2, characterized in that The fine-grained mapping of the semantic control agent module specifically includes: Extract the semantics of action instructions in text through natural language processing technology; Establish a mapping matrix between semantic labels and 3D motion parameters.

6. The method according to claim 2, characterized in that The real-time decision-making process includes: Calculate the generation quality score of the current perspective and time node through the quality evaluation function; When the score is lower than the threshold, the iterative training rounds of the region are automatically increased; Dynamically allocate computing resources to prioritize low-scoring areas.

7. A computer system comprising: one or more processing units; A memory storing computer program instructions, wherein the computer program instructions are executable by one or more processing units to cause the one or more processing units to perform the steps of the method according to any one of claims 1 to 8.

8. A computer program product comprising a computer-readable storage medium storing computer program instructions, wherein the program instructions are executable by one or more processing units to cause the one or more processing units to perform the steps of the method according to any one of claims 1 to 8.

Images