A method for generating four-dimensional scenes and an electronic device

Abstract

This invention discloses a method and electronic device for generating four-dimensional scenes, relating to the field of 4D technology. In this method, a three-dimensional representation of a target static object, an initial three-dimensional representation of a target dynamic object, and a target three-dimensional deformation amount are generated based on text descriptions of static and dynamic objects, respectively. Path points are inserted according to the spatial position descriptions of the target dynamic object at any two time points, and the three-dimensional representations of the inserted path points are obtained. A four-dimensional scene is generated based on the combined three-dimensional representations at multiple time points. Furthermore, by splitting the scene into static and dynamic objects for separate processing and then combining the three-dimensional representations, the quality of four-dimensional scene generation is improved, and the difficulty of scene generation is reduced. In addition, only the three-dimensional representation at the first time point and the three-dimensional deformation amount at the second time point need to be calculated; the three-dimensional representations of other path points are determined by the initial three-dimensional representation and the target three-dimensional deformation amount. Therefore, the computational load is reduced, and the efficiency of obtaining multiple three-dimensional representations of dynamic objects is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 4D, in particular to a four-dimensional scene generation method and an electronic device. BACKGROUND

[0002] Four-dimensional (4D) scene generation technology is a core technology of generating interactive, high-fidelity dynamic digital content by fusing three-dimensional (3D) spatial form and dynamic changes in time dimension. In related four-dimensional scene generation methods, the entire three-dimensional scene is usually supervised as a whole by relying on global text description, and then a four-dimensional scene is generated. However, since the entire three-dimensional scene is only supervised as a whole by relying on global text description, it is easy to cause the generated three-dimensional scene to have problems such as missing details and unreasonable deformation, so that the quality of the final four-dimensional scene generation is not high; in addition, directly generating a three-dimensional representation of the scene at each time based on the text description directly leads to a large amount of calculation, and the generation efficiency of the four-dimensional scene is low.

[0003] Therefore, how to improve the generation quality and generation efficiency of the four-dimensional scene is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0004] The present application provides a four-dimensional scene generation method and an electronic device to at least solve the problems of low generation quality and generation efficiency of the four-dimensional scene in the related art.

[0005] The present application provides a four-dimensional scene generation method, comprising:

[0006] Obtaining a text description of a target static object in a scene and a text description of a target dynamic object at any two time points; wherein the any two time points include a first time point and a second time point, and the first time point is located before the second time point; the text description includes an object attribute description and a spatial position description;

[0007] Generating a three-dimensional representation of the target static object based on the text description of the target static object, generating an initial three-dimensional representation of the target dynamic object based on the text description of the target dynamic object at the first time point, and generating a target three-dimensional deformation amount of the target dynamic object based on the text description of the target dynamic object at the second time point; wherein the three-dimensional representation is used to represent the spatial position and the attribute of the object;

[0008] According to the spatial position description of the target dynamic object at the any two time points, inserting a path point, and determining the three-dimensional representation of the target dynamic object at the inserted path point according to the initial three-dimensional representation and the target three-dimensional deformation amount, to obtain a plurality of three-dimensional representations of the target dynamic object;

[0009] generate a four-dimensional scene according to the combined three-dimensional representations of multiple time points.

[0010] generate a four-dimensional scene according to the combined three-dimensional representations of multiple time points.

[0011] The application further provides an electronic device, comprising a memory for storing a computer program and a processor for executing the computer program to implement the steps of any of the four-dimensional scene generation methods.

[0012] The application has the following advantages. Firstly, in the method, a text description of a target static object in a scene and text descriptions of a target dynamic object at any two time points are obtained, and then a three-dimensional representation of the target static object, an initial three-dimensional representation of the target dynamic object and a target three-dimensional deformation amount are generated. For the target dynamic object, path points are inserted according to spatial position descriptions of the target dynamic object at any two time points, and a three-dimensional representation of the inserted path points is obtained. The three-dimensional representation of the target static object and the three-dimensional representation of the target dynamic object at the same time point are combined. Finally, a four-dimensional scene is generated according to the combined three-dimensional representations of multiple time points. It can be seen that the generation of the four-dimensional scene is realized by the method. Moreover, compared with a method of relying on a global text description to perform overall supervision on the entire three-dimensional scene, in the method provided by the application, the scene is divided into a static object and a dynamic object for separate processing, and then the three-dimensional representations are combined, so that the quality of the four-dimensional scene generation is improved, and the difficulty of the scene generation is reduced. In addition, compared with a method of calculating a three-dimensional representation at each time point, in the method provided by the application, only the three-dimensional representation at the first time point and the three-dimensional deformation amount at the second time point are calculated, and the three-dimensional representations of other path points are determined according to the initial three-dimensional representation and the target three-dimensional deformation amount, so that the calculation amount is reduced, and the efficiency of obtaining the multiple three-dimensional representations of the dynamic object is improved.

[0013] In addition, the application further provides an electronic device comprising the four-dimensional scene generation method and having the same effect. BRIEF DESCRIPTION OF DRAWINGS

[0014] To make the application clearer, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative effort based on these drawings.

[0015] Figure 1 A flowchart of a four-dimensional scene generation method provided by the embodiments of the application;

[0016] Figure 2A schematic diagram of a text-driven three-dimensional Gaussian generation algorithm provided for an embodiment of the present application is shown in FIG. 1.

[0017] Figure 3 A schematic diagram of interpolation and interpolation three-dimensional representation generation based on a video interpolation pre-training model provided for an embodiment of the present application is shown in FIG. 2.

[0018] Figure 4 A schematic diagram of all-frame four-dimensional representation optimization based on a video generation pre-training model provided for an embodiment of the present application is shown in FIG. 3.

[0019] Figure 5 A flowchart of a complex four-dimensional scene generation method based on a pre-training model provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0021] It should be noted that, in the description of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0022] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 A flowchart of a four-dimensional scene generation method provided for an embodiment of the present application is shown in FIG. 4, which includes the following steps. Figure 1

[0023] S10: Obtain the text description of the target static object in the scene, and the text description of the target dynamic object at any two time instants; wherein the any two time instants include a first time instant and a second time instant, and the first time instant is located before the second time instant; the text description includes object attribute description and spatial position description;

[0024] ​S11: generating a three-dimensional representation of the target static object based on the text description of the target static object, generating an initial three-dimensional representation of the target dynamic object based on the text description of the target dynamic object at the first time, and generating a target three-dimensional deformation of the target dynamic object based on the text description of the target dynamic object at the second time; wherein the three-dimensional representation is used to represent the spatial position and attributes of the object;

[0025] S12: inserting a path point according to the spatial position description of the target dynamic object at any two time points, and determining the three-dimensional representation of the target dynamic object at the inserted path point according to the initial three-dimensional representation and the target three-dimensional deformation to obtain a plurality of three-dimensional representations of the target dynamic object;

[0026] S13: combining the three-dimensional representation of the target static object and the three-dimensional representation of the target dynamic object in the same time based on the spatial position description of the target static object and the plurality of three-dimensional representations of the target dynamic object to obtain a combined three-dimensional representation at the same time;

[0027] S14: generating a four-dimensional scene according to the combined three-dimensional representation at multiple time points.

[0028] The scene is not limited, and is determined according to actual conditions. There can be dynamic objects and static objects in the scene (referred to as a complex scene). The text description of each static object (i.e., the target static object) in the scene and the text description of each dynamic object (i.e., the target static object) at any two time points are obtained. In order to improve the efficiency of obtaining the text description and the fine-grained text description of the scene, in the implementation, obtaining the text description of the static object and the text description of the dynamic object in the scene comprises:

[0029] obtaining a pre-constructed context prompt and a text description requirement of the target scene; wherein the context prompt comprises scene description and extraction of attribute text description of the scene, attribute text description and spatial position description of the static object, and attribute text description and spatial position description of the dynamic object;

[0030] inputting the context prompt and the text description requirement of the target scene into a pre-trained language model;

[0031] outputting the text description of the static object and the text description of the dynamic object in the target scene by the pre-trained language model.

[0032] In the implementation, before outputting the text description of the static object and the text description of the dynamic object in the target scene by the pre-trained language model, it further comprises:

[0033] obtaining a pre-constructed instance of generating the text description of the static object and the text description of the dynamic object in the scene;

[0034] inputting an instance, a context prompt, and a text description requirement of a target scene into a pre-trained language model.

[0035] The above process of obtaining a text description is referred to as a process of fine-grained scene description extraction based on a pre-trained language model. In order to enable those skilled in the art to better understand the above process, the process is described in detail below with specific embodiments.

[0036] With the leap development of large model technology, the inference ability of the pre-trained language model today is extremely strong, and a more accurate and reasonable result can be inferred according to the input text modal problem. Therefore, with the help of its strong cognitive inference ability, a more comprehensive and detailed overall scene description and a variety of attribute information of static and dynamic objects contained in the scene can be inferred for the complex four-dimensional scene description input by the user. Therefore, the present application proposes a fine-grained scene description extraction method for complex four-dimensional scene generation, which specifically includes the following steps:

[0037] 1) Constructing a context prompt: by constructing a context prompt, the pre-trained language model can understand the task requirement. For the generation task of fine-grained description of a complex four-dimensional scene, the present application designs the following context prompt:

[0038] "Please analyze the following scene description and extract the text description set of the dynamic scene, including the attribute text description of the entire scene, the attribute text description and spatial position description (including three-dimensional coordinates and bounding box size) of static objects, the attribute text description and spatial position description (including three-dimensional coordinates (x, y, z) and bounding box size (length in x-axis direction, length in y-axis direction, and length in z-axis direction)) of dynamic objects, and the attribute text description and spatial position description (including three-dimensional coordinates (x, y, z) and bounding box size (length in x-axis direction, length in y-axis direction, and length in z-axis direction)) of dynamic objects. The scene description may contain multiple static objects, multiple dynamic objects, and their attributes and spatial positions. Among them, the attributes of the objects, such as color and material. For dynamic objects, the attributes at the beginning of the action, the spatial position and bounding box size at the beginning of the action, the attributes at the end of the action, and the spatial position and bounding box size at the end of the action can be described.

[0039] 2) Constructing an instance: after the rule is explained, an instance needs to be further given to enable the pre-trained language model to understand the task, and the corresponding result is output in the format specified by the user. The instance constructed by the present application is as follows:

[0040] "Assuming the user input text is "a small cat jumps from a chair to a sofa", one of your eligible outputs is: {"sceneAttributes": "This is a cozy living room, with sunlight streaming through the windows onto the light wood floor, and a faint scent of flowers in the air. The room is neatly furnished, with light wood floors.", "staticObjects": [{"name": "chair", "attributes": "a dark brown wooden chair, with a light gray cushion on the seat", "position": {"coordinates": [1, 1, 0.5], "boundingBox": {"xLength": 0.5, "yLength": 0.5, "zLength": 1}}}, {"name": "sofa", "attributes": "a white fabric sofa, soft and comfortable, with several colored cushions", "position": {"coordinates": [3.5, 3, 0.5], "boundingBox": {"xLength": 2, "yLength": 1, "zLength": 1}}}], "dynamicObjects": [{"name": "cat", "startAttributes": "an orange tabby cat, small in size, with soft fur and green eyes, sitting on the edge of the chair, ready to jump", "startPosition": {"coordinates": [1, 1, 0.5], "boundingBox": {"xLength": 0.3, "yLength": 0.3, "zLength": 0.3}}, "endAttributes": "an orange tabby cat, small in size, with soft fur and green eyes, successfully jumps onto the sofa, landing steadily in the center of the sofa", "endPosition": {"coordinates": [3.5, 3, 0.5], "boundingBox": {"xLength": 0.3, "yLength": 0.3, "zLength": 0.3}}}]}".

[0041] 3) Ask new questions: input the context hints and examples into a pre-trained language model, and the pre-trained language model will have the ability to split the four-dimensional scene into fine-grained scene text descriptions. For the user's new four-dimensional scene text description needs, you can directly ask the pre-trained language model questions to get a set of fine-grained state text descriptions for any four-dimensional scene.

[0042] The text description of the static object, the text description of the dynamic object and the text description of the four-dimensional scene by the user can be obtained by the method. In the embodiment of the application, the text description of the target static object in the scene and the text description of the target dynamic object at any two time points are obtained. The any two time points include a first time point and a second time point, and the first time point is before the second time point. The any two time points are, for example, the start time point of the action of the dynamic object and the end time point of the action of the dynamic object.

[0043] After obtaining the text description of the static object and the text description of the dynamic object, the three-dimensional representation of the dynamic object and the three-dimensional representation of the static object can be determined based on the text-driven image generation model. If the three-dimensional representation of the scene at each time point is directly generated based on the text description, the calculation amount is large, and the generation efficiency of the four-dimensional scene is low. Therefore, in the application, the initial three-dimensional representation of the target dynamic object is generated based on the text description of the target dynamic object at the first time point, and the target three-dimensional deformation amount of the target dynamic object is generated based on the text description of the target dynamic object at the second time point. The process of generating the three-dimensional representation based on the text description is referred to as the process of text-driven three-dimensional generation of the dynamic and static objects in the scene.

[0044] In some embodiments, generating the three-dimensional representation of the target static object based on the text description of the target static object comprises:

[0045] generating the initial three-dimensional representation of the target static object based on the text description of the target static object at the first time point;

[0046] generating the initial three-dimensional representation of the object based on the text description of the object (which can be a dynamic object or a static object) at the first time point comprises:

[0047] randomly initializing the three-dimensional representation;

[0048] randomly generating a camera perspective and rendering the initialized three-dimensional representation to a two-dimensional image;

[0049] adding random noise to the two-dimensional image to obtain a noisy image;

[0050] inputting the diffusion time point, the noisy image and the vector obtained by encoding the text description of the object at the first time point into the pre-trained image generation model;

[0051] generating predicted noise by the pre-trained image generation model;

[0052] calculating the loss by the loss function, and obtaining the parameter gradient by gradient backpropagation;

[0053] inputting the parameter gradient into the optimizer and adjusting the model parameters by the optimizer;

[0054] outputting the optimized model parameters for representing the three-dimensional representation to obtain the initial three-dimensional representation of the object.

[0055] Since the position of a static object does not change over time, in the embodiments of the present application, the initial three-dimensional representation of the target static object is generated based on the text description of the target static object at the first time. The initial three-dimensional representation of the target static object can represent the three-dimensional representation of the object over time. The calculation redundancy is reduced.

[0056] In some embodiments, generating the target three-dimensional deformation of the target dynamic object based on the text description of the target dynamic object at the second time comprises:

[0057] obtaining the initial three-dimensional representation of the target dynamic object, and setting the initial three-dimensional deformation;

[0058] randomly generating a camera perspective, and rendering the initial three-dimensional representation to a two-dimensional image;

[0059] adding random noise to the two-dimensional image to obtain a noisy image;

[0060] inputting the diffusion time, the noisy image, and the encoded vector of the text description of the target dynamic object at the second time into the pre-trained image generation model;

[0061] generating predicted noise through the pre-trained image generation model;

[0062] calculating loss through a loss function, and obtaining parameter gradients through gradient backpropagation;

[0063] inputting the parameter gradients into an optimizer, and adjusting the model parameters using the optimizer;

[0064] outputting the optimized model parameters for representing the three-dimensional deformation to obtain the target three-dimensional deformation of the target dynamic object.

[0065] In order to enable those skilled in the art to better understand the text-driven three-dimensional generation of dynamic and static objects in a scene described above, the following will be described again in conjunction with specific embodiments and the accompanying drawings.

[0066] After obtaining the text description set of the scene and dynamic and static objects (dynamic objects and static objects), the corresponding three-dimensional content can be generated separately, and the complete three-dimensional content of the scene can be obtained by combining them. There are various three-dimensional representation methods for three-dimensional content, which can be point cloud, mesh, neural radiance field, three-dimensional Gaussian, etc. Among them, three-dimensional Gaussian (three-dimensional Gaussian) has the advantages of strong three-dimensional expression ability and fast rendering speed, and therefore, the present application takes three-dimensional Gaussian as a representative of three-dimensional representation. Three-dimensional Gaussian refers to using a series of Gaussian kernels to represent three-dimensional data, which has shown extremely high reasoning speed and reconstruction quality in three-dimensional reconstruction tasks. Specifically, each Gaussian kernel can be represented by some parameters, including a center parameter a rotation variable a scale variable some additional parameters including opacity values and color features for rendering . is the mathematical notation for a real number set. The center parameter value, opacity values and color features for rendering in it are collectively referred to as attribute parameter values With the help of the splat-based volume rendering method, three-dimensional Gaussians can be differentially rendered into a 2D image according to a given camera perspective. Figure 2 A schematic diagram of a text-driven three-dimensional Gaussian generation algorithm provided by an embodiment of the present application is shown in the figure. When generating a three-dimensional representation of a target static object, the three-dimensional Gaussian static representation module is turned on, and the three-dimensional Gaussian offset module is turned off. When generating a three-dimensional representation of a target dynamic object, the three-dimensional Gaussian static representation module is turned off, and the three-dimensional Gaussian offset module is turned on.

[0067] 1) For static objects, and initial state moments of dynamic objects, the three-dimensional Gaussian offset can be set to 0, at which time only the three-dimensional Gaussian static representation needs to be optimized; for the end state moment of the dynamic object, in order to ensure that the main body of the dynamic object remains consistent, the parameter can be kept frozen, and only the three-dimensional Gaussian deformation corresponding to the end moment of the dynamic object needs to be optimized.

[0068] Specifically, the training steps for static objects, and initial state moments of dynamic objects are as follows:

[0069] 1) Randomly initialize the three-dimensional Gaussian representation , where is set to 0 and is not updated during training;

[0070] 2) Randomly generate a camera perspective v, render the three-dimensional Gaussian representation to a 2D image ;

[0071] 3) Add random noise to the image , obtaining a noisy image , where is a unit matrix;

[0072] 4) Let the pre-trained image generation model adopted be , and the diffusion moment , the noisy image ​and the text description of the static object or the initial state of the dynamic object encoded vector , which are input into the pre-trained image generation model to obtain the predicted noise ; wherein the pre-trained image generation model is not limited, for example, the popular Stable Diffusion text-to-image model can be used;

[0073] 5) define the loss function, i.e. SDS loss as follows: , gradient backpropagation is performed to optimize the parameters of the three-dimensional representation ; wherein, represents the mathematical expectation;

[0074] 6) set the iteration threshold , repeat steps 2) to 5) until the iteration is completed.

[0075] The parameters obtained at this time are the three-dimensional representation of the static object or the initial state of the dynamic object.

[0076] 2) For the end state of the dynamic object, only the three-dimensional deformation needs to be optimized, and the static three-dimensional representation does not need to be re-optimized to avoid inconsistencies in the main body of the dynamic object. Therefore, for the three-dimensional representation of the end state of the dynamic object , only needs to be optimized, and the remaining steps remain unchanged.

[0077] After obtaining the initial three-dimensional representation of the static and dynamic objects and the target three-dimensional deformation of the dynamic object by the above method, the three-dimensional representation of the target static object is generated based on the text description of the target static object, the initial three-dimensional representation of the target dynamic object is generated based on the text description of the first time of the target dynamic object, and the target three-dimensional deformation of the target dynamic object is generated based on the text description of the second time of the target dynamic object. Finally, the three-dimensional representation of the target dynamic object at the inserted path point is determined according to the initial three-dimensional representation and the target three-dimensional deformation. The process of inserting the path point and generating the three-dimensional representation of the path point is called the process of dynamic object path and path point three-dimensional representation interpolation generation.

[0078] In some embodiments, according to the spatial position description of the target dynamic object at any two times, the inserted path point comprises:

[0079] obtain the total number of path points and the serial number of the target path point to be inserted;

[0080] determine the path point interpolation coefficient based on the total number of path points and the serial number of the target path point to be inserted;

[0081] According to the spatial position description of the target dynamic object at any two time points and the path point interpolation coefficient, the coordinate of the path point to be inserted is determined;

[0082] According to the coordinate of the path point to be inserted, the path point is inserted.

[0083] According to the initial three-dimensional representation and the target three-dimensional deformation, the three-dimensional representation of the target dynamic object at the inserted path point is determined, so as to obtain a plurality of three-dimensional representations of the target dynamic object, including:

[0084] The product result of the target three-dimensional deformation and the path point interpolation coefficient corresponding to the path point is obtained;

[0085] According to the product result, the three-dimensional deformation corresponding to the path point is determined;

[0086] According to the result obtained by adding the initial three-dimensional representation and the three-dimensional deformation corresponding to the path point, the intermediate three-dimensional representation of the target dynamic object at the inserted path point is determined; wherein the spatial position of the target dynamic object in the intermediate three-dimensional representation is the position of the target dynamic object under its own coordinate system;

[0087] The intermediate three-dimensional representation of the inserted path point is converted into a three-dimensional representation under the world coordinate system, so as to obtain a plurality of three-dimensional representations of the target dynamic object.

[0088] In order to make the person skilled in the art better understand the process of the above-mentioned dynamic object path and path point three-dimensional representation interpolation generation, the following will be described again in combination with specific embodiments and drawings. Figure 3 An interpolation and interpolation three-dimensional representation generation based on a video interpolation pre-training model is provided for the embodiment of the present application. First, interpolation is performed according to the initial and end spatial positions of the dynamic object. Let the initial spatial position and the end spatial position be , , it is assumed that there are T states in total, that is, there are T path points in total, and T-2 path points need to be inserted between the 1st path point time and the Tth path point time. For example Figure 3 the inserted path points in the second spatial position , the third spatial position , etc. Let the path point interpolation coefficient :

[0089] then , the coordinate of the interpolated path point is: , wherein , , , so that a set of spatial sampling points uniformly distributed between the initial spatial position and the end spatial position is obtained.

[0090] Secondly, the initial three-dimensional representation of the dynamic object is interpolated according to the three-dimensional deformation at the initial time and the three-dimensional deformation at the end time, wherein the three-dimensional deformation of the dynamic object at the initial state is 0, the interpolation coefficient is the same as the path point interpolation coefficient, that is: The three-dimensional deformation corresponding to the second spatial position is represented as The three-dimensional deformation corresponding to the third spatial position is represented as . represents , 0, represents . The three-dimensional representation at each interpolation path point time can be obtained from the initial three-dimensional representation , wherein represents the three-dimensional deformation corresponding to the interpolation path point.

[0091] Finally, since the coordinate position of the three-dimensional representation obtained by interpolation adopts the object coordinate system, it is necessary to apply the spatial path point coordinates in the world coordinate system to the three-dimensional representation to update the three-dimensional representation at each interpolation path point time. Let the attribute parameters of the three-dimensional Gaussian representation to be updated at the current time be , then the updated three-dimensional Gaussian representation in the world coordinate system is: . Finally, the three-dimensional representation at each interpolation path point time in the world coordinate system is obtained.

[0092] In order to solve the unreasonable phenomenon such as penetration of dynamic objects, in some embodiments, the method for generating a four-dimensional scene further comprises:

[0093] extracting the spatial coordinate data of each dynamic object at multiple times, and obtaining the spatial position boundary information of the static object, and establishing a spatial coordinate system containing the time sequence position of the dynamic object and the fixed position of the static object;

[0094] constructing a trajectory conflict detection model based on the spatial coordinate system; wherein the model is preset with a conflict judgment rule: when the overlapping area of the spatial coordinates of any two dynamic objects at the same time exceeds a preset threshold, or the spatial coordinates of the dynamic object touch the position boundary of the static object, it is determined that there is a penetration conflict risk;

[0095] using the trajectory conflict detection model to perform time-by-time traversal detection on the complete motion trajectory of each dynamic object, and positioning the time point corresponding to the conflict risk and the related dynamic object;

[0096] ​​For the detected conflict risk, the spatial position parameters of the corresponding dynamic object before and after the conflict time point are adjusted based on the motion logic of the dynamic object.

[0097] Through the method, the adjusted dynamic object motion trajectory avoids the static object position boundary and does not penetrate with other dynamic objects, and the continuity and rationality of the motion trajectory are maintained.

[0098] In the above, the three-dimensional representation of the target static object and the plurality of three-dimensional representations of the target dynamic object are obtained, and in generating the four-dimensional scene, the three-dimensional representation of the target static object and the three-dimensional representation of the target dynamic object in the same time scene are combined based on the spatial position description of the target static object and the plurality of three-dimensional representations of the target dynamic object to obtain a combined three-dimensional representation at the same time; and the four-dimensional scene is generated according to the combined three-dimensional representations at multiple times.

[0099] When the static object and the dynamic object are combined in the three-dimensional space, the situation of overlapping between the static and dynamic objects or three-dimensional particle overlapping is inevitable, which will greatly affect the overall quality of the scene. Therefore, in order to improve the quality of the three-dimensional representation of the scene, in some embodiments, after obtaining the combined three-dimensional representation at the same time, before generating the four-dimensional scene according to the combined three-dimensional representations at multiple times, it further includes:

[0100] In the case of detecting that there is no overlapping area between objects in the same time scene, entering the step of generating the four-dimensional scene according to the combined three-dimensional representations at multiple times;

[0101] In the case of detecting that there is an overlapping area between objects in the same time scene, maintaining the three-dimensional representation of the non-overlapping area, and fusing the overlapping area to obtain a new three-dimensional representation of the overlapping area;

[0102] Updating the combined three-dimensional representation according to the three-dimensional representation of the non-overlapping area and the new three-dimensional representation of the overlapping area.

[0103] Specifically, detecting whether there is an overlapping area between objects includes:

[0104] Determining the axis-aligned bounding box of the object according to the minimum and maximum coordinates of the object in multiple dimensions;

[0105] In the case of detecting that any two bounding boxes have an overlapping area in multiple dimensions, it is determined that there is an overlapping area between the objects corresponding to the bounding boxes; otherwise, it is determined that there is no overlapping area;

[0106] And / or, obtaining the difference between the point cloud data of the object and the coordinate mean value;

[0107] Performing eigenvalue decomposition on the difference to obtain the principal component direction in multiple dimensions;

[0108] A bounding box with orientation of an object is constructed using a principal direction;

[0109] In a case where it is detected that any two bounding boxes have an overlapping region in multiple dimensions, it is determined that there is an overlapping region between objects corresponding to the bounding boxes; otherwise, it is determined that there is no overlapping region.

[0110] After it is determined that there is an overlapping region between objects, the overlapping region is fused. Specifically, fusing the overlapping region includes:

[0111] Obtaining particles having a common region in the overlapping region;

[0112] Determining a weight coefficient corresponding to any two target particles in fusion according to the opacity of the any two target particles; wherein the any two target particles are the particles having the common region;

[0113] Determining a value of an attribute parameter in a three-dimensional representation after fusion of the any two target particles according to the center and the weight coefficient of the any two target particles; wherein the attribute parameter includes a center parameter, an opacity parameter, and a color feature parameter for rendering;

[0114] Determining a value of a scale variable in the three-dimensional representation after fusion of the any two target particles according to the scale variable and the weight coefficient of the any two target particles;

[0115] Performing spherical linear interpolation processing on a rotation variable of the any two target particles based on the weight coefficient to obtain a value of the rotation variable in the three-dimensional representation after fusion of the any two target particles.

[0116] Obtaining the particles having the common region in the overlapping region includes:

[0117] Obtaining information of the particles in the overlapping region; wherein the information of the particles includes a value of a particle attribute parameter, a particle radius , and a shape parameter of the particle ;

[0118] Determining a distance between any two particles according to a difference between values of attribute parameters of the any two particles and a sum of shape parameters of the any two particles;

[0119] In a case where it is detected that the distance between the any two particles is less than or equal to the particle radius, it is determined that the any two particles are the particles having the common region.

[0120] In addition, before determining the weight coefficient corresponding to the any two target particles in fusion according to the opacity of the any two target particles, the method further includes:

[0121] Obtaining a volume of the common region of the any two target particles and a minimum value of volumes of the any two target particles.

[0122] If the ratio between the volume of the common region of any two target particles and the minimum volume is greater than a threshold, then proceed to the step of determining the weight coefficients corresponding to any two target particles during fusion based on the opacity of the two target particles.

[0123] The above process is referred to as the fusion process of three-dimensional combination of dynamic and static objects and overlapping areas. To enable those skilled in the art to better understand this process, specific embodiments are described below.

[0124] Based on the spatial positions of static objects and dynamic objects at each moment in the scene, the scene is combined to obtain a fused 3D GS representation of the scene at each moment.

[0125] When static and dynamic objects are combined in 3D space, overlap between static and dynamic objects or overlap of 3D Gaussian particles is inevitable, which can greatly affect the overall quality of the scene. Therefore, this invention proposes a two-stage 3D GS fusion method, which can effectively calculate the overlap between different objects and perform particle-level 3D GS fusion reasonably, effectively ensuring the quality of 3D GS fusion of the scene.

[0126] The first stage involves calculating the 3D bounding box or convex hull for each object, thus obtaining all overlapping regions. Both 3D bounding boxes and convex hulls can be used to calculate the spatial extent of an object. 3D bounding boxes are faster to calculate but have slightly lower accuracy, while convex hulls are more complex to calculate but offer higher accuracy in determining overlap.

[0127] Taking bounding boxes as an example, first, calculate the axis-aligned bounding box for each object: ,in, They represent the objects in the Minimum and maximum values ​​of the axis coordinates; They represent the objects in the Minimum and maximum values ​​of the axis coordinates; They represent the objects in the Minimum and maximum values ​​of axis coordinates.

[0128] Optionally, after subtracting the mean coordinates from the 3D GS point cloud, covariance matrix eigenvalue decomposition is performed to obtain the principal component directions along the three axes, thus obtaining an Oriented Bounding Box (OBB) for more accurate calculation of object overlap. Secondly, AABB / OBB is used to quickly determine whether two bounding boxes intersect (AABB can be determined using simple coordinate axis comparison; OBB can be detected on 15 axes using the Separating Axis Theorem (SAT)). If they intersect, the overlapping region of the bounding boxes is calculated.

[0129] Second stage: for the overlapping region between objects, determine whether each 3D GS particle exists overlap and perform particle level attribute fusion. First, approximate the 3D GS particle as an ellipsoid , where represents the shape parameters of the particle, i.e. the direction and scale of the Gaussian ellipsoid, represents the 3D coordinates of a point on the ellipsoid, which is the isosurface of the Gaussian at Mahalanobis distance .

[0130] Second, approximate whether two ellipsoids exist overlap, if , then it can be approximately determined that two ellipsoids exist overlap, if , then they do not overlap; third, for the overlapping 3D GS particles, calculate the overlapping region; where, represents the distance between two 3D GS particles, represents the 3D Gaussian representation of one 3D GS particle in the world coordinate system, represents the 3D Gaussian representation of another 3D GS particle in the world coordinate system; represents the shape parameters of one 3D GS particle, represents the shape parameters of another 3D GS particle.

[0131] Let the overlapping region volume be , define , represents the volume of one 3D GS particle, represents the volume of another 3D GS particle, represents the proportion of the overlapping region relative to the smaller ellipsoid, define the threshold value , if it exceeds the threshold, then perform Gaussian particle fusion. Usually can be set. Finally, for the Gaussian particles that exist overlap, perform weight-based parameter fusion. Let the weight corresponding to one 3D GS particle be: , and the weight corresponding to another 3D GS particle be: , then the fused attribute parameter value is: , the fused scale variable is: , where the rotation cannot be directly weighted and needs to use spherical linear interpolation , the fused rotation variable is: to ensure smooth interpolation rotation. In the above expressions, and respectively represent the opacity values of two three-dimensional GS particles, and respectively represent the three-dimensional coordinates of a point on two three-dimensional GS particles, and respectively represent the scale variables of two three-dimensional GS particles, and respectively represent the rotation variables of two three-dimensional GS particles. The above process is performed on all three-dimensional Gaussian particles in the overlapping region, and the fusion of the overlapping region between the objects is completed.

[0132] To further improve the generation quality of the four-dimensional scene, in some embodiments, generating the four-dimensional scene according to the combined three-dimensional representations of multiple time instants includes:

[0133] After obtaining the combined three-dimensional representation of the target time instant, a rendering image corresponding to the combined three-dimensional representation of the target time instant is obtained;

[0134] The rendering image corresponding to the combined three-dimensional representation of the target time instant and the textual description of the four-dimensional scene are input into the pre-trained video generation model for training;

[0135] In the case where the loss function meets the preset requirement, it is confirmed that the combined three-dimensional representation of the target time instant is optimized;

[0136] Based on the optimized combined three-dimensional representations of multiple time instants, the four-dimensional scene is generated.

[0137] This process is called four-dimensional representation optimization based on a pre-trained video generation model. To better understand the above process, the following describes specific embodiments in conjunction with the accompanying drawings. To improve the continuity of the entire complex four-dimensional scene, after obtaining the three-dimensional representation corresponding to each path point time instant of the dynamic object, the pre-trained video generation model is used to render the three-dimensional representation of the scene at each path time instant to obtain an image, thereby forming a video. The Score Distillation Sampling (SDS) optimization method based on the video is used to simultaneously optimize the three-dimensional deformation representation corresponding to all video frames, thereby improving the generation effect of the overall complex four-dimensional scene. Figure 4An all-frame four-dimensional representation optimization schematic diagram based on a pre-trained model for video generation is provided for an embodiment of the present application. At each path point time, an image is rendered based on the fused three-dimensional Gaussian representation, and finally a video rendered at all path point times is obtained, scene four-dimensional representation optimization based on a pre-trained video generation model is performed, and a final high-quality four-dimensional scene is obtained. Specifically, a score distillation sampling (SDS) optimization method is used to optimize all three-dimensional Gaussian representations corresponding to the video frames. The optimization process is similar to the text-driven scene dynamic and static object three-dimensional generation method, but the difference lies in that the pre-trained model is a pre-trained video generation model, and the input is multiple frames rendered by the scene fused three-dimensional Gaussian representation, so as to optimize the three-dimensional Gaussian representations corresponding to all frames at the same time. For example, the three-dimensional Gaussian representations fused at multiple times are first fused three-dimensional representation , second fused three-dimensional representation , third fused three-dimensional representation ……, and the fused three-dimensional representation at the end time , the three-dimensional Gaussian representations fused at multiple times generate a four-dimensional scene.

[0138] Finally, the three-dimensional Gaussian representation of the dynamic and static objects at each time of the entire scene is obtained, and the generation of the high-quality four-dimensional scene is completed. The three-dimensional representation optimization based on the pre-trained video generation model can effectively improve the continuity of the entire complex four-dimensional scene, and obtain a high-quality four-dimensional representation of the complex scene.

[0139] The entire process of the four-dimensional scene generation method is described above. In order to more clearly understand the entire process, the following will be described again in conjunction with the drawings. Figure 5 A flowchart of a complex four-dimensional scene generation method based on a pre-trained model is provided for an embodiment of the present application, as shown in Figure 5 , the method comprises:

[0140] S15: obtaining a text description of a user for a four-dimensional scene;

[0141] S16: scene fine-grained description extraction based on a pre-trained language model;

[0142] S17: obtaining a scene and dynamic and static object text description set;

[0143] S18: text-driven scene dynamic and static object three-dimensional generation;

[0144] S19: obtaining a static object three-dimensional representation and a dynamic object start-end three-dimensional representation;

[0145] S20: obtaining a dynamic object path and path point three-dimensional representation interpolation;

[0146] S21: obtaining a three-dimensional representation of a dynamic object path point;

[0147] S22: Four-dimensional representation optimization based on pre-trained video generation model;

[0148] S23: Generate four-dimensional scene.

[0149] Specifically, the following steps are included:

[0150] The first step: for the user input text description of the four-dimensional scene, a pre-trained language model is used to extract the text description set of the four-dimensional scene through context instruction learning, including the attribute text description of the whole scene, the spatial position description of static objects and their attribute text description, the initial and final attribute text description of dynamic objects and their initial and final spatial position description.

[0151] The second step: using a text-driven three-dimensional generation algorithm, the three-dimensional representation of static objects and the initial and final three-dimensional representation of dynamic objects are obtained. In particular, for dynamic objects, first, the initial three-dimensional representation is obtained according to the initial text state description; second, for the end state, only the offset based on the initial three-dimensional representation is optimized according to the end text state description.

[0152] The third step: for dynamic objects, a series of path point sets are obtained according to the initial and final spatial position descriptions; for each interpolated path point time, the three-dimensional deformation offset of each path point time is obtained based on the three-dimensional representation offset of the end state, so as to obtain a series of three-dimensional representations of dynamic object path points.

[0153] The fourth step: the scene static objects are combined to obtain a static three-dimensional scene according to the spatial position relationship; then, the dynamic objects are added at each path point time and rendered to obtain a video, and the scene three-dimensional representation optimization based on the pre-trained video generation model is performed to obtain the final high-quality three-dimensional scene.

[0154] The application provides a complex four-dimensional scene generation method based on a pre-trained model (including a pre-trained language model, a pre-trained image generation model and a pre-trained video generation model). First, scene understanding based on a pre-trained language model is performed on a text description about a complex four-dimensional scene input by a user, and attribute text descriptions of the entire scene, static objects and their attribute text descriptions and spatial position descriptions, dynamic objects and their initial and final attribute text descriptions and initial and final spatial position descriptions are extracted. Second, a pre-trained text-driven three-dimensional content generation algorithm is used to obtain initial three-dimensional representations of the static objects based on the attribute text descriptions of the static objects, and initial and final three-dimensional representations of the dynamic objects based on the initial and final attribute text descriptions of the dynamic objects. Third, path interpolation is performed according to the initial and final spatial positions of the dynamic objects to obtain a series of initial path points of the dynamic objects, and the three-dimensional representations of the dynamic objects obtained by interpolation are calculated at each path point. Finally, the three-dimensional representations of the dynamic objects at each path point are optimized based on the attribute text descriptions of the entire scene and the pre-trained video generation model, and high-quality four-dimensional content of the entire scene is finally obtained. By separately processing static and dynamic objects, the problem of complex four-dimensional scene generation is effectively solved, and the generation quality of the complex four-dimensional scene is significantly improved.

[0155] The above describes a four-dimensional scene generation method, and the embodiment also provides a four-dimensional scene generation device, which comprises:

[0156] An acquisition module is configured to acquire text descriptions of target static objects in a scene and text descriptions of target dynamic objects at any two time points; wherein the any two time points include a first time point and a second time point, and the first time point is located before the second time point; the text descriptions include object attribute descriptions and spatial position descriptions;

[0157] A first generation module is configured to generate a three-dimensional representation of the target static objects based on the text descriptions of the target static objects, generate an initial three-dimensional representation of the target dynamic objects based on the text descriptions of the target dynamic objects at the first time point, and generate a target three-dimensional deformation amount of the target dynamic objects based on the text descriptions of the target dynamic objects at the second time point; wherein the three-dimensional representation is used to represent the spatial position and the attribute of the object;

[0158] A determination module is configured to insert a path point according to the spatial position descriptions of the target dynamic objects at the any two time points, and determine the three-dimensional representation of the target dynamic objects at the inserted path point according to the initial three-dimensional representation and the target three-dimensional deformation amount, so as to obtain a plurality of three-dimensional representations of the target dynamic objects;

[0159] A combination module is configured to combine the three-dimensional representation of the target static objects and the three-dimensional representation of the target dynamic objects in the same time point based on the spatial position descriptions of the target static objects and the plurality of three-dimensional representations of the target dynamic objects, so as to obtain a combined three-dimensional representation in the same time point.

[0160] a second generating module configured to generate a four-dimensional scene according to the combined three-dimensional representation of the plurality of time instants.

[0161] In some embodiments, the four-dimensional scene generating apparatus further comprises:

[0162] the second generating module is triggered when it is detected that there is no overlapping region between the objects in the same time instant scene;

[0163] a maintaining and fusing module configured to maintain the three-dimensional representation of the non-overlapping region and fuse the overlapping region to obtain a new three-dimensional representation of the overlapping region when it is detected that there is an overlapping region between the objects in the same time instant scene;

[0164] an updating module configured to update the combined three-dimensional representation according to the three-dimensional representation of the non-overlapping region and the new three-dimensional representation of the overlapping region.

[0165] The features of the embodiments of the four-dimensional scene generating apparatus can be referred to the related descriptions of the embodiments of the four-dimensional scene generating method, which will not be repeated here.

[0166] The embodiments of the present application further provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-mentioned embodiments of the four-dimensional scene generating method.

[0167] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above-mentioned embodiments of the four-dimensional scene generating method when running.

[0168] In an example embodiment, the above-mentioned computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0169] The embodiments of the present application further provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the four-dimensional scene generating method.

[0170] The embodiments of the present application further provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the four-dimensional scene generating method.

[0171] Those skilled in the art will further appreciate that the units and algorithms described in connection with the examples disclosed herein can be implemented in electronic hardware, computer software, or any combination thereof. To clearly illustrate this interchangeability of hardware and software, various components will be described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0172] The above provides a detailed introduction to the four-dimensional scene generation method and the electronic device. The principles and implementation manners of the present application are described by using specific examples. The above description of the examples is only applicable to help understand the method and the core idea of the present application. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method of four-dimensional scene generation, characterized by, The method comprises the following steps: obtaining a text description of a target static object in a scene and a text description of a target dynamic object at any two time points; wherein the any two time points include a first time point and a second time point, and the first time point is located before the second time point; the text description includes an object attribute description and a spatial position description; generating a three-dimensional representation of the target static object based on the text description of the target static object, generating an initial three-dimensional representation of the target dynamic object based on the text description of the target dynamic object at the first time point, and generating a target three-dimensional deformation of the target dynamic object based on the text description of the target dynamic object at the second time point; wherein the three-dimensional representation is used to represent the spatial position and the attribute of the object; inserting a path point according to the spatial position description of the target dynamic object at the any two time points, and determining a three-dimensional representation of the target dynamic object at the inserted path point according to the initial three-dimensional representation and the target three-dimensional deformation to obtain a plurality of three-dimensional representations of the target dynamic object; combining the three-dimensional representation of the target static object and the three-dimensional representation of the target dynamic object in the same time point scene based on the spatial position description of the target static object and the plurality of three-dimensional representations of the target dynamic object to obtain a combined three-dimensional representation in the same time point; generating a four-dimensional scene according to the combined three-dimensional representations in multiple time points; inserting a path point according to the spatial position description of the target dynamic object at the any two time points comprises the following steps: obtaining a total number of pre-set path points and a serial number of a target path point to be inserted; determining a path point interpolation coefficient based on the total number of path points and the serial number of the target path point to be inserted; determining the coordinates of the path point to be inserted according to the spatial position description of the target dynamic object at the any two time points and the path point interpolation coefficient; inserting the path point according to the coordinates of the path point to be inserted; determining a three-dimensional representation of the target dynamic object at the inserted path point according to the initial three-dimensional representation and the target three-dimensional deformation to obtain a plurality of three-dimensional representations of the target dynamic object comprises the following steps: obtaining a product result of the target three-dimensional deformation and a path point interpolation coefficient corresponding to the path point; determining a three-dimensional deformation corresponding to the path point according to the product result; determining an intermediate three-dimensional representation of the target dynamic object at the inserted path point according to a result of adding the initial three-dimensional representation and the three-dimensional deformation corresponding to the path point; wherein the spatial position of the target dynamic object in the intermediate three-dimensional representation is the position of the target dynamic object in its own coordinate system; converting the intermediate three-dimensional representation of the inserted path point into a three-dimensional representation in a world coordinate system to obtain the plurality of three-dimensional representations of the target dynamic object.

2. The four-dimensional scene generation method of claim 1, wherein, After obtaining the combined three-dimensional representation in the same time point, and before generating the four-dimensional scene according to the combined three-dimensional representations in multiple time points, the method further comprises the following steps: determining an axis-aligned bounding box of the object according to the minimum and maximum coordinates of the object in multiple dimensions; and / or, obtaining a difference between the point cloud data of the object and the coordinate mean value; performing eigenvalue decomposition on the difference to obtain principal component directions in multiple dimensions; and constructing an oriented bounding box of the object by using the principal component directions. In the case of detecting that any two bounding boxes have an overlapping region in multiple dimensions, it is determined that there is an overlapping region between the objects corresponding to the bounding boxes; otherwise, it is determined that there is no overlapping region; In the case of detecting that there is no overlapping region between objects in the scene at the same time, the step of generating a four-dimensional scene according to the combined three-dimensional representation of multiple time points is entered; In the case of detecting that there is an overlapping region between objects in the scene at the same time, the three-dimensional representation of the non-overlapping region is maintained, and the overlapping region is fused to obtain a new three-dimensional representation of the overlapping region; The combined three-dimensional representation is updated according to the three-dimensional representation of the non-overlapping region and the new three-dimensional representation of the overlapping region.

3. The four-dimensional scene generation method of claim 2, wherein, Fusing the overlapping region includes: Obtaining particles that have a common region in the overlapping region; Determining a weight coefficient corresponding to any two target particles according to the opacity of the any two target particles; wherein the any two target particles are particles that have a common region; Determining the value of an attribute parameter in the three-dimensional representation after the fusion of the any two target particles according to the center of the any two target particles and the weight coefficient; wherein the attribute parameter includes a center parameter, an opacity parameter, and a color feature parameter for rendering; Determining the value of a scale variable in the three-dimensional representation after the fusion of the any two target particles according to the scale variable of the any two target particles and the weight coefficient; Performing spherical linear interpolation processing on the rotation variable of the any two target particles based on the weight coefficient to obtain the value of the rotation variable in the three-dimensional representation after the fusion of the any two target particles.

4. The four-dimensional scene generation method of claim 3, wherein, Obtaining particles that have a common region in the overlapping region includes: Obtaining information of particles in the overlapping region; wherein the information of the particles includes attribute parameter values of the particles, a particle radius, and a shape parameter of the particles; Determining the distance between any two particles according to the difference between the attribute parameter values of the any two particles and the sum of the shape parameters of the any two particles; In the case of detecting that the distance between the any two particles is less than or equal to the particle radius, it is determined that the any two particles are particles that have a common region.

5. The four-dimensional scene generation method of claim 3, wherein, Before determining the weight coefficient corresponding to the any two target particles according to the opacity of the any two target particles, it further includes: Obtaining the volume of the common region of the any two target particles and the minimum value of the volumes of the any two target particles; In the case of detecting that the ratio between the volume of the common region of the any two target particles and the minimum value of the volumes is greater than a threshold value, the step of determining the weight coefficient corresponding to the any two target particles according to the opacity of the any two target particles is entered.

6. The four-dimensional scene generation method of claim 3, wherein, Generating a four-dimensional scene according to the combined three-dimensional representation of multiple time points includes: After obtaining the combined three-dimensional representation of a target time point, obtaining a rendering image corresponding to the combined three-dimensional representation of the target time point; Inputting the rendering image corresponding to the combined three-dimensional representation of the target time point and the text description of the four-dimensional scene into a pre-trained video generation model for training; In the case of satisfying a preset requirement of a loss function, it is confirmed that the combined three-dimensional representation of the target time point is optimized. Generate a four-dimensional scene based on the optimized combination of three-dimensional representations of multiple time points.

7. The four-dimensional scene generation method of any one of claims 1 to 6, characterized in that, The text description of the static object and the text description of the dynamic object in the scene include: Obtain a pre-constructed context prompt and a text description requirement of a target scene; wherein the context prompt includes scene description, attribute text description of the scene, attribute text description and spatial position description of the static object, and attribute text description and spatial position description of the dynamic object; Obtain a pre-constructed instance of generating the text description of the static object and the text description of the dynamic object in the scene; Input the instance, the context prompt, and the text description requirement of the target scene into a pre-trained language model; Output the text description of the static object and the text description of the dynamic object in the target scene through the pre-trained language model.

8. The four-dimensional scene generation method of claim 7, wherein, Generating a three-dimensional representation of a target static object based on the text description of the target static object includes: Generating an initial three-dimensional representation of the target static object based on the text description of the target static object at the first time point; Generating an initial three-dimensional representation of the object based on the text description of the object at the first time point includes: Randomly initialize the three-dimensional representation; Randomly generate a camera perspective and render the initialized three-dimensional representation to a two-dimensional image; Add random noise to the two-dimensional image to obtain a noisy image; Input the diffusion time point, the noisy image, and the vector obtained by encoding the text description of the object at the first time point into a pre-trained image generation model; Generate predicted noise through the pre-trained image generation model; Calculate the loss through the loss function, and obtain the parameter gradient through gradient backpropagation; Input the parameter gradient into the optimizer and adjust the model parameters using the optimizer; Output the optimized model parameters for representing the three-dimensional representation to obtain the initial three-dimensional representation of the object.

9. An electronic device, comprising: It includes: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the four-dimensional scene generation method according to any one of claims 1 to 8.

Images