Scene information generation method, apparatus, device, and storage medium

Abstract

Embodiments of the present disclosure relate to a scene information generation method, apparatus, device and storage medium. The method comprises: obtaining a plurality of images of a scene in which a movable target object is located in a historical time period and trajectory control information of the target object, the trajectory control information indicating one or more target poses of the target object in a target time period; determining a historical voxel representation of the scene in the historical time period based on the plurality of images; and generating a target voxel representation of the scene after the target object operates according to the one or more target poses based on the historical voxel representation, the trajectory control information and noise data for the target time period, the target voxel representation being used to generate a trajectory of the target object operating according to the one or more target poses. In this way, the flexibility of scene information prediction can be improved to adapt to trajectory prediction requirements.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and more specifically, to methods, apparatus, devices, and storage media for generating scene information. Background Technology

[0002] Trajectory prediction technology primarily involves reasonably predicting the motion trends of surrounding dynamic objects in complex environments to assist in path planning decisions for movable target objects. Trajectory prediction typically relies on key technologies such as environmental perception, historical motion data analysis, and prediction of future scene information. Trajectory generation methods may face challenges when dealing with complex scenes; for example, scene information generation methods struggle to accurately model the impact of various environmental factors on motion trends. Therefore, improving the flexibility and generalization ability of future scene information prediction has become a crucial issue that urgently needs to be addressed. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for generating scene information is provided. The method includes: acquiring multiple images of a scene in which a movable target object is located during a historical time period and trajectory control information of the target object, the trajectory control information indicating one or more target poses of the target object during a target time period; determining a historical voxel representation of the scene during the historical time period based on the multiple images; and generating a target voxel representation of the scene after the target object operates according to one or more target poses, based on the historical voxel representation, the trajectory control information, and noise data for the target time period, the target voxel representation being used to generate the trajectory of the target object operating according to one or more target poses.

[0004] In a second aspect of this disclosure, a scene information generation apparatus is provided. The apparatus includes: an acquisition module configured to acquire multiple images of a scene in which a movable target object is located during a historical time period and trajectory control information of the target object, the trajectory control information indicating one or more target poses of the target object during a target time period; a historical voxel representation determination module configured to determine a historical voxel representation of the scene during the historical time period based on the multiple images; and a target voxel representation generation module configured to generate a target voxel representation of the scene after the target object operates according to one or more target poses, based on the historical voxel representation, the trajectory control information, and noise data for the target time period, the target voxel representation being used to generate the trajectory of the target object operating according to one or more target poses.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart illustrating an example process for generating scenario information according to some embodiments of this disclosure is shown;

[0012] Figure 3 A schematic diagram of an example architecture for generating target scene features according to some embodiments of the present disclosure is shown;

[0013] Figure 4 Some embodiments according to this disclosure are shown. Figure 3 A schematic diagram of an example architecture of a neural network;

[0014] Figure 5 A schematic diagram illustrating an example of a training scenario generation model according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A schematic structural block diagram of a scene information generation apparatus according to some embodiments of the present disclosure is shown;

[0016] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," "machine learning network," "neural network," or simply "network," and these terms are used interchangeably.

[0020] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0023] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0024] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown. References Figure 1 One or more movable objects, such as a first robot 110-1, a second robot 110-2, and a third robot 110-3, are deployed in environment 100. First robot 110-1, second robot 110-2, and third robot 110-3 can also be collectively or individually referred to as robot 110. Each robot 110 may be equipped with control devices to control its operation. For example, ... Figure 1 As shown, control devices 120-1, 120-2, and 120-3 are respectively deployed at the first robot 110-1, the second robot 110-2, and the third robot 110-3, which may also be collectively referred to as control device 120 or individually. In some embodiments, only one robot may be deployed in the environment 100, and only the movement path of this one robot needs to be planned. In some embodiments, multiple robots may be deployed in the environment 100. In this case, path planning can be performed simultaneously for multiple robots in the environment 100, or path planning can be performed only for any one robot in the environment 100.

[0027] In embodiments of this disclosure, robot 110 can be used for various suitable purposes, such as a transport robot for delivering goods, and control device 120 can be used to plan the movement path of robot 110. Robot 110 can, for example, perform parcel delivery services or warehouse management services. As an example, robot 110 can operate in indoor environments, such as warehouse environments.

[0028] exist Figure 1 In environment 100, control device 120 can be any type of microcontroller, programmable logic controller, industrial computer, single-board computer, field-programmable gate array, digital signal processor, multi-core processor, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, control device 120 can control robot 110 based on control commands it generates.

[0029] In some embodiments, the operation of robot 110 can be controlled via server 130. Server 130 can receive relevant data for various tasks and remotely control the operation of robot 110. Control device 120 collects data from robot 110 and communicates with server 130 via network 140 to send the data to server 130. Server 130 determines control commands for the robot based on the acquired data and sends the control commands to control device 120. Control device 120 controls robot 110 based on the received control commands. Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0030] In some embodiments, robot 110 can connect to and communicate with server 130 via network 140. Robot 110 can send relevant data on various tasks performed by robot 110 to server 130 in real time.

[0031] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Furthermore, although in Figure 1 The illustration shows a robot, but embodiments of this disclosure can also be applied to other types of movable objects.

[0032] As briefly mentioned earlier, with the development of intelligent perception, intelligent control, and machine learning technologies, trajectory prediction has become a key task for mobile objects to achieve efficient decision-making and safe planning in complex and dynamic environments. For example, in fields such as autonomous driving and robot perception, scene information prediction, as a crucial link between environmental perception and path decision-making, directly affects the safety and reliability of trajectory prediction. The core task of scene information prediction is to predict the environment in which a mobile object will be located in the future based on historical information, so that autonomous driving systems or robots can plan reasonable paths in advance. This, in turn, can avoid potential collisions and optimize driving efficiency.

[0033] Traditional trajectory prediction or scene generation methods typically use historical observation information as input to predict scene characteristics at one or more future time points. These methods often rely on the natural evolution of time series, and their outputs are limited by historical behavior patterns, making it difficult to flexibly respond to new control objectives. For example, when the input historical trajectory shows that a movable object is moving forward, traditional models can usually only predict its continued forward trend, and cannot flexibly generate scene changes corresponding to new behavior patterns such as "turn left" or "turn right" based on the current situation. This prediction model may be suitable for tasks with simple behavior and strong path determinism, but for real-world scenarios such as complex decision-making and multi-strategy planning, its flexibility and generalization ability are still insufficient.

[0034] In this context, predicting the 3D scene a movable object will face after performing specific actions (such as turning, accelerating, or pulling over) from visual information becomes a pressing technical challenge. This requires not only predicting the target object's state at a fixed future point in time but also simulating the scene perception results under different behavioral commands. In other words, besides temporal continuity modeling, it should also support the generation of diverse 3D scenes conditioned by future behavioral control signals (such as target pose). This capability may impact the safety and robustness of key modules such as trajectory generation, path optimization, and obstacle avoidance decisions.

[0035] In view of the above, according to embodiments of this disclosure, an improved scheme for scene information generation is proposed to at least partially solve one or more of the above-mentioned problems. In this scheme, firstly, multiple images of the scene where the movable target object is located within a historical time period and trajectory control information of the target object are acquired, the trajectory control information indicating one or more target poses of the target object within a target time period. Further, based on the multiple images, a historical voxel representation of the scene within the historical time period is determined. Finally, based on the historical voxel representation, the trajectory control information, and noise data for the target time period, a target voxel representation of the scene after the target object operates according to one or more target poses can be generated. The target voxel representation can be used to generate the trajectory of the target object operating according to one or more target poses.

[0036] In the embodiments of this disclosure, a historical voxel representation is constructed by acquiring multiple frames of images over a historical time period, and the pose of the target object is used as a trajectory control condition. This approach enables the flexible generation of corresponding future 3D scenes based on different pose commands, thereby supporting path simulation and environmental perception under various behavioral modes. This method allows for more refined control and response in scene generation, enhancing the flexibility and generalization ability of trajectory prediction, and ultimately improving the safety and adaptability of the target object in complex dynamic scenes.

[0037] Some exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0038] Figure 2 A flowchart illustrating an example process 200 for generating scenario information according to some embodiments of this disclosure is shown. The following will primarily refer to... Figure 1 The example environment described herein is intended to illustrate process 200, but it should be understood that this is merely exemplary and not intended to be limiting. Embodiments of this disclosure can also be applied to other types of environments. Furthermore, process 200 will be described relative to server 130, but this is merely exemplary. Process 200 may be executed by a remote server of the target object (e.g., server 130), or by a control device of the target object itself (e.g., control device 120), or may be performed collaboratively by a remote server and a control device.

[0039] In box 210, server 130 acquires multiple images of the scene where the movable target object is located over a historical time period, as well as trajectory control information of the target object. In embodiments of this disclosure, the target object may include various suitable types of movable objects, such as, but not limited to, autonomous driving systems (e.g., autonomous vehicles), drones, robots, etc. Such objects are capable of autonomously moving towards a set target within their environment. For example, the target object may be one or more of the first robot 110-1, the second robot 110-2, and the third robot 110-3 described above (collectively or individually referred to as robot 110). Furthermore, for safe and efficient movement, it is necessary to perceive and predict the environment in which the target object is located. Based on the results of perception and prediction, the next action of the target object can be determined (e.g., determining the direction of travel, planning a reasonable path, etc.).

[0040] In some embodiments, the scene in which the target object is located can refer to the surrounding environment experienced by the target object within a certain time range, which may include static objects, dynamic objects, etc. Static objects may include, for example, roads, obstacles, buildings, shelves, and other objects that remain still over a relatively long period of time. Dynamic objects may include, for example, people, animals, other movable objects, and other objects that may move in real time.

[0041] During the movement of a target object, information about its surrounding environment during a specific historical time period can be acquired using image acquisition devices (e.g., cameras, vision sensors) to further construct 3D understanding or predict future scenes. Such image acquisition devices can be placed in the surrounding environment or on the target object. For example, server 130 can acquire one or more T-frame images from one or more perspectives captured by a camera on the target object within a historical time period. These images record the environmental changes experienced by the target object, providing foundational data for subsequent future scene prediction. In embodiments of this disclosure, frames and moments can have a correspondence. For example, each frame captured by a camera on the target object can correspond to a specific moment within a time period.

[0042] In some embodiments, trajectory control information refers to control signals used to indicate the future actions or behavioral intentions of a target object, which may include the target pose of the target object at one or more future moments. This target pose may indicate, but is not limited to, the target object's position (e.g., x, y coordinates) and attitude (e.g., heading angle, orientation, rotation angle, etc.) in three-dimensional space. Trajectory control information can support precise (e.g., centimeter-level) position changes and angle-level attitude changes, and is applicable to high-precision path planning and behavior simulation scenarios. By using trajectory control information as a control condition for generating future scene features, a three-dimensional scene representation of the target object after performing the corresponding action can be generated.

[0043] In some examples, server 130 can determine multiple poses of a target object based on the indication of trajectory control information. Each pose can represent the state of the target object after performing a specific action (such as moving forward, turning left, turning right, etc.). By flexibly inputting different pose sequences, environmental changes under various potential behavioral paths can be simulated to facilitate path safety assessment and multi-strategy planning.

[0044] Continue to refer to Figure 2 In box 220, server 130 determines a historical voxel representation of a scene within a historical time period based on multiple images. Exemplarily, upon receiving the multiple images, server 130 can convert them into a 3D voxel representation using a 3D encoder. This 3D encoder can be used to extract spatial structure and geometric information contained in the images and model the scene in 3D voxel space. For example, the voxel representation can be a regular 3D mesh structure, where each cell can be used to represent whether a corresponding position in space is occupied, the voxel density value, semantic labels, or other attributes.

[0045] In some embodiments, the historical voxel representation may include a set of voxel representations for multiple historical moments. This set includes several voxel representations corresponding to multiple frames of images or multiple historical moments. A particular frame voxel representation in the historical voxel representation may correspond to a three-dimensional spatial feature formed after the image acquired at a certain moment has been processed by an encoder.

[0046] As an example, a historical time period can represent a continuous time interval, such as several seconds in the past. Within this time period, server 130 can continuously acquire image frames at a fixed frequency (e.g., several frames per second), with each image frame corresponding to a specific moment within that time period. The multiple moments included in the historical time period can be understood as a collection of these image frame acquisition time points. For example, one image frame corresponds to every 0.2 seconds in the past second, for a total of 5 frames. Each frame represents a specific moment in the past (e.g., t0 = 0s, t1 = 0.2s, t2 = 0.4s, ...). Furthermore, server 130 can construct a corresponding voxel representation based on the image frames corresponding to each moment, that is, each image frame generates a 3D scene representation at the corresponding moment after 3D modeling. The voxel representation of each frame reflects the 3D spatial environment in which the target object is located at the corresponding moment. Therefore, multi-frame voxel representations can be used to describe the continuous 3D modeling results of the scene in which the target object is located within a historical time period.

[0047] Alternatively or additionally, multiple images within a historical time period may not be limited to a sequence of images acquired continuously at fixed time intervals. In other words, multiple images may include images acquired at non-equidistant intervals, such as images acquired at key moments manually set according to application requirements, or images acquired based on detected events. In this case, server 130 can also construct corresponding multi-frame voxel representations based on these non-continuous or non-equidistant images, thereby supporting subsequent scene information prediction and trajectory planning.

[0048] Return to reference Figure 2 In box 230, server 130 generates a target voxel representation of the scene after the target object operates according to one or more target poses, based on historical voxel representations, trajectory control information, and noise data for the target time period. The target voxel representation is used to generate the trajectory of the target object operating according to one or more target poses, that is, the predicted trajectory after performing behavior according to the corresponding target pose, thereby supporting tasks such as path planning and behavior simulation.

[0049] Traditional 3D scene generation methods typically lack the ability to simulate scene changes in a directional manner under arbitrary control conditions, thus failing to meet the flexibility and diversity required for complex behavior-driven decision-making. To improve prediction flexibility and accuracy, trajectory control information can be introduced as a generation control condition, which can indicate the pose of one or more targets in the future.

[0050] In some embodiments, server 130 can jointly input trajectory control information, historical voxel representations, and noise data into a scene generation model. By responding conditionally to different pose commands, it generates three-dimensional voxel representations corresponding to each target pose. In this way, while maintaining the historical consistency of scene features, it can simulate environmental changes after the target object performs different actions, realizing scene modeling corresponding to multiple possible trajectories. Server 130 can not only model the scene of future single-point behavior, but also continuously generate scene representations under multi-step target poses, thereby enhancing the control response capability of tasks such as trajectory planning and decision evaluation in complex dynamic environments.

[0051] Figure 3 A schematic diagram of an example architecture 300 for generating target scene features according to some embodiments of the present disclosure is shown. In this example, the scene generation model may include a neural network 310 (e.g., a Transformer network). However, this is merely exemplary, and the scene generation model can be implemented with any suitable network architecture. For example, the scene generation model may include an encoder and a decoder, etc. Figure 3 As shown, historical scene features 301 for a specific historical time period can be determined based on historical voxel representations. For example, the voxel representations of T frames within a historical time period can be input into the encoder for feature extraction to obtain the corresponding historical scene features 301-1 to 301-T, where T is a positive integer greater than 1. These historical scene features 301 can effectively characterize the spatiotemporal information basis of the environment in which the target object is located.

[0052] In some embodiments, intermediate scene features can be generated based on historical scene features and noise data for a target time period. For example, T frames of historical scene features 301 and N frames of noise data 302 for the target time period (e.g., noise data 302-1 to 302-N, where N is a positive integer greater than 1) are uniformly encoded, and the two are concatenated in the time dimension as the input sequence, which is then input into the neural network 310. Historical scene features 301 can provide prior information about environmental evolution, while noise data 302 can provide the starting point for the generative model. As an example, in the reasoning process of the target scene features (also known as the generation process), noise data 302 can be pure noise, such as random noise sampled from a standard normal distribution. Noise data 302 can serve as the starting point for the generative model, which gradually generates the corresponding target three-dimensional voxel representation.

[0053] Continue to refer to Figure 3The input to the neural network 310 may also include trajectory control information 303. The trajectory control information may include one or more target poses of the target object in a future time period, which may include the target object's position parameters in three-dimensional space (such as translations x and y) and attitude parameters (such as orientation angle and rotation angle). By extracting features from the target pose of each frame, the corresponding trajectory control features can be obtained. Alternatively or additionally, the specific trajectory control information may be determined based on user input. The introduction of trajectory control information enables the model to explicitly respond to the future movement intentions of the target object when generating target scene features of a movable object, thereby achieving precise control over the scene generation process.

[0054] In some embodiments, the target voxel representation for the target time period can be determined based on intermediate scene features and trajectory control features. (See reference) Figure 3 The neural network 310 can determine the target scene features 304 (such as target scene features 304-1 to 304-N) for N frames based on intermediate scene features and trajectory control information 303. As an example, Figure 4 Some embodiments according to this disclosure are shown. Figure 3 A schematic diagram of an example architecture for neural network 310 is shown. In this example, neural network 310 may include a preprocessing module 311, a multilayer perceptron (MLP) 312, a combination module 313, and basic building blocks 314, etc. However, this is only exemplary, and neural network 310 may include other model layers.

[0055] like Figure 4 As shown, the input data of the neural network 310 may include N frames of trajectory control information 303 and intermediate features 305 obtained based on T frames of historical scene features 301 and N frames of noise data 302. The preprocessing module 311 can perform a dimensionality transformation (reshape) operation on the intermediate features 305 to adapt them to the modeling structure of the neural network 310. For example, the original size of the intermediate feature 305 is (T+N)×C×H×W, where C is the number of channels, and H and W are the length and width, respectively. Through the reshape operation, the temporal dimension (T+N) and the spatial dimension H×W are fused to obtain an intermediate feature 306 with a size of ((T+N)×H×W)×C.

[0056] Continue to refer to Figure 4The trajectory control feature 306 can be obtained by performing feature extraction on the trajectory control information 303. For example, the multilayer perceptron 312 performs feature extraction on the trajectory control information 303 of size N×3 to obtain a trajectory control feature 306 of size N×C that matches the dimension of the intermediate feature 305. The trajectory control information 303 can correspond to the target voxel representation to be generated. That is, to generate N frames of target voxel representation, N target poses are input, thereby extracting N trajectory control features 306. Each trajectory control feature 306 corresponds to a frame in the generated sequence, indicating the behavioral state that the target voxel should reflect in that frame. Alternatively or additionally, the number of trajectory control information 303 (which, for example, indicates the target pose) may not correspond to the number of frames of target voxel representation to be generated. For example, the trajectory control information 303 may include only a single control command or only specify the target pose of a specific frame.

[0057] Continue to refer to Figure 4 The combination module 313 combines the intermediate scene feature 305 and the trajectory control feature 306 to obtain the combined feature 307. The basic building unit 314 can obtain the target scene feature 304 for the target time period by applying an attention mechanism to the combined input feature and trajectory control feature (i.e., the combined feature 307).

[0058] As an example, the basic building block 314 may include a multi-head self-attention module, a layer normalization module, a feedforward neural network, etc., but this is only exemplary, and the basic building block 314 may include other networks or modules. The multi-head self-attention module enables global information interaction between features, and the layer normalization module can accelerate the convergence speed. The feedforward neural network can further learn complex nonlinear features. This structure can make full use of historical scene feature information and trajectory control information to generate a target scene representation. For example, the neural network 310 can output target scene features with a shape of N×C×H×W. Furthermore, the decoder can be used to recover the target scene features 304 into a complete 3D voxel grid, thereby obtaining the target voxel representation for the corresponding target time period (N frames).

[0059] It should be noted that the scale and number of the various features mentioned above, as well as the specific composition of the modules and network structures used for feature extraction, attention control, and feature fusion in the scene generation model, are merely examples. The embodiments disclosed herein can be flexibly adjusted according to specific application requirements, data scale, model accuracy requirements, and other factors. For example, the scene generation model may include more or fewer feature extraction modules, feedforward neural networks, ensemble modules, encoders, decoders, convolutional layers, etc., or be replaced with other modules or network structures with equivalent functions, thereby optimizing scene generation accuracy, computational efficiency, model capacity, and other aspects.

[0060] In some embodiments, the target voxel representation is generated using a flow-based scene generation model. Such a model can achieve high-quality generation of future 3D scenes by guiding noise data 302 to evolve gradually towards target scene features 304 along a specific path. For example, first, 3D voxel representations of historical time periods and noise data corresponding to the target time period are input, and combined with the pose control information of the target object, a time-dependent vector field, i.e., a flow field, is learned using the generative model. This flow field indicates the change of scene information from a noise distribution to a target distribution. That is, under specific conditions, how the 3D voxel features gradually approach the semantic distribution of the target scene from a randomly initialized noise state.

[0061] In some embodiments, the aforementioned process of generating target voxel representations based on historical voxel representations and trajectory control information can be based on a scene generation model architecture. This process can be performed during the inference phase. During inference, the scene generation model continuously updates noise features at multiple time steps based on the learned flow field, causing them to evolve along the flow field direction, thereby obtaining a target voxel representation corresponding to the target pose. This mechanism enables the model to generate structurally complete and controllable 3D scene representations from unstructured random noise input, thereby improving the accuracy and control responsiveness of voxel representation generation. Alternatively or additionally, this process can be performed during the training phase of the scene generation model. Some embodiments are described below for the training of the scene generation model.

[0062] Figure 5 A schematic diagram of an example architecture 500 for training a scene generation model according to some embodiments of the present disclosure is shown. Figure 5 As shown, the scene generation model may include a neural network (such as a Transformer network) 310, an encoder 320, and a decoder 330. However, this is only an example, and the scene generation model may include other model layers.

[0063] like Figure 5As shown, the training process of the scene generation model can include two stages. The goal of the first stage is to train a 3D representation learning module with an encoder-decoder structure. The core function of this module is to map the original 3D voxel representations into a compact, structured latent discrete space so that the subsequent generation model can model efficiently. In this stage, the encoder 320 is used to compress the voxel representations 511 into discrete representations (e.g., tokens), or scene features 512. The decoder 330 is then used to recover the corresponding voxel representations 513 from the scene features 512. Through this representation learning process, the modeling dimensionality and computational complexity of the subsequent model can be significantly reduced, while also improving the expressiveness of the data in spatial structure and the quality of the generated reconstruction, laying the foundation for the scene generation in the second stage.

[0064] In some embodiments, the two stages of the scene generation model training process can be decoupled. For example, the first stage of the training process is a pre-training process. The second stage of the scene generation model training process can directly utilize the pre-training results of the first stage.

[0065] Continue to refer to Figure 5 In the second phase of the training process, server 130 can utilize the scene generation model to generate a target voxel representation 504 of the scene after the target object has performed one or more target pose operations as indicated by the trajectory control information 503, based on historical scene features 501, noise data 502, and trajectory control information 503. Alternatively or additionally, historical scene features 501 can be obtained by encoding the historical voxel representation using encoder 320 trained in the first phase.

[0066] In some embodiments, noise data 502 refers to the intermediate feature representation obtained by adding noise to the ground truth voxel representation of the target time period. Server 130 can obtain ground truth training samples for the target time period, that is, the three-dimensional voxel representation of the scene where the target object is located after performing one or more target pose operations indicated by trajectory control information 503. The three-dimensional voxel representation may include the ground truth voxel representation of the target time period (N frames).

[0067] Furthermore, the server 130 can invoke the pre-trained encoder 320 to encode the aforementioned ground truth voxel representation, extract its spatial structure and semantic features, and obtain the ground truth scene feature 505 for the corresponding target time period. This feature is used to represent the ideal expression of the scene in which the target object is located under known pose control conditions, serving as the "target state" during the training phase.

[0068] In some embodiments, to simulate the transformation from noise to a target distribution, noise data with different noise intensities needs to be constructed during training. Noise data 502 can be obtained by progressively adding noise to the ground truth scene feature 505 at multiple noise-adding time steps t∈[0,1]. This noise-adding process can be represented by the following formula:

[0069]

[0070] Where x0 is the true scenario feature, ∈ t For noise with a standard normal distribution, α t This represents the noise intensity parameter that varies over time.

[0071] The resulting xt is the noise data 502 during the training process, used to simulate the process of gradually generating the target voxel representation from a randomly initialized state during model inference. The noise data 502 will be used as input to supervise the training of the generative model to learn the flow field path from the noisy state to the ground truth state, thereby achieving the mapping ability from the noisy state to the target state.

[0072] In some embodiments, target scene features 504 can be generated by progressively reducing noise in noise data 502 in multiple denoising time steps, wherein the denoising time steps in the multiple denoising time steps are configured to process the noise scene features corresponding to the noise data 502.

[0073] As an example, the core of the training process lies in teaching the scene generation model how to gradually reconstruct the target feature x0 from noisy scene features xt. To achieve this, in each denoising time step, noisy data xt and the corresponding time step t can be input into the neural network 510 to obtain a flow field f(x). t ,t), used to indicate how the noise changes to approximate the target feature. Each denoising time step t in this process can be matched one-to-one with a denoising time step t, the purpose being to learn how to implement the reverse process of the denoising operation at each time step. From the noisy feature x t Recover the low-noise feature x t-1 The denoising process can be represented by the following formula:

[0074] x t-1 =x t -ηf(x t ,t) (2)

[0075] Where x t The noise data at time step t (corresponding to the current noise scene characteristics), f(x) t,t) is the flow field learned by the model, which indicates the "denoising direction" or "evolution direction" of the current noisy scene features. η is the learning rate parameter, used to control the update magnitude of each denoising step, i.e., from x t To x t-1 The speed of evolution.

[0076] In some embodiments, server 130 can update the scene generation model for multiple denoising time steps, based on the difference between the flow field and the reference flow. In each denoising time step, the scene generation model generates a flow field based on the current noise features (representing how the current features should change to progressively denoise). To guide the model in learning the correct denoising direction, it can be trained based on the reference flow.

[0077] The reference flow refers to the ideal velocity direction or reference trend of the noise scene features changing towards the real data at a given time step. The training objective of the model is to make the obtained flow field as close as possible to the reference flow. Therefore, the difference between the two can be used to construct the loss function. By continuously optimizing this difference across all denoising time steps, the model can eventually learn a stable and effective generation strategy, thereby accurately restoring the noisy data to the target voxel representation.

[0078] In some embodiments, the reference stream includes a reference rate of change of noisy scene features to target scene features. In each of the multiple denoising time steps, server 130 can utilize the scene generation model to determine a rate vector of change of the noisy scene features to the target scene features at that denoising time step. Then, server 130 can construct a loss function based on the difference between the rate vector and the reference rate of change. Further, server 130 can update the scene generation model based on the loss function.

[0079] As an example, the reference flow can be understood as a desired rate of change vector, describing the direction and magnitude in which the noise scene features at the current time step t should change to approximate the ground truth scene features. The reference rate of change can be approximated by the gradient information of the noise scene features at that denoising time step t. It reflects the "guiding direction" of the noise feature distribution in the current state, for example, through x... t The gradient of the log probability at time step t express.

[0080] As another example, the loss function built based on the difference between the flow field and the reference flow can be expressed by the following formula:

[0081]

[0082] The loss function value used to optimize the model is obtained by calculating the average expected value of the difference between the flow field predicted by the scene generation model and the reference flow at multiple denoising time steps. This loss function value measures the deviation between the feature change direction predicted by the scene generation model under different noise intensities and the ideal denoising direction. Based on this, the server 130 can use the backpropagation algorithm to update the model parameters based on the loss function. For example, the parameters of the scene generation model can be updated by iteratively solving the problem using gradient descent or Euler methods, so that the model can learn a more accurate and stable feature change direction at different denoising time steps. This mechanism enables the trained scene generation model to recover the target voxel representation under various noise conditions and behavior control conditions, improving the overall generation quality and control response performance.

[0083] As can be more clearly understood from the description of the various embodiments of this disclosure, in one aspect, a trajectory control mechanism is introduced, in which the target pose of the movable object is input as conditional information into the scene generation model. In this way, the scene generation model possesses higher behavioral control flexibility and generation diversity, enabling it to respond to trajectory control commands with higher precision and achieve 3D scene modeling under multi-path and multi-strategy conditions. On the other hand, a flow matching mechanism is introduced, enabling the scene generation model to gradually generate a high-quality 3D voxel representation corresponding to the target pose from a noisy initial state under different behavioral control conditions. In this way, the scene generation model can not only complete the effective evolution from a random state to the target state more quickly, but also better maintain the stability and continuity of the generation process, thereby significantly improving the accuracy, controllability, and generalization ability of 3D scene generation.

[0084] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6 A schematic structural block diagram of a scene information generation apparatus 600 according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the server 130. The various modules / components in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0085] refer to Figure 6As shown, the device 600 includes an acquisition module 610 configured to acquire multiple images of the scene where the movable target object is located during a historical time period and trajectory control information of the target object, the trajectory control information indicating one or more target poses of the target object during a target time period; a historical voxel representation determination module 620 configured to determine the historical voxel representation of the scene during the historical time period based on multiple images; and a target voxel representation generation module 630 configured to generate a target voxel representation of the scene after the target object operates according to one or more target poses based on the historical voxel representation, trajectory control information, and noise data for the target time period, the target voxel representation being used to generate the trajectory of the target object operating according to one or more target poses.

[0086] In some embodiments, the target voxel representation generation module 630 is further configured to utilize an encoder to determine historical scene features for a historical time period based on historical voxel representations; generate intermediate scene features based on historical scene features and noise data for a target time period; obtain trajectory control features by feature extraction of trajectory control information; and determine the target voxel representation based on intermediate scene features and trajectory control features.

[0087] In some embodiments, the target voxel representation generation module 630 is further configured to combine intermediate scene features and trajectory control features; and to obtain target scene features for a target time period by applying an attention mechanism to the combined input features and trajectory control features; and to determine the target voxel representation based on the target scene features using a decoder.

[0088] In some embodiments, the target voxel representation is generated using a flow-field-based scene generation model, where the flow field indicates the change in scene information from a noise distribution to a target distribution.

[0089] In some embodiments, the apparatus 600 further includes a training module, and the noise data is obtained by: using an encoder to generate ground truth scene features for a target time period based on a ground truth voxel representation for the target time period; obtaining noise data by progressively adding noise to the ground truth scene features in multiple noise-adding time steps, and obtaining target scene features corresponding to the target voxel representation by: generating target scene features by progressively reducing noise in the noise data in multiple denoising time steps, wherein the denoising time steps in the multiple denoising time steps are configured to process the corresponding noise scene features, and the training module is configured to be based on the target voxel representation.

[0090] In some embodiments, the reference flow includes a reference rate of change of noisy scene features to target scene features. The training module is further configured to, at each of the multiple denoising time steps, utilize the scene generation model to determine a rate vector of change of noisy scene features to target scene features at that denoising time step; construct a loss function based on the difference between the rate vector and the reference rate of change, where the reference rate of change includes the gradient information of the noisy scene features at that denoising time step; and update the scene generation model based on the loss function.

[0091] In some embodiments, the time control information is determined based on user input.

[0092] Figure 7 A block diagram is shown of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. The electronic device 700 may, for example, be used to implement... Figure 1 The control device 120 shown. It should be understood that, Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0093] refer to Figure 7 The electronic device 700 is in the form of a general-purpose electronic device. Components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0094] Electronic device 700 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 700.

[0095] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0096] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0097] Input device 760 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0098] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0099] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0101] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0103] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is determined to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating scene information, comprising: Acquire multiple images of the scene where the movable target object is located during a historical time period and trajectory control information of the target object, wherein the trajectory control information indicates one or more target poses of the target object during the target time period; Based on the multiple images, determine the historical voxel representation of the scene in the historical time period; as well as Based on the historical voxel representation, the trajectory control information, and noise data for the target time period, a target voxel representation of the scene is generated after the target object operates according to the one or more target poses. The target voxel representation is used to generate the trajectory of the target object operating according to the one or more target poses.

2. The method of claim 1, wherein generating a target voxel representation of the scene after the target object has been manipulated according to the one or more target poses comprises: Using the encoder, based on the historical voxel representation, determine the historical scene features for the historical time period; Based on the historical scene features and the noise data for the target time period, intermediate scene features are generated; Trajectory control features are obtained by feature extraction from the trajectory control information; as well as The target voxel representation is determined based on the intermediate scene features and the trajectory control features.

3. The method of claim 2, wherein determining the target voxel representation comprises: Combine the intermediate scene features and the trajectory control features; as well as By applying an attention mechanism to the combined intermediate scene features and trajectory control features, target scene features for the target time period are obtained. as well as The target voxel representation is determined using a decoder based on the target scene features.

4. The method of claim 1, wherein the target voxel representation is generated using a flow field-based scene generation model, the flow field indicating the change of scene information from a noise distribution to a target distribution.

5. The method of claim 4, wherein the method is performed during the training of the scene generation model, and the noise data is obtained in the following manner: Using an encoder, based on the ground truth voxel representation for the target time period, ground truth scene features for the target time period are generated; The noise data is obtained by progressively adding noise to the ground truth scene features at multiple noise-adding time steps, and The target scene features corresponding to the target voxel representation are obtained in the following manner: The target scene features are generated by progressively reducing noise in the noise data in multiple denoising time steps, wherein each of the multiple denoising time steps is configured to process the corresponding noise scene features.

6. The method according to claim 5, wherein the scene generation model is trained in the following manner: For each of the multiple denoising time steps, the scene generation model is updated based on the difference between the flow field and the reference flow.

7. The method of claim 6, wherein the reference stream includes a reference rate of change of the noisy scene features to the target scene features, and wherein updating the scene generation model comprises: In each of the multiple denoising time steps, the scene generation model is used to determine the velocity vector of the change of the noisy scene features toward the target scene features in that denoising time step; Based on the difference between the velocity vector and the reference change velocity, a loss function is constructed, wherein the reference change velocity includes the gradient information of the noise scene features at the denoising time step; as well as The scene generation model is updated based on the loss function.

8. The method according to claim 1, wherein the trajectory control information is determined based on user input.

9. A scene information generation device, comprising: The acquisition module is configured to acquire multiple images of the scene where the movable target object is located during a historical time period and trajectory control information of the target object, wherein the trajectory control information indicates one or more target poses of the target object during the target time period. The historical voxel representation determination module is configured to determine the historical voxel representation of the scene in the historical time period based on the plurality of images; as well as The target voxel representation generation module is configured to generate a target voxel representation of the scene after the target object operates according to the one or more target poses, based on the historical voxel representation, the trajectory control information, and noise data for the target time period. The target voxel representation is used to generate the trajectory of the target object operating according to the one or more target poses.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images