Image data generation method, mobile device, electronic device and medium

Abstract

The invention provides an image data generation method, a mobile device, an electronic device and a medium, and the method comprises the steps: obtaining a historical image feature sequence which comprises a plurality of image features corresponding to a plurality of historical moments; a noise image feature sequence is obtained, the noise image feature sequence is located behind the historical image feature sequence in the time sequence, and the dimension of image features in the noise image feature sequence is consistent with the dimension of image features in the historical image feature sequence; the historical image feature sequence and the noise image feature sequence are input into an image feature prediction model, a predicted image feature sequence is obtained, and each image feature in the predicted image feature sequence is used for representing image information in a three-dimensional space; and generating image data based on the predicted image feature sequence. According to the technical scheme, the quality of the generated image data can be improved.

Description

Technical Field

[0001] The present application relates to the field of robot vision technology, and in particular to an image data generation method, a mobile device, an electronic device and a medium. Background Art

[0002] With the continuous advancement of science and technology, people's lives and work are gradually developing in the direction of intelligence. For example, for a target task, the scene image at the current moment of the target task can be used to predict the scene image at the future moment. In this way, the execution process of the target task can be planned based on the scene image at the future moment. However, the scene images at the future moment predicted by the current prediction method have the problem of poor quality. Summary of the invention

[0003] In view of this, embodiments of the present application provide an image data generating method, a mobile device, an electronic device, and a medium, which can improve the quality of generated image data.

[0004] In a first aspect, an embodiment of the present application provides an image data generation method, comprising: obtaining a historical image feature sequence, wherein the historical image feature sequence includes multiple image features corresponding to multiple historical moments; obtaining a noise image feature sequence, wherein the noise image feature sequence is located after the historical image feature sequence in time order, and the dimension of the image features in the noise image feature sequence is consistent with the dimension of the image features in the historical image feature sequence; inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model to obtain a predicted image feature sequence, wherein each image feature in the predicted image feature sequence is used to characterize image information in three-dimensional space; and generating image data based on the predicted image feature sequence.

[0005] In a second aspect, an embodiment of the present application provides a model training method, comprising: selecting multiple image features from an initial image feature sequence as a sample historical image feature sequence; obtaining a sample noise image feature sequence, wherein the sample noise image feature sequence is located after the sample historical image feature sequence in time order, and the dimension of the image features in the sample noise image feature sequence is consistent with the dimension of the image features in the sample historical image feature sequence; inputting the sample historical image feature sequence and the sample noise image feature sequence into an image feature prediction model to obtain a predicted sample image feature sequence, wherein each image feature in the predicted sample image feature sequence is used to characterize image information in three-dimensional space; training the image feature prediction model based on the difference between the predicted sample image feature sequence and the target image feature sequence in the initial image feature sequence to obtain a trained image feature prediction model, wherein the target image feature sequence is consistent with the predicted sample image feature sequence in the time dimension.

[0006] In a third aspect, an embodiment of the present application provides an image data generating device, comprising: an acquisition module, used to acquire a historical image feature sequence, wherein the historical image feature sequence includes multiple image features corresponding to multiple historical moments; the acquisition module, also used to acquire a noise image feature sequence, wherein the noise image feature sequence is located after the historical image feature sequence in time order, and the dimension of the image features in the noise image feature sequence is consistent with the dimension of the image features in the historical image feature sequence; a prediction module, used to input the historical image feature sequence and the noise image feature sequence into an image feature prediction model to obtain a predicted image feature sequence, wherein each image feature in the predicted image feature sequence is used to characterize image information in three-dimensional space; a generation module, used to generate image data based on the predicted image feature sequence.

[0007] In a fourth aspect, an embodiment of the present application provides a model training device, comprising: a selection module, used to select multiple image features from an initial image feature sequence as a sample historical image feature sequence; an acquisition module, used to acquire a sample noise image feature sequence, wherein the sample noise image feature sequence is located after the sample historical image feature sequence in time order, and the dimension of the image features in the sample noise image feature sequence is consistent with the dimension of the image features in the sample historical image feature sequence; a prediction module, used to input the sample historical image feature sequence and the sample noise image feature sequence into an image feature prediction model to obtain a predicted sample image feature sequence, wherein each image feature in the predicted sample image feature sequence is used to characterize image information in three-dimensional space; a training module, used to train the image feature prediction model based on the difference between the predicted sample image feature sequence and the target image feature sequence in the initial image feature sequence to obtain a trained image feature prediction model, wherein the target image feature sequence is consistent with the predicted sample image feature sequence in the time dimension.

[0008] In a fifth aspect, an embodiment of the present application provides a mobile device, comprising a control module, the control module being used to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0009] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; and a memory for storing processor executable instructions, wherein the processor is used to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0010] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0011] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor of a computer device, the computer device is enabled to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0012] In the ninth aspect, an embodiment of the present application provides a chip, comprising: a processor; and a memory for storing processor executable instructions, wherein the processor is used to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0013] The embodiment of the present application provides an image data generation method, a mobile device, an electronic device and a medium. By acquiring a historical image feature sequence and a noise image feature sequence, and inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model, a predicted image feature sequence can be obtained, and then image data can be generated based on the predicted image feature sequence. In the embodiment of the present application, the noise image feature sequence is located after the historical image feature sequence in time order, the dimension of the image feature in the noise image feature sequence is consistent with the dimension of the image feature in the historical image feature sequence, and the historical image feature sequence includes multiple image features corresponding to multiple historical moments. The image feature prediction model can process the noise image feature sequence based on the historical image feature sequence, so that the state of things represented by the predicted image feature sequence conforms to the law of development of things. Therefore, the quality of the image data generated based on the predicted image feature sequence is high, which can improve the accuracy of the decision or planning made by the robot based on the image data, and thus improve the completion effect of the task. Furthermore, since each image feature in the predicted image feature sequence is used to characterize image information in three-dimensional space, the predicted image feature sequence can be presented according to the required viewing angle, thereby further improving the quality of the final image or video, for example, avoiding the situation where objects in the image or video are occluded or deformed as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Shown is a schematic diagram of the system architecture of an image data generation system provided by an exemplary embodiment of the present application.

[0015] Figure 2 Shown is a flowchart of an image data generating method provided by an exemplary embodiment of the present application.

[0016] Figure 3 Shown is a flowchart of an image data generating method provided by another exemplary embodiment of the present application.

[0017] Figure 4Shown is a schematic diagram of an image data generation process provided by an exemplary embodiment of the present application.

[0018] Figure 5 Shown is a flowchart of a model training method provided by an exemplary embodiment of the present application.

[0019] Figure 6 Shown is a schematic diagram of the training phase and the inference phase of a model provided by an exemplary embodiment of the present application.

[0020] Figure 7 Shown is a schematic diagram of the structure of an image data generating device provided by an exemplary embodiment of the present application.

[0021] Figure 8 Shown is a schematic structural diagram of a model training device provided by an exemplary embodiment of the present application.

[0022] Fig. 9 Shown is a block diagram of an electronic device for executing an image data generating method or a model training method provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] Application Overview

[0025] With the development of intelligent technology, more and more tasks can be performed by robots. For example, some robots can complete tasks independently, while others can interact with the environment and collaborate with other robots to complete tasks. In the field of robotics, a robot world model can refer to the internal representation that a robot uses to understand and predict its surroundings. This model can help robots make decisions and plans when performing tasks, and can also be used to build realistic robot simulation environments for closed-loop evaluation and reinforcement learning.

[0026] The current robot world model can understand the physical world that the robot encounters and generate high-fidelity images or videos. However, the images or videos generated by the current robot world model have poor quality problems. For example, the objects in the image or video may not maintain the shape and color of the original objects in the real world. Especially when the original objects are blocked or presented at an undesirable angle, the appearance of the objects in the image or video may be deformed, and sometimes the objects may even disappear completely. Such poor quality images or videos may affect the robot's decision-making or planning process, and thus affect the completion of the task.

[0027] In view of the above technical problems, an embodiment of the present application provides an image data generation method, by obtaining a historical image feature sequence and a noise image feature sequence, and inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model, a predicted image feature sequence can be obtained, and then image data can be generated based on the predicted image feature sequence. In an embodiment of the present application, the noise image feature sequence is located after the historical image feature sequence in time order, the dimension of the image feature in the noise image feature sequence is consistent with the dimension of the image feature in the historical image feature sequence, and the historical image feature sequence includes multiple image features corresponding to multiple historical moments, respectively. The image feature prediction model can process the noise image feature sequence based on the historical image feature sequence, so that the state of things represented by the predicted image feature sequence conforms to the law of development of things. Therefore, the quality of the image data generated based on the predicted image feature sequence is high, so that the accuracy of the decision or planning made by the robot based on the image data can be improved, and then the completion effect of the task can be improved. Here, the image data can be an image or a video, or point cloud data, and the point cloud data can be rendered to obtain an image or a video. Furthermore, since each image feature in the predicted image feature sequence is used to characterize image information in three-dimensional space, the predicted image feature sequence can be presented according to the required viewing angle, thereby further improving the quality of the final image or video, for example, avoiding the situation where objects in the image or video are occluded or deformed as much as possible.

[0028] Exemplary Systems

[0029] Figure 1 FIG. 1 is a schematic diagram of a system architecture of an image data generation system provided by an exemplary embodiment of the present application. Figure 1 As shown, the image data generation system 100 may include a computer device 110 and a task execution device 120. The computer device 110 may be a server, or a terminal device, etc. The terminal device may be a mobile phone, a notebook, etc. The task execution device 120 may be a device for executing a task, such as a vehicle, a robot, etc.

[0030] The task execution device 120 may be equipped with a sensor 121, which may be used to collect images around the task execution device 120. In an example, the task execution device 120 may send images of multiple historical moments acquired by the sensor 121 to the computer device 110. In some cases, each historical moment may correspond to multiple images, and the multiple images may be acquired at multiple viewing angles. The computer device 110 may obtain a historical image feature sequence based on the images of multiple historical moments, generate a noise image feature sequence according to the dimensions of the image features in the historical image feature sequence, and input the historical image feature sequence and the noise image feature sequence into an image feature prediction model, so as to process the noise image feature sequence based on the historical image feature sequence using the image feature prediction model to obtain a predicted image feature sequence. The noise image feature sequence is located after the historical image feature sequence in time order, and the dimensions of the image features in the noise image feature sequence are consistent with the dimensions of the image features in the historical image feature sequence. Each image feature in the predicted image feature sequence is used to characterize image information in three-dimensional space. Further, the computer device 110 may generate image data based on the predicted image feature sequence, and the image data may be an image or a video. The computer device 110 may send the image data to the task execution device 120, and the task execution device 120 may make a decision or plan based on the image data, such as adjusting the task execution strategy to improve the task completion effect.

[0031] For example, the task execution device 120 is a vehicle, and the task to be performed by the task execution device 120 is to drive to a target location. The task execution device 120 can adjust the driving route between the current position point and the target location according to the image data; or, the task execution device 120 is a dual-arm robot, and the task to be performed by the task execution device 120 is to grab a target object. The task execution device 120 can adjust the joint angles of the robot arm according to the image data to grab the target object.

[0032] In another example, the computer device 110 may generate image data based on the predicted image feature sequence, make decisions or plans based on the image data, and then send the decision results or planning results to the task execution device 120, and the task execution device 120 may execute the task based on the decision results or planning results.

[0033] In another example, the task execution device 120 can directly obtain a historical image feature sequence based on images of multiple historical moments, generate a noise image feature sequence based on the dimensions of the image features in the historical image feature sequence, and input the historical image feature sequence and the noise image feature sequence into an image feature prediction model, so as to use the image feature prediction model to process the noise image feature sequence based on the historical image feature sequence to obtain a predicted image feature sequence. Further, the task execution device 120 can generate image data based on the predicted image feature sequence, make a decision or plan based on the image data, and execute the task based on the decision result or the planning result.

[0034] Exemplarily, the sensor 121 may be at least one of a lidar sensor, an ultrasonic sensor, a visual sensor, and the like.

[0035] It should be understood that the above application scenario examples are only shown to facilitate understanding of the spirit and principle of the present application, and the embodiments of the present application are not limited thereto. On the contrary, the embodiments of the present application can be applied to any scenario that may be applicable.

[0036] Exemplary Methods

[0037] Figure 2 Shown is a flowchart of an image data generating method provided by an exemplary embodiment of the present application. Figure 2 The method can be Figure 1 The computer device 110 or the task execution device 120 in the embodiment is executed. For the convenience of description, the following description is given by taking the computer device 110 executing the method as an example. Figure 2 As shown, the image data generating method may include the following contents.

[0038] 210: Obtain historical image feature sequence.

[0039] Specifically, the historical image feature sequence includes multiple image features corresponding to multiple historical moments. The multiple historical moments may include multiple moments before the current moment, or include moments before the current moment and the current moment.

[0040] In one example, each of the multiple historical moments may correspond to an image feature, and the image feature may represent the image information corresponding to an image collected at one viewing angle, or represent the image information corresponding to multiple images collected at multiple viewing angles. The image here may be an image obtained by collecting environmental information around the task execution device 120 using a sensor. For example, for a historical moment, multiple images at the historical moment may be collected, each of the multiple images may correspond to a viewing angle, and the image information corresponding to the multiple images may be called RGB information or three-dimensional visual information, and an image feature may be obtained based on the multiple images at the historical moment. The image features corresponding to multiple historical moments may constitute a historical image feature sequence.

[0041] In one example, each image feature may be a visual latent code. Specifically, a multi-view 3D reconstruction method may be used to reconstruct multiple images at each historical moment into 3D point cloud data, such as 3D Gaussian point cloud data, or a static 3D Gaussian sphere in the form of a point cloud. Furthermore, a visual encoder may be used to encode the reconstructed 3D point cloud data into a visual latent code, i.e., to obtain image features. The visual encoder may be an encoder based on a variational autoencoder (VAE) architecture or other suitable encoders.

[0042] In one example, when the initial moment is the current moment, the image features corresponding to the current moment may be repeatedly stacked to obtain multiple image features, that is, to obtain a historical image feature sequence.

[0043] 220: Obtain a noise image feature sequence.

[0044] Specifically, the noise image feature sequence is located after the historical image feature sequence in time order, and the dimension of the image features in the noise image feature sequence is consistent with the dimension of the image features in the historical image feature sequence.

[0045] In one example, a noise image feature sequence may be generated according to dimensions of image features in a historical image feature sequence.

[0046] In one example, the specific parameters of each image feature in the noise image feature sequence can be set according to actual needs, for example, according to at least one of the factors such as the actual task scenario, the change law of multiple image features in the historical image feature sequence over time, etc. Optionally, in other examples, the specific parameters of each image feature in the noise image feature sequence can all be set to an initial value of 0.

[0047] In one example, the first intervals between the multiple historical moments corresponding to the historical image feature sequence may be equal, and the interval between the moments corresponding to the first image feature in the noise image feature sequence and the last image feature in the historical image feature sequence may be equal to or unequal to the first interval. Alternatively, in other examples, the first intervals between the multiple historical moments corresponding to the historical image feature sequence may be equal or unequal, the second intervals between the moments corresponding to the respective image features in the noise image feature sequence may be equal or unequal, and the first interval may be equal to or unequal to the second interval.

[0048] 230: Input the historical image feature sequence and the noise image feature sequence into the image feature prediction model to obtain a predicted image feature sequence.

[0049] In one example, an image feature prediction model can be used to predict an image feature sequence at a future moment based on an existing image feature sequence. Specifically, the input of the image feature prediction model may include a historical image feature sequence and a noise image feature sequence. The image feature prediction model may process the noise image feature sequence based on the historical image feature sequence, such as denoising or other processing, to obtain a predicted image feature sequence. The predicted image feature sequence may also be referred to as a denoised image feature sequence. The historical image feature sequence and the noise image feature sequence may be input into the image feature prediction model as two independent sequences, or the historical image feature sequence and the noise image feature sequence may be spliced ​​into one sequence and the spliced ​​sequence may be input into the image feature prediction model.

[0050] Specifically, each image feature in the predicted image feature sequence is used to represent image information in three-dimensional space.

[0051] In one example, each image feature in the predicted image feature sequence can represent image information from multiple perspectives, and the image information from multiple perspectives can represent environmental information around the task execution device 120 in the task scene at a future moment, such as the position and shape of objects around the task execution device 120 from multiple perspectives.

[0052] 240: Generate image data based on the predicted image feature sequence.

[0053] Specifically, the image data may be an image or a video; or the image data may be point cloud data, and the point cloud data may be rendered to obtain an image or a video.

[0054] In one example, the predicted image feature sequence may be a visual latent code, and a visual decoder may be used to decode the visual latent code into three-dimensional point cloud data, such as three-dimensional Gaussian point cloud data. The three-dimensional point cloud data may be rendered to obtain an image or video. The visual decoder may be a visual decoder based on a VAE architecture or other suitable decoders.

[0055] In one example, the three-dimensional point cloud data can be rendered to any preset virtual camera using Gaussian splatting or other rendering methods to obtain the picture under the camera, that is, an image or video.

[0056] In one example, any image feature sequence among the historical image feature sequence, the noise image feature sequence, and the predicted image feature sequence may be a visual image feature sequence.

[0057] The embodiment of the present application provides an image data generation method, by obtaining a historical image feature sequence and a noise image feature sequence, and inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model, a predicted image feature sequence can be obtained, and then image data can be generated based on the predicted image feature sequence. In the embodiment of the present application, the noise image feature sequence is located after the historical image feature sequence in time order, the dimension of the image features in the noise image feature sequence is consistent with the dimension of the image features in the historical image feature sequence, and the historical image feature sequence includes multiple image features corresponding to multiple historical moments, respectively. The image feature prediction model can process the noise image feature sequence based on the historical image feature sequence, so that the state of things represented by the predicted image feature sequence conforms to the law of development of things. Therefore, the quality of the image data generated based on the predicted image feature sequence is high, which can improve the accuracy of the decision or planning made by the robot based on the image data, and thus improve the completion effect of the task. Furthermore, since each image feature in the predicted image feature sequence is used to characterize image information in three-dimensional space, the predicted image feature sequence can be presented according to the required viewing angle, thereby further improving the quality of the final image or video, for example, avoiding the situation where objects in the image or video are occluded or deformed as much as possible.

[0058] According to an embodiment of the present application, the image data generation method further includes: updating the historical image feature sequence using at least one image feature in the predicted image feature sequence to obtain an updated historical image feature sequence; using the updated historical image feature sequence as the historical image feature sequence, and repeating step 230.

[0059] Specifically, the predicted image feature sequence may include one or more image features, and at least one image feature in the predicted image feature sequence may be used to update the historical image feature sequence. For example, at least one image feature in the predicted image feature sequence may be added after the original historical image feature sequence, and N image features may be removed from the original historical image feature sequence, so that an updated historical image feature sequence may be obtained. Here, the number of the N image features may be equal to the number of at least one image feature in the predicted image feature sequence, so that the dimension of the updated historical image feature sequence is consistent with the dimension of the original historical image feature sequence, and thus, the updated historical image feature sequence may be further used as the input of the image feature prediction model.

[0060] Furthermore, in one example, the updated historical image feature sequence and the noise image feature sequence can be input into the image feature prediction model, and the image feature prediction model is used to process the noise image feature sequence based on the updated historical image feature sequence to obtain a new predicted image feature sequence. The new predicted image feature sequence can be located after the updated historical image feature sequence in time order. In this input of the image feature prediction model, the noise image feature sequence can be the same as the noise image feature sequence in the first input of the image feature prediction model.

[0061] In another example, step 220 and step 230 may be repeated. Specifically, a new noise image feature sequence may be generated for the updated historical image feature sequence. The dimension of the new noise image feature sequence is consistent with the dimension of the updated historical image feature sequence. The specific parameters of each image feature in the new noise image feature sequence may be related to the change pattern of multiple image features in the updated historical image feature sequence over time or other factors, and may be set according to actual needs. The updated historical image feature sequence and the new noise image feature sequence are input into the image feature prediction model, and the new noise image feature sequence is processed based on the updated historical image feature sequence using the image feature prediction model to obtain a new predicted image feature sequence.

[0062] Similarly, the current historical image feature sequence can be continuously updated using at least one image feature in the current predicted image feature sequence to obtain an updated historical image feature sequence, and the updated historical image feature sequence can be used as the historical image feature sequence to repeat step 230, or to repeat steps 220 and 230, so as to obtain the predicted image feature sequence at the required future moment.

[0063] In this embodiment, the historical image feature sequence is updated by using at least one image feature in the predicted image feature sequence to obtain an updated historical image feature sequence, and the predicted image feature sequence at subsequent moments can be obtained based on the updated historical image feature sequence. In this way, the updating process of the historical image feature sequence and the prediction process of the image feature prediction model can be repeated according to the moment when the image features actually need to be predicted, thereby improving the prediction efficiency.

[0064] According to an embodiment of the present application, a historical image feature sequence is updated using at least one image feature in a predicted image feature sequence to obtain an updated historical image feature sequence, including: updating the historical image feature sequence using the last image feature in the predicted image feature sequence to obtain an updated historical image feature sequence.

[0065] In one example, the predicted image feature sequence may include multiple image features, and the last image feature in the predicted image feature sequence is added after the original historical image feature sequence, and an image feature is removed from the original historical image feature sequence, so that an updated historical image feature sequence can be obtained. The image feature removed from the original historical image feature sequence can be an image feature located at the beginning, end, or middle of the original historical image feature sequence.

[0066] In this embodiment, by using the last image feature in the predicted image feature sequence to update the historical image feature sequence instead of using all the image features in the predicted image feature sequence to update the historical image feature sequence, the computational complexity of the prediction process can be reduced and the temporal sparsity of the historical image feature sequence can be improved, which can improve the quality of the predicted image data.

[0067] According to an embodiment of the present application, a historical image feature sequence is updated using at least one image feature in a predicted image feature sequence to obtain an updated historical image feature sequence, including: according to a first-in-first-out rule, the historical image feature sequence is updated using the last image feature in the predicted image feature sequence to obtain an updated historical image feature sequence.

[0068] In one example, when updating the original historical image feature sequence, N image features may be removed from the original historical image feature sequence, and the N image features may be located at any position in the original historical image feature sequence, such as at the beginning, end, or middle of the original historical image feature sequence. The N image features may be continuous or discontinuous in time.

[0069] For example, according to the first-in-first-out rule, the last image feature in the predicted image feature sequence can be added after the original historical image feature sequence, and the first image feature in the original historical image feature sequence can be removed, so as to obtain an updated historical image feature sequence.

[0070] In this embodiment, the historical image feature sequence is updated according to the first-in-first-out rule, which can improve the timeliness of the historical image feature sequence, and thus improve the quality of the predicted image data. In addition, the historical image feature sequence is updated using the last image feature in the predicted image feature sequence, which can reduce the amount of calculation in the prediction process and improve the sparsity of the historical image feature sequence in time, which can further improve the quality of the predicted image data.

[0071] According to an embodiment of the present application, each image feature in the historical image feature sequence includes: features corresponding to the image at each viewing angle among multiple viewing angles; and directional features corresponding to the viewing angle.

[0072] In one example, the viewing angle may represent the angle of observing objects around the task execution device 120 in the task scene, and the direction feature corresponding to the viewing angle may represent the sight direction corresponding to the viewing angle.

[0073] In one example, each image feature in the historical image feature sequence may represent features (i.e., image information) corresponding to multiple images collected at one time, and the multiple images may be collected at multiple different viewing angles. Therefore, each image feature in the historical image feature sequence may include features corresponding to images at each of the multiple viewing angles.

[0074] In another example, each image feature in the historical image feature sequence may include features corresponding to the image at each of the multiple perspectives and directional features corresponding to the perspective. For example, the features of the image collected at a certain historical moment and the directional features of the multiple perspectives used when collecting the image at the historical moment may be spliced ​​together to obtain an image feature, that is, to obtain a visual latent code Z<1,0>. The visual latent codes of multiple historical moments may constitute a historical image feature sequence. The historical image feature sequence may also be called a memory queue.

[0075] In this embodiment, by adding directional features corresponding to the viewing angle to the image features, the image feature prediction model can explicitly know the direction or viewing angle from which the observation information (i.e., each image feature in the historical image feature sequence) is observed. In this way, the generated image feature sequence at future moments (i.e., the predicted image feature sequence) can also contain certain directional features, i.e., have better spatial consistency.

[0076] According to an embodiment of the present application, step 230 may include: using an image feature prediction model to process the noise image feature sequence based on a bidirectional attention mechanism and based on a historical image feature sequence, and generating each image feature in the predicted image feature sequence.

[0077] Specifically, in the process of processing the noisy image feature sequence according to the historical image feature sequence (such as denoising or other processing), the image feature prediction model can simultaneously generate each image feature in the predicted image feature sequence based on the bidirectional attention mechanism, so that the mutual influence between each image feature in the predicted image feature sequence can be taken into account, and the quality of the predicted image feature sequence finally obtained can be improved. For example, the image feature prediction model can use the diffusion generation paradigm based on the bidirectional Transformer in the instantaneous space to implement the bidirectional attention mechanism, thereby achieving the effect of simultaneously generating each image feature in the predicted image feature sequence.

[0078] Of course, in other embodiments, the image feature prediction model may generate each image feature in the predicted image feature sequence simultaneously or sequentially based on other mechanisms, and the embodiments of the present application are not limited to this.

[0079] In this embodiment, the image feature prediction model can simultaneously generate each image feature in the predicted image feature sequence based on the bidirectional attention mechanism, so that the mutual influence between the various image features in the predicted image feature sequence can be taken into account, thereby improving the quality of the predicted image feature sequence finally obtained.

[0080] Furthermore, in some embodiments, the image feature prediction model may use a block autoregressive generation paradigm based on a causal transformer over a longer time series, which is equivalent to, in terms of space and time, relative to a previously output predicted image feature sequence, the predicted image feature sequence currently output by the image feature prediction model may be generated based on a unidirectional attention mechanism. The image feature prediction model uses a block autoregressive generation paradigm based on a causal transformer over a longer time series, and uses a diffusion generation paradigm based on a bidirectional transformer in instantaneous space. In this way, the advantages of these two generation paradigms can be organically combined to achieve high-quality generation of four-dimensional spatial data. Here, the four-dimensional spatial data may include a time dimension and three spatial dimensions.

[0081] According to one embodiment of the present application, step 230 may include: inputting the historical image feature sequence and the noise image feature sequence into the image feature prediction model, and based on the task text encoding and / or action latent code, controlling the image feature prediction model to process the noise image feature sequence based on the historical image feature sequence to obtain a predicted image feature sequence.

[0082] In one example, the input of the image feature prediction model may include control conditions in addition to the historical image feature sequence and the noise image feature sequence. The control conditions may be used to control the image feature prediction model to generate an image feature sequence that satisfies the control conditions, and then a corresponding picture may be generated based on the image feature sequence.

[0083] In one example, the control condition may be used to characterize the specific content of the task to be performed. For example, the specific content of the task to be performed may be: using a robotic arm to pick up a red ball on the table.

[0084] In one example, the control condition can be described by a task text. For example, the specific content of the task to be performed is described by the task text, and then the task text is encoded into a task text encoding using a text encoder, and the task text encoding can be input into the image feature prediction model as a control condition. In this example, the task scenario corresponding to the task to be performed can be: planning the corresponding action picture according to the textual instructions, that is, visual planning. In one example, the task text encoding can be a text embedding vector, and the text encoder can be a text encoder based on the contrastive language-image pre-training (CLIP) architecture or other suitable encoders.

[0085] In one example, the control condition can be described by an action instruction. For example, the specific content of the task to be performed is described by an action instruction, and then the action encoder is used to encode the action instruction into an action latent code, which can be input into the image feature prediction model as a control condition. In this example, the action instruction can be a parameter such as the joint rotation angle of the robot arm, and the task scenario corresponding to the task to be performed can be: planning the corresponding action picture according to the action instruction, that is, simulation evaluation. The task to be performed can be used to test whether the action instruction is correct, that is, the image feature prediction model is equivalent to an evaluator, which is used to test whether the action picture generated based on the output image feature sequence can perform the task to be performed. In one example, the action encoder can be an encoder based on the VAE architecture or other suitable encoder.

[0086] In one example, the task text encoding and the action latent code can be simultaneously input into the image feature prediction model, and the image feature prediction model is controlled to process the noise image feature sequence based on the historical image feature sequence to obtain a predicted image feature sequence.

[0087] In this embodiment, by inputting control conditions into the image feature prediction model, the image feature prediction model can be applied to different task scenarios, thereby improving the flexibility of the image feature prediction model.

[0088] According to an embodiment of the present application, each image feature in the historical image feature sequence is used to characterize three-dimensional Gaussian point cloud data, the image feature prediction model includes a four-dimensional scene model, and the image feature prediction model is used to predict changes in three-dimensional Gaussian point cloud data over time.

[0089] In one example, the image feature prediction model includes a four-dimensional scene model, which can be used to predict a four-dimensional image feature sequence, which can be used to present a task scene. The four dimensions in the four-dimensional scene model can include a time dimension and three spatial dimensions.

[0090] In one example, both the historical image feature sequence and the noise image feature sequence can be four-dimensional data, and each image feature in the historical image feature sequence and the noise image feature sequence can be three-dimensional data, such as being used to characterize three-dimensional Gaussian point cloud data. The predicted image feature sequence output by the image feature prediction model can be used to predict changes in three-dimensional Gaussian point cloud data over time.

[0091] In one example, the four-dimensional scene model may also be a four-dimensional world model, which may adopt a deep learning network structure or other network structures. For example, the four-dimensional world model may be a model based on a multi-layer transformer architecture, which may be used to generate a four-dimensional spatiotemporal sequence, i.e., a four-dimensional image feature sequence. In one example, the four-dimensional world model may be used in the embodied field.

[0092] In this embodiment, the existing three-dimensional Gaussian point cloud data can be processed by the four-dimensional scene model to obtain predicted three-dimensional Gaussian point cloud data, and the image data generated based on the predicted three-dimensional Gaussian point cloud data can have a certain logic, that is, the state of things presented by the image data can be more in line with the law of development of things. In addition, since the four-dimensional scene model is four-dimensional, the output image feature sequence is also four-dimensional, so the image features corresponding to the appropriate viewing angle can be selected from the output image feature sequence for decoding and rendering to obtain a picture, or the data corresponding to the appropriate viewing angle can be selected from the decoded three-dimensional Gaussian point cloud data for rendering to obtain a picture, which can improve the quality of the final picture (such as an image or video), for example, it can avoid the situation where the objects in the picture are blocked or deformed as much as possible.

[0093] Figure 3 Shown is a flowchart of an image data generating method provided by another exemplary embodiment of the present application. Figure 3 The embodiment is Figure 2 For the example of the embodiment, in order to avoid repetition, the same points can be referred to the description in the above embodiment, which will not be repeated here. Figure 3 As shown, the image data generating method may include the following contents.

[0094] 310: Obtain a historical image feature sequence.

[0095] In one example, for a historical moment when an image needs to be collected, images from multiple perspectives can be collected at that historical moment to obtain multiple images, and the directional features corresponding to each perspective can be recorded. Furthermore, a multi-perspective 3D reconstruction method can be used to reconstruct multiple images at each historical moment into 3D point cloud data, such as a static 3D Gaussian sphere in the form of a point cloud, and then a visual encoder can be used to encode the reconstructed 3D Gaussian sphere into a visual latent code, and the visual latent code and the corresponding directional features can be spliced ​​to obtain an image feature Z<i,0> Here, Z<i,0> The i in Z can represent the sequence number. When i=1, it means that the image feature is in the initial historical image feature sequence.<i,0> 0 in can indicate that no noise is added.

[0096] In one example, an initial historical image feature sequence, ie, an initial memory queue, may be constructed by repeatedly stacking m copies of the image features (or referred to as visual latent codes).

[0097] 320: Generate a noise image feature sequence according to the dimension of the image features in the historical image feature sequence.

[0098] In one example, c dimensions and Z can be initialized<i,0> Uniform Gaussian noise Z<i+1,T> , get the noise image feature sequence, and concatenate the noise image feature sequence to the end of the historical image feature sequence.

[0099] 330: Inputting the historical image feature sequence and the noise image feature sequence into the image feature prediction model, based on the task text encoding or the action latent code, controlling the image feature prediction model to process the noise image feature sequence based on the historical image feature sequence to obtain a predicted image feature sequence.

[0100] In one example, the spliced ​​historical image feature sequence and the noise image feature sequence can be input into the image feature prediction model, and the task text encoding or action latent code can be input into the image feature prediction model as a control condition. In one example, the image feature prediction model can use the diffusion reverse diffusion generative paradigm to transform the Z in the noise image feature sequence into the Z in the noise image feature sequence after d rounds of iterations.<i+1,T> Denoising for Z<i+1,0> , we can get the predicted image feature sequence, that is, we can get c noise-free three-dimensional visual latent codes.

[0101] 340: Generate image data based on the predicted image feature sequence.

[0102] In one example, Z<i+1,0> The three-dimensional visual latent code in the image is decoded by a visual decoder into three-dimensional point cloud data at each moment, such as a three-dimensional Gaussian sphere. The three-dimensional Gaussian sphere is rendered to any preset virtual camera using the Gaussian splash method to obtain the image or video under the camera, that is, the image data.

[0103] 350: According to the first-in-first-out rule, the historical image feature sequence is updated using the last image feature in the predicted image feature sequence to obtain an updated historical image feature sequence.

[0104] In one example, the last image feature in the predicted image feature sequence is added after the original historical image feature sequence, and the first image feature in the original historical image feature sequence is removed, so that an updated historical image feature sequence can be obtained.

[0105] Further, steps 320 to 340 may be repeatedly executed until a preset special end of sentence (EOS) is generated. For example, when step 320 is repeatedly executed, the c dimensions and Z may be initialized.<i+1,0> Uniform Gaussian noise Z<i+2,T> , obtain a new noise image feature sequence, and splice the new noise image feature sequence to the end of the updated historical image feature sequence queue.

[0106] Figure 4 It shows the image data generation process that can use the block autoregressive generation paradigm based on causal Transformer on a longer time series, and the diffusion generation paradigm based on bidirectional Transformer on the instantaneous space.

[0107] like Figure 4 As shown, Z<1,0> can represent an image feature in the initial historical image feature sequence, and Z<1,0> can be reused to form a historical image feature sequence. Figure 4 One Z<1,0> is omitted. Z<2,T> can represent Gaussian noise when i=2. Figure 4 Two Gaussian noises are shown in the figure, and these two Gaussian noises can constitute a noise image feature sequence. Z<1,0> and Z<2,T> are input into the generative model (image feature prediction model), and the generative model can denoise the noise image feature sequence according to a certain step size. In one example, T steps of noise are added in the process of adding noise, and in the process of denoising, denoising can be performed according to T steps. For example, after the denoising process in the tth step (t is greater than or equal to 0 and less than or equal to T), the image feature Z<2,t> can be obtained, and after T steps of denoising, the image feature Z<2,0> can be obtained, that is, the denoising is completed. Figure 4 The three-dimensional point cloud data represented by the two image features Z<2, t> in may be different.

[0108] Similarly, after obtaining the predicted image feature sequence, that is, after obtaining two Z<2,0>, Z<1,0>, Z<2,0> and Z<3,T> can be input into the generation model to start a new round of image feature sequence prediction process. Z<3,T> can represent Gaussian noise when i=3, Figure 4 Two Gaussian noises are shown in FIG. 1 , which can constitute a noise image feature sequence. The generative model can perform denoising on the noise image feature sequence according to a certain step length. After the denoising process in the tth step (t is greater than or equal to 0 and less than or equal to T), the image feature Z<3, t> can be obtained. After the denoising process in T steps, the image feature Z<3, 0> can be obtained, that is, the denoising is completed. Figure 4 The three-dimensional point cloud data represented by the two image features Z<3, t> in t can be different. Similarly, the image feature sequence prediction process can be performed multiple times until the EOS is generated and terminated.

[0109] In some embodiments, during the image data generation process, a K / V cache method can be used to store data, such as storing historical image feature sequences. The K / V cache method can make the previously calculated data no longer need to be calculated again, thereby improving computing efficiency.

[0110] The present application also provides a model training method, such as Figure 5 As shown, the model training method may include the following contents.

[0111] 510: Select multiple image features from the initial image feature sequence as a sample historical image feature sequence.

[0112] In one example, the initial image feature sequence may be an existing complete image feature sequence, which may be used to train an image feature prediction model. For example, multiple image features are selected from the front section of the initial image feature sequence to form a sample historical image feature sequence. The multiple image features may be continuous or discontinuous in time, and the number of the multiple image features may be set according to actual needs.

[0113] In one example, multiple image features may be selected from an initial image feature sequence according to a certain rule or randomly as a sample historical image feature sequence.

[0114] 520: Obtain a sample noise image feature sequence.

[0115] In one example, the sample noise image feature sequence is located after the sample historical image feature sequence in time order, and the dimension of the image features in the sample noise image feature sequence is consistent with the dimension of the image features in the sample historical image feature sequence. In one example, the sample noise image feature sequence can be generated according to the dimension of the image features in the sample historical image feature sequence. The method for generating the sample noise image feature sequence is similar to the method for generating the noise image feature sequence described above, and will not be described here to avoid repetition.

[0116] 530: Inputting the sample history image feature sequence and the sample noise image feature sequence into the image feature prediction model to obtain a predicted sample image feature sequence.

[0117] In one example, an image feature prediction model may be used to perform denoising or other processing on a sample noise image feature sequence based on a sample historical image feature sequence to obtain a predicted sample image feature sequence.

[0118] In one example, each image feature in the predicted sample image feature sequence is used to represent image information in three-dimensional space.

[0119] In one example, the image feature prediction model processes the sample noise image feature sequence based on the sample historical image feature sequence to obtain a predicted sample image feature sequence. This process is similar to the process in which the above-mentioned image feature prediction model processes the noise image feature sequence based on the historical image feature sequence to obtain a predicted image feature sequence. To avoid repetition, it will not be repeated here.

[0120] In one example, during the training process, the input of the image feature prediction model may include a sample historical image feature sequence and a sample noise image feature sequence. Further, the input of the image feature prediction model may also include a control condition. The specific content of the control condition can be found in the description of the above embodiment. To avoid repetition, it will not be repeated here.

[0121] 540: Training the image feature prediction model based on the difference between the predicted sample image feature sequence and the target image feature sequence in the initial image feature sequence to obtain a trained image feature prediction model.

[0122] In one example, the target image feature sequence is consistent with the predicted sample image feature sequence in the time dimension. Based on the difference between the predicted sample image feature sequence and the target image feature sequence, the parameters of the image feature prediction model can be adjusted to obtain a trained image feature prediction model. The trained image feature prediction model can be used to execute the above-mentioned image data generation method. In one example, the execution subject of the model training method can be the above-mentioned computer device 110, task execution device 120 or other device.

[0123] The embodiment of the present application provides a model training method, by inputting a sample historical image feature sequence and a sample noise image feature sequence into an image feature prediction model, a predicted sample image feature sequence is obtained, and thus a trained image feature prediction model can be obtained based on the difference between the predicted sample image feature sequence and the target image feature sequence. The state of things represented by the predicted image feature sequence output by the image feature prediction model obtained by the model training method conforms to the law of development of things. Therefore, the quality of the image data generated based on the predicted image feature sequence is high, which can improve the accuracy of the decision or planning made by the robot based on the image data, and thus improve the completion effect of the task. Furthermore, since each image feature in the predicted image feature sequence is used to represent the image information in three-dimensional space, the predicted image feature sequence can be presented according to the required viewing angle, thereby further improving the quality of the final image or video, for example, it can avoid the situation where objects in the image or video are blocked or deformed as much as possible.

[0124] In some embodiments, multiple image features are selected from the front section of the initial image feature sequence to form a sample historical image feature sequence, and the multiple image features may be discontinuous in time, that is, the sample historical image feature sequence is sparse in time. Using the sparse sample historical image feature sequence to train the image feature prediction model can improve the generalization of the model.

[0125] Figure 6 FIG. 1 is a schematic diagram of the training phase and the reasoning phase of a model provided by an exemplary embodiment of the present application. Figure 6 As shown in the figure, during the training phase, two image features can be randomly selected from the initial image feature sequence as the sample historical image feature sequence, that is, corresponding to Figure 6 According to the dimension of the image features in the sample historical image feature sequence, four Gaussian noises can be generated as well as A sample noise image feature sequence is obtained. The sample historical image feature sequence and the sample noise image feature sequence are input into an image feature prediction model, which may also be referred to as a generation model. The generation model may output a predicted sample image feature sequence (also referred to as a denoised sample image feature sequence). The predicted sample image feature sequence may include four image features V1, V2, V3, and V4. Based on the difference between the predicted sample image feature sequence and the target image feature sequence, a loss may be calculated. Based on the loss, the parameters of the generation model may be adjusted.

[0126] In the inference stage, based on the two image features and According to the dimension of the image features in the historical image feature sequence, four Gaussian noises can be generated. as well as The noise image feature sequence is obtained. The historical image feature sequence and the noise image feature sequence are input into the generation model, and a predicted image feature sequence can be output. The predicted image feature sequence can include four image feature sequences. as well as The time length corresponding to the predicted image feature sequence can be regarded as a time window. This inference can be the i-th inference, so it can be represented as Chunk i. By updating the original historical image feature sequence, an updated historical image feature sequence can be obtained, which includes two image features: and The updated historical image feature sequence can be used for the i+1th reasoning, that is, Chunk i+1. Similarly, four copies of Gaussian noise can be generated for the updated historical image feature sequence: as well as The updated historical image feature sequence and the noise image feature sequence are input into the generation model to output a predicted image feature sequence, which may include four image feature sequences. And EOS. EOS can be a preset special end latent code. Generating EOS can indicate the termination of the reasoning iteration process.

[0127] Exemplary Devices

[0128] Figure 7 FIG. 1 is a schematic diagram of the structure of an image data generating device provided by an exemplary embodiment of the present application. Figure 7 As shown, the image data generating device 700 includes: an acquisition module 710 , a prediction module 720 , and a generation module 730 .

[0129] The acquisition module 710 is used to acquire a historical image feature sequence, wherein the historical image feature sequence includes multiple image features corresponding to multiple historical moments. The acquisition module 710 is also used to acquire a noise image feature sequence, wherein the noise image feature sequence is located after the historical image feature sequence in time order, and the dimension of the image features in the noise image feature sequence is consistent with the dimension of the image features in the historical image feature sequence. The prediction module 720 is used to input the historical image feature sequence and the noise image feature sequence into the image feature prediction model to obtain a predicted image feature sequence, wherein each image feature in the predicted image feature sequence is used to characterize image information in three-dimensional space. The generation module 730 is used to generate image data based on the predicted image feature sequence.

[0130] The embodiment of the present application provides an image data generating device, which can obtain a predicted image feature sequence by acquiring a historical image feature sequence and a noise image feature sequence, and inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model, thereby obtaining a predicted image feature sequence, and then generating image data based on the predicted image feature sequence. In the embodiment of the present application, the noise image feature sequence is located after the historical image feature sequence in time order, the dimension of the image features in the noise image feature sequence is consistent with the dimension of the image features in the historical image feature sequence, and the historical image feature sequence includes multiple image features corresponding to multiple historical moments, respectively. The image feature prediction model can process the noise image feature sequence based on the historical image feature sequence, so that the state of things represented by the predicted image feature sequence conforms to the law of development of things. Therefore, the quality of the image data generated based on the predicted image feature sequence is high, which can improve the accuracy of the decision or planning made by the robot based on the image data, and then improve the completion effect of the task. Furthermore, since each image feature in the predicted image feature sequence is used to characterize image information in three-dimensional space, the predicted image feature sequence can be presented according to the required viewing angle, thereby further improving the quality of the final image or video, for example, avoiding the situation where objects in the image or video are occluded or deformed as much as possible.

[0131] According to one embodiment of the present application, the image data generating device 700 also includes: an updating module 740, which is used to: update the historical image feature sequence using at least one image feature in the predicted image feature sequence to obtain an updated historical image feature sequence; use the updated historical image feature sequence as the historical image feature sequence, repeat the step of obtaining the noise image feature sequence, and input the historical image feature sequence and the noise image feature sequence into the image feature prediction model to obtain the predicted image feature sequence.

[0132] According to an embodiment of the present application, the updating module 740 is used to update the historical image feature sequence using the last image feature in the predicted image feature sequence to obtain an updated historical image feature sequence.

[0133] According to an embodiment of the present application, the updating module 740 is used to update the historical image feature sequence according to the first-in-first-out rule using the last image feature in the predicted image feature sequence to obtain an updated historical image feature sequence.

[0134] According to an embodiment of the present application, each image feature in the historical image feature sequence includes: features corresponding to the image at each viewing angle among multiple viewing angles; and directional features corresponding to the viewing angle.

[0135] According to an embodiment of the present application, the prediction module 720 is used to: utilize the image feature prediction model to process the noise image feature sequence based on the bidirectional attention mechanism and based on the historical image feature sequence, and simultaneously generate each image feature in the predicted image feature sequence.

[0136] According to one embodiment of the present application, the prediction module 720 is used to: input the historical image feature sequence and the noise image feature sequence into the image feature prediction model, and based on the task text encoding and / or action latent code, control the image feature prediction model to process the noise image feature sequence based on the historical image feature sequence to obtain a predicted image feature sequence.

[0137] According to an embodiment of the present application, each image feature in the historical image feature sequence is used to characterize three-dimensional Gaussian point cloud data, the image feature prediction model includes a four-dimensional scene model, and the image feature prediction model is used to predict changes in three-dimensional Gaussian point cloud data over time.

[0138] It should be understood that the operations and functions of the acquisition module 710, the prediction module 720, the generation module 730, and the update module 740 in the above embodiment can refer to the above Figure 2 or Figure 3 To avoid repetition, the description of the image data generating method provided in the embodiment will not be repeated here.

[0139] Figure 8 FIG. 1 is a schematic diagram of the structure of a model training device provided by an exemplary embodiment of the present application. Figure 8 As shown, the model training device 800 includes: a selection module 810, an acquisition module 820, a prediction module 830, and a training module 840.

[0140] The selection module 810 is used to select multiple image features from the initial image feature sequence as the sample historical image feature sequence. The acquisition module 820 is used to acquire the sample noise image feature sequence, wherein the sample noise image feature sequence is located after the sample historical image feature sequence in time order, and the dimension of the image features in the sample noise image feature sequence is consistent with the dimension of the image features in the sample historical image feature sequence. The prediction module 830 is used to input the sample historical image feature sequence and the sample noise image feature sequence into the image feature prediction model to obtain a predicted sample image feature sequence, wherein each image feature in the predicted sample image feature sequence is used to characterize the image information in three-dimensional space. The training module 840 is used to train the image feature prediction model based on the difference between the predicted sample image feature sequence and the target image feature sequence in the initial image feature sequence to obtain a trained image feature prediction model, wherein the target image feature sequence is consistent with the predicted sample image feature sequence in the time dimension.

[0141] The embodiment of the present application provides a model training device, which obtains a predicted sample image feature sequence by inputting a sample history image feature sequence and a sample noise image feature sequence into an image feature prediction model, so that a trained image feature prediction model can be obtained based on the difference between the predicted sample image feature sequence and the target image feature sequence. The state of things represented by the predicted image feature sequence output by the image feature prediction model obtained by the model training method conforms to the law of development of things. Therefore, the quality of the image data generated based on the predicted image feature sequence is high, which can improve the accuracy of the decision or planning made by the robot based on the image data, and thus improve the completion effect of the task. Furthermore, since each image feature in the predicted image feature sequence is used to represent the image information in three-dimensional space, the predicted image feature sequence can be presented according to the required viewing angle, thereby further improving the quality of the final image or video, for example, it can avoid the situation where objects in the image or video are blocked or deformed as much as possible.

[0142] It should be understood that the operations and functions of the selection module 810, the acquisition module 820, the prediction module 830, and the training module 840 in the above-mentioned embodiment can refer to the above-mentioned Figure 5 To avoid repetition, the description of the model training method provided in the embodiment will not be repeated here.

[0143] An embodiment of the present application also provides a mobile device, which includes a control module, and the control module is used to execute the image data generation method provided in any of the above embodiments.

[0144] In some embodiments, the mobile device may be a robot, such as a dual-arm robot, or may be a vehicle or other device that can be used to perform a specific task.

[0145] The operation and functions of the mobile device provided in the embodiments of the present application can refer to the above Figure 2 or Figure 3 To avoid repetition, the description of the image data generating method provided in the embodiment will not be repeated here.

[0146] Fig. 9 The block diagram of an electronic device 900 for executing an image data generation method or a model training method provided by an exemplary embodiment of the present application is shown. The electronic device 900 may specifically be a server, a mobile device, a control device of a mobile device, a server interacting with a mobile device, or other device.

[0147] Reference Fig. 9, the electronic device 900 includes a processing component 910, which further includes one or more processors, and a memory resource represented by a memory 920 for storing instructions executable by the processing component 910, such as an application. The application stored in the memory 920 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 910 is configured to execute instructions to perform the above-mentioned image data generation method or model training method.

[0148] The electronic device 900 may also include a power supply component configured to perform power management of the electronic device 900, a wired or wireless network interface configured to connect the electronic device 900 to a network, and an input / output (I / O) interface. The electronic device 900 may be operated based on an operating system stored in the memory 920, such as Windows Server 2008. TM , Mac OSX TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0149] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the above-mentioned electronic device 900, enables the above-mentioned electronic device 900 to execute an image data generation method or a model training method.

[0150] A computer program product includes a computer program. When the computer program is executed by a processor of a computer device, the computer device is enabled to execute the image data generation method or model training method provided in any of the above embodiments.

[0151] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, and will not be described one by one here.

[0152] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0157] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program check codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0158] It should be noted that, in the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0160] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating image data, characterized in that: include: Acquire a historical image feature sequence, wherein the historical image feature sequence includes a plurality of image features corresponding to a plurality of historical moments respectively; Acquire a noise image feature sequence, wherein the noise image feature sequence is located after the historical image feature sequence in time order, and the dimension of the image features in the noise image feature sequence is consistent with the dimension of the image features in the historical image feature sequence; Inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model to obtain a predicted image feature sequence, wherein each image feature in the predicted image feature sequence is used to represent image information in a three-dimensional space; Image data is generated based on the predicted sequence of image features.

2. The image data generating method according to claim 1, characterized in that: Also includes: Using at least one image feature in the predicted image feature sequence to update the historical image feature sequence to obtain an updated historical image feature sequence; The updated historical image feature sequence is used as the historical image feature sequence, and the steps of obtaining the noise image feature sequence and inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model to obtain a predicted image feature sequence are repeated.

3. The image data generating method according to claim 2, characterized in that: The updating of the historical image feature sequence by using at least one image feature in the predicted image feature sequence to obtain an updated historical image feature sequence includes: The historical image feature sequence is updated using the last image feature in the predicted image feature sequence to obtain the updated historical image feature sequence.

4. The image data generating method according to claim 2, characterized in that: The updating of the historical image feature sequence by using at least one image feature in the predicted image feature sequence to obtain an updated historical image feature sequence includes: According to the first-in-first-out rule, the historical image feature sequence is updated using the last image feature in the predicted image feature sequence to obtain the updated historical image feature sequence.

5. The image data generating method according to claim 1, characterized in that: Each image feature in the historical image feature sequence includes: features corresponding to the image at each viewing angle among multiple viewing angles; and directional features corresponding to the viewing angle.

6. The image data generating method according to claim 1, characterized in that: The step of inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model to obtain a predicted image feature sequence comprises: The image feature prediction model is used to process the noise image feature sequence based on a bidirectional attention mechanism and based on the historical image feature sequence, and each image feature in the predicted image feature sequence is generated.

7. The image data generating method according to claim 1, characterized in that: The step of inputting the historical image feature sequence and the noise image feature sequence into an image feature prediction model to obtain a predicted image feature sequence comprises: The historical image feature sequence and the noise image feature sequence are input into the image feature prediction model, and based on the task text encoding and / or action latent code, the image feature prediction model is controlled to process the noise image feature sequence based on the historical image feature sequence to obtain the predicted image feature sequence.

8. The image data generating method according to any one of claims 1 to 7, characterized in that: Each image feature in the historical image feature sequence is used to characterize three-dimensional Gaussian point cloud data. The image feature prediction model includes a four-dimensional scene model. The image feature prediction model is used to predict changes in three-dimensional Gaussian point cloud data over time.

9. A model training method, characterized in that: include: Selecting multiple image features from the initial image feature sequence as a sample historical image feature sequence; Acquire a sample noise image feature sequence, wherein the sample noise image feature sequence is located after the sample historical image feature sequence in time order, and the dimension of the image features in the sample noise image feature sequence is consistent with the dimension of the image features in the sample historical image feature sequence; Inputting the sample historical image feature sequence and the sample noise image feature sequence into an image feature prediction model to obtain a predicted sample image feature sequence, wherein each image feature in the predicted sample image feature sequence is used to characterize image information in a three-dimensional space; The image feature prediction model is trained based on the difference between the predicted sample image feature sequence and the target image feature sequence in the initial image feature sequence to obtain a trained image feature prediction model, wherein the target image feature sequence is consistent with the predicted sample image feature sequence in the time dimension.

10. A mobile device, characterized in that: It comprises a control module, wherein the control module is used to execute the image data generating method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor, Wherein, the processor is used to execute the image data generation method described in any one of claims 1 to 8 or the model training method described in claim 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the image data generation method described in any one of claims 1 to 8 or the model training method described in claim 9.

13. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor of a computer device, the computer device is enabled to execute the image data generation method described in any one of claims 1 to 8 or the model training method described in claim 9.

Images