Image data generation method, model training method, electronic equipment and medium

Abstract

The invention provides an image data generation method, a model training method, electronic equipment and a medium, and the method comprises the steps: obtaining the initial image data of a target scene, and enabling the initial image data to be used for representing the state of a target object in the target scene at a current moment; template image data are obtained, the template image data are generated based on the motion trail parameters, the motion trail parameters are predicted trail parameters for the target object, the template image data comprise a template object, and the template image data are used for presenting the process that the template object moves along a trail represented by the motion trail parameters; and generating predicted image data according to the initial image data and the template image data, the predicted image data being used for presenting the process of the target object moving along the trajectory in the target scene. According to the technical scheme, the acquisition cost of the training sample can be reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image data generation method, a model training method, an electronic device, and a medium. Background Art

[0002] With the continuous advancement of science and technology, people's lives and work are gradually moving towards intelligence. For example, for a specific scene, image data of the scene at the current moment can be collected and input into a model. The model can then predict the state of the scene at future moments, thereby helping intelligent agents make decisions. However, the cost of obtaining training samples for this model is high. Summary of the Invention

[0003] In view of this, embodiments of the present application provide an image data generation method, a model training method, an electronic device, and a medium, which can reduce the cost of acquiring training samples.

[0004] In a first aspect, an embodiment of the present application provides an image data generation method, comprising: obtaining initial image data of a target scene, wherein the initial image data is used to characterize the state of a target object in the target scene at a current moment; obtaining template image data, wherein the template image data is generated based on motion trajectory parameters, the motion trajectory parameters are trajectory parameters predicted for the target object, the template image data includes a template object, and the template image data is used to present the process of the template object moving along the trajectory represented by the motion trajectory parameters; generating predicted image data based on the initial image data and the template image data, wherein the predicted image data is used to present the process of the target object moving along the trajectory in the target scene.

[0005] In the second aspect, an embodiment of the present application provides a model training method, comprising: inputting sample initial image data and sample template image data into a model to obtain sample predicted image data output by the model, wherein the sample initial image data includes image data at the current moment in the sample image sequence, the sample template image data includes a template object, and the sample template image data is used to present the process of the template object moving along the trajectory represented by the sample motion trajectory parameters, and the sample motion trajectory parameters are used to represent the motion trajectory of the sample target object in the target image data in the sample image sequence, and the time period corresponding to the target image data is after the current moment; the model is trained based on the difference between the sample predicted image data and the target image data to obtain an image generation model.

[0006] In a third aspect, an embodiment of the present application provides an image data generating device, comprising: a first acquisition module for acquiring initial image data of a target scene, wherein the initial image data is used to characterize the state of a target object in the target scene at a current moment; a second acquisition module for acquiring template image data, wherein the template image data is generated based on motion trajectory parameters, the motion trajectory parameters are trajectory parameters predicted for the target object, the template image data includes a template object, and the template image data is used to present the process of the template object moving along the trajectory represented by the motion trajectory parameters; a generation module for generating predicted image data based on the initial image data and the template image data, wherein the predicted image data is used to present the process of the target object moving along the trajectory in the target scene.

[0007] In a fourth aspect, an embodiment of the present application provides a model training device, comprising: an input module for inputting sample initial image data and sample template image data into a model to obtain sample predicted image data output by the model, wherein the sample initial image data includes image data at the current moment in the sample image sequence, the sample template image data includes a template object, and the sample template image data is used to present the process of the template object moving along the trajectory represented by the sample motion trajectory parameters, the sample motion trajectory parameters are used to represent the motion trajectory of the sample target object in the target image data in the sample image sequence, and the time period corresponding to the target image data is after the current moment; a training module for training the model based on the difference between the sample predicted image data and the target image data to obtain an image generation model.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is used to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0010] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor of a computer device, the computer device is able to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0011] In an eighth aspect, an embodiment of the present application provides a chip comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is used to execute the image data generation method described in the first aspect or the model training method described in the second aspect.

[0012] Embodiments of the present application provide an image data generation method, a model training method, an electronic device, and a medium that can generate template image data based on the predicted motion trajectory parameters of a target object in a target scene. The template image data is used to present the process of the template object moving along the trajectory represented by the motion trajectory parameters, thereby visualizing the motion trajectory parameters, that is, concretizing or intuitively presenting high-dimensional or abstract motion trajectory parameters. Furthermore, embodiments of the present application can generate predicted image data based on the initial image data and template image data at the current moment, thereby reducing the gap between the motion trajectory parameters and the image data, that is, converting the motion trajectory parameters into pixel-level template image data. Based on the pixel-level template image data and the initial image data at the current moment, the predicted image data is obtained, which can improve the success rate of predicted image data generation and the degree of consistency between the trajectory of the target object in the predicted image data and the trajectory represented by the motion trajectory parameters, thereby improving the accuracy and authenticity of the predicted image data. In addition, training an action prediction model based on this predicted image data can improve the reliability of the model's training samples and reduce the cost of acquiring training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Shown is a schematic diagram of the system architecture of an image data generation system provided by an exemplary embodiment of the present application.

[0014] Figure 2 Shown is a flowchart of an image data generation method provided by an exemplary embodiment of the present application.

[0015] Figure 3 Shown is a schematic diagram of template image data provided by an exemplary embodiment of the present application.

[0016] Figure 4 Shown is a flowchart of an image data generating method provided by another exemplary embodiment of the present application.

[0017] Figure 5 Shown is a flowchart of an image data generation method provided by another exemplary embodiment of the present application.

[0018] Figure 6 Shown is a schematic diagram of an image data generation process provided by an exemplary embodiment of the present application.

[0019] Figure 7Shown is a flow chart of a model training method provided by an exemplary embodiment of the present application.

[0020] Figure 8 Shown is a schematic structural diagram of an image data generating device provided by an exemplary embodiment of the present application.

[0021] Figure 9 Shown is a schematic structural diagram of a model training device provided by an exemplary embodiment of the present application.

[0022] Figure 10 Shown is a block diagram of an electronic device for executing an image data generation method or a model training method provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] Application Overview

[0025] With the development of intelligent technology, more and more tasks can be performed by intelligent agents such as robots. For example, for a specific scene, the robot can collect image data at the current moment and input the image data into a model. The model can then predict the state of the scene at a future moment, thereby helping the robot make decisions. Specifically, in some cases, the state of the scene at a future moment may include the motion trajectory parameters of the robot in the scene at a future moment. That is, the model (action prediction model) can predict the robot's motion trajectory parameters at a future moment. Controlling the robot's motion based on the predicted motion trajectory parameters can realize the robot's decision-making process, such as the execution process of completing the target task.

[0026] To improve the prediction accuracy of these models, they require extensive training. These training samples can include image data (or video data) that depicts the specific motion trajectory of an object. These training samples typically require acquisition from real devices, which can be expensive.

[0027] To address the above technical issues, embodiments of the present application provide an image data generation method that can generate template image data based on the predicted motion trajectory parameters of a target object in a target scene. The template image data is used to present the process of the template object moving along the trajectory represented by the motion trajectory parameters, thereby visualizing the motion trajectory parameters, that is, concretizing or intuitively presenting high-dimensional or abstract motion trajectory parameters. Furthermore, embodiments of the present application can generate predicted image data based on the initial image data and template image data at the current moment, thereby reducing the gap between the motion trajectory parameters and the image data, that is, converting the motion trajectory parameters into pixel-level template image data. Based on the pixel-level template image data and the initial image data at the current moment, the predicted image data is obtained, which can improve the success rate of predicted image data generation and improve the degree of consistency between the trajectory of the target object in the predicted image data and the trajectory represented by the motion trajectory parameters, thereby improving the accuracy and authenticity of the predicted image data. In addition, training an action prediction model based on the predicted image data can improve the reliability of the model's training samples and reduce the cost of obtaining training samples.

[0028] Exemplary Systems

[0029] Figure 1 FIG. 1 is a schematic diagram of the system architecture of an image data generation system provided by an exemplary embodiment of the present application. Figure 1 As shown, the image data generation system 100 may include a computer device 110 and a task execution device 120. The computer device 110 may be a server or a terminal device, such as a mobile phone or a laptop. The task execution device 120 may be a device for executing a task, such as a vehicle or a robot.

[0030] In one example, the task execution device 120 may be equipped with a sensor 121, which can be used to collect image data when the task execution device 120 performs a specific task in a target scene. Furthermore, the task execution device 120 may store motion trajectory parameters of a target object in the target scene when performing a specific task. For example, the target object may include a portion of the task execution device 120 that participates in the execution of the specific task. In one example, the target scene may be a scene in which a robot performs a target task, and the target object may be a portion of the robot used to perform the target task. For example, when the target task is a grasping task, the target object may be the robot's gripper; when the target task is a dancing task, the target object may include at least one of the robot's arms, grippers, or legs. In one example, the aforementioned image data and corresponding motion trajectory parameters may be collected during the process of controlling the robot to perform the target task via teleoperation, or acquired through other suitable means.

[0031] In one example, the task execution device 120 may send the image data and the corresponding motion trajectory parameters to the computer device 110 .

[0032] In one example, image data at a certain moment (the current moment) in the aforementioned image data can be used as the initial image data, and the motion trajectory parameters corresponding to image data in a target time period after the initial image data in the image data can be used as the original motion trajectory parameters. The initial image data can be used to characterize the state of the target object at the current moment. The original motion trajectory parameters are used to characterize the trajectory of the target object in the target time period, which is after the current moment.

[0033] In one example, the original motion trajectory parameters may be expanded, that is, the original motion trajectory parameters may be adjusted to obtain new motion trajectory parameters. The new motion trajectory parameters may be trajectory parameters predicted or designed for the target object.

[0034] In one example, computer device 110 may generate template image data based on the new motion trajectory parameters. The template image data includes a template object, and the template image data is used to represent the process of the template object moving along the trajectory represented by the new motion trajectory parameters. Furthermore, computer device 110 may generate predicted image data based on the initial image data and the template image data, wherein the predicted image data is used to represent the process of the target object moving along the trajectory represented by the new motion trajectory parameters in the target scene.

[0035] Furthermore, based on the new motion trajectory parameters and the predicted image data corresponding to the new motion trajectory parameters, a training sample for the motion prediction model can be obtained. The motion prediction model is then trained based on the training sample, so that the trained motion prediction model can predict the motion trajectory parameters of the task execution device 120 in the future based on the image data at the current moment. Specifically, the task execution device 120 can be deployed with the motion prediction model 122. For example, if the task execution device 120 is a robot, the robot can be controlled to perform a corresponding action or task based on the predicted motion trajectory parameters.

[0036] In one example, a world model may be deployed on the computer device 110 , and the computer device 110 may use the world model to generate predicted image data according to the initial image data and the template image data.

[0037] Exemplarily, the sensor 121 may be at least one of a lidar sensor, an ultrasonic sensor, a visual sensor, and the like.

[0038] It should be understood that the above application scenario examples are only provided to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited thereto. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0039] Exemplary Methods

[0040] Figure 2 Shown is a flowchart of an image data generation method provided by an exemplary embodiment of the present application. Figure 2 The method can be Figure 1 The computer device 110 or other device in the embodiment is executed. For the convenience of description, the following description is made by taking the computer device 110 executing the method as an example. Figure 2 As shown, the image data generating method may include the following contents.

[0041] 210: Acquire initial image data of the target scene.

[0042] Specifically, the initial image data is used to represent the state of the target object in the target scene at the current moment.

[0043] In one example, the initial image data may include image data at the current moment. For example, the image data at the current moment may include an image of the target scene at the current moment. Alternatively, in other examples, the image data at the current moment may include image features or latent codes corresponding to the image at the current moment. The meanings of other image data herein are similar to those of the image data at the current moment and are not further described here to avoid repetition.

[0044] In one example, the target scene may represent the scene covered by the initial image data. For example, the target scene may include a scene in which a robot is performing a target task, such as grasping, dancing, cleaning, or other tasks. Specifically, the initial image data may represent the state of a target object in the target scene at the current moment.

[0045] In one example, a target object can be an object in a target scene that needs to be observed and controlled. For example, the target object can include a part of a robot used to perform a target task. Specifically, for different target tasks, the target object can include at least one of the following robot parts: a gripper, an arm, a leg, a head, or a waist.

[0046] 220: Acquire template image data, wherein the template image data is generated based on motion trajectory parameters, and the motion trajectory parameters are trajectory parameters predicted for the target object.

[0047] Specifically, the template image data includes a template object, and the template image data is used to present a process in which the template object moves along a trajectory represented by the motion trajectory parameters.

[0048] In one example, the motion trajectory parameters may represent the motion trajectory of the target object within a certain period of time after the current moment. For example, the motion trajectory parameters may include motion trajectory parameters of the end or center of the target object.

[0049] In one example, a template object can be pre-set, corresponding to the target object. The template object is used to visualize motion trajectory parameters. For example, the template object can be presented within template background image data according to the motion trajectory parameters, thereby obtaining template image data. Specifically, the motion trajectory parameters may include state parameters corresponding to N time instants. The state parameters corresponding to N time instants are used as the state parameters of the template object to visualize the template object, thereby obtaining image data corresponding to N frames of images. Therefore, the template image data can be used to present the process of the template object moving along the trajectory represented by the motion trajectory parameters.

[0050] In one example, the initial image data may be truly captured image data, and the motion trajectory parameters of the target object in the target scene may be obtained through specific algorithm fitting or manually set.

[0051] 230: Generate predicted image data based on the initial image data and the template image data.

[0052] Specifically, the predicted image data is used to present a process in which the target object moves along a trajectory in the target scene.

[0053] In one example, the trajectory represented by the motion trajectory parameters can be more accurately assigned to the target object in the initial image data through the template image data, so that the predicted image data can be obtained.

[0054] In one example, the predicted image data is generated based on motion trajectory parameters, rather than actual data. This predicted image data can be used as training samples for other models. For example, a motion prediction model can be trained based on the predicted image data and motion trajectory parameters, reducing the cost of acquiring training samples for the motion prediction model.

[0055] Embodiments of the present application provide a method for generating image data that can generate template image data based on predicted motion trajectory parameters for a target object in a target scene. The template image data is used to present the process of the template object moving along the trajectory represented by the motion trajectory parameters, thereby visualizing the motion trajectory parameters, i.e., concretizing or intuitively presenting high-dimensional or abstract motion trajectory parameters. Furthermore, embodiments of the present application can generate predicted image data based on the initial image data and template image data at the current moment, thereby reducing the gap between the motion trajectory parameters and the image data, i.e., converting the motion trajectory parameters into pixel-level template image data. Based on the pixel-level template image data and the initial image data at the current moment, the predicted image data is obtained, which can improve the success rate of predicted image data generation and the degree of consistency between the trajectory of the target object in the predicted image data and the trajectory represented by the motion trajectory parameters, thereby improving the accuracy and authenticity of the predicted image data. In addition, training an action prediction model based on the predicted image data can improve the reliability of the model's training samples and reduce the cost of acquiring training samples.

[0056] According to one embodiment of the present application, the image data generation method may further include: obtaining original motion trajectory parameters corresponding to the initial image data, wherein the original motion trajectory parameters are used to characterize the trajectory of the target object in a target time period, and the target time period is located after the current moment; adjusting the original motion trajectory parameters to obtain motion trajectory parameters.

[0057] In one example, the original motion trajectory parameters may be the trajectory parameters of a target object captured during a target period of time after the current moment. For example, the initial image data and original motion trajectory parameters may be obtained by controlling the robot's motion via teleoperation. In another example, the robot may use a trained model to predict the motion trajectory parameters during the target period of time based on the initial image data to obtain the original motion trajectory parameters.

[0058] In one example, the start time of the target period may be the current time, or may be after the current time.

[0059] In one example, the original motion trajectory parameters may be adjusted, for example, manually or by using a specific algorithm, to obtain adjusted motion trajectory parameters. The adjusted motion trajectory parameters are the motion trajectory parameters mentioned in step 220.

[0060] In one example, the start time of the time period corresponding to the motion trajectory parameters can be the current time corresponding to the initial image data or after the current time. Furthermore, the time period corresponding to the motion trajectory parameters can be the same as the target time period, or partially overlap with the target time period. For example, the end time of the time period corresponding to the motion trajectory parameters can be before or after the end time of the target time period, and the start time of the time period corresponding to the motion trajectory parameters can be before or after the start time of the target time period. It should be understood that the time period corresponding to the motion trajectory parameters and the target time period can be selected according to actual needs, and the embodiments of the present application do not impose specific limitations on this.

[0061] In one example, each motion trajectory parameter in this application can be represented by a motion feature sequence. The motion feature sequence may include motion features of the target object at multiple moments. The motion features may include state parameters. The specific content of the state parameters can be found in the description below.

[0062] In this embodiment, a large number of motion trajectory parameters can be obtained by adjusting the original motion trajectory parameters. These motion trajectory parameters are generated based on the original motion trajectory parameters and do not actually exist. Moreover, these motion trajectory parameters can be different from each other. Based on the large number of motion trajectory parameters, a large amount of predicted image data can be obtained. Based on the predicted image data and the corresponding motion trajectory parameters, training samples for the motion prediction model can be obtained. This can improve the efficiency of obtaining training samples and reduce the cost of obtaining training samples. In addition, based on the motion trajectory parameters, template image data can be first obtained, and then predicted image data can be generated based on the initial image data and the template image data. This can improve the accuracy and authenticity of the predicted image data, thereby improving the reliability of the model's training samples.

[0063] According to one embodiment of the present application, the image data generation method may also include: selecting sample image data from the predicted image data, and selecting sample motion trajectory parameters corresponding to the sample image data from the motion trajectory parameters, wherein the time period corresponding to the sample motion trajectory parameters is after the moment corresponding to the sample image data; and training the robot motion prediction model using the sample image data and the sample motion trajectory parameters to obtain a trained robot motion prediction model.

[0064] In one example, training samples for a motion prediction model can be obtained based on the predicted image data and corresponding motion trajectory parameters. Specifically, sample image data is selected from the predicted image data. The sample image data can be image data at the first moment in a target period, or image data from a first period. The first moment can be any moment before the end of the target period. The first period can be a portion of the target period, the end of which can be before the end of the target period, and the portion can include multiple moments.

[0065] Furthermore, the sample motion trajectory parameters may be partial motion trajectory parameters. The time or time period corresponding to the sample motion trajectory parameters may be located after the first time or time period. The sample image data and the sample motion trajectory parameters may serve as training samples for the motion prediction model.

[0066] In one example, a motion prediction model can be used to predict the robot's motion trajectory parameters at a subsequent moment or time period based on image data collected by the robot at the current moment. This allows the robot to control its motion according to the predicted motion trajectory parameters and thereby complete the target task. In the field of robot control, this motion prediction model can be referred to as a robot motion prediction model. Of course, this motion prediction model can also be used in other intelligent device control fields, such as intelligent vehicle control.

[0067] Similarly, the motion prediction model can be trained using sample image data and sample motion trajectory parameters to obtain a trained motion prediction model. Specifically, the motion prediction model can predict motion trajectory parameters for subsequent moments or periods based on the sample image data. The parameters of the motion prediction model can be adjusted based on the difference between the predicted motion trajectory parameters and the sample motion trajectory parameters, thereby obtaining a trained motion prediction model.

[0068] In this embodiment, training samples for the motion prediction model are obtained based on predicted image data and corresponding motion trajectory parameters, which can improve the efficiency of training sample acquisition and reduce the cost of training sample acquisition. Furthermore, template image data can be first obtained based on the motion trajectory parameters, and then predicted image data can be generated based on the initial image data and the template image data. This improves the accuracy and authenticity of the predicted image data, thereby increasing the reliability of the model's training samples and improving the model's training effectiveness.

[0069] According to an embodiment of the present application, the motion trajectory parameters are trajectory parameters predicted for the target object based on the initial image data using the robot motion prediction model. The image data generation method further includes: evaluating the accuracy of the robot motion prediction model based on the predicted image data.

[0070] In one example, the action prediction model may be a pre-trained model. The solution of this embodiment may be used to evaluate the performance, such as accuracy, of the action prediction model.

[0071] In one example, the initial image data may be image data captured by the robot at the current moment. The initial image data is input into the motion prediction model, which may output predicted motion trajectory parameters. In one example, the start time of the time period corresponding to the motion trajectory parameters may be the current moment corresponding to the initial image data, or a time period after the current moment.

[0072] During the actual application of the motion prediction model, the robot controls its motion based on the trajectory parameters predicted by the motion prediction model. If the motion prediction model performs well, such as with high accuracy, the robot can successfully perform the target task by controlling its motion based on the trajectory parameters predicted by the model. If the motion prediction model performs poorly, such as with low accuracy, the robot may not be able to successfully perform the target task by controlling its motion based on the trajectory parameters predicted by the model. Therefore, in some cases, it is useful to evaluate the performance of the motion prediction model before putting it into practical use.

[0073] In some embodiments, the motion prediction model can be deployed to a real machine (such as a robot) and run to evaluate the accuracy of the motion prediction model. This evaluation method has high labor and time costs. In some embodiments, a simulation environment can be constructed to evaluate the accuracy of the motion prediction model in the simulation environment. Since there are still certain differences between the simulation environment and the real environment, the reliability or accuracy of the evaluation results obtained by this evaluation method is low.

[0074] In this embodiment, corresponding predicted image data can be generated based on the initial image data and the motion trajectory parameters predicted by the motion prediction model. The predicted image data is visual, equivalent to bringing the predicted motion trajectory parameters into the target scene. Therefore, the accuracy of the motion prediction model can be evaluated based on the motion trajectory of the target object in the predicted image data, thereby reducing the labor and time costs of model evaluation and improving the reliability or accuracy of the evaluation results. In addition, template image data can be first obtained based on the motion trajectory parameters, and then predicted image data can be generated based on the initial image data and the template image data. This can improve the accuracy and authenticity of the predicted image data, thereby further improving the reliability or accuracy of the evaluation results.

[0075] According to an embodiment of the present application, the image data generation method further includes: updating the terminal image data in the predicted image data to the initial image data, and repeating the steps of acquiring the template image data and generating the predicted image data.

[0076] The solution of this embodiment can be used to further evaluate the accuracy of the action prediction model.

[0077] In one example, after generating template image data based on motion trajectory parameters predicted by a motion prediction model, and generating predicted image data based on the initial image data and the template image data, the terminal image data in the predicted image data can be updated to the initial image data. In a new round of image data generation, the initial image data can be input into the motion prediction model to obtain motion trajectory parameters corresponding to the initial image data. Based on the motion trajectory parameters, corresponding template image data can be generated, and then new predicted image data can be generated based on the initial image data and the template image data. The steps of updating the terminal image data in the new predicted image data to the initial image data, obtaining motion trajectory parameters, generating template image data, and generating predicted image data are repeated until the predicted image data obtained in a certain round of image data generation process indicates that the target task has failed to execute or the target task has been executed.

[0078] In one example, if the predicted image data obtained in a certain round of image data generation process shows that the target task fails to be executed, it indicates that the accuracy of the action prediction model is low; if the predicted image data obtained in a certain round of image data generation process shows that the target task is completed, it indicates that the accuracy of the action prediction model is high.

[0079] In one example, the end image data may include image data at an end moment in a time period corresponding to the predicted image data.

[0080] In this embodiment, template image data is obtained based on motion trajectory parameters, predicted image data is generated based on the initial image data and the template image data, and the accuracy of the motion prediction model is evaluated based on the predicted image data. This can reduce the labor cost and time cost of model evaluation and improve the reliability or accuracy of the evaluation results.

[0081] According to one embodiment of the present application, the motion trajectory parameters include state parameters of the target object at each of multiple moments, and the state parameters include position parameters and posture parameters. Step 220 may include: projecting the template object into the template background image data according to the state parameters to obtain template image data.

[0082] In one example, the state parameters of the target object at each moment may include the position parameters and attitude parameters of the target object. For example, the position parameters may include three-dimensional position parameters corresponding to the x, y, and z coordinate axes, and the attitude parameters may include rotation angle parameters of the target object around the x, y, and z coordinate axes. The rotation angle parameters are also three-dimensional, so the state parameters can be six-dimensional parameters.

[0083] In one example, the motion trajectory parameters may include multiple state parameters, and the multiple state parameters may correspond one-to-one to multiple moments. In one example, the template object may be a symbol, graphic, or other carrier that can be used to represent the state parameters in the image data (template image data). In one example, the background image represented by the template background image data may include a solid color background image, such as a white background image, a black background image, or a background image of another color.

[0084] For example, Figure 3 As shown, the template object may be circular, and the template background image data may include a black background image. In one example, the size of the image represented by the template background image data may be consistent with the size of the image represented by the predicted image data. Furthermore, the size of the image represented by the template background image data may be consistent with or inconsistent with the size of the image represented by the initial image data.

[0085] In one example, projecting the template object onto the template background image data according to the state parameters to obtain the template image data is equivalent to projecting the template object onto the two-dimensional canvas according to the posture parameters. Figure 3 As shown, in the black background image, the center position of the circle can be determined by the position parameter in the state parameter, that is, the position parameter in the state parameter can be reflected by the center position of the circle. In one example, the posture parameter in the state parameter can be represented by the posture of multiple coordinate axes in the template image data, such as by the posture of three coordinate axes, where the three coordinate axes can represent the x, y, and z axes respectively. Figure 3 The starting point of each coordinate axis in can be connected to the center position of the template object.

[0086] Specifically, the state parameters at each moment can be mapped to the black background image through the template object, that is, the template object is projected into the template background image data according to the state parameters. Figure 3 The position of the center of the middle circle represents the projected position of the part (target object) on the robot used to perform the target task at the current viewing angle. Taking the robot's gripper as an example, in order to reflect the rotation angle (posture parameter) of the gripper at the current moment, the three-dimensional spatial position of the gripper (the three-dimensional spatial position of the end of the gripper) can be used as the center, and the x, y, and z coordinate axes of the end coordinate system of the gripper can be extended by a preset length, such as 10 cm, and the coordinates of the extension points obtained after the extension of the preset length are recorded. Then, the position of the gripper and the line connecting each extension point are projected to the current viewing angle to obtain three coordinate axes or three lines in the black background image. When the gripper rotates, the x, y, and z axes of the end coordinate system also change, so that the line connecting the gripper and the extension point projected onto the black background image also changes.

[0087] Here, the preset length is set to facilitate the user to see the three coordinate axes, so the preset length can be set according to actual needs.

[0088] In other examples, the template object may be a regular shape such as a rectangle, a triangle, or other irregular shapes. In one example, the shape of the template object may be different at different times, and the posture parameter in the state parameter is reflected by the shape of the template object.

[0089] In this embodiment, the template object is projected onto template background image data based on state parameters to generate template image data. This allows for visualization of the motion trajectory parameters, thereby concretizing or visualizing high-dimensional or abstract motion trajectory parameters. This reduces the gap between the motion trajectory parameters and the image data, converting them into pixel-level template image data, thereby improving the accuracy and authenticity of the predicted image data.

[0090] Furthermore, according to an embodiment of the present application, the target object includes a gripper of the robot, and the state parameter also includes a parameter of the opening and closing degree of the gripper.

[0091] In one example, the degree of opening and closing of the gripper during movement can be continuously changing. To improve the accuracy of the predicted image data, the gripper's opening and closing degree parameter can also be mapped to the template background image data. For example, the opening and closing degree parameter can be characterized by the color depth of the template object, that is, the color depth of the template object can represent the degree of opening and closing of the gripper. When the template object is circular, the color depth of the circle can represent the current degree of opening and closing of the gripper. For example, the darker the color, the tighter the gripper is closed, and the lighter the color, the looser the gripper is closed.

[0092] Further, if Figure 3 As shown, the black background image includes two circles. The circle of one color can represent the left gripper, and the circle of the other color can represent the right gripper, that is, the template objects corresponding to the left and right grippers can be distinguished by different colors.

[0093] In this embodiment, the opening and closing degree parameters of the gripper are represented by the color depth of the template object, and different grippers are represented by template objects of different colors, which can improve the visualization effect of the motion trajectory parameters and thereby improve the accuracy and authenticity of the predicted image data.

[0094] According to an embodiment of the present application, the motion trajectory parameters include absolute motion trajectory parameters, wherein step 230 may include: obtaining relative motion trajectory parameters corresponding to the absolute motion trajectory parameters; and generating predicted image data according to the initial image data, the template image data, and the relative motion trajectory parameters.

[0095] In one example, the absolute motion trajectory parameters may include the state parameters of the target object at each moment in a specific coordinate system. The relative motion trajectory parameters may include the difference between the state parameters at each moment and the state parameters at the previous moment. In one example, the difference state parameters at the initial moment may be an initial value, such as 0.

[0096] In one example, after obtaining the absolute motion trajectory parameters, the state parameters at the previous moment can be subtracted from the state parameters at the current moment to obtain the difference state parameters at the current moment. The relative motion trajectory parameters may include the difference state parameters at each moment.

[0097] In this embodiment, the initial image data may be equivalent to the initial environmental condition input, the template image data may be equivalent to the pixel-level condition input, and the relative motion trajectory parameters may be equivalent to the global condition input. Specifically, the absolute motion trajectory parameters may be used for projection to obtain the template image data, that is, this embodiment may visualize the absolute motion trajectory parameters. The relative motion trajectory parameters may be used to illustrate the method of constructing coordinates. The relative motion trajectory parameters may better reflect dynamic content, such as high-speed motion and motion speed, and therefore may express more information. Generating predicted image data based on the initial image data, the template image data, and the relative motion trajectory parameters may further improve the accuracy and authenticity of the predicted image data.

[0098] According to one embodiment of the present application, predicted image data is generated based on initial image data, template image data and relative motion trajectory parameters, including: generating intermediate image features based on the initial image data and template image data; performing cross-attention calculation on the intermediate image features and relative motion trajectory parameters to obtain predicted image data.

[0099] In one example, the intermediate image features may include image features before or during the generation of the predicted image data. The relative motion trajectory parameters may be interacted with the intermediate image features through a cross-attention module.

[0100] For example, the motion trajectory parameters include state parameters at 10 moments, so that predicted image data at 10 moments can be generated to obtain 10 frames of images. The predicted image data at 10 moments can be generated simultaneously. In the process of generating the predicted image data at 10 moments, 10 intermediate image features can be generated first, and then the predicted image data can be generated. In the process of generating the predicted image data at 10 moments, the overall relative motion trajectory parameters (such as including 10 difference state parameters) can be interacted with the overall intermediate image features (such as 10 intermediate image features), such as performing cross-attention calculation.

[0101] Specifically, the number of difference state parameters included in the relative motion trajectory parameters can be the same as the number of intermediate image features, and the difference state parameters included in the relative motion trajectory parameters can correspond one-to-one with the intermediate image features. The relative motion trajectory parameters can be represented by an action feature sequence, and the overall intermediate image features can be represented by an image feature sequence. Cross-attention calculation is performed between the two sequences.

[0102] For example, attention calculation can be performed based on the query vector group of the action feature sequence and the value vector of the image feature sequence to obtain the image feature sequence after interaction, and the predicted image data can be obtained based on the image feature sequence after interaction.

[0103] Alternatively, attention calculation can be further performed based on the query vector group of the image feature sequence and the value vector of the action feature sequence to obtain the action feature sequence after interaction. Predicted image data can be obtained based on the image feature sequence after interaction and the action feature sequence after interaction.

[0104] It should be understood that the specific process of performing cross-attention calculation on the intermediate image features and relative motion trajectory parameters to obtain predicted image data can be set according to actual needs.

[0105] Specifically, each feature sequence may correspond to multiple query vectors and multiple value vectors.

[0106] In one example, the query vector can be used to represent the degree to which the current element in a feature sequence pays attention to other elements, while the value vector can carry representation information for each element. Based on the query vector of the current element and the value vector of another element, the relationship between the current element and the other element can be calculated, thereby updating the representation vector of the other element. This process can be considered an attention calculation process.

[0107] In this embodiment, by performing cross-attention calculation on the intermediate image features and relative motion trajectory parameters to obtain predicted image data, the constraint ability of the motion trajectory conditions on the generated image data or pictures can be further enhanced, thereby further improving the accuracy and authenticity of the predicted image data.

[0108] According to an embodiment of the present application, the predicted image data is generated based on the initial image data and the template image data using an image generation model.

[0109] In one example, the image generation model can be a world model. A world model can refer to the internal representation that a robot uses to understand and predict its surroundings. This model can help the robot make decisions and plans when performing tasks, and can also be used to build realistic robot simulation environments for closed-loop evaluation and reinforcement learning. In one example, a world model can use a neural network, such as a variational autoencoder (VAE), a recurrent neural network (RNN), or a transformer, to encode and decode environmental states.

[0110] In one example, the world model may be a model built based on a diffusion model.

[0111] In this embodiment, initial image data and template image data can serve as inputs to the world model, or initial image data, template image data, and relative motion trajectory parameters can serve as inputs to the world model. The world model can output predicted image data that conforms to physical laws and scene interaction logic. The world model can be used to expand the training samples of the motion prediction model. Specifically, the predicted image data generated by the world model can be used to generate training samples, reducing the cost of acquiring training samples. Alternatively, the world model can be used to evaluate the accuracy of the motion prediction model. Specifically, the world model can be used as an evaluation tool, such as a simulator, to evaluate the accuracy of the motion prediction model in place of a simulation environment, thereby reducing model evaluation costs.

[0112] Figure 4 Shown is a flowchart of an image data generation method provided by another exemplary embodiment of the present application. Figure 4 The embodiment is Figure 2 For the example of embodiment, in order to avoid repetition, the same points can be referred to the description in the above embodiment, which will not be repeated here. Figure 4 As shown, the image data generating method may include the following contents.

[0113] 410: Acquire initial image data of the target scene, and acquire original motion trajectory parameters corresponding to the initial image data.

[0114] Specifically, the original motion trajectory parameters are used to characterize the trajectory of the target object during a target period, which is located after the current moment. For details about the target scene, target object, target period, initial image data, and original motion trajectory parameters, please refer to the relevant descriptions in the above embodiments. To avoid repetition, they are not further described here.

[0115] 420: Adjust the original motion trajectory parameters to obtain motion trajectory parameters.

[0116] Specifically, the specific contents of the motion trajectory parameters and the motion trajectory parameters obtained by adjusting the original motion trajectory parameters can be found in the relevant descriptions in the above embodiments, and will not be repeated here to avoid repetition.

[0117] 430: Generate template image data based on the motion trajectory parameters.

[0118] Specifically, the template image data and the specific content of generating the template image data can refer to the relevant description in the above embodiments, and will not be described again here to avoid repetition.

[0119] 440: Obtain relative motion trajectory parameters corresponding to the motion trajectory parameters, and generate predicted image data according to the initial image data, the template image data, and the relative motion trajectory parameters.

[0120] Specifically, the motion trajectory parameters here may be absolute motion trajectory parameters. The specific contents of generating the predicted image data may refer to the relevant description in the above embodiment, and will not be described again here to avoid repetition.

[0121] 450: Select sample image data from the predicted image data, and select sample motion trajectory parameters corresponding to the sample image data from the motion trajectory parameters.

[0122] Specifically, the time period corresponding to the sample motion trajectory parameters is after the time corresponding to the sample image data. The determination method of the sample image data and the sample motion trajectory parameters can be referred to the relevant description in the above embodiment, and will not be repeated here to avoid repetition.

[0123] 460: Using sample image data and sample motion trajectory parameters to train the robot motion prediction model to obtain a trained robot motion prediction model.

[0124] Specifically, the specific content of the action prediction model can be found in the relevant description in the above embodiment, and will not be repeated here to avoid repetition.

[0125] The predicted image data generated in this embodiment can be used to generate training samples for the motion prediction model.

[0126] Figure 5 Shown is a flowchart of an image data generation method provided by another exemplary embodiment of the present application. Figure 5 The embodiment is Figure 2 For the example of embodiment, in order to avoid repetition, the same points can be referred to the description in the above embodiment, which will not be repeated here. Figure 5 As shown, the image data generating method may include the following contents.

[0127] 510: Acquire initial image data of the target scene, and acquire motion trajectory parameters of the target object in the target scene.

[0128] Specifically, the initial image data is used to represent the state of the target object at the current moment, and the motion trajectory parameters are trajectory parameters predicted for the target object. The motion trajectory parameters can be trajectory parameters predicted for the target object based on the initial image data using a robot motion prediction model.

[0129] It should be understood that the relevant contents of the target scene, target object, initial image data, motion trajectory parameters and action prediction model can be found in the relevant descriptions in the above embodiments. To avoid repetition, they will not be repeated here.

[0130] 520: Generate template image data based on the motion trajectory parameters.

[0131] Specifically, the template image data and the specific content of generating the template image data can refer to the relevant description in the above embodiments, and will not be described again here to avoid repetition.

[0132] 530: Obtain relative motion trajectory parameters corresponding to the motion trajectory parameters, and generate predicted image data according to the initial image data, the template image data, and the relative motion trajectory parameters.

[0133] Specifically, the motion trajectory parameters here may be absolute motion trajectory parameters. The specific contents of generating the predicted image data may refer to the relevant description in the above embodiment, and will not be described again here to avoid repetition.

[0134] like Figure 6 As shown, the relative motion trajectory parameters can be obtained based on the absolute motion trajectory parameters, and the relative motion trajectory parameters can be input into the world model (based on the Transformer generation model). Template image data can be generated based on the motion trajectory parameters, corresponding to Figure 6 The three template images shown in the figure can be collectively referred to as a motion trajectory map. The motion trajectory map can be input into the world model. In one example, the motion trajectory map can be passed through a visual encoder to obtain corresponding image features or latent codes, and the image features or latent codes are input into the world model. The initial image data is also input into the world model. Figure 6 The image corresponding to the initial image data in shows the scene of the gripper grabbing the bottle. This image can be passed through the visual encoder to obtain the corresponding image features or latent codes to input into the world model. The world model can generate predicted image data based on the input data, and the predicted image data can be passed through the visual decoder to obtain the corresponding multi-frame images, such as Figure 6 The model generates an image showing the gripper putting the bottle outside the box. For example, the target task can be to put the bottle outside the box. Figure 6 The model-generated image shown confirms that the bottle was successfully placed outside the box, which confirms that the action prediction model has high accuracy.

[0135] 540: Evaluating the Accuracy of Robotic Motion Prediction Models Based on Predicted Image Data.

[0136] Specifically, the specific content of the action prediction model can be found in the relevant description in the above embodiment, and will not be repeated here to avoid repetition.

[0137] The predicted image data generated in this embodiment can be used to evaluate the accuracy of the robot motion prediction model.

[0138] The present application also provides a model training method, such as Figure 7 As shown, the model training method may include the following contents.

[0139] 710: Input the sample initial image data and the sample template image data into the model to obtain the sample predicted image data output by the model.

[0140] Specifically, the sample initial image data includes image data at the current moment in the sample image sequence, and the sample template image data includes a template object. The sample template image data is used to represent the process of the template object moving along a trajectory represented by the sample motion trajectory parameters. The sample motion trajectory parameters are used to represent the motion trajectory of the sample target object in the target image data in the sample image sequence, and the time period corresponding to the target image data is located after the current moment. The sample initial image data is similar to the initial image data described above.

[0141] In one example, the sample image sequence and sample motion trajectory parameters may be pre-collected data. For example, the sample image sequence and sample motion trajectory parameters may be obtained by controlling the movement of a robot via teleoperation. In another example, the robot may perform a target task using a trained motion prediction model, and the sample image sequence and sample motion trajectory parameters may be collected during the execution of the target task. In another example, the sample image sequence and sample motion trajectory parameters may be obtained using other suitable methods. The time axis of the sample image sequence may correspond to the time axis of the sample motion trajectory parameters.

[0142] In one example, the image sequence may include image data at multiple moments. The sample initial image data includes image data at a current moment in the sample image sequence, where the current moment may be a moment before the end moment of a time period corresponding to the sample image sequence.

[0143] The sample template image data is similar to the template image data described above. The specific generation process for the sample template image data can be found in the description of the aforementioned embodiments. For example, the sample template image data includes a template object, and the sample template image data is used to represent the process of the template object moving along a trajectory represented by the sample motion trajectory parameters. The sample motion trajectory parameters are used to represent the motion trajectory of the sample target object in the target image data.

[0144] In one example, the target image data is data from a portion of a sample image sequence, which may occur after the current moment. The sample target object, similar to the target object described above, may represent an object that needs to be observed and controlled within the scene covered by the sample initial image data. For example, the sample target object may include a part of a robot used to perform a target task. For details about the sample target object, please refer to the description of the target object in the above embodiments.

[0145] The sample predicted image data is similar to the predicted image data described above, and its details can be found in the relevant descriptions of the above embodiments. In one example, the time period corresponding to the sample predicted image data can be the same as the time period corresponding to the target image data.

[0146] In one example, the model may be the aforementioned image generation model, specifically, the model of the aforementioned image generation model before training.

[0147] In an example, the sample motion trajectory parameter may be a sample absolute motion trajectory parameter. The sample absolute motion trajectory parameter is similar to the above-mentioned absolute motion trajectory parameter. For details, reference may be made to the above-mentioned description of the absolute motion trajectory parameter.

[0148] Furthermore, the sample relative motion trajectory parameters can be derived based on the sample absolute motion trajectory parameters. The sample relative motion trajectory parameters are similar to the relative motion trajectory parameters described above. For details, please refer to the above description of the relative motion trajectory parameters. In one example, the sample initial image data, the sample template image data, and the sample relative motion trajectory parameters can be input into the model to obtain the sample predicted image data as output by the model.

[0149] 720: Train the model based on the difference between the sample predicted image data and the target image data to obtain an image generation model.

[0150] In one example, the sample predicted image data is generated by the model, while the target image data is originally collected. This means the target image data can serve as the annotated data. The greater the difference between the sample predicted image data and the target image data, the more the model needs further training; the smaller the difference, the better the model has learned. The trained model is called the image generation model.

[0151] The embodiment of the present application provides a model training method that can generate sample template image data based on sample motion trajectory parameters. The sample template image data is used to present the process of the template object moving along the trajectory represented by the sample motion trajectory parameters, so that the motion trajectory parameters can be visualized, that is, the high-dimensional or abstract motion trajectory parameters can be concretized or intuitively presented. Furthermore, the embodiment of the present application can generate sample predicted image data based on the sample initial image data and the sample template image data at the current moment, thereby reducing the gap between the motion trajectory parameters and the image data, that is, converting the sample motion trajectory parameters into pixel-level sample template image data. Training the model based on the sample template image data can improve the model's ability to learn fine information in the input data, that is, the sample template image data can be equivalent to pixel-level conditional input, which is more conducive to model learning and can improve the accuracy of the trained image generation model.

[0152] In some embodiments, the sample initial image data, the sample template image data, and the sample relative motion trajectory parameters can be input into the model to obtain the sample predicted image data output by the model. For example, the model can generate sample intermediate image features based on the sample initial image data and the sample template image data, and perform cross-attention calculation on the sample intermediate image features and the sample relative motion trajectory parameters to obtain sample predicted image data. It should be understood that this cross-attention calculation is similar to the cross-attention calculation between the intermediate image features and the relative motion trajectory parameters mentioned above. Through the cross-attention calculation, the constraint ability of the motion trajectory conditions on the generation of image data or pictures can be further enhanced, thereby further improving the accuracy of the trained image generation model.

[0153] Exemplary devices

[0154] Figure 8 FIG. 1 is a schematic diagram of the structure of an image data generating device provided by an exemplary embodiment of the present application. Figure 8 As shown, the image data generating device 800 includes: a first acquiring module 810 , a second acquiring module 820 , and a generating module 830 .

[0155] The first acquisition module 810 is configured to acquire initial image data of a target scene, wherein the initial image data is used to represent the state of a target object in the target scene at the current moment. The second acquisition module 820 is configured to acquire template image data, wherein the template image data is generated based on motion trajectory parameters, which are parameters of a predicted trajectory for the target object. The template image data includes the template object, and the template image data is used to represent the process of the template object moving along the trajectory represented by the motion trajectory parameters. The generation module 830 is configured to generate predicted image data based on the initial image data and the template image data, wherein the predicted image data is used to represent the process of the target object moving along the trajectory in the target scene.

[0156] Embodiments of the present application provide an image data generation device that can generate template image data based on predicted motion trajectory parameters for a target object in a target scene. The template image data is used to present the process of the template object moving along the trajectory represented by the motion trajectory parameters, thereby visualizing the motion trajectory parameters, that is, concretizing or intuitively presenting high-dimensional or abstract motion trajectory parameters. Furthermore, embodiments of the present application can generate predicted image data based on the initial image data and template image data at the current moment, thereby reducing the gap between the motion trajectory parameters and the image data, that is, converting the motion trajectory parameters into pixel-level template image data. Based on the pixel-level template image data and the initial image data at the current moment, the predicted image data is obtained, which can improve the success rate of predicted image data generation and the degree of consistency between the trajectory of the target object in the predicted image data and the trajectory represented by the motion trajectory parameters, thereby improving the accuracy and authenticity of the predicted image data. In addition, training an action prediction model based on this predicted image data can improve the reliability of the model's training samples and reduce the cost of obtaining training samples.

[0157] According to one embodiment of the present application, the first acquisition module 810 is further used to: obtain original motion trajectory parameters corresponding to the initial image data, wherein the original motion trajectory parameters are used to characterize the trajectory of the target object in the target time period, and the target time period is located after the current moment; adjust the original motion trajectory parameters to obtain motion trajectory parameters.

[0158] According to one embodiment of the present application, the image data generating device 800 also includes a training module 840, which is used to: select sample image data from the predicted image data, and select sample motion trajectory parameters corresponding to the sample image data from the motion trajectory parameters, wherein the time period corresponding to the sample motion trajectory parameters is after the moment corresponding to the sample image data; use the sample image data and the sample motion trajectory parameters to train the robot motion prediction model to obtain a trained robot motion prediction model.

[0159] According to an embodiment of the present application, the motion trajectory parameters are trajectory parameters predicted for the target object based on the initial image data using the robot motion prediction model. The image data generating device 800 also includes an evaluation module 850 for evaluating the accuracy of the robot motion prediction model based on the predicted image data.

[0160] According to an embodiment of the present application, the image data generating device 800 further includes an updating module 860 for updating the terminal image data in the predicted image data to the initial image data, and repeating the steps of acquiring the template image data and generating the predicted image data.

[0161] According to one embodiment of the present application, the motion trajectory parameters include state parameters of the target object at each of the multiple moments, where the state parameters include position parameters and posture parameters. The second acquisition module 820 is configured to project the template object onto the template background image data based on the state parameters to obtain the template image data.

[0162] According to an embodiment of the present application, the target object includes a gripper of the robot, and the state parameter further includes a parameter of the opening and closing degree of the gripper.

[0163] According to one embodiment of the present application, the opening and closing degree parameter is represented by the color depth of the template object in the template image data, and the posture parameter is represented by the posture of multiple coordinate axes in the template image data, and the starting point of each coordinate axis in the multiple coordinate axes is connected to the center position of the template object.

[0164] According to an embodiment of the present application, the motion trajectory parameters include absolute motion trajectory parameters, wherein the generation module 830 is used to: obtain relative motion trajectory parameters corresponding to the absolute motion trajectory parameters; and generate predicted image data based on the initial image data, the template image data, and the relative motion trajectory parameters.

[0165] According to one embodiment of the present application, the generation module 830 is used to: generate intermediate image features based on the initial image data and the template image data; perform cross-attention calculation on the intermediate image features and the relative motion trajectory parameters to obtain predicted image data.

[0166] According to an embodiment of the present application, the predicted image data is generated based on the initial image data and the template image data using an image generation model.

[0167] It should be understood that the operations and functions of the first acquisition module 810, the second acquisition module 820, the generation module 830, the training module 840, the evaluation module 850 and the update module 860 in the above embodiment can refer to the above Figure 2 or Figure 4 or Figure 5 To avoid repetition, the description of the image data generating method provided in the embodiment will not be repeated here.

[0168] Figure 9 The figure shows a schematic diagram of the structure of a model training device provided by an exemplary embodiment of the present application. Figure 9 As shown, the model training device 900 includes: an input module 910 and a training module 920.

[0169] Input module 910 is used to input sample initial image data and sample template image data into the model to obtain sample predicted image data output by the model, wherein the sample initial image data includes image data at the current moment in the sample image sequence, the sample template image data includes a template object, and the sample template image data is used to represent the process of the template object moving along a trajectory represented by sample motion trajectory parameters. The sample motion trajectory parameters are used to represent the motion trajectory of the sample target object in the target image data in the sample image sequence, and the time period corresponding to the target image data is located after the current moment. Training module 920 is used to train the model based on the difference between the sample predicted image data and the target image data to obtain an image generation model.

[0170] The embodiment of the present application provides a model training device that can generate sample template image data based on sample motion trajectory parameters. The sample template image data is used to present the process of the template object moving along the trajectory represented by the sample motion trajectory parameters, so that the motion trajectory parameters can be visualized, that is, the high-dimensional or abstract motion trajectory parameters can be concretized or intuitively presented. Furthermore, the embodiment of the present application can generate sample predicted image data based on the sample initial image data and the sample template image data at the current moment, thereby reducing the gap between the motion trajectory parameters and the image data, that is, converting the sample motion trajectory parameters into pixel-level sample template image data. Training the model based on the sample template image data can improve the model's ability to learn fine information in the input data, that is, the sample template image data can be equivalent to pixel-level conditional input, which is more conducive to model learning and can improve the accuracy of the trained image generation model.

[0171] It should be understood that the operations and functions of the input module 910 and the training module 920 in the above embodiment can refer to the above Figure 7 To avoid repetition, the description of the model training method provided in the embodiment will not be repeated here.

[0172] Figure 10 FIG2 is a block diagram of an electronic device 1000 for executing an image data generation method or a model training method according to an exemplary embodiment of the present application. The electronic device 1000 may be a server, a mobile device, a control device for a mobile device, a server interacting with a mobile device, or other device.

[0173] Reference Figure 10 The electronic device 1000 includes a processing component 1010, which further includes one or more processors, and a memory resource represented by a memory 1020 for storing instructions executable by the processing component 1010, such as an application. The application stored in the memory 1020 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1010 is configured to execute the instructions to perform the above-mentioned image data generation method or model training method.

[0174] The electronic device 1000 may further include a power supply component configured to perform power management of the electronic device 1000, a wired or wireless network interface configured to connect the electronic device 1000 to a network, and an input / output (I / O) interface. The electronic device 1000 may be operated based on an operating system stored in the memory 1020, such as Windows Server 2003. TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0175] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the above-mentioned electronic device 1000, enables the above-mentioned electronic device 1000 to execute an image data generation method or a model training method.

[0176] A computer program product includes a computer program. When the computer program is executed by a processor of a computer device, the computer device is enabled to execute the image data generation method or model training method provided in any of the above embodiments.

[0177] All of the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, and will not be described in detail here.

[0178] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0179] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program check codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] It should be noted that, in the description of this application, the terms "first," "second," "third," etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0185] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0186] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating image data, characterized in that: include: Acquiring initial image data of a target scene, wherein the initial image data is used to represent a state of a target object in the target scene at a current moment; Acquiring template image data, wherein the template image data is generated based on motion trajectory parameters, the motion trajectory parameters are trajectory parameters predicted for the target object, the template image data includes a template object, and the template image data is used to present a process of the template object moving along the trajectory represented by the motion trajectory parameters; Predicted image data is generated according to the initial image data and the template image data, wherein the predicted image data is used to present a process in which the target object moves along the trajectory in the target scene.

2. The image data generation method according to claim 1, wherein: Also includes: Acquiring original motion trajectory parameters corresponding to the initial image data, wherein the original motion trajectory parameters are used to characterize the trajectory of the target object in a target time period, and the target time period is located after the current moment; The original motion trajectory parameters are adjusted to obtain the motion trajectory parameters.

3. The image data generating method according to claim 1, wherein: Also includes: Selecting sample image data from the predicted image data, and selecting sample motion trajectory parameters corresponding to the sample image data from the motion trajectory parameters, wherein a time period corresponding to the sample motion trajectory parameters is after a time instant corresponding to the sample image data; The robot motion prediction model is trained using the sample image data and the sample motion trajectory parameters to obtain a trained robot motion prediction model.

4. The image data generating method according to claim 1, wherein: The motion trajectory parameters are trajectory parameters predicted for the target object based on the initial image data using a robot motion prediction model, wherein the image data generation method further includes: The accuracy of the robot motion prediction model is evaluated based on the predicted image data.

5. The image data generating method according to claim 4, wherein: Also includes: The terminal image data in the predicted image data is updated to the initial image data, and the steps of acquiring the template image data and generating the predicted image data are repeated.

6. The image data generating method according to claim 1, wherein: The motion trajectory parameters include state parameters of the target object at each of a plurality of moments, wherein the state parameters include position parameters and posture parameters. Wherein, the obtaining of template image data includes: The template object is projected onto template background image data according to the state parameters to obtain the template image data.

7. The image data generating method according to claim 6, wherein: The target object includes a gripper of the robot, and the state parameter also includes an opening and closing degree parameter of the gripper.

8. The image data generating method according to claim 7, wherein: The opening and closing degree parameter is represented by the color depth of the template object in the template image data, and the posture parameter is represented by the posture of multiple coordinate axes in the template image data, and the starting point of each coordinate axis in the multiple coordinate axes is connected to the center position of the template object.

9. The image data generating method according to claim 1, wherein: The motion trajectory parameters include absolute motion trajectory parameters, The step of generating predicted image data according to the initial image data and the template image data includes: Obtaining relative motion trajectory parameters corresponding to the absolute motion trajectory parameters; The predicted image data is generated according to the initial image data, the template image data and the relative motion trajectory parameters.

10. The image data generating method according to claim 9, wherein: Generating the predicted image data according to the initial image data, the template image data, and the relative motion trajectory parameters includes: generating intermediate image features according to the initial image data and the template image data; A cross-attention calculation is performed on the intermediate image features and the relative motion trajectory parameters to obtain the predicted image data.

11. The image data generating method according to any one of claims 1 to 10, characterized in that: The predicted image data is generated based on the initial image data and the template image data using an image generation model.

12. A model training method, characterized in that: include: Inputting sample initial image data and sample template image data into a model to obtain sample predicted image data output by the model, wherein the sample initial image data includes image data at a current moment in a sample image sequence, the sample template image data includes a template object, the sample template image data is used to present a process of the template object moving along a trajectory represented by sample motion trajectory parameters, the sample motion trajectory parameters are used to represent the motion trajectory of a sample target object in target image data in the sample image sequence, and the time period corresponding to the target image data is located after the current moment; The model is trained based on the difference between the sample predicted image data and the target image data to obtain an image generation model.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor, Wherein, the processor is used to execute the image data generation method described in any one of claims 1 to 11 or the model training method described in claim 12.

14. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which is used to execute the image data generation method according to any one of claims 1 to 11 or the model training method according to claim 12.

15. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by the processor of a computer device, the computer device is enabled to execute the image data generation method described in any one of claims 1 to 11 or the model training method described in claim 12.

Images