Method and apparatus for cross-object type training data generation based on differentiable particleization

Abstract

The application provides a cross-object-type training data generation method and device based on differentiable particleization. The method comprises the following steps: acquiring initial scene data collected by a robot performing an operation task; performing unified particleization modeling on the initial scene data based on a differentiable particle engine to generate a first particle system model corresponding to a first operation object; adjusting the first particle system model to obtain a second particle system model adapted to a second operation object; performing full-time sequence dynamics simulation on the second particle system model to generate simulation particle motion data of the second operation object under the operation task; and integrating the simulation particle motion data and the initial scene data to generate a cross-object-type robot operation training data set. The application can avoid collecting data for each object separately, significantly reduce data collection costs and workload, and realize efficient construction of cross-object-type training data in the robot operation field.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology or image data processing technology, and in particular to a method and apparatus for generating cross-object type training data based on differentiable particles. Background Technology

[0002] In the field of intelligent robot operation, robot training requires support from a large amount of operational data covering various types of objects. In real-world scenarios, data collection, annotation, and processing must be carried out separately for each type of object. Current technologies typically employ a method of collecting and constructing dedicated training data for each object separately to meet the training needs of robot operation.

[0003] In existing technologies, data collection and annotation are usually carried out separately for each type of manipulated object, such as rigid, flexible, and liquid. Moreover, data collection in real-world scenarios is easily affected by factors such as environmental complexity and differences in object shape. This not only makes the overall data collection workload huge, the cycle lengthy, and the cost high, but also results in the operation data of different scenarios and different objects being independent and difficult to reuse, making it impossible to efficiently build a unified training dataset covering multiple types of objects.

[0004] How to reduce data collection costs and workload while efficiently generating training data for robots across object types has become a key issue that urgently needs to be addressed. Summary of the Invention

[0005] This application provides a method and apparatus for generating cross-object type training data based on differentiable particleization. By performing unified particleization representation modeling on the initial scene data of the robot operation scenario, a particle system model corresponding to the first operation object is obtained. The model is then adjusted to adapt to different types of second operation objects. Full-time dynamic simulation is performed on the adjusted particle system model to generate simulated particle motion data. Finally, the simulation data is fused with the initial scene data to generate a cross-object type robot operation training dataset. This avoids collecting data separately for each object, significantly reduces data collection costs and workload, and achieves efficient construction of cross-object type training data.

[0006] In a first aspect, embodiments of this application provide a method for generating cross-object type training data based on differentiable particles, the method comprising: Acquire initial scene data collected by the robot in the robot operation scene when the robot performs operation tasks. The initial scene data includes multi-view image data, robot action sequence, and first type label corresponding to the first operation object. Based on a differentiable particle engine, the initial scene data is modeled using a unified particle representation to generate a first particle system model corresponding to the first manipulated object. The first particle system model is adjusted to obtain a second particle system model adapted to the second manipulated object; A full-time dynamics simulation is performed on the second particle system model to generate simulated particle motion data of the second manipulated object under the operation task. The simulated particle motion data is integrated with the initial scene data to generate a robot operation training dataset that spans multiple object types.

[0007] Secondly, embodiments of this application provide a cross-object type training data generation device based on differentiable particles. The device includes an acquisition module, a model generation module, a simulation module, and a data processing module, wherein: The acquisition module is used to acquire initial scene data collected by the robot in the robot operation scene when the robot performs operation tasks. The initial scene data includes multi-view image data, robot action sequence, and a first type label corresponding to the first operation object. The model generation module is used to perform unified particle-based representation modeling on the initial scene data based on a differentiable particle engine, and generate a first particle system model corresponding to the first operated object; and adjust the first particle system model to obtain a second particle system model adapted to the second operated object. The simulation module is used to perform full-time dynamics simulation on the second particle system model and generate simulated particle motion data of the second manipulated object under the operation task. The data processing module is used to integrate the simulated particle motion data with the initial scene data to generate a robot operation training dataset that spans multiple object types.

[0008] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing steps in any method of the first aspect of this application.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in any method of the first aspect of this application.

[0010] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in any method of the first aspect of this application. The computer program product may be a software installation package.

[0011] By implementing the embodiments of this application, a unified particle-based representation model can be performed on the initial scene data of the robot operation scenario to obtain a particle system model corresponding to the first operation object. This model is then adjusted to adapt to different types of second operation objects. A full-time dynamic simulation is performed on the adjusted particle system model to generate simulated particle motion data. Finally, the simulated data is fused with the initial scene data to generate a cross-object type robot operation training dataset. Compared to existing methods that collect and construct dedicated training data for each object separately, this application avoids collecting data for each object separately, significantly reducing data collection costs and workload, and achieving efficient construction of cross-object type training data. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a system architecture diagram of a cross-object type training data generation system provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a simulation layer provided in an embodiment of this application; Figure 3 This is an application scenario diagram of a cross-object type training data generation system provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 5 This is a flowchart illustrating a cross-object type training data generation method based on differentiable particleization provided in an embodiment of this application. Figure 6 This is a schematic diagram of a process for constructing a second particle system model provided in an embodiment of this application; Figure 7 This is a schematic flowchart of a dynamic simulation of a second particle system model provided in an embodiment of this application; Figure 8This is a block diagram of the functional modules of a cross-object type training data generation device based on differentiable particleization provided in an embodiment of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0015] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0016] It should be understood that the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document indicates that the preceding and following related objects are in an "or" relationship. In the embodiments of this application, "multiple" refers to two or more.

[0017] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0018] In this application embodiment, "connection" refers to various connection methods such as direct connection or indirect connection to realize communication between devices. This application embodiment does not limit this in any way.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] In the field of intelligent robot operation, robot training requires support from a large amount of operational data covering various types of objects. In real-world scenarios, data collection, annotation, and processing must be carried out separately for each type of object. Current technologies typically employ a method of collecting and constructing dedicated training data for each object separately to meet the training needs of robot operation.

[0021] In existing technologies, data collection and annotation are usually carried out separately for each type of manipulated object, such as rigid, flexible, and liquid. Moreover, data collection in real-world scenarios is easily affected by factors such as environmental complexity and differences in object shape. This not only makes the overall data collection workload huge, the cycle lengthy, and the cost high, but also results in the operation data of different scenarios and different objects being independent and difficult to reuse, making it impossible to efficiently build a unified training dataset covering multiple types of objects.

[0022] How to efficiently generate training data for robots across object types while reducing data collection costs and workload has become a key issue that urgently needs to be addressed.

[0023] To address the aforementioned issues, this application provides a method and apparatus for generating cross-object type training data based on differentiable particle modeling. The method involves acquiring initial scene data collected during robot operation tasks, performing unified particle-based representation modeling on the initial scene data using a differentiable particle engine to generate a first particle system model corresponding to a first operating object, adjusting the first particle system model to obtain a second particle system model adapted to a second operating object, performing full-time dynamics simulation on the second particle system model to generate simulated particle motion data, and then integrating the simulated particle motion data with the initial scene data to generate a cross-object type robot operation training dataset.

[0024] It is evident that by using a unified particle representation for different objects and generating training data through cross-type simulation, it is not necessary to collect data separately for each object, which effectively reduces data collection costs and workload, and enables the efficient construction of cross-object type training data, providing high-quality and highly generalizable training datasets for robot operation.

[0025] For easier understanding, please refer to Figure 1 , Figure 1This is a system architecture diagram of a cross-object type training data generation system provided in this application embodiment. The system includes a perception layer, a modeling layer, a simulation layer, and an integration layer. The modules at each level work together to achieve efficient generation of cross-object type robot operation training data based on differentiable particles.

[0026] The perception layer acquires raw image data and environmental point cloud data of the robot's operation scene through multi-view RGBD cameras, and acquires robot action sequences (including joint angles, movement speed, and execution commands) through robot joint encoders and motion controllers. Simultaneously, an image recognition module, combined with an object type database, identifies and obtains the first type label and operation task identification information corresponding to the operated object. Data frame alignment is achieved based on hardware timestamps to ensure temporal consistency of image data, action data, and label information. Then, distortion correction and Gaussian noise reduction are performed on the raw image data to obtain standardized image data; downsampling, outlier removal, and foreground segmentation are performed on the environmental point cloud data to extract the effective point cloud of the first operated object; Kalman filtering is performed on the robot action sequence to remove motion jitter interference, resulting in a smooth and reliable robot action sequence; finally, the standardized image data, optimized robot action sequence, first type label, and task identification information are integrated to form clean and standardized initial scene data.

[0027] Among them, the modeling layer is based on a differentiable particle engine. It performs unified particle-based representation processing on the initial scene data output by the perception layer, extracts the geometric features and physical correlation information of the manipulated object, and constructs the particle system model corresponding to the first manipulated object. This realizes unified representation modeling of different types of objects and lays the foundation for cross-object type adaptation.

[0028] For easier understanding, please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of a simulation layer provided in an embodiment of this application. The simulation layer includes a type adaptation module, a parameter configuration module, a dynamics solution module, and a data augmentation module.

[0029] The type adaptation module obtains the type label of the second object being operated on, and adjusts and adapts the attributes of the particle system model generated by the modeling layer based on a preset object type sample database to obtain a particle system model adapted to the second object being operated on, thereby realizing model transfer across object types.

[0030] The parameter configuration module quantizes and maps the corresponding dynamic simulation parameters based on the physical properties of the second manipulated object, adaptively configures the constraint parameters and collision parameters required for the simulation, and selects a dynamic solution algorithm that matches the object type to ensure the accuracy and rationality of the simulation process.

[0031] The dynamics solution module inputs the robot action sequence and the adapted particle system model into the differentiable dynamics simulation model, performs full-time iterative solution through a preset solution algorithm, records the particle motion state, and generates original particle motion trajectory data.

[0032] The data augmentation module perturbs and enhances physical properties within a preset range, generates multiple sets of differentiated simulation parameters, performs multiple augmentation simulations based on different parameters, generates multiple sets of augmented particle motion trajectory data, and enriches the diversity of training data.

[0033] It is evident that through the collaborative work of various modules in the simulation layer, model adaptation and simulation data generation across object types can be completed efficiently without the need to collect real data separately for each object, greatly improving the efficiency of training data generation and providing core support for the construction of training datasets across object types.

[0034] The integration layer verifies the validity of the simulated particle motion data output by the simulation layer, removes abnormal data, and then integrates the verified simulation data with the initial scene data collected by the perception layer to complete the data standardization process. Finally, it generates a robot operation training dataset that crosses object types, thus completing the construction and optimization of the training data.

[0035] It is evident that through the coordinated efforts of the perception layer, modeling layer, simulation layer, and integration layer, the entire process from raw data acquisition, unified modeling, cross-type simulation to data integration is automated. This reduces data acquisition costs and workload while efficiently constructing high-quality cross-object type training datasets, providing ample data support for robot operation training.

[0036] For easier understanding, please refer to Figure 3 , Figure 3 This is an application scenario diagram of a cross-object type training data generation system provided in this application embodiment. In this scenario, a robot performs an operation task on a first manipulated object. The cross-object type training data generation system collects multi-view image data, robot action sequences, and the first type label and operation task identification information of the first manipulated object from the scene through a perception layer, forming initial scene data. Then, through collaborative computation of the perception layer, modeling layer, simulation layer, and integration layer, the system completes data preprocessing, differentiable particle modeling, cross-object type model adaptation, full-time dynamic simulation, simulation data augmentation and validity verification, data association integration and standardization, and finally outputs a cross-object type robot operation training dataset. This dataset, based on a single operation task, generates operation training samples adapted to multiple different types of manipulated objects. It can be directly used for training robot operation-related models, achieving low-cost and high-efficiency expansion of training data and providing sufficient data support for robot cross-object type generalization operations.

[0037] The following is combined Figure 4 The electronic devices in the embodiments of this application will be described. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 4 As shown, the electronic device includes one or more processors, a memory, a communication interface, and one or more programs. The processor is connected to the memory and the communication interface via an internal communication bus.

[0038] The processor can be used for: Acquire initial scene data collected by the robot in the robot operation scene when the robot performs operation tasks. The initial scene data includes multi-view image data, robot action sequence, and first type label corresponding to the first operation object. Based on a differentiable particle engine, the initial scene data is modeled using a unified particle representation to generate a first particle system model corresponding to the first manipulated object. The first particle system model is adjusted to obtain a second particle system model adapted to the second manipulated object; A full-time dynamics simulation is performed on the second particle system model to generate simulated particle motion data of the second manipulated object under the operation task. The simulated particle motion data is integrated with the initial scene data to generate a robot operation training dataset that spans multiple object types.

[0039] The one or more programs are stored in the aforementioned memory and configured to be executed by the aforementioned processor, and the one or more programs include instructions for performing any step in the above method embodiments.

[0040] The processor can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, cells, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication unit can be a communication interface, transceiver, transceiver circuit, etc., and the storage unit can be a memory.

[0041] The memory can be volatile or non-volatile, or a combination of both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0042] It is understood that the electronic device may include more or fewer structural elements than those shown in the block diagram above, such as a power module, physical buttons, a Wi-Fi module, a speaker, a Bluetooth module, sensors, a display module, etc., without limitation. It is understood that the electronic device may incorporate elements such as... Figure 1 The system architecture described above.

[0043] After understanding the software and hardware architecture of this application, the following will be combined with... Figure 5 This application describes a method for generating cross-object type training data based on differentiable particleization in an embodiment of the present application. Figure 5 This is a flowchart illustrating a cross-object type training data generation method based on differentiable particleization provided in this application embodiment, specifically including the following steps: Step S501: Obtain initial scene data collected by the robot in the robot operation scene when performing operation tasks. The initial scene data includes multi-view image data, robot action sequence, and first type label corresponding to the first operation object.

[0044] In a specific embodiment, the robot performs an operation task on a first manipulated object in the operation scenario (such as grasping a rigid block, folding flexible fabric, pouring liquid containers, etc.). On the hardware side, scene images and environmental point clouds are acquired through a multi-view RGBD camera, and robot motion sequences are acquired synchronously through robot joint encoders and motion controllers. Then, hardware timestamps are used to align data frames from devices such as multi-view cameras and joint encoders to ensure the temporal consistency of multi-source data such as images and motions. On the algorithm side, distortion correction, Gaussian noise reduction, and foreground segmentation are performed on the acquired raw image data to remove background interference, and Kalman filtering and smoothing are performed on the robot motion sequence to eliminate motion jitter noise.

[0045] Initial scene data includes, but is not limited to: multi-view image data, robot motion sequences, and the first type label of the first manipulated object; additionally, operation task identification information, environmental point cloud data, sensor timestamp information, robot joint state data, etc., can be added according to the needs of the actual application scenario.

[0046] Step S502: Based on the differentiable particle engine, perform unified particle-based representation modeling on the initial scene data to generate the first particle system model corresponding to the first manipulated object.

[0047] Among them, the differentiable particle engine is a differentiable computational framework for high-precision physical simulation and parameter optimization. Its core is to discretize the three-dimensional scene and target object into a large number of particle units with physical properties. The particles are used as the basic carrier to represent the physical states of the object, such as position, velocity, mass, stiffness, and viscosity. The interaction, motion evolution and deformation process between particles are described through differentiable physical constraints, force rules and dynamic iterative formulas, so as to realize unified modeling and simulation of different types of objects such as rigid bodies, flexible bodies, cloth and fluids.

[0048] Compared to traditional physics simulation engines, the core feature of the differentiable particle engine lies in the complete differentiability of the entire physics solution process. It supports gradient-based backpropagation and automatic parameter optimization, enabling fine-tuning and adaptive updates of dynamic parameters, particle distribution, and constraint strength directly during simulation iterations. This engine can stably reproduce the rigid displacement and rotation of rigid objects, as well as accurately simulate the deformation, flow, collision, and contact behaviors of flexible objects, cloth, fluids, and other objects with large deformations and no fixed topology. It provides underlying computational support for unified representation, dynamic simulation, parameter optimization, and robot operation data generation across different object types.

[0049] In one possible embodiment, the step of performing unified particle-based representation modeling on the initial scene data based on a differentiable particle engine to generate a first particle system model corresponding to the first manipulated object includes: performing foreground segmentation and correction registration processing on the multi-view image data to obtain standardized image data; extracting depth point cloud data of the first manipulated object from the standardized image data; generating an initial particle set based on a preset particle generation function and the depth point cloud data; optimizing the particle distribution in the initial particle set using K-nearest neighbor uniformity constraints to obtain a first particle set densely covering the surface and interior of the first manipulated object, the first particle set containing the 3D position coordinates of multiple particles; and constructing a first particle system model corresponding to the first manipulated object based on the first type label, a pre-built object type sample database, and the first particle set.

[0050] The foreground segmentation and correction registration process for the image mainly involves first separating the foreground region of the first object from the multi-view image using semantic segmentation or threshold segmentation algorithms to remove irrelevant background interference; then combining the in-camera distortion coefficients to complete image distortion correction; and finally registering and aligning images from different viewpoints using multi-view geometric constraints to eliminate geometric deviations caused by viewpoint differences, thereby obtaining standardized image data with geometric accuracy and clear targets.

[0051] Specifically, using depth point cloud data as input, an initial particle set is generated using the particle generation function from the Particle-based Object Manipulation (PROMPT) method, as shown below: G: V; Where G is a preset particle generation function used to convert a 3D depth point cloud into a 3D particle representation. The depth point cloud data of the first manipulated object is extracted from the standardized image. V is the generated initial particle set, which is represented as the three-dimensional particle coordinate set V=(x,y,z), where x and y are the horizontal and vertical coordinates corresponding to the particle's projection on the image plane, and z is the particle's depth coordinate, which together constitute the particle's position in three-dimensional space.

[0052] Furthermore, to ensure that particles uniformly and completely cover the surface and interior of the object, the initial particle distribution is optimized using the K-nearest neighbor Chamfer loss (K-nearest neighbor chamfer loss). This loss is composed of bidirectional nearest neighbor distances, and the specific formula is as follows: ; in, The K-nearest neighbor Chamfer loss value is used to measure the distance between the particle reprojection point set and the image sampling point set, optimizing the particle distribution uniformity. For the 2D point set of particles reprojected onto the image plane, For all The set that constitutes this is the complete set of 2D points that the particles reproject onto the image plane. Let M be a set of 2D points uniformly sampled from a masked image of objects of different types. The set constitutes the entire set of 2D points uniformly sampled from the cross-type object mask image, where K is the number of nearest neighbors (here, 100) used to determine the reference nearest neighbor range for each point. For the corresponding K nearest neighbors in the point set (if ,but Let M be the K nearest neighbors; if ,but for (K nearest neighbors) The L2 norm squared between two points is used to penalize sparse neighbor points through bidirectional summation, ensuring that the particle set completely covers the surface and interior of different types of objects.

[0053] It can be seen that by minimizing the above loss function and iteratively updating the particle positions, a first set of particles with uniform distribution and geometric alignment is finally obtained.

[0054] In one possible embodiment, constructing the first particle system model corresponding to the first manipulated object based on the first type label, a pre-built object type sample database, and the first particle set includes: obtaining a first set of physical attributes and a set of inter-particle connections corresponding to the first manipulated object from the object type sample database based on the first type label; the object type sample database pre-stores multiple sets of physical attributes and multiple inter-particle connections corresponding to multiple different types of objects, including fixed connections for rigid objects, elastic constraints for flexible objects, and no-connection connections for liquids; adding particle-level physical attributes to each particle in the first particle set based on the first set of physical attributes, including velocity, mass, coefficient of friction, stiffness, and elastic modulus; and integrating the first particle set, the particle-level physical attributes, and the first inter-particle connections to obtain the first particle system model.

[0055] Among them, the object type sample database is a pre-built standardized reference library. It stores the physical attribute sets corresponding to various object types and the connection relationships between particles. The physical attribute parameters are collected by physical measurement equipment such as force gauges and viscometers. Then, combined with the robot's standard operation tasks and the gradient backpropagation of the differentiable particle engine, the measured parameters are calibrated and optimized online. Finally, standard physical attribute parameters adapted to the simulation scenario are formed. They can be quickly matched and called according to the object type label, providing unified and accurate basic parameter support for particle system modeling.

[0056] Specifically, the specific parameters of the physical property set differ for different types of manipulated objects. For example, the physical property set of rigid objects focuses on rigidity-related parameters such as stiffness and friction coefficient, while flexible objects focus on flexibility-related parameters such as elastic modulus and flexible stiffness, and liquids focus on fluid-related parameters such as particle mass and viscosity coefficient. The set is used as a whole to define the physical properties of the object and ensure that the particle system model can accurately reproduce the motion and deformation laws of the object.

[0057] In a specific embodiment, particle-level physical attributes are added to each particle in the first particle set according to the first set of physical attributes, including: determining the mass of a single particle and assigning it to the corresponding particle based on the object density and the volume of space occupied by the particle in the set of physical attributes; configuring local structural stiffness for the particle based on the stiffness parameter in the set; setting the particle deformation recovery capability based on the elastic modulus; configuring the friction characteristics between the particle and the external environment and other particles based on the friction coefficient; and simultaneously assigning the initial motion velocity in the set to each particle, so that each particle fully carries the mass, velocity, friction coefficient, stiffness and elastic modulus matching the object type, thereby realizing the fine configuration of particle-level physical attributes.

[0058] Among these, the inter-particle connections are the core rules defining the interaction patterns of particles in the first particle set. They directly determine the simulation effects of the particle system model on different types of objects, corresponding one-to-one with the type of object being manipulated. Specifically, the fixed connections corresponding to rigid objects fix the relative positions and spacing between particles, ensuring that the overall particle set does not deform; the elastic constraints corresponding to flexible objects allow for a certain range of relative displacement between particles by simulating elastic forces; and the unconnected connections corresponding to liquids mean that particles have no fixed constraints and interact only through collisions and pressure fields.

[0059] For example, by combining the configuration process of particle-level physical properties, a single particle can be represented as a complete physical state including position, motion, mass, and material properties. A single particle can be represented as: v=(x,y,z,X,Y,Z,m,k,E,μi); Where v represents the state of a single particle, (x,y,z) are the 3D position coordinates of the particle, X,Y,Z are the velocity components of the particle in the three coordinate axes, m is the particle mass, which is obtained by uniformly distributing the object density and the particle's spatial volume, k is the stiffness parameter of the particle, E is the elastic modulus of the particle, which is used to characterize the particle's deformation recovery ability, and μi is the particle-level friction coefficient, which is used to characterize the frictional characteristics between the particle and the outside world and other particles. Furthermore, by integrating the first particle set, particle-level physical properties, and inter-particle connections, the overall state of the first particle system model is constructed. The first particle system model can be represented as: S=(V,R,μ); Where S represents the complete system state of the first particle system model, V represents the first particle set, which consists of all individual particles v, R represents the connection relationship between the first particles, which is divided into the following categories according to the object type: fixed connection relationship R_fixed for rigid objects, elastic constraint connection relationship R_spring for flexible objects, and no connection relationship R_none for liquids, μ represents the system-level friction coefficient, which is determined by the first physical property set and serves as a unified representation of the friction characteristics of the overall object. It is consistent with the particle-level friction coefficient μi, that is, under normal circumstances, the system-level friction coefficient μ and the particle-level friction coefficient μi are equal in value.

[0060] As can be seen, in this embodiment, by forming a first particle system model that includes particle geometric position, motion state, mass, stiffness, elastic modulus, friction characteristics and inter-particle interaction constraints, the physical properties, motion laws and deformation behavior of different types of objects can be accurately reproduced.

[0061] Step S503: Adjust the first particle system model to obtain a second particle system model adapted to the second manipulated object.

[0062] Specifically, the core idea of ​​cross-type adjustment of the first particle system model is based on the principle of cross-type object physical property transfer. Combined with the type difference of the second operating object, the connection relationship and physical property parameters between particles in the model are adjusted. There is no need to rebuild the particle set. By adapting to the mechanical properties of different types of objects, a second particle system model adapted to the second operating object can be quickly obtained.

[0063] For easier understanding, please refer to Figure 6 , Figure 6 This is a flowchart illustrating the construction of a second particle system model according to an embodiment of this application. The step of adjusting the first particle system model to obtain a second particle system model adapted to the second manipulated object includes: A1. Obtain the second type label of the second operation object; A2. Obtain the set of second physical attributes and the connection relationship between second particles corresponding to the second operated object from the object type sample database according to the second type label; A3. Update the particle-level physical properties in the first particle system model according to the second set of physical properties, and update the first inter-particle connection relationship in the first particle system model according to the second inter-particle connection relationship; A4. Verify the consistency between the updated first particle system model and the shape of the second manipulated object based on the preset point cloud distance matching algorithm; A5. After the shape consistency verification is passed, the updated first particle system model is determined as the second particle system model of the second manipulated object.

[0064] Specifically, based on the second type of label, the physical attribute set of the object to be adapted to the second operation, as well as the corresponding inter-particle connection relationship, are retrieved from the pre-built and calibrated optimized object type sample database to provide standard parameters for model updates.

[0065] Specifically, based on the connection relationship between the second set of physical properties and the second particle, the parameters and constraints of the first particle system model are replaced to achieve cross-type conversion. The key conversion rules and formulas are as follows: First, update the inter-particle connections: If the first manipulated object is a rigid rod and the second manipulated object is a flexible rope, then the fixed connection is replaced with an elastic connection, and the interparticle forces are constructed based on Hooke's Law formula below: F= kΔx; Where F is the spring force between particles, k is the spring constant, which is determined according to the rigidity characteristics of the flexible rope, ranging from 100 to 1000 N / m, and Δx is the difference between the actual spacing and the initial spacing between particles, used to characterize tensile / compression deformation. If the first manipulated object is a liquid and the second manipulated object is cloth, then the unconnected relationship between particles is reconstructed into a regular neighborhood mass-spring system. That is, two types of elastic constraints are established between adjacent particles, so that loose particles form a stable cloth sheet structure. Among them, the tensile constraint force and shear constraint force between particles are constructed based on the following formulas: F_stretch=k_s(||Δx|| L_0); Where F_stretch is the inter-particle stretching constraint force, used to limit the excessive stretching of cloth particles along the connecting line direction, k_s is the stretching spring coefficient, ||Δx|| is the current actual distance between adjacent particles, and L_0 is the initial reference length of adjacent particles; F_shear=k_sh(||Δx_diag|| L_d0); Where F_shear is the inter-particle shear constraint force, used to suppress the slippage and misalignment of particles in the diagonal direction of the fabric, avoid distortion and twisting of the fabric surface, and maintain the stability of the in-plane structure of the fabric; k_sh is the shear spring coefficient; ||Δx_diag|| is the current actual distance between diagonally adjacent particles; and L_d0 is the initial reference length of diagonally adjacent particles. Then, the particle-level physical properties are updated: based on the second set of physical properties, the particle mass m, stiffness k, elastic modulus E, friction coefficient μi and other parameters are updated uniformly to make the particle physical properties consistent with the second type of object being operated on. Finally, the consistency between the updated first particle system model and the shape of the second manipulated object is verified based on a preset point cloud distance matching algorithm. Specifically, the deviation between the updated particle point cloud and the target shape of the second manipulated object is calculated to ensure that the shape of the particle set is not distorted. The distance calculation formula used for shape consistency verification is as follows: d_avg= ; Where d_avg is the average Euclidean distance of the particle point cloud, used to quantify the deviation between the updated model and the shape of the target object in the second operation. When d_avg is less than a preset threshold, the shape consistency check is considered passed. N is the total number of particles. Let i be the updated 3D coordinates of the i-th particle. The coordinates of the point cloud corresponding to the shape of the second operation object target.

[0066] It is understood that this application only provides one method for verifying the shape consistency. Other methods, including but not limited to point cloud Hausdorff distance verification, nearest neighbor distance deviation verification, Chamfer distance (bevel distance) verification, etc., can also be used. As long as the accurate determination of the shape consistency between the updated particle system model and the second manipulated object can be achieved, they are all within the protection scope of this application.

[0067] As can be seen, in this embodiment, different types of manipulated objects can be quickly adapted simply by updating the physical property parameters and inter-particle connection constraints. This ensures a high degree of matching between the model and the shape and physical characteristics of the second manipulated object, while also significantly improving modeling efficiency. It effectively solves the problem of cumbersome and time-consuming traditional cross-type modeling processes, and realizes efficient reuse and rapid switching of simulation models for different types of objects.

[0068] Step S504: Perform a full-time dynamics simulation on the second particle system model to generate simulated particle motion data of the second manipulated object under the operation task.

[0069] Among them, the full-time dynamics simulation takes the force balance and motion equations of the particle system as the core principle, and continuously iterates the solution of the second particle system model throughout the entire operation task cycle, updating the position, velocity and interaction force state of each particle at each moment, so as to completely reproduce the real motion and deformation process of the object in the task process, and finally output the time-continuous simulation particle motion data.

[0070] For easier understanding, please refer to Figure 7 , Figure 7 This is a flowchart illustrating a dynamic simulation of a second particle system model provided in an embodiment of this application. The step of performing a full-time dynamic simulation of the second particle system model to generate simulated particle motion data of the second manipulated object under the operational task includes: B1. Determine the initial dynamic parameters based on the second set of physical properties; B2. Initialize and configure the preset differentiable dynamic simulation model according to the initial dynamic parameters, and configure a dynamic solution algorithm that matches the second manipulated object for the differentiable dynamic simulation model; B3. Input the robot action sequence and the second particle system model into the differentiable dynamics simulation model, and obtain the simulated particle motion data output by the differentiable dynamics simulation model through full-time iterative solution.

[0071] The initial dynamic parameters are obtained by quantization mapping of the second set of physical properties and are adaptively adjusted according to the object type. The core purpose is to transform the standardized parameters (such as density, stiffness, etc.) that "describe the inherent characteristics of the object" in the second set of physical properties into "dynamic calculation parameters" that can be directly iteratively solved by the differentiable dynamic simulation model and used to calculate the particle motion state.

[0072] Specifically, based on the type label of the second operated object, the corresponding core parameters are extracted from the second physical attribute set, and the standardized physical parameters are transformed into specific dynamic parameters according to the preset quantization mapping formula (adapted to different object types) to complete the initial configuration; among them, the initial dynamic parameters specifically include: elastic coefficients (such as tensile elastic coefficient, shear elastic coefficient), fluid dynamic parameters (such as viscosity coefficient, fluid diffusion coefficient), internal constraint parameters (such as constraint stiffness threshold, particle neighborhood radius), and collision contact parameters (such as collision recovery coefficient, contact friction coefficient).

[0073] For example, for flexible rope-like objects, the spring coefficient k and elastic modulus E are extracted from the second set of physical properties. Using the mapping formula k_elastic=k×E / L0 (L0 being the initial particle spacing), they are transformed into the tensile elastic coefficient k_elastic. Simultaneously, the spring coefficient is mapped to the constraint stiffness threshold in the internal constraint parameters, ensuring the tensile constraint solution during flexible rope simulation. For liquid-like objects, the density ρ and viscosity parameter η are extracted from the second set of physical properties. The density ρ is mapped to the contact mass coefficient in the collision contact parameters, and the viscosity parameter η is directly quantized into the viscosity coefficient in the fluid dynamics parameters. Furthermore, combined with the characteristics of liquid particles, the fluid diffusion coefficient is determined, achieving accurate simulation of liquid flow and collision behavior.

[0074] Among them, the differentiable dynamics simulation model is a physical simulation model built on a differentiable computational architecture and oriented towards cross-type objects. It uses particles as the basic representation unit, encapsulates the physical properties of objects into differentiable dynamic parameters, and uses a differentiable dynamics solution algorithm adapted to the object type to achieve state iteration. Its core features are that the simulation process is differentiable throughout, supports gradient backpropagation and automatic parameter optimization, and can accurately output the temporal motion states such as particle position, velocity, and deformation gradient according to the robot's motion input. At the same time, it is compatible with the unified simulation and data generation of multiple types of objects such as rigid, flexible, cloth, and fluid, providing a differentiable, high-precision, and highly generalizable simulation computational foundation for robot operation strategy optimization and training dataset construction.

[0075] Furthermore, the preset differentiable dynamic simulation model is initialized and configured according to the initial dynamic parameters, specifically by uniformly encapsulating the initial dynamic parameters into a cross-type dynamic parameter set. The dynamic model is initialized based on a differentiable particle engine, and its core iterative formula is shown below: ; in, This represents the particle system state at time t+1, including the position, velocity, and physical properties of all particles. Let be the state of the particle system at time t. The robot motion input at time t corresponds to the timing motion commands in the robot motion sequence mentioned earlier, including specific operational parameters such as the magnitude of the force applied by the gripper, the motion speed, and the displacement. The adaptive cross-type dynamic parameter set is composed of the aforementioned initial dynamic parameter mapping. f() is a differentiable dynamic state transition function, the specific calculation logic of which is implemented by the subsequently selected dynamic solution algorithm, and is used to complete the iterative solution from the current system state and robot input to the system state at the next moment.

[0076] The dynamic solution algorithm is adaptively selected based on the type and characteristics of the second manipulated object to ensure simulation accuracy and stability. For objects without fixed topology, such as liquids and flexible ropes, or those that are easily flowable and deformable, the moving least squares mass point method is used to solve the problem, taking into account both particle continuity and deformation degrees of freedom. For objects with a mesh structure, such as cloth, that require simultaneous consideration of tensile and shear constraints, the mass-spring system algorithm is used to iteratively solve for the elastic constraint forces and motion states between particles.

[0077] Specifically, the moving least squares matter point method uses particles as the basic computational unit, maps particle information to the background grid to solve for momentum, force and acceleration, and then smoothly transfers the grid calculation results back to the particles through moving least squares interpolation. It combines the advantages of Lagrange particle description and Eulerian grid calculation, and can stably solve the continuous motion and deformation process of easily deformable objects such as liquids and flexible ropes under the condition of no fixed grid topology.

[0078] Specifically, the mass-spring system algorithm treats the fabric as a discrete mechanical system composed of particles and springs. Based on Hooke's law, it calculates the tensile and shear constraints between particles in real time, and iteratively updates the particle position and velocity using Newton's second law. This reproduces the flexible dynamic characteristics of the fabric under external forces, such as deformation, swaying, and wrinkling, and can efficiently meet the dynamic solution requirements of objects with regular neighborhood constraints.

[0079] In one possible embodiment, the step of inputting the robot action sequence and the second particle system model into the differentiable dynamics simulation model, and obtaining the simulated particle motion data output by the differentiable dynamics simulation model through full-time iterative solution, includes: based on the differentiable dynamics simulation model, iteratively solving the input information using the dynamics solution algorithm, performing benchmark simulation and recording the particle motion state to generate original particle motion trajectory data; updating the configuration of the differentiable dynamics simulation model multiple times; iteratively solving the input information based on the multiple updated differentiable dynamics simulation model, performing multiple enhanced simulations and recording the particle motion state to generate multiple sets of enhanced particle motion trajectory data; and outputting the original particle motion trajectory data and the multiple sets of enhanced particle motion trajectory data as the simulated particle motion data.

[0080] Specifically, the iterative solution of the input information using a dynamic solution algorithm requires selecting the appropriate solution algorithm based on the type of the second manipulated object and calculating iteratively time-by-time using a specific formula, as detailed below: If the second manipulated object is a liquid or a flexible rope, the Moving Least Squares Material Point Method (MLS-MPM) is used as the dynamic solution algorithm. It solves the problem through three iterative steps: Particle to Mesh (P2G), Mesh Operation (GO), and Mesh to Particle (G2P). The core formulas and parameters are explained below: Particle-to-Mesh (P2G): Updates the particle deformation gradient, describing the particle deformation state, with the following formula: ; in, Let be the deformation gradient of particle p at time n+1; I is the identity matrix; Δt is the simulation time step, ranging from 0.001 to 0.01 s; Let be the velocity gradient of particle p at time n; Let be the deformation gradient of particle p at time n; Mesh Operation (GO): Normalizes the mesh node velocity, providing the basis for particle velocity updates. The formula is: ; in, Let be the normalized velocity of grid node i at time n+1; Let be the momentum of grid node i at time n+1; Let be the cumulative mass of grid node i at time n+1; Mesh to Particle (G2P): Interpolates the mesh node velocities to the particles, updating the particle velocities. The formula is: ; in, Let be the velocity of particle p at time n+1; Let p be the interpolation weight of particle p with respect to grid node i; Let be the final velocity of grid node i at time n+1.

[0081] If the second manipulated object is cloth, the mass-spring system algorithm is used as the dynamic solution algorithm. The particle's position and velocity are updated by iteratively calculating the net force of the springs acting on the particle. The core formulas and parameters are explained below: ; in, Let p be the position of particle p at time n+1; Let p be the position of particle p at time n; Let be the velocity of particle p at time n; The resultant force of the spring on particle p at time n (including tensile constraint force F_stretch and shear constraint force F_shear). Δt is the mass of particle p; Δt is the simulation time step (values ​​range from 0.001 to 0.01 s).

[0082] As can be seen, in this embodiment, during the iterative solution process, the robot action sequence and dynamic parameter set φ are input synchronously, and the position, velocity, deformation gradient and other states of the particles are updated at each time step to complete the baseline simulation. Subsequently, the configuration of the differentiable dynamic simulation model is updated multiple times, and the iterative solution process of the corresponding type is repeated to complete multiple enhanced simulations, and finally the original and multiple sets of enhanced particle motion trajectory data are generated.

[0083] In one possible embodiment, the step of updating the configuration of the differentiable dynamics simulation model multiple times includes: within a preset physical property adjustment range, performing multiple perturbations on multiple physical properties in the second physical property set to generate multiple differentiated third physical property sets; determining multiple sets of enhanced dynamic parameters according to the multiple sets of third physical property sets; and updating the configuration of the differentiable dynamics simulation model multiple times according to the multiple sets of enhanced dynamic parameters.

[0084] Specifically, the configuration update of the differentiable dynamics simulation model is not a simple repetition, but rather a simulation of the motion state of objects under different physical property scenarios by changing key dynamic parameters. This ensures that the model always adapts to the target object type, and generates particle motion data covering different scenarios through simulation of multiple sets of differentiated parameters, avoiding the simulation limitations caused by a single parameter. At the same time, it provides a diverse parameter basis for subsequent enhanced simulations, ensuring that the final output simulation data can comprehensively reflect the motion laws of objects under different physical conditions, thereby improving the reliability and applicability of the simulation results.

[0085] As can be seen, in this embodiment, the inherent properties of objects are transformed into dynamic parameters that can be used in simulation through scientific parameter mapping. Combined with a solution algorithm that adapts to the object type, the simulation accuracy is ensured. Furthermore, parameter perturbation and multi-round simulation enhancement cover different physical scenarios, effectively avoiding the limitations of a single parameter. Finally, comprehensive and reliable simulation data are output, while taking into account both simulation efficiency and result realism. This meets the simulation needs of various flexible and rigid objects and improves the applicability and robustness of dynamic simulation.

[0086] Step S505: Integrate the simulated particle motion data with the initial scene data to generate a robot operation training dataset that spans object types.

[0087] Among them, the cross-object type robot operation training dataset is a structured dataset formed after physical validity verification, data pairing and association and standardization processing. It integrates three core data dimensions: simulated particle motion data (including particle position, velocity and other state information), robot action sequence, task identifier and object type label, covering operation scenarios of different object types. It can be directly used for training robot operation models and helps robots master the operation logic and rules of cross-type objects.

[0088] In one possible embodiment, integrating the simulated particle motion data with the initial scene data to generate a robot operation training dataset across object types includes: performing physical validity verification on the simulated particle motion data, removing particle motion data with physical deformation anomalies, particle penetration, or force imbalance; pairing and associating the verified simulated particle motion data with the robot action sequence in the initial scene data to form robot operation training samples; and integrating and standardizing the robot operation training samples with the task identification information of the operation task and the second type label of the second operation object to obtain the robot operation training dataset across object types.

[0089] Specifically, the integration and standardization process reuses unified operation task annotation information, enabling the same task logic to adapt to various object types such as rigid, flexible, and liquid, achieving a universal expression of cross-type data. On the other hand, by combining the type labels of the second operation object, the original simulation data and the enhanced data are uniformly classified and merged, and the data validity is screened based on differentiable physical characteristics, eliminating invalid samples with excessive dynamic errors and abnormal physical behavior. Finally, it is converted into a standardized format that is directly compatible with deep learning frameworks, thus forming a robot operation training dataset with unified structure, physical reliability, and coverage of multiple object types.

[0090] As can be seen, in this embodiment, the particle system model is adjusted to adapt to different types of second manipulated objects, and a full-time dynamic simulation is carried out on the adjusted particle system model to generate simulated particle motion data. Finally, the simulation data is fused with the initial scene data to generate a robot operation training dataset across object types. This avoids collecting data separately for each object, significantly reduces data collection costs and workload, and achieves efficient construction of training data across object types.

[0091] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the electronic device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0092] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0093] When dividing each function into modules according to its corresponding function. Figure 8 This is a functional block diagram of a cross-object type training data generation device based on differentiable particle modeling provided in this application embodiment. The cross-object type training data generation device 800 based on differentiable particle modeling includes an acquisition module 810, a model generation module 820, a simulation module 830, and a data processing module 840, wherein: The acquisition module 810 is used to acquire initial scene data collected by the robot in the robot operation scene when the robot performs operation tasks. The initial scene data includes multi-view image data, robot action sequence, and a first type label corresponding to the first operation object. The model generation module 820 is used to perform unified particle-based representation modeling on the initial scene data based on a differentiable particle engine to generate a first particle system model corresponding to the first operated object; and to adjust the first particle system model to obtain a second particle system model adapted to the second operated object. The simulation module 830 is used to perform a full-time dynamics simulation on the second particle system model and generate simulated particle motion data of the second manipulated object under the operation task. The data processing module 840 is used to integrate the simulated particle motion data with the initial scene data to generate a robot operation training dataset that spans object types.

[0094] Optionally, in the aspect of performing unified particle-based representation modeling on the initial scene data based on a differentiable particle engine to generate a first particle system model corresponding to the first manipulated object, the model generation module 820 is specifically used for: Foreground segmentation and correction registration are performed on the multi-view image data to obtain standardized image data; Extract the depth point cloud data of the first manipulated object from the standardized image data; An initial particle set is generated based on the preset particle generation function and the depth point cloud data. The particle distribution in the initial particle set is optimized by using K-nearest neighbor homogenization constraints to obtain a first particle set that densely covers the surface and interior of the first operation object. The first particle set contains the 3D position coordinates of multiple particles. The first particle system model corresponding to the first manipulated object is constructed based on the first type label, the pre-built object type sample database, and the first particle set.

[0095] Optionally, in constructing the first particle system model corresponding to the first manipulated object based on the first type label, the pre-built object type sample database, and the first particle set, the model generation module 820 is specifically used for: Based on the first type label, the first physical attribute set and the first inter-particle connection relationship corresponding to the first operated object are obtained from the object type sample database. The object type sample database pre-stores multiple physical attribute sets and multiple inter-particle connection relationships corresponding to multiple different types of objects. The inter-particle connection relationship includes the fixed connection relationship corresponding to rigid objects, the elastic constraint relationship corresponding to flexible objects, and the no-connection relationship corresponding to liquids. Particle-level physical properties are added to each particle in the first particle set according to the first set of physical properties. The particle-level physical properties include motion velocity, particle mass, coefficient of friction, stiffness and elastic modulus. By integrating the first particle set, the particle-level physical properties, and the inter-particle connections, the first particle system model is obtained.

[0096] Optionally, in adjusting the first particle system model to obtain a second particle system model adapted to the second manipulating object, the model generation module 820 is specifically used for: Obtain the second type label of the object being operated on; Based on the second type label, obtain the second set of physical attributes and the connection relationship between the second particles corresponding to the second operated object from the object type sample database; Update the particle-level physical properties in the first particle system model according to the second set of physical properties, and update the first inter-particle connection relationship in the first particle system model according to the second inter-particle connection relationship; The consistency between the updated first particle system model and the shape of the second manipulated object is verified based on a preset point cloud distance matching algorithm. After the shape consistency verification is passed, the updated first particle system model is determined as the second particle system model of the second manipulated object.

[0097] Optionally, in performing a full-time dynamics simulation on the second particle system model to generate simulated particle motion data of the second manipulated object under the operation task, the simulation module 830 is specifically used for: Initial dynamic parameters are determined based on the second set of physical properties. The initial dynamic parameters are obtained by quantization mapping of the second set of physical properties and are adaptively adjusted according to the object type. The initial dynamic parameters include one or more of the following: elastic coefficients, hydrodynamic parameters, internal constraint parameters, and collision contact parameters. The preset differentiable dynamic simulation model is initialized and configured according to the initial dynamic parameters; and, Configure a dynamic solution algorithm that matches the second manipulated object for the differentiable dynamic simulation model. The dynamic solution algorithm includes the moving least squares material point method for liquids or flexible ropes, and the mass-spring system algorithm for cloth. The robot motion sequence and the second particle system model are input into the differentiable dynamics simulation model. After full-time iterative solution, the simulated particle motion data output by the differentiable dynamics simulation model is obtained.

[0098] Optionally, in the process of inputting the robot action sequence and the second particle system model into the differentiable dynamics simulation model, and obtaining the simulated particle motion data output by the differentiable dynamics simulation model through full-time iterative solution, the simulation module 830 is specifically used for: Based on the differentiable dynamics simulation model, the input information is iteratively solved by the dynamics solution algorithm to perform benchmark simulation and record the particle motion state, thereby generating original particle motion trajectory data. The configuration of the differentiable dynamics simulation model was updated multiple times; The input information is iteratively solved based on the updated differentiable dynamics simulation model, and multiple enhanced simulations are performed and the particle motion state is recorded to generate multiple sets of enhanced particle motion trajectory data. The original particle motion trajectory data and the multiple sets of enhanced particle motion trajectory data are output as the simulated particle motion data.

[0099] Optionally, regarding the multiple configuration updates of the differentiable dynamics simulation model, the simulation module 830 is specifically used for: Within the preset physical attribute adjustment range, multiple physical attributes in the second physical attribute set are perturbed and enhanced multiple times to generate multiple differentiated third physical attribute sets; Multiple sets of enhanced dynamic parameters are determined based on the aforementioned sets of third physical properties; The configuration of the differentiable dynamic simulation model is updated multiple times based on the multiple sets of enhanced dynamic parameters.

[0100] Optionally, in integrating the simulated particle motion data with the initial scene data to generate a robot operation training dataset across object types, the data processing module 840 is specifically used for: The simulated particle motion data is physically validated, and particle motion data with abnormal physical deformation, particle penetration, or force imbalance are removed. The verified simulated particle motion data is paired and associated with the robot action sequence in the initial scene data to form robot operation training samples; The robot operation training samples are integrated and standardized with the task identification information of the operation task and the second type label of the second operation object to obtain the robot operation training dataset across object types.

[0101] As can be seen, by performing unified particle-based representation modeling on the initial scene data of the robot operation scenario, a particle system model corresponding to the first operation object is obtained. This model is then adjusted to adapt to different types of second operation objects. Full-time dynamic simulation is carried out on the adjusted particle system model to generate simulated particle motion data. Finally, the simulation data is fused with the initial scene data to generate a robot operation training dataset across object types. This avoids collecting data separately for each object, significantly reduces data collection costs and workload, and achieves efficient construction of training data across object types.

[0102] It should be noted that the specific implementation of each operation can be described in the corresponding description of the method embodiments shown above. The cross-object type training data generation device 800 based on differentiable particleization can be used to execute the above method embodiments of this application, and will not be described again here.

[0103] This application also provides a computer-readable storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0104] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may include an electronic device.

[0105] It should be noted that, for the sake of simplicity, the above embodiments are all described as a series of actions. Those skilled in the art should understand that this application is not limited to the described order of actions, as some steps in the embodiments of this application can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of this application.

[0106] In the above embodiments, the descriptions of each embodiment in this application have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0107] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0108] The steps of the methods or algorithms described in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, EPROM, electrically erasable programmable read-only memory (EEPROM), registers, hard disk, portable hard disk, read-only optical disk (CD-ROM), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Furthermore, the ASIC can reside in a terminal device or management device. Alternatively, the processor and storage medium can exist as discrete components in the terminal device or management device.

[0109] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0110] The modules / units included in the various devices and products described in the above embodiments can be software modules / units, hardware modules / units, or a combination of both. For example, for devices and products applied to or integrated into a chip, all modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs that run on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits. For devices and products applied to or integrated into a chip module, all modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware methods such as circuits. The implementation is achieved through a software program that runs on the processor integrated within the chip module. The remaining modules / units (if any) can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into terminal equipment, each of their modules / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components within the terminal equipment. Alternatively, at least some modules / units can be implemented through a software program that runs on the processor integrated within the terminal equipment, while the remaining modules / units (if any) can be implemented using hardware methods such as circuits.

[0111] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above descriptions are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A method for generating cross-object type training data based on differentiable particleization, characterized in that, The method includes: Acquire initial scene data collected by the robot in the robot operation scene when the robot performs operation tasks. The initial scene data includes multi-view image data, robot action sequence, and first type label corresponding to the first operation object. Based on a differentiable particle engine, the initial scene data is modeled using a unified particle representation to generate a first particle system model corresponding to the first manipulated object. The first particle system model is adjusted to obtain a second particle system model adapted to the second manipulated object; A full-time dynamics simulation is performed on the second particle system model to generate simulated particle motion data of the second manipulated object under the operation task. The simulated particle motion data is integrated with the initial scene data to generate a robot operation training dataset that spans multiple object types.

2. The method according to claim 1, characterized in that, The process of using a differentiable particle engine to perform unified particle-based representation modeling on the initial scene data, generating a first particle system model corresponding to the first manipulated object, includes: Foreground segmentation and correction registration are performed on the multi-view image data to obtain standardized image data; Extract the depth point cloud data of the first manipulated object from the standardized image data; An initial particle set is generated based on the preset particle generation function and the depth point cloud data. The particle distribution in the initial particle set is optimized by using K-nearest neighbor homogenization constraints to obtain a first particle set that densely covers the surface and interior of the first operation object. The first particle set contains the 3D position coordinates of multiple particles. The first particle system model corresponding to the first manipulated object is constructed based on the first type label, the pre-built object type sample database, and the first particle set.

3. The method according to claim 2, characterized in that, The step of constructing the first particle system model corresponding to the first manipulated object based on the first type label, the pre-built object type sample database, and the first particle set includes: Based on the first type label, the first physical attribute set and the first inter-particle connection relationship corresponding to the first operated object are obtained from the object type sample database. The object type sample database pre-stores multiple physical attribute sets and multiple inter-particle connection relationships corresponding to multiple different types of objects. The inter-particle connection relationship includes the fixed connection relationship corresponding to rigid objects, the elastic constraint relationship corresponding to flexible objects, and the no-connection relationship corresponding to liquids. Particle-level physical properties are added to each particle in the first particle set according to the first set of physical properties. The particle-level physical properties include motion velocity, particle mass, coefficient of friction, stiffness and elastic modulus. By integrating the first particle set, the particle-level physical properties, and the inter-particle connections, the first particle system model is obtained.

4. The method according to claim 3, characterized in that, The adjustment of the first particle system model to obtain a second particle system model adapted to the second manipulated object includes: Obtain the second type label of the object being operated on; Based on the second type label, obtain the second set of physical attributes and the connection relationship between the second particles corresponding to the second operated object from the object type sample database; Update the particle-level physical properties in the first particle system model according to the second set of physical properties, and update the first inter-particle connection relationship in the first particle system model according to the second inter-particle connection relationship; The consistency between the updated first particle system model and the shape of the second manipulated object is verified based on a preset point cloud distance matching algorithm. After the shape consistency verification is passed, the updated first particle system model is determined as the second particle system model of the second manipulated object.

5. The method according to claim 4, characterized in that, The step of performing a full-time dynamics simulation on the second particle system model to generate simulated particle motion data of the second manipulated object under the operation task includes: Initial dynamic parameters are determined based on the second set of physical properties. The initial dynamic parameters are obtained by quantization mapping of the second set of physical properties and are adaptively adjusted according to the object type. The initial dynamic parameters include one or more of the following: elastic coefficients, hydrodynamic parameters, internal constraint parameters, and collision contact parameters. The preset differentiable dynamic simulation model is initialized and configured according to the initial dynamic parameters; and, Configure a dynamic solution algorithm that matches the second manipulated object for the differentiable dynamic simulation model. The dynamic solution algorithm includes the moving least squares material point method for liquids or flexible ropes, and the mass-spring system algorithm for cloth. The robot motion sequence and the second particle system model are input into the differentiable dynamics simulation model. After full-time iterative solution, the simulated particle motion data output by the differentiable dynamics simulation model is obtained.

6. The method according to claim 5, characterized in that, The process involves inputting the robot action sequence and the second particle system model into the differentiable dynamics simulation model, and then performing a full-time iterative solution to obtain the simulated particle motion data output by the differentiable dynamics simulation model, including: Based on the differentiable dynamics simulation model, the input information is iteratively solved by the dynamics solution algorithm to perform benchmark simulation and record the particle motion state, thereby generating original particle motion trajectory data. The configuration of the differentiable dynamics simulation model was updated multiple times; The input information is iteratively solved based on the updated differentiable dynamics simulation model, and multiple enhanced simulations are performed and the particle motion state is recorded to generate multiple sets of enhanced particle motion trajectory data. The original particle motion trajectory data and the multiple sets of enhanced particle motion trajectory data are output as the simulated particle motion data.

7. The method according to claim 6, characterized in that, The multiple configuration updates to the differentiable dynamics simulation model include: Within the preset physical attribute adjustment range, multiple physical attributes in the second physical attribute set are perturbed and enhanced multiple times to generate multiple differentiated third physical attribute sets; Multiple sets of enhanced dynamic parameters are determined based on the aforementioned sets of third physical properties; The configuration of the differentiable dynamics simulation model is updated multiple times based on the multiple sets of enhanced dynamic parameters.

8. The method according to claim 6, characterized in that, The process of integrating the simulated particle motion data with the initial scene data to generate a robot operation training dataset across object types includes: The simulated particle motion data is physically validated, and particle motion data with abnormal physical deformation, particle penetration, or force imbalance are removed. The verified simulated particle motion data is paired and associated with the robot action sequence in the initial scene data to form robot operation training samples; The robot operation training samples are integrated and standardized with the task identification information of the operation task and the second type label of the second operation object to obtain the robot operation training dataset across object types.

9. A cross-object type training data generation device based on differentiable particleization, characterized in that, The device includes an acquisition module, a model generation module, a simulation module, and a data processing module, wherein: The acquisition module is used to acquire initial scene data collected by the robot in the robot operation scene when the robot performs operation tasks. The initial scene data includes multi-view image data, robot action sequence, and a first type label corresponding to the first operation object. The model generation module is used to perform unified particle-based representation modeling on the initial scene data based on a differentiable particle engine, and generate a first particle system model corresponding to the first operated object; and adjust the first particle system model to obtain a second particle system model adapted to the second operated object. The simulation module is used to perform full-time dynamics simulation on the second particle system model and generate simulated particle motion data of the second manipulated object under the operation task. The data processing module is used to integrate the simulated particle motion data with the initial scene data to generate a robot operation training dataset that spans multiple object types.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-8.

Images