Multi-modal simulation data generation method and system for dense stacking scenarios

Abstract

The embodiment of the specification provides a multi-modal simulation data generation method and system for a dense stacking scene, wherein the method comprises the following steps: generating a physically stable simulation scene through a plurality of parallel physical simulation processes, and outputting posture information of a plurality of objects in the simulation scene; based on the posture information, an initial position without collision is allocated to the plurality of objects by using a constructive placement algorithm; the initialized simulation scene is subjected to parallel execution of RGB image rendering and ideal depth map rendering, and the ideal depth map is subjected to structured sensorization processing to generate a sensorized depth map and a corresponding invalidity mask; the RGB image, the ideal depth map, the sensorized depth map and the invalidity mask are aligned, and instance labeling is generated based on the posture information to pack a multi-modal simulation data set.

Description

Technical Field

[0001] This document relates to the fields of computer vision and robot simulation technology, and in particular to a method and system for generating multimodal simulation data for densely stacked scenarios. Background Technology

[0002] In industrial automation tasks (such as electronic component sorting and bin grasping), the performance of deep learning models highly depends on large-scale, high-quality, and fully labeled RGB-D data. However, real-world data acquisition is costly, time-consuming, and difficult to cover extreme scene distributions such as dense stacking and strong occlusion. Currently, simulation data generation has become an important technical approach to solve these problems. Among the authorized patents in China, CN115903541B proposes a method for generating visual algorithm simulation datasets based on twin scenes. This patent automatically generates RGB-D images and annotation information by constructing a proportional robotic arm grasping simulation scene and randomly configuring scene enhancement, representing a relatively advanced existing technical solution in this field.

[0003] The current level of automation in data generation still has significant technical limitations when dealing with high-contact, densely stacked industrial scenarios: 1. Efficiency bottleneck of physical simulation architecture: Its physical simulation still relies on the traditional architecture, which faces irregular memory access and data dependency branching problems in high contact-intensive scenarios, resulting in a significant decrease in simulation throughput; 2. High collision resolution cost in initialization method: The random configuration scene enhancement strategy generates a large number of collisions during the dense stacking initialization process, which need to be "resolved" through time-consuming physics solutions, resulting in high time complexity of the initialization process; 3. Insufficient modeling of sensor noise: Insufficient modeling of the structured noise characteristics of depth sensors (such as occlusion holes, femtopixels, and distance-related errors) limits the performance of simulation-to-real-world transfer. Coupling of rendering processes: Using a unified rendering process makes it difficult to simultaneously ensure the realism of RGB appearance and depth noise. Summary of the Invention

[0004] This specification provides one or more embodiments of a method for generating multimodal simulation data for densely stacked scenarios, including: S1. Generate a physically stable simulation scene through multiple parallel physical simulation processes, and output the attitude information of multiple objects in the simulation scene; S2. Based on the posture information, a constructive placement algorithm is used to assign collision-free initial positions to the multiple objects; S3. For the initialized simulation scene, RGB image rendering and ideal depth map rendering are performed in parallel, and the ideal depth map is processed by structured sensorization to generate a sensorized depth map and the corresponding invalidity mask. S4. Align the RGB image, the ideal depth map, the sensorized depth map, and the invalidity mask, generate instance annotations based on the pose information, and package them into a multimodal simulation dataset.

[0005] Furthermore, the step of generating a physically stable simulation scene through multiple parallel physical simulation processes and outputting the attitude information of multiple objects in the simulation scene specifically includes: The main process maintains a task queue containing scene configuration information. Each queue entry contains a scene ID, an object list, and physical parameter information. Multiple worker processes retrieve tasks from the task queue in parallel and independently initialize and run their respective physical simulation worlds; The simulations of each physical simulation world proceed in parallel until the preset physical stability judgment conditions are met. Once the scene stabilizes, export the 6D pose, velocity, contact information, and metadata of each object.

[0006] Furthermore, the constructive placement algorithm is an initialization method based on hexagonal dense stacking and constrained perturbation. The specific steps of assigning collision-free initial positions to the multiple objects using the constructive placement algorithm based on the attitude information include: Calculate the radius of the bounding sphere for each object, and determine the maximum radius and the average radius; Based on the maximum radius and the average radius, the interlayer spacing and planar grid spacing of the layered stack are determined; Initial planar positions are assigned to objects sequentially on the layered hexagonal grid points in descending order of object size, and random planar perturbations within a limited range are applied. Real-time collision detection is performed to ensure that there are no collisions at the time of initialization.

[0007] Furthermore, the RGB image rendering employs 3D Gaussian Splatting technology; the ideal depth map rendering employs Z-buffer technology.

[0008] Furthermore, the step of performing structured sensorization processing on the ideal depth map to generate a sensorized depth map and a corresponding invalidity mask specifically includes: The ideal depth map is quantized and its effective range is clipped. Perform baseline occlusion hole detection and generate occlusion mask; For non-invalid pixels, inject systematic bias and heteroscedastic noise related to depth values ​​and surface incident angles; Determine whether the boundary structure has failed based on depth gradient information, and generate a boundary failure mask; The occlusion mask, the boundary failure mask, and the out-of-range invalid mask are combined to generate the final invalidity mask.

[0009] Furthermore, the injected noise model is defined by the following formula: ; ; in The angle between the surface normal and the line of sight.

[0010] Furthermore, the instance annotation includes an instance segmentation mask and a bounding box; The method further includes recycling and regenerating samples whose invalidity mask ratio exceeds a preset threshold.

[0011] This specification provides one or more embodiments of a multimodal simulation data generation system for densely stacked scenarios, including: Physics simulation module: used to generate physically stable simulation scenes through multiple parallel physics simulation processes, and output the attitude information of multiple objects in the simulation scene; Collision-free initialization module: used to assign collision-free initial positions to the multiple objects based on the posture information using a constructive placement algorithm; The separate rendering and sensorization module is used to perform RGB image rendering and ideal depth map rendering in parallel on the initialized simulation scene, and to perform structured sensorization processing on the ideal depth map to generate a sensorized depth map and the corresponding invalidity mask. Data Packaging Module: This module aligns the RGB image, the ideal depth map, the sensorized depth map, and the invalidity mask, generates instance annotations based on the pose information, and packages them into a multimodal simulation dataset.

[0012] This specification provides one or more embodiments of an electronic device, including: Processor; and, A memory is configured to store computer-executable instructions, which, when executed, cause the processor to implement the steps of the above-described method for generating multimodal simulation data for densely stacked scenarios.

[0013] This specification provides one or more embodiments of a storage medium for storing computer-executable instructions that, when executed, implement the steps of the above-described method for generating multimodal simulation data for densely stacked scenarios.

[0014] This invention employs multi-world CPU parallel simulation instead of traditional GPU centralized solving, effectively avoiding performance crashes under irregular dense contact conditions and achieving a nearly one-order-of-magnitude increase in scene generation throughput, ensuring efficient and stable production in high-density scenes. The constructive collision-free initialization method, through hexagonal grid dense stacking and constrained perturbation, achieves a strictly collision-free initial layout in linear time complexity, completely avoiding the exponential iteration costs of traditional methods and greatly accelerating simulation convergence. By decoupling the RGB and depth generation processes and introducing a structured depth sensor module that integrates four key noise types—quantization, occlusion, heteroscedasticity noise, and boundary failure—it generates highly realistic depth data with clear failure annotations while maintaining high-fidelity RGB rendering throughput, significantly reducing the domain gap between simulation and real sensor data. This solution achieves a significant performance improvement of approximately 18.7% in tasks such as instance segmentation, demonstrating excellent simulation realism and efficient sim-to-real transfer capabilities.

[0015] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for generating multimodal simulation data for densely stacked scenarios, provided for one or more embodiments of this specification; Figure 2 A schematic diagram illustrating the composition of a multimodal simulation data generation system for densely stacked scenarios, provided for one or more embodiments of this specification; Figure 3 This is a schematic diagram of the structure of an electronic device provided for one or more embodiments of this specification. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0019] Method Implementation Examples According to embodiments of the present invention, a method for generating multimodal simulation data for densely stacked scenarios is provided. Figure 1 A flowchart illustrating a method for generating multimodal simulation data for densely stacked scenarios, provided in one or more embodiments of this specification, is shown below. Figure 1 As shown, the multimodal simulation data generation method for densely stacked scenarios according to an embodiment of the present invention specifically includes: S1. Generate a physically stable simulation scene through multiple parallel physical simulation processes, and output the attitude information of multiple objects in the simulation scene.

[0020] A main process acts as the scheduling center, maintaining a thread-safe queue of scenes to be generated. Each task entry in the queue encapsulates parameters for a complete sub-scene, including the scene ID, a list of object IDs to be placed, a random seed, container geometry, and physical parameters such as gravity and friction coefficient. Subsequently, multiple worker processes are launched, pulling tasks from the task queue in parallel. After acquiring a task, each worker process independently initializes a completely new MuJoCo physics simulation world, sets gravity according to the configuration, loads object models, and constructs its own physical environment, achieving complete isolation and parallel execution between simulation instances.

[0021] In each target world, the simulation process progresses incrementally with a fixed step size. The physics solver employs an adaptive strategy to handle dense contact and utilizes soft constraints and friction cone approximation to ensure numerical stability and computational efficiency. The dynamic convergence state of the scene is continuously monitored. By calculating and tracking the norm of the generalized velocities of all rigid bodies, the simulation stops when the motion velocities of all objects meet the preset physical stability criteria. Metadata such as the 6D pose, linear and angular velocities, contact point information, and the random seed used in the simulation are automatically exported for each object in the stable scene. Here, the set of rigid bodies is denoted as... , No. The generalized velocity of the rigid body is The condition for determining scene stability is that the following conditions are met within multiple consecutive simulation steps: ; When the motion speed of all objects is below a small threshold within multiple consecutive simulation steps, the scene is considered to have reached a physically stable state.

[0022] S2. Based on the posture information, a constructive placement algorithm is used to assign collision-free initial positions to the multiple objects.

[0023] The constructed placement algorithm is an initialization method based on hexagonal grid dense stacking and constrained perturbation. It performs geometric analysis on all objects to be placed, and determines the maximum and average radii of all objects by calculating the simplified bounding sphere radius of each object. Based on the maximum and average radii, the layer spacing and planar grid spacing for the layered stacking are determined. The layer spacing and grid constants for the hexagonal grid stacking are as follows: ; in The radius of the maximum enclosing sphere, The average radius, For safety margin, planar disturbance vector The two components satisfy .

[0024] Initial planar positions are assigned to objects on the layered hexagonal grid points in descending order of their bounding sphere radius, and random planar perturbations are applied within a finite range. Simultaneously, the minimum ball spacing is detected in real time for real-time collision detection, ensuring the initialization time... There was no collision.

[0025] In addition, optimizations were made for special cases such as boundary regions and slender objects with abnormal aspect ratios. The sampling method was changed from global hexagonal grid to local Poisson disk sampling to find suitable positions and ensure high initialization success rate and scene rationality.

[0026] S3. For the initialized simulation scene, perform RGB image rendering and ideal depth map rendering in parallel, and perform structured sensorization processing on the ideal depth map to generate a sensorized depth map and the corresponding invalidity mask.

[0027] During data acquisition, deterministic spherical sampling is used to obtain multiple viewpoints, with each viewpoint sharing fixed camera intrinsics. In the rendering stage, data is generated in parallel from multiple predefined viewpoints for each scene. The generation of the RGB appearance utilizes advanced 3D Gaussian Splatting technology to achieve high-fidelity appearance rendering and ideal depth maps. The generation is performed using Z-buffer technology.

[0028] The ideal depth map undergoes structured sensorization processing to generate a sensorized depth map and a corresponding invalidity mask. This process is divided into four stages, including: With ideal depth As input, the ideal depth map is quantized and its effective range is clipped: ; Perform baseline occlusion hole detection and generate occlusion mask. ; For non-invalid pixels, inject systematic bias and heteroscedastic noise related to the depth value and surface incident angle; the injected noise model is defined by the following formula: ; ; in The angle between the surface normal and the line of sight; Determining whether a boundary structure fails based on depth gradient information, when... At that time, with probability Mark the pixels as invalid to generate a boundary failure mask. ; By fusing the occlusion mask, the boundary failure mask, and the out-of-range invalid mask, the final output is the sensorized depth. and invalidity mask .

[0029] S4. Align the RGB image, the ideal depth map, the sensorized depth map, and the invalidity mask, generate instance annotations based on the pose information, and package them into a multimodal simulation dataset.

[0030] RGB images, ideal depth, sensor-based depth, and invalid masks are pixel-level aligned. Instance segmentation masks and bounding box annotations are generated based on the object's 6D pose and camera parameters. Data is organized according to a standard directory structure, with each sample accompanied by a metadata JSON file. A quality audit is then performed; samples with invalid mask ratios exceeding a preset threshold are recycled and regenerated. Finally, a simulation dataset with various data types is packaged and generated.

[0031] The following is a specific embodiment of a method for generating multimodal simulation data for densely stacked scenarios: System Environment and Data Preparation: The hardware environment requires at least a single GPU with 24GB of VRAM, with 8 GPUs recommended for parallel processing to support large-scale training and faster convergence. The software environment requires Python 3.10, PyTorch 2.0, and CUDA 11.7 or later. The detection and segmentation framework uses the Mask2Former pipeline as its base, integrating the MAG and CC-DPE modules. The training data uses the ECCD large-scale RGB-D dataset, covering scenes of weakly textured, small-sized, and strongly occluded stacked electronic components, providing pixel-aligned RGB, ideal depth, SDS sensorized depth, and invalid masks. For transfer learning evaluation, real-world test sets from devices such as RealSense can be collected, retaining sensor invalidation flags for weak supervision or evaluation. All runtime environments use fixed versions of environment lock files, and the randomization process uses a unified master seed and offsets by process to ensure reproducibility.

[0032] Input specification and preprocessing: The input consists of RGB and depth values ​​of the same size and pixel alignment. RGB values ​​are normalized by mean and variance according to ImageNet statistics; depth values ​​are first cropped to the specified size. Then linearly scaled to Three types of priors are derived from deep learning. Sobel gradient magnitude maps characterize regions with drastic depth changes, invalid masks identify pixels with depth failures, and boundary sharpness masks highlight boundary regions with high local variance. Data augmentation employs random horizontal flipping, scale jittering, and color jittering, and applies consistent geometric transformations to RGB, depth, and prior data to maintain alignment.

[0033] Multi-world CPU parallel simulation: The main process initializes a thread-safe scene queue. Each task in the queue contains a scene ID, a list of object IDs to be placed (the number n is random between 50 and 100), a random seed, container geometry, and physical parameters. Sixteen worker processes run in parallel. After each process retrieves a task from the queue, it independently executes the following steps: Initialize a MuJoCo world, set the gravity vector to [0, 0, -9.81] m / s², and the simulation step size Δt = 0.002 s.

[0034] Call the HSCFI algorithm in phase two to obtain the initial poses of all objects without collisions and place them into the world.

[0035] Run a physical simulation. After each simulation step, calculate the generalized velocity norm of all rigid bodies in the scene. Let the set of rigid bodies be... , No. The generalized velocity of the rigid body is When continuous Step to satisfy: ; The scene is then determined to be stable.

[0036] Once the scene is stable, export the 6D pose (position and quaternion), bounding box, and metadata (including random seed, physical parameters, etc.) of each object, and use this data as input for the next stage.

[0037] Collision-free initialization of the simulation scene: The preprocessing stage calculates the radius of the object's bounding sphere, determining the maximum and average radii. The interlayer spacing and grid constant of the hexagonal grid stack are as follows: ; in The radius of the maximum enclosing sphere, The average radius, For safety margins.

[0038] Place objects on the hexagonal grid in descending order of radius, and apply a planar perturbation to each object. Simultaneously, the minimum ball spacing is detected in real time to ensure... Strictly collision-free operation is maintained at all times. If a collision occurs, the perturbation vector is resampled, with a maximum of 5 attempts. If the operation still fails, the object is postponed to the next layer.

[0039] Local Poisson disk sampling optimization is used for boundary regions and slender body special cases.

[0040] Separate RGB-D rendering and depth sensorization: In this stage, data is generated for each stable scene from 6 different perspectives (elevation range [-30°, 30°], azimuth angle every 60°).

[0041] Multiple viewpoints are acquired using deterministic spherical sampling, with each viewpoint sharing a fixed camera intrinsic parameter.

[0042] The RGB rendering pass uses 3D Gaussian Splatting technology to achieve high-fidelity appearance rendering. It loads the object poses and high-precision meshes of the scene and renders them at a specified viewpoint. On the RTX 3090 hardware in this embodiment, this process achieves a throughput of approximately 71 FPS, with an output RGB image PSNR of up to 42.1. The depth rendering pass uses a Z-buffer to generate an ideal depth map. .

[0043] The structured depth sensor module performs four-stage processing on the ideal depth: With ideal depth For input, perform quantization and cropping: ; Baseline occlusion hole detection and generation of occlusion mask .

[0044] Noise and system bias related to injection distance and incident angle for non-invalid pixels: ; ; in The angle between the surface normal and the line of sight.

[0045] Determine if the boundary structure has failed: When At that time, with probability Mark pixels as invalid and generate boundary failure masks based on depth gradient magnitude. .

[0046] Final output sensor depth and invalidity mask .

[0047] Data alignment, annotation, and packing: Pixel-level alignment of RGB images, ideal depth, sensorized depth, and invalid masks.

[0048] Instance annotation generation: Based on the exported 6D pose of the object and camera parameters, the geometric model of each object is projected onto the image plane to generate instance segmentation masks and bounding box annotations in COCO format.

[0049] Data Packaging: Each sample (corresponding to one viewpoint of a scene) contains the following files: RGB image (PNG), ideal depth map (32-bit EXR), sensorized depth map (32-bit EXR), invalid mask (8-bit PNG), instance segmentation mask (PNG), and annotation file (JSON).

[0050] The system automatically calculates metrics such as the visible pixel ratio and invalid mask ratio for each sample. If the invalid mask ratio exceeds 30%, the sample is marked as low quality and a regeneration is triggered. All operation logs are recorded for traceability.

[0051] Implementation Results: Following the workflow of this embodiment, an ECCD dataset containing over 100,000 RGB-D samples was generated in approximately 48 hours. After training an instance segmentation model using this dataset, the average precision (AP) on a real-world industrial test set improved from 45.5% to 52.6%, a relative improvement of approximately 18.7%, demonstrating the significant effectiveness of this invention in improving the efficiency and realism of simulation data.

[0052] The beneficial effects of this invention are as follows: By employing multi-world CPU parallel simulation instead of traditional GPU centralized solving, performance crashes under irregular dense contact are effectively avoided, achieving a nearly one-order-of-magnitude increase in scene generation throughput and ensuring efficient and stable production in high-density scenes. A constructive collision-free initialization method, through hexagonal grid dense stacking and constrained perturbation, achieves a strictly collision-free initial layout in linear time complexity, completely avoiding the exponential iteration costs of traditional methods and greatly accelerating simulation convergence. By decoupling the RGB and depth generation processes and introducing a structured depth sensor module that integrates four key noise types—quantization, occlusion, heteroscedasticity noise, and boundary failure—high-fidelity RGB rendering throughput is maintained while generating highly realistic depth data with clear failure annotations, significantly reducing the domain gap between simulation and real sensor data. This solution achieves a significant performance improvement of approximately 18.7% in tasks such as instance segmentation, demonstrating excellent simulation realism and efficient sim-to-real transfer capabilities.

[0053] System Implementation Examples According to embodiments of the present invention, a multimodal simulation data generation system for densely stacked scenarios is provided. Figure 2 This specification provides a schematic diagram illustrating the composition of a multimodal simulation data generation system for densely stacked scenarios, as shown in one or more embodiments. Figure 2 As shown, the multimodal simulation data generation system for densely stacked scenarios according to an embodiment of the present invention specifically includes: Physical simulation module 20: used to generate a physically stable simulation scene through multiple parallel physical simulation processes, and output the attitude information of multiple objects in the simulation scene; Collision-free initialization module 22: used to assign collision-free initial positions to the plurality of objects based on the posture information using a constructive placement algorithm; Separate rendering and sensorization module 24: used to perform RGB image rendering and ideal depth map rendering in parallel on the initialized simulation scene, and to perform structured sensorization processing on the ideal depth map to generate a sensorized depth map and a corresponding invalidity mask; Data packaging module 26: used to align the RGB image, the ideal depth map, the sensorized depth map and the invalidity mask, generate instance annotations based on the pose information, and package them into a multimodal simulation dataset.

[0054] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0055] Device Example 1 This invention provides an electronic device, such as... Figure 3As shown, it includes: a memory 30, a processor 32, and a computer program stored in the memory 30 and executable on the processor 32. When the computer program is executed by the processor 32, it performs the following method steps: S1. Generate a physically stable simulation scene through multiple parallel physical simulation processes, and output the attitude information of multiple objects in the simulation scene; S2. Based on the posture information, a constructive placement algorithm is used to assign collision-free initial positions to the multiple objects; S3. For the initialized simulation scene, RGB image rendering and ideal depth map rendering are performed in parallel, and the ideal depth map is processed by structured sensorization to generate a sensorized depth map and the corresponding invalidity mask. S4. Align the RGB image, the ideal depth map, the sensorized depth map, and the invalidity mask, generate instance annotations based on the pose information, and package them into a multimodal simulation dataset.

[0056] Device Example 2 This invention provides a computer-readable storage medium storing an information transmission implementation program. When executed by a processor 32, the program performs the following method steps: S1. Generate a physically stable simulation scene through multiple parallel physical simulation processes, and output the attitude information of multiple objects in the simulation scene; S2. Based on the posture information, a constructive placement algorithm is used to assign collision-free initial positions to the multiple objects; S3. For the initialized simulation scene, RGB image rendering and ideal depth map rendering are performed in parallel, and the ideal depth map is processed by structured sensorization to generate a sensorized depth map and the corresponding invalidity mask. S4. Align the RGB image, the ideal depth map, the sensorized depth map, and the invalidity mask, generate instance annotations based on the pose information, and package them into a multimodal simulation dataset.

[0057] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating multimodal simulation data for densely stacked scenarios, characterized in that, include: S1. Generate a physically stable simulation scene through multiple parallel physical simulation processes, and output the attitude information of multiple objects in the simulation scene; S2. Based on the posture information, a constructive placement algorithm is used to assign collision-free initial positions to the multiple objects; S3. For the initialized simulation scene, RGB image rendering and ideal depth map rendering are performed in parallel, and the ideal depth map is processed by structured sensorization to generate a sensorized depth map and the corresponding invalidity mask. S4. Align the RGB image, the ideal depth map, the sensorized depth map, and the invalidity mask, generate instance annotations based on the pose information, and package them into a multimodal simulation dataset.

2. The method according to claim 1, characterized in that, The process of generating a physically stable simulation scene through multiple parallel physical simulation processes and outputting the attitude information of multiple objects in the simulation scene specifically includes: The main process maintains a task queue containing scene configuration information. Each queue entry contains a scene ID, an object list, and physical parameter information. Multiple worker processes retrieve tasks from the task queue in parallel and independently initialize and run their respective physical simulation worlds; The simulations of each physical simulation world proceed in parallel until the preset physical stability judgment conditions are met. Once the scene is stable, export the 6D pose, velocity, contact information, and metadata of each object.

3. The method according to claim 1, characterized in that, The constructive placement algorithm is an initialization method based on hexagonal dense stacking and constrained perturbation. The specific steps of assigning collision-free initial positions to the multiple objects using the constructive placement algorithm based on the pose information include: Calculate the radius of the bounding sphere for each object, and determine the maximum radius and the average radius; Based on the maximum radius and the average radius, the interlayer spacing and planar grid spacing of the layered stack are determined; Initial planar positions are assigned to objects sequentially on the layered hexagonal grid points in descending order of object size, and random planar perturbations within a limited range are applied. Real-time collision detection is performed to ensure that there are no collisions at the time of initialization.

4. The method according to claim 1, characterized in that, The RGB image rendering uses 3D Gaussian Splatting technology; the ideal depth map rendering uses Z-buffer technology.

5. The method according to claim 1, characterized in that, The process of performing structured sensorization on the ideal depth map to generate a sensorized depth map and a corresponding invalidity mask specifically includes: The ideal depth map is quantized and its effective range is clipped. Perform baseline occlusion hole detection and generate occlusion mask; For non-invalid pixels, inject systematic bias and heteroscedastic noise related to depth values ​​and surface incident angles; Determine whether the boundary structure has failed based on depth gradient information, and generate a boundary failure mask; The occlusion mask, the boundary failure mask, and the out-of-range invalidity mask are combined to generate the final invalidity mask.

6. The method according to claim 5, characterized in that, The injected noise model is defined by the following formula: ； ； in The angle between the surface normal and the line of sight.

7. The method according to claim 1, characterized in that, The instance annotation includes an instance segmentation mask and a bounding box; The method further includes recycling and regenerating samples whose invalidity mask ratio exceeds a preset threshold.

8. A multimodal simulation data generation system for densely stacked scenarios, characterized in that, include: Physics simulation module: used to generate physically stable simulation scenes through multiple parallel physics simulation processes, and output the attitude information of multiple objects in the simulation scene; Collision-free initialization module: used to assign collision-free initial positions to the multiple objects based on the posture information using a constructive placement algorithm; The separate rendering and sensorization module is used to perform RGB image rendering and ideal depth map rendering in parallel on the initialized simulation scene, and to perform structured sensorization processing on the ideal depth map to generate a sensorized depth map and the corresponding invalidity mask. Data Packaging Module: This module aligns the RGB image, the ideal depth map, the sensorized depth map, and the invalidity mask, generates instance annotations based on the pose information, and packages them into a multimodal simulation dataset.

9. An electronic device, characterized in that, include: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to implement the steps of the multimodal simulation data generation method for densely stacked scenarios as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, Used to store computer-executable instructions, which, when executed, implement the steps of the multimodal simulation data generation method for densely stacked scenarios as described in any one of claims 1 to 7.

Images