High-reality simulation data set generation method and system

Abstract

The invention relates to a method and a system for generating a high-reality simulation data set. The method comprises the following steps: creating or importing a three-dimensional model of a target object and a laboratory scene background in a Blender; a Blender geometric node system is utilized to construct a scattering network, the scattering network receives the surface of the laboratory scene background as input, and object instances are scattered on the scattering network; selecting Cycles of Blender as a rendering engine and initializing the Cycles, rendering each object instance on a walking network, and generating and outputting an RGB image, a depth map, an instance segmentation map and an annotation file; performing domain adaptation translation through a pre-training model based on CycleGAN, and generating and outputting an image after reality enhancement; and matching and packaging the image after reality enhancement with the depth map, the instance segmentation map and the automatically generated annotation file to generate a final data set.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and robot simulation technology, specifically, to a method and system for generating high-fidelity simulation datasets. This disclosure can be used to efficiently and automatically generate training data for robot vision tasks in laboratory environments with high realism and accurate multimodal annotation. Background Technology

[0002] In the field of robot vision, the performance of deep learning models heavily relies on large-scale, high-quality training datasets. For robot operation tasks in laboratory environments (such as object grasping and instrument operation), acquiring such data through actual filming is costly, time-consuming, and difficult to cover all possible scene changes and object poses.

[0003] Currently, generating synthetic data using 3D modeling and rendering software (such as Blender and Unity) is an effective alternative. For example, existing technology CN120339747A discloses a scheme that generates test image data based on UE5 and AirSim, inputs it into the model under test to obtain recognition results, analyzes the model's recognition performance to identify weaknesses, and then generates targeted training image data from simulation playback files. This technical solution utilizes a simulation platform to generate data and achieve targeted enhancement, filtering data based on model test feedback. However, existing technology CN120339747A does not offer a solution on how to fundamentally improve the diversity, visual realism, and perspective realism of the generated data itself.

[0004] The existing technologies have the following main technical defects: (1) Poor scene generation flexibility: The placement of objects depends on manual or fixed scripts, making it difficult to efficiently and realistically simulate the complex situation of random scattering and stacking of objects in the laboratory environment, resulting in insufficient diversity of generated data.

[0005] (2) Limited image realism: Traditional rendering engines have gaps in terms of materials, lighting and noise compared to the real world, and existing domain adaptation methods often cannot stably and effectively learn and transfer complex texture and noise features in real images.

[0006] (3) Non-integrated annotation process: Image rendering and annotation generation are usually carried out in separate steps or require post-processing. The process is cumbersome, making it difficult to ensure accurate synchronization between annotation and image pixels, and it is impossible to achieve end-to-end automation.

[0007] (4) Poor system adaptability: Most existing solutions are customized for specific tasks and lack a configurable and scalable general framework to quickly adapt to different laboratory scenarios and vision task requirements.

[0008] Currently, there is no good technical solution in the market to solve the above-mentioned technical problems. Summary of the Invention

[0009] To address the problems existing in the prior art, this disclosure proposes a method and system for generating high-fidelity simulation datasets. This addresses the technical issues of insufficient scene generation diversity, limited image realism, non-integrated annotation processes, and poor system adaptability in the prior art. The technical solution adopted in this disclosure is as follows: In a first aspect, this disclosure provides a method for generating a high-fidelity simulation dataset, the method comprising: S100: Create or import a 3D model of the target object and a laboratory scene background into Blender; S200: A controllable random scattering network is constructed using Blender's geometry node system. The scattering network receives the surface of the laboratory scene background as input and scatters object instances on the scattering network. S300: Selects Blender's Cycles as the rendering engine and initializes it. Renders each object instance on the scattering network, generating and outputting RGB images, depth maps, instance segmentation maps, and annotation files. S400: Unsupervised domain-adaptive translation is performed using a pre-trained CycleGAN-based model to generate and output an enhanced image; the domain-adaptive translation is used to bridge the distribution differences between the simulated image (source domain) and the real image (target domain), thereby enhancing the realism of the image on the scattering network (domain adaptation); S500: Match and package the enhanced image with the depth map, instance segmentation map, and automatically generated annotation file to generate the final dataset.

[0010] Preferably, the CycleGAN-based pre-trained model includes: generator : Used to learn how to convert simulated images into realistic images; generator : Used to learn how to convert real images into simulated images; Discriminator Used to determine whether the input image is a real laboratory image or an image created by [unclear]. Generated "fake" real images; Discriminator Used to determine whether the input image is the original simulation image or a result of [other methods]. The generated "fake" simulation image.

[0011] Preferably, the generator adopts a U-Net structure to preserve the structural information of the input image (such as object contours and segmentation boundaries).

[0012] Preferably, the discriminator adopts a PatchGAN structure, which performs real and fake identification on local areas of the image, helping to generate more refined textures.

[0013] Preferably, during the training process of the CycleGAN-based pre-trained model, the generator... Automatically learn real image features and inject those features into the input simulation image; The real image features include noise characteristics present in the real image (such as sensor noise), light reflection model, color deviation, and texture details.

[0014] A second aspect of this disclosure provides a high-realism simulation dataset generation system, the system comprising: The Scene Building module is used to create or import 3D models of target objects and laboratory scene backgrounds into Blender; The object scattering module is used to construct a controllable random scattering network using Blender's geometry node system. The scattering network receives the surface of the laboratory scene background as input and scatters object instances on the scattering network. The rendering initialization module is used to select Blender's Cycles as the rendering engine and initialize it, render each object instance on the scattering network, and generate and output RGB images, depth maps, instance segmentation maps and annotation files; The domain adaptation translation module is used to perform unsupervised domain adaptation translation using a CycleGAN-based pre-trained model, generating and outputting images with enhanced realism. The dataset generation module is used to match and package the enhanced image with the depth map, instance segmentation map and automatically generated annotation file to generate the final dataset.

[0015] In a third aspect, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the high-realism simulation dataset generation method described above.

[0016] In a fourth aspect, this disclosure provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the high-realism simulation dataset generation method described above.

[0017] The beneficial effects of this disclosure are as follows: This disclosure proposes a method and system for generating high-realism simulation datasets. It constructs physically based rendering through procedural scene building to generate structurally perfect labeled data, and then uses a GAN network for domain adaptation translation to approximate the style of simulated images to real images, thereby generating high-quality, highly realistic training datasets. This disclosure deeply integrates procedural 3D content generation technology, perspective simulation, and deep learning-based image domain adaptation technology to construct an end-to-end automated data generation pipeline. First, using the procedural capabilities of a 3D rendering engine, a highly randomized laboratory scene is constructed, and camera equipment is set up to capture test image data in this scene, generating batches of original simulation sequences with highly diverse perspectives and perfect annotations. Then, a generative adversarial network is introduced and optimized. Through unsupervised learning, the visual style (such as texture, noise, and lighting) of the simulated images is systematically transferred to the real image domain, significantly improving the visual realism of the generated images. Finally, a solution is formed that can simulate the real workflow of robots and quickly produce high-quality, highly realistic, multimodal labeled data.

[0018] Compared with the prior art, this disclosure has the following advantages: (A) Enhancing data diversity and realism: By procedurally distributing geometric nodes, objects are highly randomized in terms of scale, position, and pose. Combined with the domain adaptability of GAN, the realism of the generated images and the diversity of data distribution are significantly improved.

[0019] (B) Automated annotation: All annotation information is generated during the rendering process, avoiding tedious post-processing, ensuring the accuracy of annotation, and greatly improving the efficiency of data generation.

[0020] (C) Enhanced system flexibility and adaptability: The entire process is driven by parameterized configuration, and users can quickly generate datasets for different laboratory scenarios and vision tasks by modifying the configuration, making the system highly adaptable.

[0021] (D) Overcoming the domain difference problem between simulation and reality: By introducing and optimizing the loss function of CycleGAN, the training process is stabilized, enabling the generator to effectively learn and inject real-world noise, lighting and texture features, effectively narrowing the domain gap between simulation data and real data. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0023] Figure 1 This is a flowchart of a method for generating a highly realistic simulation dataset, as provided in Embodiment 1 of this disclosure.

[0024] Figure 2 This is an architecture diagram of the CycleGAN-based pre-trained model described in Embodiment 1 of this disclosure.

[0025] Figure 3 This is the RGB image of the first example scene.

[0026] Figure 4 This is the depth map of the first example scene.

[0027] Figure 5 This is the instance segmentation graph for the first example scenario.

[0028] Figure 6 This is the RGB image of the second example scene.

[0029] Figure 7 This is the depth map for the second example scene.

[0030] Figure 8 This is the instance segmentation graph for the second example scenario.

[0031] Figure 9 This is an architecture diagram of a high-fidelity simulation dataset generation system provided in Embodiment 2 of this disclosure. Detailed Implementation

[0032] The present disclosure will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other.

[0033] The following detailed descriptions are exemplary and intended to provide further detailed explanation of this disclosure. Unless otherwise specified, all technical terms used in this disclosure have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure.

[0034] Example 1: like Figure 1 As shown, this disclosure provides a method for generating a highly realistic simulation dataset, the method including steps S100 to S500.

[0035] S100: Create or import 3D models of target objects and laboratory scene backgrounds into Blender.

[0036] S200: A controllable random scattering network is constructed using Blender's geometry node system. The scattering network receives the surface of the laboratory scene background as input and scatters object instances on the scattering network.

[0037] Further, S200 includes the following steps S201 to S203.

[0038] S201 Density Control: A controllable random scattering network is constructed using Blender's geometric node system. A density factor is set for the scattering network to control the number of object instances appearing per unit area of ​​the scattering network.

[0039] S202 Random Transformation: Randomly rotate (e.g., around the X, Y, Z axes), scale (between a set minimum and maximum value, such as 0.8, 1.5), and offset the position of each object instance in the scattering network to simulate the phenomenon of different sizes and postures of object instances when naturally placed.

[0040] S203 Stacking Simulation: The physics engine simulates or presets rules on the scattering network to allow object instances scattered on the scattering network to stack under certain conditions, increasing the complexity of the laboratory scene background.

[0041] S300: Selects Blender's Cycles as the rendering engine and initializes it. It renders each object instance on the scattering network and generates and outputs RGB images, depth maps, instance segmentation maps, and annotation files.

[0042] Further, S300 includes the following steps S301 to S303: S301: Select Blender's Cycles as the rendering engine and initialize it.

[0043] The reason is that the Cycles' physically based path tracing can generate images with realistic lighting and shadows. During the initialization of the rendering engine, appropriate lighting (such as area lights and HDRI ambient lights) and material properties are set.

[0044] S302: When rendering through the rendering engine, RGB images, depth maps and instance segmentation maps are output synchronously in each rendering process (by assigning a unique color ID to each object instance, a segmentation mask is generated during rendering).

[0045] S303: Call the Blender Python API to automatically extract the bounding box, category label, and 6D pose (including 3D position + 3D rotation) of each object instance during rendering, and store them as a label file.

[0046] Furthermore, the annotation files adopt standard annotation formats (such as COCO and YOLO).

[0047] S400: Unsupervised domain-adaptive translation is performed using a CycleGAN-based pre-trained model to generate and output images with enhanced realism.

[0048] The domain-adaptive translation is used to bridge the distribution differences between simulated images (source domain) and real images (target domain), enhancing the realism of images on the scattering network (domain adaptation). The reason for performing unsupervised domain-adaptive translation based on the pre-trained model of the CycleGAN architecture is that the CycleGAN architecture does not require paired simulated-real image data.

[0049] Furthermore, such as Figure 2 As shown, the CycleGAN-based pre-trained model includes: generator : Used to learn how to convert simulated images into realistic images; generator : Used to learn how to convert real images into simulated images; Discriminator Used to determine whether the input image is a real laboratory image or an image created by [unclear]. Generated "fake" real images; Discriminator Used to determine whether the input image is the original simulation image or a result of [other methods]. The generated "fake" simulation image.

[0050] Furthermore, the generator adopts a U-Net structure to preserve the structural information of the input image (such as object contours and segmentation boundaries).

[0051] Furthermore, the discriminator adopts a PatchGAN structure, which distinguishes between real and fake local regions of the image, thus helping to generate more refined textures.

[0052] Furthermore, the generator Discriminator The adversarial loss function uses the least squares loss (LSGAN) formula, as shown below: (1) In equation (1), It is a set of simulated images. It is a set of real images. and They are , The sample. , .

[0053] Furthermore, the generator Discriminator The cycle consistency loss function is calculated using the L1 norm, as shown below: (2) The cycle consistency loss function is used to ensure that the image undergoes two domain transformations (such as...). Afterwards, it can basically restore the original state to prevent the generator from making destructive modifications to the input image.

[0054] Furthermore, the generator Discriminator The identity loss function is calculated using the L1 norm, as shown below: ;(3).

[0055] The identity loss function is used to encourage the generator not to make modifications when it receives the target domain image as input, which helps to stabilize the color distribution.

[0056] Furthermore, the generator Discriminator The formula for calculating the total loss function is as follows: (4) In equation (4), and These are hyperparameters used to balance the weights of various loss functions.

[0057] Furthermore, the overall optimization objective of the generator is to minimize the total loss function.

[0058] Furthermore, the aforementioned , They are 10 and 0.5 respectively.

[0059] Furthermore, during the training process of the CycleGAN-based pre-trained model, the generator... Automatically learn real image features and inject those features into the input simulation image.

[0060] Furthermore, the real image features include noise characteristics present in the real image (such as sensor noise), light reflection model, color deviation, and texture details.

[0061] As is well known, the core advantage of GANs in implicit learning through adversarial training is that they do not require manual specification of the noise formula. After training, the noise formula is fixed. The weights can be used to process all newly rendered simulation images, achieving automated realism enhancement.

[0062] S500: Match and package the enhanced image with the depth map, instance segmentation map, and automatically generated annotation file to generate the final dataset.

[0063] The final dataset, generated through matching and packaging, contains highly realistic simulated RGB images, depth data, pixel-level segmentation data, and instance information (bounding boxes, pose), which can be directly used to supervise the training of object detection, instance segmentation, and 6D pose estimation algorithms in robot vision tasks.

[0064] To further illustrate the working principle of the method described in this disclosure, the implementation environment is set to "Blender 3.6 or later (3D modeling and rendering software), PyTorch 2.0 or later (deep learning framework), and Python 3.10 (programming language)". The following example steps can be used to implement the method described in this disclosure: Step (100): Create or import the target object 3D model and laboratory scene background in Blender. Specifically, create or import a planar model (ground.obj) with physical materials or textures as the target object 3D model in Blender, and create a dedicated model asset library as the laboratory scene background.

[0065] The model asset library contains common visual task targets, such as Lego bricks (Legobrick.blend), apples (Apple.blend), and car models (car_model.blend). These target object 3D models all come with preset physical materials or textures.

[0066] This step is performed in the Blender environment and is designed to automate the generation of highly randomized lab scenes.

[0067] Step (200): Construct a controllable random scattering network using Blender's geometry node system. The scattering network receives the surface of the laboratory scene background as input and scatters object instances across the network. Step (200) is the core step in achieving scene diversity, specifically including: Step (201): Select the object on the plane and add a geometry node modifier to it. In the geometry node editor, construct a scattering network. First, use the "Distribute Points on Surface" node and set its "Density Factor" to 0.05. This "Density Factor" controls the number of object instances appearing per unit area of ​​the scattering network, thus determining the sparsity of the objects.

[0068] Step (202): Connect the output of the distribution points to the "Instantiate on Point" node. In the "Instance" input of this node, link an asset browser or collection, point to the model asset library created in step S11, and set it to random selection mode to ensure that the types of objects instantiated are different each time a scene is generated, thereby achieving random transformation.

[0069] In this step, to achieve random transformations, a "Random Value" node is used to drive the rotation and scaling inputs of the "Instantiate at Point" node. For rotation, three "Random Value" nodes are set, corresponding to the X, Y, and Z axes respectively, with a minimum value of -3.14 radians and a maximum value of 3.14 radians. For scaling, a single "Random Value" node is set, with a minimum value of 0.8 and a maximum value of 1.5, and uniform scaling is selected. Alternatively, different random ranges can be set for the X, Y, and Z axes to produce non-uniform scaling.

[0070] Step (203) uses this scattering network to automatically generate a laboratory scene on the tabletop each time parameters are executed or adjusted, with random distribution of object types, positions, rotations and scales.

[0071] Step (300): Select Blender's Cycles as the rendering engine and initialize it. Render each object instance on the scattering network, generating and outputting RGB images, depth maps, instance segmentation maps, and annotation files. This step can perform data generation operations using Blender's rendering and Python API, specifically including: Step (301): In Blender's rendering properties panel, set the rendering engine to Cycles; in the scene's world properties, load an HDRI environment map to provide natural ambient lighting, and you can add additional area lights for fill lighting. Set the rendering resolution to 1024×1024 pixels.

[0072] Step (302): In the view layer properties, enable the "Depth" and "Object Index" channels. In the compositor, configure the output node to synchronously save and output the following files during a single rendering process: RGB image: Standard color rendering result, saved as PNG format.

[0073] Depth map: Records the Z depth value of each pixel and saves it as a PNG file.

[0074] Instance segmentation image: Based on the "object index" channel, each object instance is presented as a mask of a unique color and saved as a PNG format.

[0075] The rendering effect will be demonstrated below using the first and second example scenarios. Figure 3 , Figure 4 , Figure 5 These are the RGB image, depth map, and instance segmentation map of the first example scene that are saved and output, respectively. Figure 6 , Figure 7 , Figure 8 These are the RGB image, depth map, and instance segmentation map of the second example scene that were saved and output. (Through...) Figures 3 to 8 As can be seen, this disclosure has good rendering performance.

[0076] Step (303): Write a Blender Python script to execute before and after rendering. Before rendering, the script assigns a unique ID to each target object in the scene. After rendering, iterates through all objects, obtaining their category name, position, and rotation. For each object, the script calculates its 2D bounding box in the current camera view. This is done by transforming the eight vertices of the object's 3D bounding box to 2D screen coordinates using the camera projection matrix, and then taking the minimum and maximum values ​​of these coordinates in the X and Y directions. Finally, the category ID, 2D bounding box ([x_min, y_min, width, height]), and 3D position and rotation quaternions (6D pose) of all objects are written to a JSON file. The structure of this JSON file is compatible with the COCO dataset format.

[0077] Step (400): Unsupervised domain adaptation translation is performed using a CycleGAN-based pre-trained model to generate and output an enhanced image. The domain adaptation translation is used to bridge the distribution differences between the simulated image (source domain) and the real image (target domain), enhancing the realism of the image on the scattering network (domain adaptation). The reason for using a CycleGAN-based pre-trained model for unsupervised domain adaptation translation is that the CycleGAN architecture does not require paired simulated-real image data.

[0078] The domain-adaptive translation can be implemented in the PyTorch environment to perform domain adaptation on the original simulated RGB image generated and output in step (300).

[0079] Step (401) Data and Model Preparation: All original simulated RGB images obtained in step (300) will be used as source domain data. Real images containing laboratory and indoor scenes were selected from the publicly available LVIS dataset as the target domain data. As described in Part 3, S31, initialize two generators ( , ) and two discriminators ( , The weights of the generator. , The U-Net architecture is adopted, and its encoder-decoder architecture can effectively preserve the structural information of the input image. The specific structure is shown in Table 1.

[0080] Table 1 - Structural Parameters of the Generator

[0081] The discriminator ( , The 70×70 PatchGAN structure is adopted. The PatchGAN structure can distinguish between real and fake local regions of the image, which helps to generate more refined textures. The specific structure is shown in Table 2.

[0082] Table 2 - Structural parameters of the discriminator

[0083] The training process of the CycleGAN-based pre-trained model may include: Step (A): and The images were uniformly scaled to 256×256 pixels and the pixel values ​​were normalized to... .

[0084] Step (B): Train according to the loss function defined in Part 3, S32.

[0085] Specifically, the total loss is:

[0086] in, =10, .

[0087] Step (C): Use the Adam optimizer, setting the learning rate to 0.0002 and the momentum parameter... , Batch size is 1. Training lasts for 200 epochs. After training, the generator is saved. The final weight.

[0088] Step (402) Domain Adaptation Translation: For any newly generated simulated RGB image in step (300), load the trained... Model.

[0089] Step (403) Generate and output the enhanced image: After preprocessing the image, input it into the model, and the output is the enhanced image with added real-world noise, texture, and lighting features—that is, generate and output the enhanced image. Save these enhanced images to replace the original simulated RGB image. The enhanced image is the image that has been enhanced.

[0090] Step (500): Match and package the enhanced image with the depth map, instance segmentation map and automatically generated annotation file to generate the final dataset.

[0091] The enhanced RGB image output from step (400), the depth map output from step (300), and the instance segmentation map are mapped one-to-one according to their filenames. The JSON annotation file generated in step (303) is associated with the enhanced image. Since the GAN processing does not change the geometry and position of objects in the image, the original annotations remain valid for the enhanced image. Finally, all image files (RGB, depth, segmentation) and annotation files (JSON) are organized according to a standard directory structure and packaged to generate the final dataset that can be directly used for training deep learning models.

[0092] As illustrated in the above examples, this disclosure achieves an end-to-end automated process from procedural scene generation, multimodal data synchronous acquisition, automated annotation to image realism enhancement, enabling efficient and flexible generation of high-quality simulation datasets that meet the visual task requirements of laboratory robots.

[0093] Example 2: like Figure 9 As shown, in a second aspect, this disclosure provides a high-fidelity simulation dataset generation system, the system comprising: Scene building module 100 is used to create or import 3D models of target objects and laboratory scene backgrounds in Blender; The object scattering module 200 is used to construct a controllable random scattering network using Blender's geometry node system. The scattering network receives the surface of the laboratory scene background as input and scatters object instances on the scattering network. The rendering initialization module 300 is used to select Blender's Cycles as the rendering engine and initialize it, render each object instance on the scattering network, and generate and output RGB images, depth maps, instance segmentation maps and annotation files. Domain Adaptive Translation Module 400 is used to perform unsupervised domain adaptive translation using a CycleGAN-based pre-trained model, generating and outputting images with enhanced realism. The dataset generation module 500 is used to match and package the enhanced image with the depth map, instance segmentation map and automatically generated annotation file to generate the final dataset.

[0094] The domain-adaptive translation is used to bridge the distribution differences between simulated images (source domain) and real images (target domain), enhancing the realism of images on the scattering network (domain adaptation). The reason for performing unsupervised domain-adaptive translation based on the pre-trained model of the CycleGAN architecture is that the CycleGAN architecture does not require paired simulated-real image data.

[0095] The scene construction module 100, object scattering module 200, rendering initialization module 300, domain adaptation translation module 400, and fault prediction module 500 in the system correspond to S100, S200, S300, S400, and S500 in the method, respectively.

[0096] It is worth noting that the system described in Example 2 is only one system implementation of the high-fidelity simulation dataset generation method described in Example 1, and does not limit the high-fidelity simulation dataset generation method described in Example 1 to depend on the system described in Example 2.

[0097] Example 3: Embodiment 3 of this disclosure provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the high-fidelity simulation dataset generation method as described in Embodiment 1. Alternatively, a high-fidelity simulation dataset generation system as described in Example 2 can be implemented.

[0098] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules, or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0099] Example 4: Embodiment 4 of this disclosure provides an electronic device including a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the high-realism simulation dataset generation method described in Embodiment 1.

[0100] Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0101] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0106] In summary, the high-realism simulation dataset generation method and system provided in embodiments 1-4 of this disclosure generate structurally perfect labeled data through procedural scene construction and physically based rendering. Then, a GAN network is used for domain adaptation translation to approximate the style of the simulated images to those of real images, thereby generating a high-quality, highly realistic training dataset. This disclosure deeply integrates procedural 3D content generation technology, perspective simulation, and deep learning-based image domain adaptation technology to construct an end-to-end automated data generation pipeline. First, the procedural capabilities of the 3D rendering engine are used to construct a highly randomized laboratory scene, and camera equipment is set up to capture test image data in this scene, generating batches of original simulation sequences with highly diverse perspectives and perfect annotations. Then, a generative adversarial network is introduced and optimized to systematically transfer the visual style (such as texture, noise, and lighting) of the simulated images to the real image domain through unsupervised learning, significantly improving the visual realism of the generated images. Finally, a solution is formed that can simulate the real workflow of a robot and quickly produce high-quality, highly realistic, multimodal labeled data.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit them. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this disclosure. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this disclosure should be covered within the protection scope of the claims of this disclosure.

Claims

1. A method for generating a highly realistic simulation dataset, characterized in that, The method includes: S100: Create or import a 3D model of the target object and a laboratory scene background into Blender; S200: A controllable random scattering network is constructed using Blender's geometry node system. The scattering network receives the surface of the laboratory scene background as input and scatters object instances on the scattering network. S300: Selects Blender's Cycles as the rendering engine and initializes it. Renders each object instance on the scattering network, generating and outputting RGB images, depth maps, instance segmentation maps, and annotation files. S400: Unsupervised domain-adaptive translation is performed using a pre-trained CycleGAN-based model to generate and output images with enhanced realism; the domain-adaptive translation is used to bridge the distribution differences between simulated images and real images, thereby enhancing the realism of images on the scattering network. S500: Match and package the enhanced image with the depth map, instance segmentation map, and automatically generated annotation file to generate the final dataset.

2. The method for generating a high-fidelity simulation dataset as described in claim 1, characterized in that, S200 includes the following steps: S201 Density Control: A controllable random scattering network is constructed using Blender's geometric node system, and a density factor is set for the scattering network to control the number of object instances appearing per unit area of ​​the scattering network. S202 Random Transformation: Randomly rotate, scale, and offset each object instance in the scattering network to simulate the phenomenon of different sizes and postures of object instances when naturally placed. S203 Stacking Simulation: The physics engine simulates or presets rules on the scattering network to allow object instances scattered on the scattering network to stack under certain conditions, increasing the complexity of the laboratory scene background.

3. The method for generating a high-fidelity simulation dataset as described in claim 1, characterized in that, The S300 includes the following steps: S301: Select Blender's Cycles as the rendering engine and initialize it; S302: When rendering through the rendering engine, RGB images, depth maps and instance segmentation maps are output synchronously in each rendering process; S303: Call the Blender Python API to automatically extract the bounding box, category label, and 6D pose in camera coordinates for each object instance during rendering, and store them as a label file; The annotation file adopts the standard annotation format.

4. The method for generating a high-fidelity simulation dataset as described in claim 1, characterized in that, The CycleGAN-based pre-trained model includes: generator : Used to learn how to convert simulated images into realistic images; generator : Used to learn how to convert real images into simulated images; Discriminator Used to determine whether the input image is a real laboratory image or an image created by [unclear]. Generated "fake" real images; Discriminator Used to determine whether the input image is the original simulation image or a result of [other methods]. The generated "fake" simulation image.

5. The method for generating a high-fidelity simulation dataset as described in claim 4, characterized in that, The generator adopts a U-Net structure.

6. The method for generating a high-fidelity simulation dataset as described in claim 4, characterized in that, The discriminator adopts the PatchGAN structure.

7. The method for generating a high-fidelity simulation dataset as described in claim 4, characterized in that, The generator Discriminator The adversarial loss function uses the least squares loss LSGAN formula, as shown below: ； (1) In equation (1), It is a set of simulated images. It is a set of real images. and They are , The sample; The generator Discriminator The cycle consistency loss function is calculated using the L1 norm, as shown below: ；(2) The cycle consistency loss function is used to ensure that the image undergoes two domain transformations (such as...). Afterwards, it can basically restore the original state to prevent the generator from making destructive modifications to the input image; The generator Discriminator The identity loss function is calculated using the L1 norm, as shown below: ；(3) The identity loss function is used to encourage the generator not to make modifications when it receives the target domain image as input; The generator Discriminator The formula for calculating the total loss function is as follows: ； (4) In equation (4), and These are hyperparameters used to balance the weights of various loss functions.

8. The method for generating a high-fidelity simulation dataset as described in claim 7, characterized in that, The overall optimization objective of the generator is to minimize the total loss function; The , They are 10 and 0.5 respectively.

9. The method for generating a high-fidelity simulation dataset as described in claim 4, characterized in that, During the training process of the CycleGAN-based pre-trained model, the generator... Automatically learn real image features and inject those features into the input simulation image; The real image features include noise characteristics, light reflection models, color deviations, and texture details present in the real image.

10. A high-fidelity simulation dataset generation system, characterized in that, The system includes: Scene building module (100) is used to create or import 3D models of target objects and laboratory scene backgrounds in Blender; The object scattering module (200) is used to construct a controllable random scattering network using Blender's geometry node system. The scattering network receives the surface of the laboratory scene background as input and scatters object instances on the scattering network. The rendering initialization module (300) is used to select Blender's Cycles as the rendering engine and initialize it, render each object instance on the scattering network, and generate and output RGB images, depth maps, instance segmentation maps and annotation files. Domain Adaptive Translation Module (400) is used to perform unsupervised domain adaptive translation using a CycleGAN-based pre-trained model, generating and outputting images with enhanced realism. The dataset generation module (500) is used to match and package the enhanced image with the depth map, instance segmentation map and automatically generated annotation file to generate the final dataset.

Images