Virtual Data Generation Method Based on Deep Learning Large Model Visual Algorithm

Abstract

This application relates to the fields of computer vision, deep learning data generation and processing technology, and provides a method for generating virtual data based on a deep learning large-scale visual algorithm. The method involves segmenting the original image according to a preset visual large-scale model and a specified category, adding each segmented foreground object and its corresponding segmentation information as a separate template to a template library; selecting foreground objects from a background library and generating a synthetic image by image fusion with randomly selected background images from the background library; achieving fusion by placing the foreground objects at predetermined positions in the background images and calculating the corresponding gradient fields to obtain a fused image; post-processing the fused image to eliminate seams or artifacts to obtain a synthetic image; and post-processing the synthetic image using a deep learning large-scale model to integrate the processed synthetic image and its corresponding annotation information to generate a virtual dataset, which is stored in a directory structure containing images, annotation information, and metadata.

Description

Technical Field

[0001] This application relates to the fields of computer vision, deep learning data generation and processing technology, and in particular to a method for generating virtual data based on a deep learning large model visual algorithm. Background Technology

[0002] In the field of computer vision, the training of deep learning algorithms heavily relies on large-scale, diverse labeled data. Traditional data acquisition and labeling are labor-intensive and inefficient, becoming a bottleneck restricting the rapid iteration of algorithms. Among existing solutions, data augmentation techniques (such as geometric transformations and color adjustments) struggle to generate complex, realistic scenes, easily leading to models learning incorrect features. While virtual data generation based on game engines can simulate real-world scenes, it requires professional 3D modeling skills and significant time investment, making it unsuitable for small to medium-sized scenarios with limited resources. Furthermore, traditional image segmentation methods require pre-training of segmentation models, presenting a "chicken-or-egg" problem that prevents them from being directly used to automatically generate foreground objects with labeled categories.

[0003] Therefore, a method is urgently needed to solve at least one of the above problems. Summary of the Invention

[0004] This application provides a virtual data generation method based on a deep learning large-scale model for visual algorithms. It aims to address the shortcomings of existing technologies where image segmentation based on large models (such as Meta's SAM model), while capable of zero-shot segmentation, can only segment all objects in an image without discrimination. It cannot filter foreground objects according to specified categories and automatically label category information, resulting in virtual data lacking semantic annotation and making it difficult to directly use for supervised learning training. Furthermore, existing virtual data synthesis methods lack intelligent background matching mechanisms and cannot automatically select suitable backgrounds based on the semantic attributes of foreground objects, leading to insufficient scene rationality in the generated data.

[0005] In a first aspect, embodiments of this application provide a method for generating virtual data using a visual algorithm based on a large deep learning model; the method includes:

[0006] The original image is segmented according to the preset visual model and the specified category to generate foreground objects and corresponding foreground object masks; each segmented foreground object and its corresponding segmentation information are added to the template library as a separate template, and the segmentation information includes the foreground object's corresponding outline, bounding box and labeled category.

[0007] A background library containing diverse backgrounds is constructed, which includes background images with different scenes, lighting conditions, and texture features. A foreground object is selected from the background library and synthesized by image fusion technology with a randomly selected background image from the background library. The foreground object is placed at a predetermined position in the background image, and the corresponding gradient field is calculated to achieve fusion and obtain a fused image. The fused image is then post-processed to eliminate seams or artifacts to obtain a composite image.

[0008] Post-processing of synthetic images using a large deep learning model adjusts lighting, shadows, and texture consistency to enhance realism.

[0009] The processed synthetic images and their corresponding annotation information are integrated to generate a virtual dataset, which is stored in a directory structure containing images, annotation information, and metadata for use in visual algorithm training and testing. The annotation information includes object category, location, bounding box, and outline.

[0010] In some embodiments, the step of segmenting the original image according to a preset visual model and a specified category to generate foreground objects and corresponding foreground object masks includes: inputting a specified category into the visual model, the visual model segmenting foreground objects of that category in the original image, generating foreground object masks, and extracting the contour information, bounding box information, and labeled category of the foreground objects.

[0011] In some embodiments, adding each segmented foreground object and its corresponding segmentation information as a separate template to the template library includes: when the number of foreground objects segmented from the original image is greater than 1, adding each segmented foreground object and its corresponding contour information, bounding box information and annotation category as an independent template to the foreground object template library one by one.

[0012] In some embodiments, constructing a background library containing diverse backgrounds includes: acquiring images under different geographical locations, times, and environmental conditions; performing resolution and quality filtering and preprocessing on the images; adjusting brightness, contrast, and color balance; acquiring images under different lighting conditions, including sunlight, cloudy days, nighttime, and indoor lighting; selecting images with rich textures and details and classifying them according to scene type; establishing a labeling and indexing management system; and constructing the background library.

[0013] In some embodiments, the step of selecting a foreground object from a background library and generating a composite image by image fusion technology with a background image randomly selected from the background library includes: scaling and perspective transformation of the foreground object to adjust its size according to the perspective relationship and proportion of the background image, and placing the adjusted foreground object at a predetermined position in the background image.

[0014] In some embodiments, the step of obtaining a fused image by placing a foreground object at a predetermined position in a background image and calculating the corresponding gradient field includes: using a Poisson fusion algorithm, first initializing the placement of the foreground object at a predetermined position in the background image, calculating the gradient fields of the background image and the corresponding source image, and using numerical methods to solve the Poisson equation to obtain the fused image.

[0015] In some embodiments, post-processing the fused image to eliminate seams or artifacts and obtain a composite image includes: smoothing and thinning the fused image, eliminating visible seams or artifacts in the corresponding fused area by means of Gaussian blur or manual adjustment, and obtaining a composite image.

[0016] In some embodiments, the post-processing of the synthesized image using a deep learning large model to adjust lighting, shadows, and texture consistency to enhance realism includes: inputting prompts to the deep learning large model, which then adjusts the lighting direction and intensity, shadows, and texture consistency of the synthesized image.

[0017] In some embodiments, the step of integrating the processed synthetic image and its corresponding annotation information to generate a virtual dataset includes: designing a directory structure that includes a first directory for storing synthetic images, a second directory for storing annotation information in JSON or XML format, and a third directory for storing metadata; and integrating the synthetic image and its corresponding annotation information such as object category, location, bounding box, and contour into the directory structure.

[0018] In some embodiments, the method further includes using a semantic matching algorithm to analyze the semantic correlation between the labeled category of the foreground object and the scene category of the background image, and selecting a matching background image from the background library based on the scene category semantic correlation to synthesize with the foreground object, so as to improve the diversity and realism of the generated virtual data.

[0019] This application provides a method for generating virtual data based on a deep learning large-scale visual algorithm. The large-scale model automatically segments specified foreground categories and generates labeled information, eliminating the need for manual labeling and saving labeling costs. Combined with a background library, it rapidly synthesizes data, improving generation efficiency compared to traditional methods. A Poisson fusion algorithm achieves seamless stitching of foreground and background, while the large-scale model adjusts lighting, shadows, and textures to make the synthesized image closely resemble a realistic photograph. The background library covers multiple scenes, lighting, and textures, and a semantic matching algorithm intelligently selects backgrounds, enhancing data diversity. No professional 3D modeling or algorithm training capabilities are required; the entire process can be automated simply by calling the large-scale model interface. This method is suitable for small and medium-sized enterprises and research scenarios, promoting the widespread adoption of AI vision technology.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic flowchart illustrating the steps of a method for generating virtual data based on a deep learning large model visual algorithm, provided in one embodiment of this application.

[0023] Figure 2 This is a schematic block diagram of a visual algorithm virtual data generation system based on a large deep learning model, provided in one embodiment of this application.

[0024] Figure 3 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0028] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0029] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] In the field of computer vision, the training of deep learning algorithms heavily relies on large-scale, diverse labeled data. Traditional data acquisition and labeling are labor-intensive and inefficient, becoming a bottleneck restricting the rapid iteration of algorithms. Among existing solutions, data augmentation techniques (such as geometric transformations and color adjustments) struggle to generate complex, realistic scenes, easily leading to models learning incorrect features. While virtual data generation based on game engines can simulate real-world scenes, it requires professional 3D modeling skills and significant time investment, making it unsuitable for small to medium-sized scenarios with limited resources. Furthermore, traditional image segmentation methods require pre-training of segmentation models, presenting a "chicken-or-egg" problem that prevents them from being directly used to automatically generate foreground objects with labeled categories.

[0032] In existing technologies, image segmentation based on large models (such as Meta's SAM model) has zero-shot segmentation capabilities, but it can only segment all objects in an image without discrimination. It cannot filter the foreground according to a specified category and automatically label the category information, resulting in the generated virtual data lacking semantic annotation and making it difficult to use directly for supervised learning training. At the same time, existing virtual data synthesis methods lack intelligent background matching mechanisms and cannot automatically select an appropriate background based on the semantic attributes of foreground objects, resulting in insufficient scene rationality of the generated data.

[0033] Please refer to Figure 1 This application provides a method for generating virtual data based on a deep learning large-scale visual algorithm, applied to computer devices. It should be noted that all information involved in the method provided in this application is extracted with the authorization of the relevant users and in accordance with relevant regulations, and will not infringe on user privacy.

[0034] The provided method for generating virtual data based on a deep learning large-scale visual algorithm includes steps S101 to S104. Details are as follows:

[0035] Step S101. Segment the original image according to the preset visual large model and the specified category to generate foreground objects and corresponding foreground object masks; add each segmented foreground object and its corresponding segmentation information as a separate template to the template library. The segmentation information includes the foreground object's corresponding contour, bounding box, and labeled category.

[0036] Specifically, a large visual model (such as Meta's SAM model) is used to segment the original image, and target foreground objects are extracted by filtering according to specified categories, generating corresponding foreground object masks. The segmentation results are stored in a structured manner as reusable templates, solving the "startup problem" of traditional segmentation methods that require pre-trained models.

[0037] Visual large-scale model initialization involves loading a pre-trained segmentation large-scale model (such as SAM) and combining it with category cues (such as text labels "car" and "pedestrian") or a classification model to achieve semantic-aware segmentation of a specified category. Example: By specifying the target category through SAM's Prompt mechanism (such as bounding box selection or text input), the model outputs the contour mask of objects of that category.

[0038] Foreground segmentation and filtering involves performing full-image segmentation on the input raw image (such as a natural scene photo or a web image) to extract the initial masks of all objects. A category classifier (such as a lightweight CNN or a large model with built-in classification capabilities) is then used to filter the segmented objects by category, retaining only foreground objects that match a preset category (e.g., only retaining the "vehicle" category).

[0039] Template structured storage generates templates for each valid foreground object that include the following information: Foreground image: The object region cropped from the original image (background removed). Mask: A binary mask that marks the object outline and bounding box. Category label: Preset semantic labels (e.g., "car" or "truck"). Templates are stored in a template library and can be retrieved by metadata such as category, size, and pose.

[0040] Step S102. Construct a background library containing diverse backgrounds, the background library including background images with different scenes, lighting conditions and texture features; select a foreground object from the background library and generate a composite image by image fusion technology with a randomly selected background image from the background library; achieve fusion by placing the foreground object at a predetermined position in the background image and calculating the corresponding gradient field to obtain a fused image, and perform post-processing on the fused image to eliminate seams or artifacts to obtain a composite image.

[0041] Specifically, a diverse background library is constructed, and synthetic images are generated by selecting semantically unrelated or weakly correlated random backgrounds and combining them with gradient field fusion technology, thus solving the problem of semantic mismatch between background and foreground in traditional methods.

[0042] Background library construction involves collecting or generating background images that include different scenes (e.g., city, countryside, indoor), lighting conditions (e.g., daytime, nighttime, cloudy), and texture features (e.g., grass, concrete, wood grain). Background images are preprocessed by removing redundant objects, standardizing sizes, and storing scene metadata (e.g., "outdoor-road-sunny").

[0043] Foreground-background matching and fusion is achieved through random background selection by randomly choosing a background image from a background library or filtering suitable backgrounds based on simple semantic rules (such as avoiding extreme size mismatches).

[0044] Foreground placement and coordinate calculation determine the position coordinates of the foreground object in the background image randomly or according to preset rules (such as center area, edge area). The overlap area between the foreground bounding box and the background image is calculated to avoid exceeding the boundaries.

[0045] Gradient field fusion employs Poisson fusion or gradient domain fusion techniques to eliminate seams in the synthesized image by matching the gradient fields of the foreground edge and the background.

[0046] The mathematical implementation involves solving the Poisson equation to make the gradient of the foreground edge consistent with the background. The formula is as follows: 2 f= v; where f is the fused pixel value and v is the gradient field of the background image.

[0047] Post-processing uses Gaussian blur, color balancing, or edge feathering to smooth the blending boundaries and eliminate artifacts.

[0048] Step S103. Post-process the synthesized image using a large deep learning model to adjust lighting, shadows, and texture consistency to enhance realism.

[0049] Specifically, deep learning large models are used to adjust the lighting, shadows, and texture consistency of synthetic images, solving the problem of insufficient scene realism generated by traditional data augmentation techniques (such as geometric transformations).

[0050] Lighting and shadow matching uses lighting estimation models (such as DeepLighting) to analyze the light direction, intensity, and color temperature of the background image, generating lighting parameters (such as ambient light and directional light). Then, 3D rendering techniques or image generation models (such as StyleGAN and Stable Diffusion) are used to add shadows and reflections to foreground objects that match the background. Example: If the background is side-lit, the shadow direction of the foreground object must match the shadow direction of objects in the background.

[0051] Texture consistency adjustment employs conditional generative models (such as cGAN), taking background texture features (such as roughness and color distribution) as input, and generating foreground surface textures that match the background. Style transfer techniques (such as Neural Style Transfer) are then used to transfer the texture style of the background to the foreground object, enhancing overall consistency.

[0052] Global realism optimization uses a discriminator network (such as the discriminator in GAN) to score the realism of the synthesized image and inversely optimizes the fusion parameters (such as transparency and contrast).

[0053] Step S104. Integrate the processed synthetic image and the corresponding annotation information to generate a virtual dataset, which is stored in a directory structure containing images, annotation information and metadata for use in visual algorithm training and testing; the annotation information includes object category, location, bounding box and outline.

[0054] Specifically, the synthesized images and their annotation information are organized into a standard dataset format to support supervised learning training of visual algorithms and solve the problem of lack of semantic annotation in existing virtual data.

[0055] The annotation information generation process extracts annotation information for each foreground object from the template library, including: category labels such as "pedestrian" and "bicycle"; location and geometric information such as bounding box coordinates (x1, y1, x2, y2) and contour vertex coordinate sequence; and metadata such as lighting conditions during compositing, background scene type, and foreground pose (e.g., "car - left front 45 degrees").

[0056] The dataset directory structure organizes files according to standard dataset formats (such as COCO and VOC). Data validation and cleaning are performed through manual sampling or automated validation scripts (such as checking whether the bounding boxes and outlines are aligned and whether the category labels are correct) to ensure annotation quality. Samples with poor fusion effects and semantic contradictions (such as "fish" appearing in a desert background) are filtered out.

[0057] In some embodiments, the step of segmenting the original image according to a preset visual model and a specified category to generate foreground objects and corresponding foreground object masks includes: inputting a specified category into the visual model, the visual model segmenting foreground objects of that category in the original image, generating foreground object masks, and extracting the contour information, bounding box information, and labeled category of the foreground objects.

[0058] Inputting a specified category involves feeding textual cues (such as "car" or "pedestrian") into an improved visual large model (such as Prompt-SAM), which then analyzes the original image based on category semantics.

[0059] Automatic segmentation and annotation identifies all foreground objects belonging to a specified category in the original image using a model, generates corresponding binary masks (foreground as 1, background as 0), and extracts the following information:

[0060] Contour information is obtained by mask edge detection to obtain the pixel-level contour coordinates of the object.

[0061] The bounding box is calculated by taking the smallest bounding rectangle of the mask and obtaining the coordinates [x, y, width, height].

[0062] The category is labeled by directly associating it with the specified category entered (e.g., category ID=1 corresponds to "car").

[0063] The output results return the segmented foreground object image, mask, and structured annotation information, which can be used for subsequent synthesis.

[0064] In some embodiments, adding each segmented foreground object and its corresponding segmentation information as a separate template to the template library includes: when the number of foreground objects segmented from the original image is greater than 1, adding each segmented foreground object and its corresponding contour information, bounding box information and annotation category as an independent template to the foreground object template library one by one.

[0065] Multi-object scene detection involves automatically identifying the independent mask of each object when multiple foreground objects of a specified category are segmented in the original image (such as multiple cars in an image).

[0066] Independent template generation extracts the contour, bounding box, and labeled category for each segmented object, storing them as independent samples in the foreground object template library. For example: Sample 1: Car A, contour coordinate set 1, bounding box 1, category = car. Sample 2: Car B, contour coordinate set 2, bounding box 2, category = car. This supports multi-object data generation in complex scenes, improving dataset diversity.

[0067] In some embodiments, constructing a background library containing diverse backgrounds includes: acquiring images under different geographical locations, times, and environmental conditions; performing resolution and quality filtering and preprocessing on the images; adjusting brightness, contrast, and color balance; acquiring images under different lighting conditions, including sunlight, cloudy days, nighttime, and indoor lighting; selecting images with rich textures and details and classifying them according to scene type; establishing a labeling and indexing management system; and constructing the background library.

[0068] Multi-source data acquisition involves collecting images of different scenes (urban, rural, indoor, etc.), different lighting conditions (sunlight, night, cloudy), and different textures (grass, concrete, wood grain). These images can be obtained through public datasets (such as COCO, ImageNet) or through self-taken photos.

[0069] Preprocessing involves filtering high-resolution images (e.g., ≥1024×1024 pixels), adjusting brightness, contrast, and color balance, and removing blurry or noisy images.

[0070] Images are categorized and managed by scene type (e.g., "traffic scene" or "indoor home"), lighting type (e.g., "nighttime" or "strong light"), and texture features (e.g., "rough surface" or "smooth surface"). An indexing system (e.g., a database or folder hierarchy) is established to facilitate quick retrieval of matching background images.

[0071] In some embodiments, the step of selecting a foreground object from a background library and generating a composite image by image fusion technology with a background image randomly selected from the background library includes: scaling and perspective transformation of the foreground object to adjust its size according to the perspective relationship and proportion of the background image, and placing the adjusted foreground object at a predetermined position in the background image.

[0072] Perspective and Proportion Adjustment: By analyzing the perspective angles (e.g., top-down, eye-level) and object proportions (e.g., the size of a car in the background) of the background image, geometric transformations (e.g., affine transformations, perspective transformations) are used to adjust the size and angle of foreground objects to match the background scene. Example: If the background is a city street (eye-level view), the foreground car needs to be scaled to match the actual proportions of a car as seen from the human eye, and perspective transformations are used to simulate the effect of objects appearing larger when closer and smaller when farther away. Placement: Place the adjusted foreground object in a reasonable area of ​​the background image (e.g., on a street, on a table), avoiding obscuring key background elements or creating an unreasonable layout.

[0073] In some embodiments, the step of obtaining a fused image by placing a foreground object at a predetermined position in a background image and calculating the corresponding gradient field includes: using a Poisson fusion algorithm, first initializing the placement of the foreground object at a predetermined position in the background image, calculating the gradient fields of the background image and the corresponding source image, and using numerical methods to solve the Poisson equation to obtain the fused image.

[0074] Initial placement involves overlaying the foreground object mask onto the target location of the background image to form the initial composite image.

[0075] Gradient field calculation involves calculating the gradient field of the foreground object (reflecting pixel change trends, such as edge direction) and the gradient field of the corresponding region in the background image.

[0076] Solving the Poisson equation using the finite difference method or the Gauss-Seidel iteration method ensures that the pixel values ​​in the fusion region satisfy the constraint that "the gradient field of the fused image is consistent with the gradient field of the foreground." The formula is: Δf = ΔS (within the fusion region), where f is the fused image, S is the foreground source image, and Δ is the Laplacian operator. Post-processing involves Gaussian blurring or manual adjustment of the fusion result to eliminate edge seams.

[0077] In some embodiments, post-processing the fused image to eliminate seams or artifacts and obtain a composite image includes: smoothing and thinning the fused image, eliminating visible seams or artifacts in the corresponding fused area by means of Gaussian blur or manual adjustment, and obtaining a composite image.

[0078] Use a Gaussian filter (such as a 3×3 kernel) to blur the blending boundaries, reducing harsh edges caused by pixel abrupt changes. For areas where automatic processing is ineffective (such as complex texture junctions), manually retouch them using image editing tools to ensure the blended area matches the background texture.

[0079] By eliminating visible artifacts from compositing, foreground objects blend naturally with the background, enhancing visual realism.

[0080] In some embodiments, the post-processing of the synthesized image using a deep learning large model to adjust lighting, shadows, and texture consistency to enhance realism includes: inputting prompts to the deep learning large model, which then adjusts the lighting direction and intensity, shadows, and texture consistency of the synthesized image.

[0081] Input prompts are provided by feeding descriptive prompts (such as "A real car is on a city street, the sun is shining from the upper left, and the tires are in shadow") into generative large models such as Stable Diffusion.

[0082] Model processing adjusts the lighting direction, shadow shape, and texture details (such as vehicle reflections and ground reflections) of the synthesized image based on prompts from a large model, enhancing scene realism. Whether to enable this step depends on computing resources and scene requirements; it can be omitted for simple scenes to improve efficiency.

[0083] In some embodiments, the step of integrating the processed synthetic image and its corresponding annotation information to generate a virtual dataset includes: designing a directory structure that includes a first directory for storing synthetic images, a second directory for storing annotation information in JSON or XML format, and a third directory for storing metadata; and integrating the synthetic image and its corresponding annotation information such as object category, location, bounding box, and contour into the directory structure.

[0084] The directory structure includes: images / (first directory): stores composite images (e.g., JPEG format), named according to the rule "scene_001.jpg"; annotations / (second directory): stores annotation files (JSON / XML format), each file corresponding to one image, containing object category, bounding box, contour coordinates, etc.; and metadata / (third directory): stores simulated shooting parameter data (e.g., virtual camera focal length, aperture), composite timestamps, etc.

[0085] In some embodiments, the method further includes using a semantic matching algorithm to analyze the semantic correlation between the labeled category of the foreground object and the scene category of the background image, and selecting a matching background image from the background library based on the scene category semantic correlation to synthesize with the foreground object, so as to improve the diversity and realism of the generated virtual data.

[0086] Semantic analysis uses Natural Language Processing (NLP) models (such as BERT) to extract semantic vectors of foreground object categories (such as "bird") and background scene labels (such as "forest" and "sky"), and calculates cosine similarity to assess relevance. Example: "Bird" has a higher similarity to "sky" than "bird" and "indoors," so a sky background is preferred for composition.

[0087] Dynamic filtering selects highly matched backgrounds (e.g., similarity > 0.7) from the background library based on semantic relevance scores, replacing random selection and avoiding unreasonable scenarios such as "fish in the desert," thus improving data authenticity.

[0088] In some embodiments, to address the problem of "mismatch between the pose of the foreground object and the logic of the background scene" in existing data synthesis (such as "a cup standing upright in the center of the floor"), a pose estimation + semantic matching intelligent algorithm is introduced. The pose of the foreground object is automatically adjusted according to the semantic attributes of the background scene, making the spatial logic of the foreground and background more consistent, thereby improving the realism of the virtual data and the rationality of the scene.

[0089] Pose feature extraction uses MediaPipe Objectron (a 3D object pose estimation model) to extract 3D pose features of the foreground object, such as the tilt angle and orientation of the "cup". For example, the pose features of the "cup" include "upright" (tilted 0°), "tilted" (tilted 45°), and "sideways" (tilted 90°).

[0090] Background semantic analysis uses CLIP (a multimodal model) to analyze the scene category (such as "desk", "floor", "sofa") and scene attributes (such as "horizontal plane" and "open space") of background images. For example, the scene attributes of "desk" are "horizontal, confined space", and the scene attributes of "floor" are "horizontal, open space".

[0091] Pose-Context Matching Rules: Construct a pose-scene adaptation rule library (generated through manual annotation or machine learning). For example: a "cup" should be in an "upright" pose in an "office desk" scene (which conforms to the everyday logic of "placed on a desktop"); a "cup" should be in a "tilted" or "sideways" pose in a "floor" scene (which conforms to the logic of "falling to the ground"); and a "chair" should be in a "facing the table" pose in a "meeting room" scene (which conforms to the logic of "meeting scene").

[0092] Pose Adaptive Adjustment: If the foreground pose does not match the background scene, use affine transformation or elastic deformation algorithms to adjust the pose of the foreground object. For example, rotate an "upright cup" 45° in a "floor" background to a "tilted" pose, and adjust its position to a "corner" (consistent with the "natural position of falling").

[0093] The adjusted foreground and background were composited, and the realism was verified using NIQE (No Reference Image Quality Assessment Model) to ensure that the blending was natural after the pose adjustment.

[0094] This solution addresses the problem of "conflict in pose logic between foreground and background" in existing solutions. By combining pose estimation and semantic matching with an intelligent algorithm, the pose of foreground objects is made consistent with the spatial logic of the background scene. The generated virtual data is more in line with human cognition, thus improving the model's learning effect on "scene semantics".

[0095] In some embodiments, to address the error propagation problem in existing segmentation annotations (such as bounding boxes and masks) (e.g., segmentation mask vulnerabilities causing annotation bounding box offsets), an intelligent annotation optimization algorithm is introduced to automatically detect and correct annotation errors, improve the accuracy and consistency of annotations, and solve the quality bottleneck in the "segmentation-annotation" link.

[0096] Labeling error detection includes: Bounding box error: Calculate the intersection-union ratio (IoU) between the bounding box and the segmentation mask. If IoU < 0.8, the bounding box is considered to be offset; Mask error: Use connected component analysis to detect gaps in the mask (e.g., the "cup" mask is missing a handle) or redundant regions (e.g., containing background pixels); Class error: Use CLIP to verify whether the class of the segmented object is consistent with the labeled class (e.g., "dog" is mislabeled as "cat").

[0097] Labeling error correction includes: Bounding box correction: Using the YOLOv8 bounding box regression head, with the segmentation mask as the supervision signal, the bounding box coordinates are optimized (e.g., adjusting the bounding box of the "cup" to more accurately surround the mask region); Mask correction: Using the U-Net variant mask refinement network (input original image, initial mask, bounding box), mask gaps are filled (e.g., learning the texture features of the "cup handle" to fill in missing parts); Class correction: If there is a class error, the object is re-segmented using Prompt-SAM, and the labeled class is updated.

[0098] Annotation consistency verification: After correction, verify again: the IoU between the bounding box and the mask is ≥0.9 (ensuring the bounding box completely surrounds the mask); the pixel values ​​(0 / 1) of the mask are completely consistent with the bounding box region (no overflow or omission); the category label is consistent with the object's appearance (e.g., the mask region for "red cup" is red). Replace the original annotations with the corrected annotations (bounding box, mask, category) and store them in the dataset's annotations directory.

[0099] This solution addresses the problem of segmentation errors being directly transmitted to annotations in existing methods. By using an intelligent annotation optimization algorithm, it automatically corrects annotation errors, improves the annotation quality of the dataset, and helps enhance the training performance of visual algorithms (such as object detection).

[0100] In some embodiments, to address the problem of insufficient diversity in existing static background libraries (such as the lack of corresponding background images for "rare birds"), generative AI (such as Stable Diffusion XL and DALL·E 3) is introduced to dynamically generate background images, and semantic matching algorithms are combined to adaptively select the optimal background, thereby achieving dynamic expansion and intelligent matching of the background library.

[0101] Background requirements analysis determines the semantic features of the required background (such as "warm indoor scene" or "sunny windowsill") based on the category (e.g., "cat") and attributes (e.g., "white, long-haired") of the foreground object.

[0102] Dynamic background generation uses Stable Diffusion XL, employing semantic features of foreground objects as cue words to generate background images that meet specific requirements. For example, the cue word could be: "a white long-haired cat sitting on a soft brown sofa in a sunny living room, with a coffee table nearby." Adjusting the CFG scale (to control the consistency between the generated image and the cue word) and steps (to control the generated details) enhances the realism of the background.

[0103] Background quality assessment uses NIQE to evaluate the sharpness and realism of generated backgrounds, and filters out background images with NIQE scores <30 to add to the background library, enabling dynamic expansion of the background library.

[0104] By combining a static background library with dynamically generated backgrounds, a semantic matching algorithm is used to select the background that best matches the foreground object. For example, for "white long-haired cat," the dynamically generated "soft sofa" background (with high semantic relevance) is preferred; if there are not enough dynamically generated backgrounds, the background of the "bedroom" scene is selected from the static background library.

[0105] The foreground and selected background are composited, and realism enhancement steps (such as adjusting lighting and shadows) are used to ensure the realism of the composite image. The matching degree between the background and foreground is verified by manual spot checks.

[0106] This solution addresses the problem of insufficient diversity in the static background library in existing solutions. It dynamically generates backgrounds through generative AI, enabling dynamic expansion of the background library. Simultaneously, by combining semantic matching algorithms, it adaptively selects the optimal background, improving the semantic relevance between the background and foreground, and making the generated virtual data more consistent with the logic of real-world scenarios.

[0107] In some embodiments, to address the problem that existing solutions only generate static image data, video processing and trajectory tracking intelligent algorithms are introduced to generate virtual data of video sequences containing dynamic foreground and background, and automatically label target trajectory information, thereby expanding the application scenarios of virtual data (such as target tracking and action recognition).

[0108] Video foreground segmentation uses Video SAM (Video Segmentation Large Model) to segment foreground objects of a specified category (such as "pedestrians") in the original video, resulting in a sequence of foreground masks and bounding boxes for each frame.

[0109] Foreground motion estimation uses the Lucas-Kanade optical flow method to calculate the motion trajectory of foreground objects (e.g., "a pedestrian walks from the left to the right side of the video").

[0110] Static background dynamization involves using Optical Flow to generate dynamic backgrounds (such as "street moving from left to right") from static background images (such as "street"), simulating the camera following the movement of pedestrians.

[0111] Generative dynamic backgrounds use Stable Video Diffusion (a video generation model) to generate dynamic backgrounds (such as "pedestrians are walking on the street, and cars are slowly driving in the background") by using the motion trajectory of foreground objects as cue words.

[0112] Video data synthesis combines a sequence of video frames of foreground objects with a sequence of dynamic background videos, using inter-frame Poisson fusion technology to avoid gaps between frames (e.g., when a pedestrian moves, the trees in the background should move backward to maintain logical consistency).

[0113] Trajectory annotation generation: Using DeepSORT (a multi-object tracking algorithm), combined with the bounding box sequence of foreground objects, target trajectory information (such as the bounding box coordinates and trajectory ID of the target in each frame) is generated.

[0114] Video dataset construction: Design the directory structure of the video dataset, including videos (stores video sequences), frames (stores keyframes), annotations (stores trajectory annotations), and metadata (stores video metadata).

[0115] This solution addresses the problem of existing solutions being unable to generate video data. By employing video segmentation, dynamic background generation, and trajectory tracking algorithms, it generates virtual data of video sequences containing dynamic foreground and background, and automatically annotates trajectory information. This expands the application scenarios of virtual data and helps improve the training effect of video vision algorithms (such as target tracking).

[0116] In some embodiments, to address the problem of insufficient semantic information in existing schemes (containing only categories, bounding boxes, and masks), multimodal large models (such as BLIP-2 and CLIP) are introduced to extract fine-grained attributes of foreground objects, and the background selection and scene adaptation are optimized by combining attribute information, thereby improving the semantic richness and scene rationality of virtual data.

[0117] Fine-grained attribute extraction: Use BLIP-2 (multimodal model) to extract fine-grained attributes of foreground objects, including: visual attributes, such as: color (e.g., "red"), shape (e.g., "circle"), texture (e.g., "smooth"), and state (e.g., "on");

[0118] Semantic attributes include: purpose (e.g., "used for drinking water") and contextual association (e.g., "commonly found on office desks"). For example, fine-grained attributes of "cup" include "red, round, smooth, open, used for drinking water, common on office desks".

[0119] Attribute-background matching is achieved by building an attribute-background adaptation rule library (generated through manual annotation or machine learning). For example, "red cup" is suitable for "white desk" (the color contrast is obvious and matches everyday scenes); "open laptop" is suitable for "conference room table" (high scene relevance); and "smooth metal ball" is suitable for "laboratory countertop" (texture and scene adaptation).

[0120] Scene adaptation optimization: Adjust the scene details of the composite image based on the fine-grained attributes of the foreground objects to enhance realism. For example, if the attribute of the "cup" is "filled with water", add a "water reflection" effect; if the attribute of the "laptop" is "open", add a "keyboard backlight" effect; if the attribute of the "metal ball" is "smooth", adjust the background lighting direction to make the metal ball produce a "mirror reflection" effect.

[0121] Attribute annotation integration: The extracted fine-grained attributes are added to the annotation information to expand the semantic richness of the annotation.

[0122] Dataset semantic augmentation integrates fine-grained attribute annotations into the dataset to train fine-grained classification (e.g., classifying “red cup” versus “blue cup”) or attribute recognition (e.g., recognizing “open laptop” versus “closed laptop”) algorithms.

[0123] This approach addresses the issue of insufficient semantic annotation in existing solutions by extracting fine-grained attributes through a multimodal large model, thereby expanding the semantic richness of the annotations. Simultaneously, it optimizes background selection and scene adaptation by combining attribute information, improving the scene rationality of the virtual data. The generated dataset is more suitable for training visual algorithms that require fine-grained semantic understanding.

[0124] In some embodiments, training visual deep learning algorithms requires a large amount of diverse data. Traditional data acquisition and annotation processes are not only time-consuming and labor-intensive, but also costly, and limit the rapid iteration and optimization of algorithms.

[0125] There are several approaches to solving this problem: data augmentation and generating virtual data. Data augmentation increases the diversity of existing data by transforming it, such as rotating, scaling, cropping, and adjusting colors. However, inappropriate data augmentation or the inability to generate data with the same complexity as real-world scenes can lead to the model learning incorrect features, thus impairing model performance. For example, excessive geometric transformations or color adjustments can cause images to lose their realism, making it difficult for the model to learn useful information from the augmented data. Regarding virtual data generation, current methods mainly include scene simulation based on game engines. While these methods can simulate real-world scenes to some extent and are used by large companies such as Nvidia and Tesla as a way to generate virtual data for training AI vision models in their practical applications, creating high-quality 3D models and scenes requires specialized skills and a significant time investment. This method is also difficult to apply to resource-constrained and small-scale applications, hindering the widespread adoption of AI vision technology.

[0126] The purpose of this patented method is to address the high cost and low efficiency issues faced in data acquisition and annotation during current visual algorithm training. Specifically, the invention aims to: rapidly generate large amounts of virtual data by combining foreground object matting with varied backgrounds, reducing the time and cost of manual annotation and data acquisition; ensure the generated data has visual realism; and enhance the diversity of the dataset through diverse background and foreground combinations to meet the needs of different scenarios. It provides a simple and easy-to-operate method, lowering the technical threshold and promoting the widespread application of virtual data generation technology.

[0127] This invention provides a virtual data generation method, particularly suitable for rapidly generating large amounts of realistic and diverse training data in visual algorithm training. This method combines foreground object segmentation based on a large visual model with varied backgrounds to generate virtual data in batches on a large scale, ensuring both data realism and meeting diversity requirements. The specific technical solution includes the following steps:

[0128] 1. Foreground Object Segmentation: Traditional segmentation algorithms require training a segmentation model first. For most practical applications, this is a chicken-and-egg problem. Unlike previous methods, this method uses an improved image segmentation technique based on state-of-the-art large-scale vision models, which can automatically separate foreground objects of any specified category from the original image without pre-training.

[0129] 2. Background Library Construction: Build an image library containing a variety of backgrounds.

[0130] 3. Data synthesis: The foreground objects obtained from the previous segmentation are randomly or regularly combined with images from the background library to generate new virtual scene images.

[0131] 4. Enhanced Realism: By adjusting the lighting, shadows, and textures of the synthesized image through image processing techniques, the realism of the image is enhanced.

[0132] 5. Dataset Construction: The synthesized images and their corresponding annotation information are integrated into a dataset for training the vision algorithm.

[0133] The specific implementation steps corresponding to the embodiment include:

[0134] Step 1: Foreground Object Segmentation: The technical solution employs an improved version of the SegmentAnything Model (SAM) visual large-scale model released by Meta AI to automatically segment foreground objects in the image. The SAM visual large-scale model is an advanced image segmentation model released by Meta AI. It was trained on 11 million images with over 1 billion masks, achieving powerful zero-shot generalization capabilities. In other words, it can segment any object in any image or video, even if those objects never appeared during training.

[0135] However, directly using it in scenarios involving the generation of virtual data presents a problem: our ultimate goal is to generate automatically labeled data with object categories. SAM, however, indiscriminately segments all objects in an image and cannot label or classify the segmented objects. Therefore, we improved the SAM model, calling it Prompt-SAM. This model can automatically segment foreground objects of a specified category based on textually specified categories, generate foreground object masks, and achieve high-precision segmentation of specified foreground objects. The segmentation information, including contour information, bounding box information, and labeled categories (see the automatic dataset construction section in step five for specific definitions and examples of these information), is added to a foreground object template library, laying the foundation for effectively generating virtual data useful for training deep vision models.

[0136] When the number of foreground objects segmented from the image is greater than 1, we add each segmented object as a separate foreground object template to the template library.

[0137] Step Two: Background Library Construction: The goal of building a background library for image segmentation is to create a diverse and comprehensive collection of images to generate realistic and varied backgrounds when synthesizing virtual data. The following are the detailed steps for building the background library:

[0138] 1. Diverse scene acquisition: Collect images from different geographical locations, times, and environmental conditions to ensure a high degree of diversity in the background images in the background library.

[0139] 2. Image Quality Control: Ensure that the image resolution and quality meet the compositing requirements. High resolution helps generate more detailed composite images. Preprocess the image, such as adjusting brightness, contrast, and color balance, to guarantee image quality.

[0140] 3. Illumination Condition Coverage: Collect images under different illumination conditions, such as sunlight, cloudy days, nighttime, and indoor lighting, to simulate various illumination environments.

[0141] 4. Rich textures and details: Select images with rich textures and details, such as different ground surfaces, walls, vegetation, etc. These details help to improve the realism of the composite image.

[0142] 5. Scene Type Classification: Collected images are classified into categories such as city, indoor, rural, beach, and mountain, so that specific background types can be selected based on the type of foreground objects during compositing. For example, trees generally do not appear in a home environment.

[0143] 6. Data Management: Establish an effective management system to label and index background images for easy and quick retrieval and use.

[0144] 7. Utilize public datasets: Use existing public background image datasets, such as ImageNet and COCO, as a supplement to the background library.

[0145] Step 3: Data Synthesis: For each foreground object, randomly select a background image from the background library and composite the foreground object into the background image. The synthesis process can be achieved using image fusion techniques, such as the Poisson fusion algorithm. The Poisson fusion algorithm seamlessly blends the foreground object into the selected background image. The core idea of ​​the Poisson fusion algorithm is to solve the Poisson equation, which is driven by the gradient field of the source image, to ensure that the pixels in the fusion region are as consistent as possible with the gradient field of the source image. Poisson fusion can be expressed by the following mathematical formula: Δf = Δ(Sf); where f is the fused image, S is the source image, and Δ is the Laplacian operator used to calculate the gradient field of the image.

[0146] Fusion process:

[0147] 1. Initialization: Place the foreground object at a predetermined position in the background image.

[0148] 2. Gradient calculation: Calculate the gradient fields of the source image and the background image.

[0149] 3. Solve the Poisson equation: Use numerical methods (such as the finite difference method or the Gauss-Seidel iterative method) to solve the Poisson equation and obtain the fused image.

[0150] 4. Post-processing: Smooth and refine the fused image to eliminate any visible seams or artifacts.

[0151] 5. Iterative optimization: If the fusion effect is not ideal, the fusion process can be iteratively optimized by adjusting the gradient field of the source image or the selection of the background image.

[0152] Step Four: Realism Enhancement: Post-processing of the synthesized image, including adjusting lighting direction and intensity, adding shadows, and enhancing texture consistency, to improve the realism of the synthesized image. This step can be achieved by calling the StableDiffusion large model or its variants to post-process the image's realism, and the use of prompts can greatly enhance the image's realism. However, because the StableDiffusion algorithm is relatively resource-intensive, and practical experience has shown that this step is not necessary in some scenarios, it is generally considered optional.

[0153] Step 5: Automatic Dataset Construction: All synthesized images and their annotation information (such as object category, location, bounding box, or outline) are integrated into a dataset. This dataset can be used to train and test vision algorithms, such as object detection and image classification tasks.

[0154] Dataset directory structure design facilitates data management and access by creating a clear directory structure. It typically includes: images / : stores all composite images; annotations / : stores image annotation information, in formats such as JSON or XML; metadata / : stores image metadata, such as shooting parameters, timestamps, etc.

[0155] For each synthesized image, these foreground objects are generated by our algorithm, so their annotation information, including object category, location, bounding box, or outline, is already known. Therefore, we can easily integrate this information to generate annotated data structures. For example, a JSON format for annotation information might look like this:

[0156] {

[0157] "image_id": "image_001.jpg",

[0158] "annotations": [

[0159] {

[0160] "category_id": 1,

[0161] "bbox": [x, y, width, height],

[0162] "score": 0.98

[0163] },

[0164] {

[0165] "category_id": 2,

[0166] "polygon": [[x1, y1], [x2, y2], [x3, y3], ...],

[0167] "score": 0.95

[0168] } ]

[0170] This data can then be used to train deep vision models.

[0171] This application provides a method for generating virtual data based on a deep learning large-scale visual algorithm. The large-scale model automatically segments specified foreground categories and generates labeled information, eliminating the need for manual labeling and saving labeling costs. Combined with a background library, it rapidly synthesizes data, improving generation efficiency compared to traditional methods. A Poisson fusion algorithm achieves seamless stitching of foreground and background, while the large-scale model adjusts lighting, shadows, and textures to make the synthesized image closely resemble a realistic photograph. The background library covers multiple scenes, lighting, and textures, and a semantic matching algorithm intelligently selects backgrounds, enhancing data diversity. No professional 3D modeling or algorithm training capabilities are required; the entire process can be automated simply by calling the large-scale model interface. This method is suitable for small and medium-sized enterprises and research scenarios, promoting the widespread adoption of AI vision technology.

[0172] Please see Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a deep learning-based large-scale visual algorithm virtual data generation system 200 provided in this application embodiment. The deep learning-based large-scale visual algorithm virtual data generation system 200 is used to execute the steps of the deep learning-based large-scale visual algorithm virtual data generation method shown in the above embodiments. The deep learning-based large-scale visual algorithm virtual data generation system 200 can be a single server or a server cluster, or it can be a terminal, such as a handheld terminal, laptop computer, wearable device, or robot.

[0173] like Figure 2 As shown, the virtual data generation system 200 based on deep learning large-scale visual algorithms includes:

[0174] Image segmentation unit 201 is used to segment the original image according to the preset visual large model and the specified category, generate foreground objects and corresponding foreground object masks; and add each segmented foreground object and its corresponding segmentation information as a separate template to the template library. The segmentation information includes the foreground object's corresponding contour, bounding box and labeled category.

[0175] Background construction unit 202 is used to construct a background library containing diverse backgrounds, the background library including background images with different scenes, lighting conditions and texture features; select a foreground object from the background library and generate a composite image by image fusion technology with a randomly selected background image from the background library; obtain a fused image by placing the foreground object at a predetermined position in the background image and calculating the corresponding gradient field; and perform post-processing on the fused image to eliminate seams or artifacts to obtain a composite image.

[0176] Image processing unit 203 is used to post-process the synthesized image using a large deep learning model to adjust the lighting, shadow and texture consistency to enhance realism.

[0177] The data generation unit 204 is used to integrate the processed synthetic image and the corresponding annotation information to generate a virtual dataset, which is stored in a directory structure containing images, annotation information and metadata for visual algorithm training and testing; the annotation information includes object category, location, bounding box and outline.

[0178] In some embodiments, the step of segmenting the original image according to a preset visual model and a specified category to generate foreground objects and corresponding foreground object masks includes: inputting a specified category into the visual model, the visual model segmenting foreground objects of that category in the original image, generating foreground object masks, and extracting the contour information, bounding box information, and labeled category of the foreground objects.

[0179] In some embodiments, adding each segmented foreground object and its corresponding segmentation information as a separate template to the template library includes: when the number of foreground objects segmented from the original image is greater than 1, adding each segmented foreground object and its corresponding contour information, bounding box information and annotation category as an independent template to the foreground object template library one by one.

[0180] In some embodiments, constructing a background library containing diverse backgrounds includes: acquiring images under different geographical locations, times, and environmental conditions; performing resolution and quality filtering and preprocessing on the images; adjusting brightness, contrast, and color balance; acquiring images under different lighting conditions, including sunlight, cloudy days, nighttime, and indoor lighting; selecting images with rich textures and details and classifying them according to scene type; establishing a labeling and indexing management system; and constructing the background library.

[0181] In some embodiments, the step of selecting a foreground object from a background library and generating a composite image by image fusion technology with a background image randomly selected from the background library includes: scaling and perspective transformation of the foreground object to adjust its size according to the perspective relationship and proportion of the background image, and placing the adjusted foreground object at a predetermined position in the background image.

[0182] In some embodiments, the step of obtaining a fused image by placing a foreground object at a predetermined position in a background image and calculating the corresponding gradient field includes: using a Poisson fusion algorithm, first initializing the placement of the foreground object at a predetermined position in the background image, calculating the gradient fields of the background image and the corresponding source image, and using numerical methods to solve the Poisson equation to obtain the fused image.

[0183] In some embodiments, post-processing the fused image to eliminate seams or artifacts and obtain a composite image includes: smoothing and thinning the fused image, eliminating visible seams or artifacts in the corresponding fused area by means of Gaussian blur or manual adjustment, and obtaining a composite image.

[0184] In some embodiments, the post-processing of the synthesized image using a deep learning large model to adjust lighting, shadows, and texture consistency to enhance realism includes: inputting prompts to the deep learning large model, which then adjusts the lighting direction and intensity, shadows, and texture consistency of the synthesized image.

[0185] In some embodiments, the step of integrating the processed synthetic image and its corresponding annotation information to generate a virtual dataset includes: designing a directory structure that includes a first directory for storing synthetic images, a second directory for storing annotation information in JSON or XML format, and a third directory for storing metadata; and integrating the synthetic image and its corresponding annotation information such as object category, location, bounding box, and contour into the directory structure.

[0186] In some embodiments, the method further includes using a semantic matching algorithm to analyze the semantic correlation between the labeled category of the foreground object and the scene category of the background image, and selecting a matching background image from the background library based on the scene category semantic correlation to synthesize with the foreground object, so as to improve the diversity and realism of the generated virtual data.

[0187] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the virtual data generation system based on deep learning large models and its modules described above can be found in the corresponding contents of the various embodiments of the virtual data generation method based on deep learning large models, and will not be repeated here.

[0188] The aforementioned method for generating virtual data based on deep learning large-scale visual algorithms can be implemented as a computer program, which can be used in various ways, such as... Figure 2 It runs on the device shown.

[0189] Please see Figure 3 , Figure 3This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application. The computer device includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and internal memory.

[0190] The storage medium can store operating devices and computer programs. The computer program includes program instructions that, when executed, cause the processor to perform any virtual data generation method based on a deep learning-based large-scale visual algorithm.

[0191] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0192] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any virtual data generation method based on a deep learning large-scale visual algorithm.

[0193] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0194] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0195] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0196] The original image is segmented according to the preset visual model and the specified category to generate foreground objects and corresponding foreground object masks; each segmented foreground object and its corresponding segmentation information are added to the template library as a separate template, and the segmentation information includes the foreground object's corresponding outline, bounding box and labeled category.

[0197] A background library containing diverse backgrounds is constructed, which includes background images with different scenes, lighting conditions, and texture features. A foreground object is selected from the background library and synthesized by image fusion technology with a randomly selected background image from the background library. The foreground object is placed at a predetermined position in the background image, and the corresponding gradient field is calculated to achieve fusion and obtain a fused image. The fused image is then post-processed to eliminate seams or artifacts to obtain a composite image.

[0198] Post-processing of synthetic images using a large deep learning model adjusts lighting, shadows, and texture consistency to enhance realism.

[0199] The processed synthetic images and their corresponding annotation information are integrated to generate a virtual dataset, which is stored in a directory structure containing images, annotation information, and metadata for use in visual algorithm training and testing. The annotation information includes object category, location, bounding box, and outline.

[0200] In some embodiments, the step of segmenting the original image according to a preset visual model and a specified category to generate foreground objects and corresponding foreground object masks includes: inputting a specified category into the visual model, the visual model segmenting foreground objects of that category in the original image, generating foreground object masks, and extracting the contour information, bounding box information, and labeled category of the foreground objects.

[0201] In some embodiments, adding each segmented foreground object and its corresponding segmentation information as a separate template to the template library includes: when the number of foreground objects segmented from the original image is greater than 1, adding each segmented foreground object and its corresponding contour information, bounding box information and annotation category as an independent template to the foreground object template library one by one.

[0202] In some embodiments, constructing a background library containing diverse backgrounds includes: acquiring images under different geographical locations, times, and environmental conditions; performing resolution and quality filtering and preprocessing on the images; adjusting brightness, contrast, and color balance; acquiring images under different lighting conditions, including sunlight, cloudy days, nighttime, and indoor lighting; selecting images with rich textures and details and classifying them according to scene type; establishing a labeling and indexing management system; and constructing the background library.

[0203] In some embodiments, the step of selecting a foreground object from a background library and generating a composite image by image fusion technology with a background image randomly selected from the background library includes: scaling and perspective transformation of the foreground object to adjust its size according to the perspective relationship and proportion of the background image, and placing the adjusted foreground object at a predetermined position in the background image.

[0204] In some embodiments, the step of obtaining a fused image by placing a foreground object at a predetermined position in a background image and calculating the corresponding gradient field includes: using a Poisson fusion algorithm, first initializing the placement of the foreground object at a predetermined position in the background image, calculating the gradient fields of the background image and the corresponding source image, and using numerical methods to solve the Poisson equation to obtain the fused image.

[0205] In some embodiments, post-processing the fused image to eliminate seams or artifacts and obtain a composite image includes: smoothing and thinning the fused image, eliminating visible seams or artifacts in the corresponding fused area by means of Gaussian blur or manual adjustment, and obtaining a composite image.

[0206] In some embodiments, the post-processing of the synthesized image using a deep learning large model to adjust lighting, shadows, and texture consistency to enhance realism includes: inputting prompts to the deep learning large model, which then adjusts the lighting direction and intensity, shadows, and texture consistency of the synthesized image.

[0207] In some embodiments, the step of integrating the processed synthetic image and its corresponding annotation information to generate a virtual dataset includes: designing a directory structure that includes a first directory for storing synthetic images, a second directory for storing annotation information in JSON or XML format, and a third directory for storing metadata; and integrating the synthetic image and its corresponding annotation information such as object category, location, bounding box, and contour into the directory structure.

[0208] In some embodiments, the method further includes using a semantic matching algorithm to analyze the semantic correlation between the labeled category of the foreground object and the scene category of the background image, and selecting a matching background image from the background library based on the scene category semantic correlation to synthesize with the foreground object, so as to improve the diversity and realism of the generated virtual data.

[0209] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the virtual data generation method based on a deep learning large model visual algorithm provided in any embodiment of this application.

[0210] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0211] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating virtual data based on a deep learning large-scale visual algorithm, characterized in that, include: The original image is segmented based on a pre-defined visual model and a specified category to generate foreground objects and their corresponding foreground object masks. Each segmented foreground object and its corresponding segmentation information are added to the template library as a separate template. The segmentation information includes the foreground object's corresponding outline, bounding box, and annotation category. Construct a background library containing diverse backgrounds, including background images with different scenes, lighting conditions, and texture features; A foreground object is selected from the background library and synthesized by image fusion technology with a randomly selected background image from the background library. The foreground object is placed at a predetermined position in the background image, and the corresponding gradient field is calculated to achieve fusion and obtain a fused image. The fused image is then post-processed to eliminate seams or artifacts to obtain a synthetic image. Post-processing of synthetic images using a large deep learning model adjusts lighting, shadows, and texture consistency to enhance realism. The processed synthetic images and their corresponding annotation information are integrated to generate a virtual dataset, which is stored in a directory structure containing images, annotation information, and metadata for use in visual algorithm training and testing. The annotation information includes object category, location, bounding box, and outline.

2. The method according to claim 1, characterized in that, The step of segmenting the original image based on a preset visual large model and a specified category to generate foreground objects and corresponding foreground object masks includes: Input the specified category into the visual big model. The visual big model segments the foreground objects of that category in the original image, generates a foreground object mask, and extracts the contour information, bounding box information, and labeled category of the foreground objects.

3. The method according to claim 1, characterized in that, The step of adding each segmented foreground object and its corresponding segmentation information as a separate template to the template library includes: When the number of foreground objects segmented from the original image is greater than 1, each segmented foreground object and its corresponding contour information, bounding box information and annotation category are treated as independent templates and added to the foreground object template library one by one.

4. The method according to claim 1, characterized in that, The construction includes a diverse background library, including: Acquire images under different geographical locations, times, and environmental conditions; perform resolution and quality screening and preprocessing on the images; and adjust brightness, contrast, and color balance. Images under different lighting conditions, including sunlight, cloudy days, nighttime, and indoor lighting, are acquired. Images with rich textures and details are selected and classified according to scene type. A labeling and indexing management system is established to construct the background library.

5. The method according to claim 1, characterized in that, The step of selecting a foreground object from a background library and generating a composite image by image fusion technology with a randomly selected background image from the background library includes: Based on the perspective and proportions of the background image, the size of the foreground object is adjusted by scaling and perspective transformation, and then the adjusted foreground object is placed in a predetermined position in the background image.

6. The method according to claim 1, characterized in that, The step of placing a foreground object at a predetermined position in the background image and calculating the corresponding gradient field to achieve fusion and obtain a fused image includes: The Poisson fusion algorithm is adopted. First, the foreground object is placed at a predetermined position in the background image. Then, the gradient fields of the background image and the corresponding source image are calculated. Finally, the Poisson equation is solved numerically to obtain the fused image.

7. The method according to claim 1, characterized in that, The post-processing of the fused image to eliminate seams or artifacts and obtain a composite image includes: The fused image is smoothed and refined by using methods such as Gaussian blur or manual adjustment to eliminate visible seams or artifacts in the corresponding fusion area, thus obtaining the composite image.

8. The method according to claim 1, characterized in that, The post-processing of the synthesized image using a large deep learning model to adjust lighting, shadows, and texture consistency to enhance realism includes: By inputting prompts into a large deep learning model, the model adjusts the lighting direction and intensity, shadow and texture consistency of the synthesized image.

9. The method according to claim 1, characterized in that, The process of integrating the processed synthetic image and its corresponding annotation information to generate a virtual dataset includes: The design includes a directory structure with a first directory storing composite images, a second directory storing annotation information in JSON or XML format, and a third directory storing metadata. The composite images and their corresponding annotation information, such as object categories, locations, bounding boxes, and outlines, are integrated into the directories corresponding to the directory structure.

10. The method according to claim 1, characterized in that, The method also includes using a semantic matching algorithm to analyze the semantic correlation between the labeled category of the foreground object and the scene category of the background image, and selecting a matching background image from the background library based on the scene category semantic correlation to synthesize with the foreground object, so as to improve the diversity and realism of the generated virtual data.

Images