Virtual data generation method combining 3D foreground object and real background picture

Abstract

The application relates to the technical field of three-dimensional modeling and image processing, and provides a virtual data generation method combining a 3D foreground object and a real background picture. The method comprises the following steps: obtaining a 3D model constructed based on a light-weight 3D modeling technology, and performing simplification polygon and optimization texture mapping optimization processing on the 3D model; obtaining a background picture; adjusting the size of a corresponding foreground object of the 3D model according to the perspective and proportion of the background picture, so that the foreground object and the background perspective relationship are matched; simulating corresponding lighting conditions of the background picture by using a preset lighting model, and applying the simulated lighting conditions to the foreground object to obtain a composite image; enhancing the texture details of the foreground object by using a detail enhancement algorithm on the composite image, performing color correction to make the color distribution consistent with the background picture, and simulating noise in real shooting; and integrating known foreground object labeling information for each composite image to generate corresponding virtual data.

Description

Technical Field

[0001] This application relates to the field of 3D modeling and image processing technology, and in particular to a method for generating virtual data by combining 3D foreground objects with real background images. Background Technology

[0002] In the field of virtual data generation, existing technologies mainly face bottlenecks in two major technical approaches:

[0003] 1. Although full 3D scene rendering technology can simulate real scenes, it relies on high-end hardware and professional engines, resulting in a single system costing hundreds of thousands of dollars. Furthermore, rendering a single image takes tens of minutes to several hours, making it difficult to meet the efficiency and cost requirements for large-scale data generation.

[0004] While 2.2D image fusion technology is relatively low-cost, it has fundamental flaws: First, the lighting conditions of the foreground object and the real background cannot be automatically matched, resulting in obvious light and shadow breaks in the synthesized image; second, it lacks geometric correction methods for perspective relationships, and simple superposition will cause distortion of object size and perspective; third, a lot of manual post-processing is required to make up for the lack of realism, which will increase hidden costs.

[0005] Therefore, a method is urgently needed to solve at least one of the above problems. Summary of the Invention

[0006] This application provides a virtual data generation method that combines 3D foreground objects with real background images, aiming to solve the problem that there is no solution in the prior art that reduces the cost of foreground construction through lightweight 3D modeling, achieves perspective consistency correction based on vanishing line algorithm, and achieves light and shadow fusion through dynamic lighting simulation.

[0007] In a first aspect, embodiments of this application provide a method for generating virtual data that combines a 3D foreground object with a real background image; the method includes:

[0008] A 3D model constructed using lightweight 3D modeling technology is obtained. The 3D model is then simplified in polygons and optimized in texture mapping to reduce the amount of rendering computation. The model categories corresponding to the 3D model include at least pedestrians, vehicles, and animals. Background images are obtained by filtering from multiple preset real-world scene images based on preset diversity requirements. The multiple real-world scene images cover different geographical, climatic, and temporal conditions.

[0009] The size of the foreground object in the 3D model is adjusted according to the perspective and proportion of the background image. The corresponding camera extrinsic parameters are obtained by calculating the vanishing line of the background image, and the angle of the foreground object is adjusted using the camera extrinsic parameters to match the perspective relationship between the foreground object and the background. After matching, the size and angle of the foreground object are saved so that the corresponding size and angle of the foreground object can be loaded when using the background image to randomly place the virtual foreground object. The camera extrinsic parameters include the position and the orientation angle of the 3D degrees of freedom.

[0010] The lighting conditions corresponding to the background image are simulated using a preset lighting model, and the simulated lighting conditions are applied to the foreground object to obtain a composite image; the lighting conditions include the lighting effects under different times and weather conditions.

[0011] The synthetic image is enhanced with a detail enhancement algorithm to improve the texture details of the foreground objects, and color correction is performed to make the color distribution consistent with the background image. Noise is simulated in real shooting, including lens flare and noise. For each synthetic image, the known foreground object annotation information is integrated to generate corresponding virtual data. The foreground object annotation information includes at least one or more of the following: object category, position, bounding box, and outline.

[0012] In some embodiments, obtaining a 3D model constructed based on lightweight 3D modeling technology includes: selecting the 3D model from a preset lightweight 3D model resource library, or training to generate the 3D model.

[0013] In some embodiments, the simplification of polygons and optimization of texture mapping of the 3D model includes: simplifying the polygons of the 3D model using a mesh simplification algorithm to reduce the number of polygons; and optimizing the texture mapping of the 3D model by adjusting the texture resolution and compressing the texture mapping format to reduce the amount of computation during rendering while maintaining the texture and details of the 3D model.

[0014] In some embodiments, the step of filtering from a plurality of preset real-world scene images to obtain a background image based on preset diversity requirements includes: classifying and filtering from the plurality of real-world scene images according to different geographical regions, climate conditions, and time conditions, and selecting a background image that meets the requirements.

[0015] In some embodiments, adjusting the size of the foreground object corresponding to the 3D model based on the perspective and proportion of the background image includes: analyzing the perspective relationship of the background image, identifying reference objects in the background image, calculating the ratio of the actual size of the object in the background image to the image size, and adjusting the size of the foreground object corresponding to the 3D model according to the ratio so that the proportion of the foreground object is consistent with the proportion of the background scene corresponding to the background image.

[0016] In some embodiments, obtaining the corresponding camera extrinsic parameters by calculating the vanishing lines of the background image includes: performing edge detection on the background image and extracting straight lines using methods such as Hough transform; finding parallel line pairs among the detected straight lines using clustering algorithms; calculating the corresponding intersection point as the vanishing point for each line pair; connecting vanishing points in different directions to form a vanishing line; and constructing an equation using the properties of the vanishing line to solve for the camera extrinsic parameters in combination with known camera intrinsic parameters.

[0017] In some embodiments, adjusting the angle of the foreground object using camera extrinsic parameters to match the perspective relationship between the foreground object and the background includes: performing rotation and translation transformations on the 3D model of the foreground object based on the position and orientation angle of the 3D degrees of freedom in the acquired camera extrinsic parameters, and projecting the transformed foreground object onto the background image plane to match the perspective relationship between the foreground object and the background image.

[0018] In some embodiments, the step of simulating the lighting conditions corresponding to the background image using a preset lighting model and applying the simulated lighting conditions to the foreground object to obtain a composite image includes: analyzing the lighting characteristics of the background image, such as the lighting direction and color temperature; using the physical lighting model in the rendering engine to simulate the lighting effects under different time and weather conditions corresponding to the background image; applying the simulated lighting effects to the foreground object; adjusting the material, shadow, and other parameters of the foreground object to make the lighting effects of the foreground object consistent with the background image; and obtaining a composite image.

[0019] In some embodiments, the step of using detail enhancement algorithms to enhance the texture details of foreground objects and performing color correction to make the color distribution consistent with the background image includes: using detail enhancement algorithms such as image sharpening to enhance the corresponding texture details of foreground objects in the composite image; and using histogram matching algorithms or color transfer algorithms to perform color correction on the composite image to make the color distribution of the composite image consistent with the color distribution of the background image, thereby enhancing the overall harmony.

[0020] In some embodiments, the step of integrating known foreground object annotation information to generate corresponding virtual data for each synthesized image includes: extracting foreground object category information from the metadata of the 3D model to calculate foreground object annotation information for the position, bounding box coordinates, or contour coordinates of the foreground object in the synthesized image; and integrating the corresponding image identifier and foreground object annotation information to generate structured data in a preset format, wherein the foreground object annotation information includes object category, position, bounding box, or contour.

[0021] This application provides a virtual data generation method that combines 3D foreground objects with real background images. By performing lightweight 3D modeling only on the foreground objects, the high hardware and labor costs of full-scene 3D rendering are avoided, thus improving modeling efficiency. A vanishing line algorithm is used to achieve perspective consistency between the foreground and background, and dynamic lighting simulation is used to eliminate light and shadow faults. The realism of the synthesized image is improved through subjective evaluation. The real background covers multiple geographical, climatic, and temporal conditions. With the flexible use of the foreground model library, a diverse dataset of millions of images can be generated quickly. Since the parameters of the foreground objects are known, structured labeled data containing category, location, and contour can be automatically generated, significantly saving manual labeling costs.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic flowchart illustrating the steps of a virtual data generation method combining 3D foreground objects and real background images, provided in an embodiment of this application.

[0025] Figure 2 This is a schematic diagram illustrating the principle of a virtual data generation method combining 3D foreground objects and real background images, provided in an embodiment of this application.

[0026] Figure 3 This is a schematic block diagram of a virtual data generation system that combines 3D foreground objects with real background images, provided in one embodiment of this application.

[0027] Figure 4 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0031] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0032] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0033] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0034] In the field of virtual data generation, existing technologies mainly face bottlenecks in two major technical approaches:

[0035] 1. Although full 3D scene rendering technology can simulate real scenes, it relies on high-end hardware and professional engines, resulting in a single system costing hundreds of thousands of dollars. Furthermore, rendering a single image takes tens of minutes to several hours, making it difficult to meet the efficiency and cost requirements for large-scale data generation.

[0036] While 2.2D image fusion technology is relatively low-cost, it has fundamental flaws: First, the lighting conditions of the foreground object and the real background cannot be automatically matched, resulting in obvious light and shadow breaks in the synthesized image; second, it lacks geometric correction methods for perspective relationships, and simple superposition will cause distortion of object size and perspective; third, a lot of manual post-processing is required to make up for the lack of realism, which will increase hidden costs.

[0037] There is currently no solution to address the problem of multi-technology collaborative approach that uses lightweight 3D modeling to reduce foreground construction costs, vanishing line algorithm to achieve perspective consistency correction, and dynamic lighting simulation to achieve light and shadow fusion.

[0038] Please refer to Figure 1 This application provides a method for generating virtual data by combining 3D foreground objects with real background images, applied to computer devices. It should be noted that all information involved in the method provided in this application is extracted with the authorization of the relevant user and in accordance with relevant regulations, and will not infringe on user privacy.

[0039] The provided method for generating virtual data by combining 3D foreground objects with real background images includes steps S101 to S104. Details are as follows:

[0040] Step S101. Obtain a 3D model constructed based on lightweight 3D modeling technology. Perform polygon simplification and texture mapping optimization on the 3D model to reduce the amount of rendering computation. The model category corresponding to the 3D model includes at least pedestrians, vehicles and animals. According to the preset diversity requirements, filter from multiple preset real scene images to obtain background images. Multiple real scene images cover different geographical, climatic and time conditions.

[0041] Specifically, lightweight 3D modeling reduces the computational complexity of 3D models by simplifying polygonal meshes and optimizing texture mapping, thus solving the high cost problem of full 3D rendering. The models cover common target categories such as pedestrians, vehicles, and animals, supporting diverse needs for subsequent scene compositing.

[0042] Background image selection is based on preset geographical, climatic, and temporal conditions (such as city / countryside, sunny / rainy day, day / night) to select background images from a real-world image library, providing diverse and realistic environmental bases for compositing.

[0043] 3D model optimization uses mesh simplification algorithms (such as Quadric Edge Collapse) to reduce the number of polygons while retaining key model features; it employs texture compression techniques (such as ETC1 and ASTC) to reduce texture map resolution, and at the same time reduces the amount of sampling computation during rendering by merging multiple layers of materials through texture baking; it establishes a standardized model library, which is stored by category (pedestrians / vehicles / animals) and supports fast access.

[0044] Background image preprocessing involves building a background image library containing multi-source data and labeling the metadata of each image (geographic location, weather, shooting time, etc.). Based on task requirements (such as autonomous driving data generation), background images that meet the criteria are filtered through keyword retrieval or machine learning classification models (such as ResNet), for example, filtering images of the "city road - cloudy day - evening" scene.

[0045] Step S102. Adjust the size of the foreground object in the 3D model according to the perspective and proportion of the background image. Obtain the corresponding camera extrinsic parameters by calculating the vanishing line of the background image, and use the camera extrinsic parameters to adjust the angle of the foreground object so that the perspective relationship between the foreground object and the background is matched. After matching, save the corresponding foreground object size and angle so that the corresponding foreground object size and angle can be loaded when using the background image to randomly place the foreground virtual object. The camera extrinsic parameters include the position and the orientation angle of the 3D degrees of freedom.

[0046] Specifically, by analyzing the perspective relationships (vanishing lines, vanishing points) of the background image, calculating the camera's extrinsic parameters (position, orientation angle), and adjusting the size and angle of the 3D foreground object to make it consistent with the background perspective, the problem of object size and perspective distortion in 2D fusion is solved.

[0047] Vanishing line detection involves performing edge detection on the background image (such as using the Canny operator) to extract straight line features (such as road edges and building outlines); then using the RANSAC algorithm to fit the straight lines, grouping collinear lines, calculating the intersection point (vanishing point) of each group of lines, and determining the perspective coordinate system of the background image (such as a single vanishing point or double vanishing point model).

[0048] The camera extrinsic parameter calculation is based on the mapping relationship between the vanishing point coordinates and the world coordinate system. The rotation matrix and translation vector (extrinsic parameters) of the camera are solved. For example, the Hartley-Zisserman algorithm is used to solve the Perspective-n-Point (PnP) problem. The scaling ratio of the 3D model (to match the size of the background object) and the rotation angle (to match the camera's viewpoint) are adjusted by the extrinsic parameters so that the foreground object presents the correct perspective relationship in the background.

[0049] The parameter storage and reuse feature stores the adjusted foreground object size, angle, and corresponding background image ID in the database, supporting quick retrieval during subsequent compositing and enabling random placement of foreground objects (such as pedestrians and vehicles in different locations) under the same background.

[0050] Step S103. Use a preset lighting model to simulate the lighting conditions corresponding to the background image, and apply the simulated lighting conditions to the foreground object to obtain a composite image; the lighting conditions include the lighting effects under different times and weather conditions.

[0051] Specifically, by analyzing the lighting conditions of the background image (such as lighting direction, intensity, color temperature, and weather effects), the same conditions are simulated using a preset lighting model and applied to the 3D foreground object, thus solving the problem of light and shadow discontinuity caused by lighting mismatch in 2D fusion.

[0052] Illumination parameter estimation is based on the color distribution of the background image (such as RGB histogram, color temperature detection) and shadow direction to estimate ambient light intensity, main light source direction and color (such as warm tones in the morning and cool tones at noon); deep learning models (such as LightNet) are used to predict the illumination environment parameters of the background image, including direct light direction, ambient light radiance, cloud cover degree, etc.

[0053] The lighting model application creates virtual light sources corresponding to the lighting conditions in the rendering engine (such as directional light to simulate the sun and ambient light to simulate diffuse reflection of the sky); it applies physically based rendering (PBR) materials to 3D foreground objects, calculates diffuse reflection, specular reflection and shadow effects according to lighting parameters, and ensures consistency with the background lighting; it supports dynamic adjustment of lighting parameters (such as simulating the gradient from day to dusk) to generate composite images with different lighting conditions under the same background.

[0054] Step S104. Apply a detail enhancement algorithm to the synthesized image to enhance the texture details of the foreground objects, perform color correction to make the color distribution consistent with the background image, and simulate noise in real shooting, including lens flare and noise; for each synthesized image, integrate the known foreground object annotation information to generate corresponding virtual data; the foreground object annotation information includes at least one or more of the following: object category, position, bounding box, and outline.

[0055] Specifically, post-processing enhances the realism of the synthesized images, including enhancing the texture details of foreground objects, matching background color distribution, simulating shooting noise, and integrating annotation information to generate labeled virtual data.

[0056] Detail enhancement and color correction employ super-resolution algorithms (such as ESRGAN) or unsharp masking to enhance the texture details of foreground objects, resolving detail blurring that may result from lightweight modeling; color transfer algorithms (such as Color Transfer) or histogram matching are used to adjust the color distribution of foreground objects to match the background, eliminating compositing artifacts.

[0057] Noise simulation simulates sensor noise by adding Gaussian noise or salt and pepper noise, and adjusts the noise intensity according to the shooting time (such as increasing noise density in low-light environments); optical defects such as lens flare and vignetting are generated by texture overlay or ray tracing to enhance the realism of the image.

[0058] Virtual data generation integrates the annotation information of 3D models (category, 2D / 3D bounding boxes, contour masks, etc.) and binds them to the synthetic images for storage; it generates dataset files containing metadata (such as lighting parameters, camera extrinsic parameters) to support the training and validation of computer vision tasks such as object detection and semantic segmentation.

[0059] In some embodiments, obtaining a 3D model constructed based on lightweight 3D modeling technology includes: selecting the 3D model from a preset lightweight 3D model resource library, or training to generate the 3D model.

[0060] Specifically, model acquisition methods include: Resource library selection: Selecting suitable models (such as pedestrians, vehicles, animals, etc.) from a pre-set lightweight 3D model resource library. The resource library needs to undergo market research to ensure that the models are cost-effective and diverse. Autonomous generation: Generating models through autonomous modeling or training (such as manually creating models using 3D modeling software or generating models based on deep learning) to meet specific scenario requirements. Lightweight modeling reduces the construction cost of foreground objects and improves model acquisition efficiency.

[0061] In some embodiments, the simplification of polygons and optimization of texture mapping of the 3D model includes: simplifying the polygons of the 3D model using a mesh simplification algorithm to reduce the number of polygons; and optimizing the texture mapping of the 3D model by adjusting the texture resolution and compressing the texture mapping format to reduce the amount of computation during rendering while maintaining the texture and details of the 3D model.

[0062] Polygon simplification employs mesh simplification algorithms (such as Quadric Edge Collapse Decimation) to reduce the number of polygons in the model, remove redundant details, retain key outlines and features, and reduce rendering computation.

[0063] Texture optimization reduces memory usage and rendering time while preserving texture details by adjusting texture map resolution (e.g., compressing high-resolution textures to medium resolution) and using efficient texture formats (e.g., ETC, ASTC).

[0064] In some embodiments, the step of filtering from a plurality of preset real-world scene images to obtain a background image based on preset diversity requirements includes: classifying and filtering from the plurality of real-world scene images according to different geographical regions, climate conditions, and time conditions, and selecting a background image that meets the requirements.

[0065] The filtering dimensions include: Geographic region: covering different geographical locations such as cities, rural areas, highways, and city blocks. Climate conditions: including different weather scenarios such as sunny days, rainy days, snowy days, and foggy days. Time conditions: covering daytime, dusk, nighttime, and different seasons (such as spring, summer, autumn, and winter) lighting environments. The process involves searching multiple image libraries according to the above dimensions, removing blurry and low-quality images, retaining high-resolution images with clear scene characteristics, and verifying copyright legality.

[0066] In some embodiments, adjusting the size of the foreground object corresponding to the 3D model based on the perspective and proportion of the background image includes: analyzing the perspective relationship of the background image, identifying reference objects in the background image, calculating the ratio of the actual size of the object in the background image to the image size, and adjusting the size of the foreground object corresponding to the 3D model according to the ratio so that the proportion of the foreground object is consistent with the proportion of the background scene corresponding to the background image.

[0067] Perspective analysis identifies reference objects in a background image (such as buildings or vehicles of known height) and calculates the perspective relationships (e.g., objects appear larger when closer and smaller when farther away) by comparing the actual size of the reference objects with the pixel size of the image.

[0068] Size adjustment scales the 3D model according to the background ratio (e.g., shrinking the distant pedestrian model to match the pixel size of the real pedestrian in the background) to ensure that the foreground object is in the same proportion as the background scene after compositing, thus avoiding geometric distortion.

[0069] In some embodiments, obtaining the corresponding camera extrinsic parameters by calculating the vanishing lines of the background image includes: performing edge detection on the background image and extracting straight lines using methods such as Hough transform; finding parallel line pairs among the detected straight lines using clustering algorithms; calculating the corresponding intersection point as the vanishing point for each line pair; connecting vanishing points in different directions to form a vanishing line; and constructing an equation using the properties of the vanishing line to solve for the camera extrinsic parameters in combination with known camera intrinsic parameters.

[0070] Edge detection and line extraction are performed by applying the Canny edge detection algorithm to the background image to extract edge contours, and then using the Hough Transform to detect line segments.

[0071] Parallel line pair identification uses a slope-based clustering algorithm (such as grouping straight lines by slope, and considering lines within a threshold as parallel line pairs) to filter out multiple groups of parallel lines (such as road edges and building outlines).

[0072] Vanishing point calculation involves taking the intersection of the extensions of each pair of parallel lines and using it as the vanishing point. Theoretically, parallel lines in the same direction correspond to the same vanishing point.

[0073] Vanishing lines are constructed by connecting vanishing points in different directions (such as horizontal and vertical directions) to form vanishing lines, which are used to represent the perspective direction of a scene.

[0074] The extrinsic parameters are solved by combining the known camera intrinsic parameters (focal length, principal point coordinates) with the geometric relationship of the vanishing line to construct a system of equations to solve for the camera extrinsic parameters (position, rotation angle).

[0075] In some embodiments, adjusting the angle of the foreground object using camera extrinsic parameters to match the perspective relationship between the foreground object and the background includes: performing rotation and translation transformations on the 3D model of the foreground object based on the position and orientation angle of the 3D degrees of freedom in the acquired camera extrinsic parameters, and projecting the transformed foreground object onto the background image plane to match the perspective relationship between the foreground object and the background image.

[0076] Adjust the angle of foreground objects using camera extrinsics to match background perspective.

[0077] The transformation matrix is ​​applied to perform coordinate transformations on the 3D model based on the camera's position (translation vector) and 3D orientation angle (rotation matrix) from its extrinsic parameters:

[0078] Rotation adjusts the model to the same direction as the background camera's viewpoint using a rotation matrix (e.g., simulating a tilt angle when shooting from left to right).

[0079] Translation involves moving the model to a target location in the background scene (such as the middle of a road or next to a sidewalk).

[0080] Projection matching ensures that the perspective relationships (such as near objects appearing larger and far objects appearing smaller, and occlusion relationships) of the model are consistent with the background by projecting the transformed 3D model onto the 2D background image plane through a perspective projection matrix, thus avoiding perspective distortion.

[0081] In some embodiments, the step of simulating the lighting conditions corresponding to the background image using a preset lighting model and applying the simulated lighting conditions to the foreground object to obtain a composite image includes: analyzing the lighting characteristics of the background image, such as the lighting direction and color temperature; using the physical lighting model in the rendering engine to simulate the lighting effects under different time and weather conditions corresponding to the background image; applying the simulated lighting effects to the foreground object; adjusting the material, shadow, and other parameters of the foreground object to make the lighting effects of the foreground object consistent with the background image; and obtaining a composite image.

[0082] Lighting feature analysis extracts features such as lighting direction (e.g., the main light source comes from the upper left), color temperature (e.g., warm tones indicate dusk), and shadow intensity from background images through image analysis algorithms (e.g., color channel statistics, gradient direction).

[0083] Physical lighting simulation uses the physical lighting model of a rendering engine (such as Unity or Blender) to create light sources that match the background (such as directional lights to simulate sunlight and point lights to simulate streetlights), and sets the light color, intensity, and attenuation parameters.

[0084] Material and shadow adjustments adjust the material properties of foreground objects (such as metallic reflection and fabric diffuse reflection) based on the background lighting, and generate real-time shadows to make the lighting and shadow effects of foreground objects (such as shadow direction and soft / hard edges) consistent with the background.

[0085] In some embodiments, the step of using detail enhancement algorithms to enhance the texture details of foreground objects and performing color correction to make the color distribution consistent with the background image includes: using detail enhancement algorithms such as image sharpening to enhance the corresponding texture details of foreground objects in the composite image; and using histogram matching algorithms or color transfer algorithms to perform color correction on the composite image to make the color distribution of the composite image consistent with the color distribution of the background image, thereby enhancing the overall harmony.

[0086] Lighting feature analysis extracts features such as lighting direction (e.g., the main light source comes from the upper left), color temperature (e.g., warm tones indicate dusk), and shadow intensity from background images through image analysis algorithms (e.g., color channel statistics, gradient direction).

[0087] Physical lighting simulation uses the physical lighting model of a rendering engine (such as Unity or Blender) to create light sources that match the background (such as directional lights to simulate sunlight and point lights to simulate streetlights), and sets the light color, intensity, and attenuation parameters.

[0088] Material and shadow adjustments adjust the material properties of foreground objects (such as metallic reflection and fabric diffuse reflection) based on the background lighting, and generate real-time shadows to make the lighting and shadow effects of foreground objects (such as shadow direction and soft / hard edges) consistent with the background.

[0089] In some embodiments, the step of integrating known foreground object annotation information to generate corresponding virtual data for each synthesized image includes: extracting foreground object category information from the metadata of the 3D model to calculate foreground object annotation information for the position, bounding box coordinates, or contour coordinates of the foreground object in the synthesized image; and integrating the corresponding image identifier and foreground object annotation information to generate structured data in a preset format, wherein the foreground object annotation information includes object category, position, bounding box, or contour.

[0090] The annotation information extraction obtains the object category (such as category ID corresponding to pedestrians or vehicles) from the 3D model metadata, and calculates the model's position, bounding box (BBox) coordinates (x, y, width, height) or contour coordinates (polygon vertices) in the 2D image through projection transformation.

[0091] Structured data generation integrates image file names and annotation information (category, location, bounding box, outline) into a preset format (such as JSON) to generate labeled virtual data, which can be directly used for deep learning model training, avoiding the cost of manual annotation.

[0092] In some embodiments, existing virtual data generation technologies primarily employ full 3D scenes combined with high-end 3D rendering engines to simulate real-world scenarios. This technology is widely used in training autonomous driving models. However, the full 3D modeling method suffers from drawbacks such as high cost, long development cycle, and high resource consumption, limiting its application in large-scale data generation scenarios.

[0093] Using 3D rendering technology requires high-performance hardware and specialized software tools, which results in high implementation costs. Market research data shows that the cost of a complete 3D rendering system can reach hundreds of thousands of dollars.

[0094] High-end 3D rendering engines typically take tens of minutes to several hours to generate a high-quality image, which poses a time challenge for generating large-scale datasets.

[0095] Compared to 3D rendering techniques, 2D image fusion technology generates virtual data at a lower cost by synthesizing foreground objects with background images. However, this technique has some significant limitations. In 2D image fusion, the lighting conditions of foreground objects and backgrounds are often inconsistent, leading to reduced realism in the synthesized image. Research shows that lighting mismatch is one of the main factors affecting the quality of synthesized images. Due to the different shooting angles and distances between foreground objects and background images, simple fusion can result in geometric size and viewpoint distortions, affecting the accuracy of model training. Although 2D image fusion technology is low-cost, the generated datasets have limitations in terms of diversity and realism, potentially requiring significant post-processing to improve dataset quality.

[0096] This invention aims to overcome the limitations of existing technologies and proposes a method that reduces data generation costs while maintaining a high degree of realism.

[0097] This invention reduces the workload of 3D modeling and increases scene diversity by using 3D modeling of foreground objects and realistic images for the background. Foreground object modeling utilizes existing 3D resource libraries to select or construct the required foreground object models, which typically have low acquisition costs and high diversity. Real-life background images are selected to ensure the realism and complexity of the scene, thereby improving the quality of the synthesized image.

[0098] This invention proposes two main methods to increase the realism of synthesized images, including: Lighting simulation: Simulating the lighting conditions of the background scene using a rendering engine, and lighting the foreground virtual objects to achieve consistency in lighting with the background. Viewpoint correction: (e.g., ...) Figure 2 As shown, the Vanishing Lines algorithm and calibration process are used to analyze the background image and obtain the calibration parameters of the virtual camera to ensure the consistency of the virtual object with the real background in terms of viewpoint.

[0099] In this method, lightweight 3D modeling techniques are employed for foreground object modeling. A foreground object library is constructed by selecting or creating representative and diverse 3D models. These models cover various categories, such as pedestrians, vehicles, and animals, to adapt to different scene requirements. During modeling, emphasis is placed on texture and detail processing to ensure high realism when composited into background images. Market research is conducted to select cost-effective and high-quality 3D model resource libraries for rapid acquisition of required foreground object models. The acquired 3D models undergo necessary optimization, including polygon simplification and texture mapping optimization, to reduce computational load during rendering and improve compositing efficiency. Background image acquisition is a crucial step in this method. A large number of high-quality real-world scene images are obtained through collaboration with image library providers. These images cover different geographical, climatic, and temporal conditions to ensure dataset diversity. Background images meeting the requirements are selected from multiple image libraries based on the diversity of scene needs. All acquired background images are ensured to have legal usage rights to avoid subsequent copyright disputes. When compositing foreground objects and background images, precise adjustments to the object's geometry and angles are necessary to ensure its plausibility and realism within the virtual scene. The size of the foreground object is adjusted based on the perspective and proportions of the background image to maintain consistency with the background scene. By calculating the vanishing lines of the background image, the camera's extrinsic parameters (including its position and 3DOF orientation) are calculated, and these parameters are used to adjust the angle of the foreground object to match its perspective with the background. Once the foreground object's perspective matches the background, its parameters can be saved. Later, when using the same background image, these parameters can be imported to randomly position the foreground virtual object. This ensures that the perspective of the foreground and background in the generated image matches, thus producing image data suitable for training visual models.

[0100] The basic steps for calculating camera calibration parameters using the Vanishing Lines algorithm are as follows: 1. Image Acquisition: First, capture images of the scene containing parallel lines. Scenes with clearly defined parallel lines, such as checkerboard patterns or building exteriors, are typically used. 2. Edge and Line Detection: Perform edge detection on the image, then extract straight lines using methods such as Hough Transform. 3. Parallel Line Identification: Among the detected lines, identify pairs of parallel lines. These parallel lines tend to intersect in the image. 4. Vanishing Point Calculation: For each pair of parallel lines, calculate their intersection point, which is the vanishing point. Theoretically, the extensions of parallel lines intersect at infinity, but on the image plane, they intersect within a finite distance. 5. Vanishing Line Determination: Connect the vanishing points in different directions to form vanishing lines. At least two orthogonal vanishing lines are usually required. 6. Intrinsic Parameter Matrix Calculation: Using the properties of vanishing lines, equations can be constructed to solve for the camera's intrinsic parameter matrix. The intrinsic parameter matrix contains information such as focal length and principal point coordinates. 7. Estimate extrinsic parameters: Based on intrinsic parameters and feature points in the image, the camera's extrinsic parameters, such as position and orientation, can be further estimated. 8. Optimization: Finally, nonlinear optimization methods (such as the Levenberg-Marquardt algorithm) are typically used to fine-tune all parameters to minimize reprojection error.

[0101] To improve the blending of foreground objects with the background image, this method employs advanced lighting simulation technology. It simulates the lighting conditions of the background scene using a rendering engine and then applies these simulations to the foreground objects. The lighting simulation utilizes the lighting model within the rendering engine to simulate lighting effects under different times and weather conditions. The simulated lighting effects are then applied to the foreground objects, ensuring realistic lighting and shadow effects across various background environments.

[0102] To further enhance the realism of the synthesized data, this method employs a series of realism enhancement techniques, including but not limited to detail enhancement, color correction, and noise simulation. The algorithm enhances the texture details of foreground objects, making them clearer and more realistic in the synthesized image. Color correction is applied to the synthesized image to ensure its color distribution matches the background image, enhancing overall harmony. Simulation of noise that may occur during real-world shooting, such as lens flare and noise, further enhances the realism of the synthesized image.

[0103] For each synthesized image, these foreground objects are generated by our algorithm, so their annotation information, including object category, location, bounding box, or outline, is already known. Therefore, we can easily integrate this information to generate annotated data structures. For example, a JSON format for annotation information might look like this:

[0104] {

[0105] "image_id": "image_001.jpg",

[0106] "annotations": [

[0107] {

[0108] "category_id": 1,

[0109] "bbox": [x, y, width, height],

[0110] "score": 0.98

[0111] },

[0112] {

[0113] "category_id": 2,

[0114] "polygon": [[x1, y1], [x2, y2], [x3, y3], ...],

[0115] "score": 0.95

[0116] } ]

[0118] This data can then be used to train deep vision models.

[0119] By implementing the technical solution of this invention, the bottleneck problem of data acquisition and annotation in visual algorithm training can be effectively solved, and the research and development and optimization process of visual algorithms can be accelerated.

[0120] In some embodiments, to address the problem that existing foreground object placement relies on random or manual adjustment and is difficult to generate "high-value" training data (such as objects in occluded, small-scale, special angle, or other difficult-to-recognize scenes), reinforcement learning is proposed to optimize the placement and angle of foreground objects, thereby generating a more diverse dataset that better meets the needs of model training.

[0121] The state space includes features of the background image (such as scene type: road / park / shopping mall; reference object size: such as street lamp height, tree spacing; lighting conditions: such as day / night), the current state of the foreground object (such as position coordinates, angle, size), and the diversity indicators of the dataset (such as the object angle distribution and position distribution in the current dataset).

[0122] The motion space includes adjusting the position of the foreground object (x / y coordinate offset, ranging from 10% to 90% of the background image), adjusting the angle (roll / pitch / yaw, ranging from -30° to 30°), and adjusting the size (scaling ratio, ranging from 0.8 to 1.2 times).

[0123] The reward function design includes: perspective consistency reward, which grants a positive reward (+1) if the perspective relationship between the foreground object and the background matches (the error calculated by the vanishing line algorithm is less than the threshold); otherwise, a negative reward (-0.5) is given. Hard sample reward, which grants an additional positive reward (+1.5) if the foreground object is in a "hard-to-recognize scene" (e.g., partial occlusion: the object is 20%-50% obscured by trees in the background; small scale: the object occupies less than 5% of the image; special angle: such as a pedestrian photographed from below), encouraging the generation of hard samples. Diversity reward, which calculates the distribution difference between the currently placed foreground object and existing objects in the dataset (e.g., the entropy of the angular distribution, the variance of the positional distribution). The greater the difference, the greater the positive reward (+0.5 to +1), ensuring the diversity of the dataset.

[0124] The agent is trained using the Proximal Policy Optimization (PPO) algorithm, learning the optimal placement strategy through interaction with the environment (background image + foreground object placement module). During training, every 1000 synthesized images generated, the dataset quality is evaluated using a dataset evaluation module (e.g., calculating dataset coverage: coverage of object categories, positions, and angles; hard sample ratio: the proportion of images in difficult-to-identify scenes). This feedback adjusts the weights of the reward function (e.g., increasing the weight of hard sample rewards) to optimize the agent's learning objective.

[0125] The trained agent can automatically adjust the position (e.g., standing in the shadow of a streetlamp), angle (e.g., facing the street), and size (e.g., matching the size of pedestrians in the background) of foreground objects (e.g., pedestrians) based on the features of the input background image (e.g., "city streets at night"), generating synthetic images that meet the requirements of "hard samples + diversity". The generated dataset is used to train an object detection model (e.g., YOLOv8). The reward function of the agent is optimized by using the detection accuracy of the model (especially the detection rate for hard samples), forming a closed loop of "data generation - model training - policy optimization".

[0126] In some embodiments, to address the problem that existing annotation information relies on 3D model metadata and is difficult to generate fine-grained semantic annotations (such as the color, action, attributes, and scene interaction of objects), a multimodal large model is proposed to automatically enrich the annotation information of foreground objects, thereby improving the detail and practicality of the annotations.

[0127] Collect synthetic images (foreground objects + background pictures), 3D model metadata (such as object category, size), and fine-grained annotations in real scenes (such as "a red sedan with license plate number Jing A12345 parked in a roadside parking space") to construct an "image-metadata-fine-grained annotation" dataset.

[0128] Fine-tune using GPT-4V (or its open-source alternative models such as LLaVA-1.5), input synthetic images and 3D model metadata (such as "object category: sedan; size: 4.5m × 1.8m × 1.5m"), and output fine-grained annotations (such as "color: red; license plate number: Jing A12345; status: parked by the roadside; scene interaction: the front of the car faces the street"). During training, introduce a visual feature enhancement module to fuse the visual features extracted from the synthetic images by the CLIP model with the text metadata, enabling the model to better understand the details in the images (such as the text on the license plate and scratches on the car body).

[0129] For the contour annotation of foreground objects, use a deep learning-driven instance segmentation model (such as an improved version of Mask2Former), combined with the output of a multimodal large model (such as "red sedan"), to automatically segment the contour of the foreground object. During model training, introduce an annotation consistency loss to compare the segmented contour with the projected contour of the 3D model to ensure the accuracy of the contour. For complex scenes (such as a pedestrian carrying an item), the model will automatically segment the contours of the "pedestrian" and the "item" (such as a handbag) to generate multi-instance annotations.

[0130] For time-series synthetic images (such as a pedestrian walking towards the camera from a distance), use a recursive multimodal model (such as LSTM + GPT-4V), and automatically update the annotation of the current frame (such as "the pedestrian is 3m away from the camera, walking speed is 1.2m / s, and the amplitude of arm swing increases") by memorizing the annotation information of the previous frame (such as "the pedestrian is 5m away from the camera, walking speed is 1m / s") to achieve dynamic tracking of annotation information.

[0131] Use an annotation review model (such as a BERT-based text classification model) to verify the quality of the annotations for the generated fine-grained annotations, checking for logical errors (such as "a sedan in a snowy scene has no snow accumulation") or semantic contradictions (such as "a sedan parked by the roadside is in a driving state"). For annotations that fail the verification, return them to the multimodal large model to regenerate to ensure the accuracy of the annotations.

[0132] This application provides a virtual data generation method that combines 3D foreground objects with real background images. By performing lightweight 3D modeling only on the foreground objects, the high hardware and labor costs of full-scene 3D rendering are avoided, thus improving modeling efficiency. A vanishing line algorithm is used to achieve perspective consistency between the foreground and background, and dynamic lighting simulation is used to eliminate light and shadow faults. The realism of the synthesized image is improved through subjective evaluation. The real background covers multiple geographical, climatic, and temporal conditions. With the flexible use of the foreground model library, a diverse dataset of millions of images can be generated quickly. Since the parameters of the foreground objects are known, structured labeled data containing category, location, and contour can be automatically generated, significantly saving manual labeling costs.

[0133] Please see Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a virtual data generation system 200 combining 3D foreground objects and real background images, provided in an embodiment of this application. This virtual data generation system 200 is used to execute the steps of the virtual data generation method combining 3D foreground objects and real background images shown in the above embodiments. The virtual data generation system 200 can be a single server or a server cluster, or it can be a terminal, such as a handheld terminal, a laptop computer, a wearable device, or a robot.

[0134] like Figure 3 As shown, the virtual data generation system 200, which combines 3D foreground objects with real background images, includes:

[0135] The model acquisition unit 201 is used to acquire a 3D model constructed based on lightweight 3D modeling technology, and to perform polygon simplification and texture mapping optimization on the 3D model to reduce the amount of rendering computation. The model category corresponding to the 3D model includes at least pedestrians, vehicles and animals. According to the preset diversity requirements, the background image is obtained by filtering from multiple preset real scene images. The multiple real scene images cover different geographical, climatic and time conditions.

[0136] The size adjustment unit 202 is used to adjust the size of the foreground object corresponding to the 3D model according to the perspective and proportion of the background image. It obtains the corresponding camera extrinsic parameters by calculating the vanishing line of the background image, and uses the camera extrinsic parameters to adjust the angle of the foreground object so that the perspective relationship between the foreground object and the background is matched. After matching, the corresponding foreground object size and foreground object angle are saved so that the corresponding foreground object size and foreground object angle can be loaded when using the background image to randomly place the foreground virtual object. The camera extrinsic parameters include the position and the orientation angle of the 3D degrees of freedom.

[0137] The image acquisition unit 203 is used to simulate the lighting conditions corresponding to the background image using a preset lighting model, and apply the simulated lighting conditions to the foreground object to obtain a composite image; the lighting conditions include the lighting effects under different times and weather conditions.

[0138] The data generation unit 204 is used to enhance the texture details of the foreground objects by using a detail enhancement algorithm on the synthesized image, perform color correction to make the color distribution consistent with the background image, and simulate noise in real shooting, including lens flare and noise. For each synthesized image, the known foreground object annotation information is integrated to generate corresponding virtual data. The foreground object annotation information includes at least one or more of the following: object category, position, bounding box, and outline.

[0139] In some embodiments, obtaining a 3D model constructed based on lightweight 3D modeling technology includes: selecting the 3D model from a preset lightweight 3D model resource library, or training to generate the 3D model.

[0140] In some embodiments, the simplification of polygons and optimization of texture mapping of the 3D model includes: simplifying the polygons of the 3D model using a mesh simplification algorithm to reduce the number of polygons; and optimizing the texture mapping of the 3D model by adjusting the texture resolution and compressing the texture mapping format to reduce the amount of computation during rendering while maintaining the texture and details of the 3D model.

[0141] In some embodiments, the step of filtering from a plurality of preset real-world scene images to obtain a background image based on preset diversity requirements includes: classifying and filtering from the plurality of real-world scene images according to different geographical regions, climate conditions, and time conditions, and selecting a background image that meets the requirements.

[0142] In some embodiments, adjusting the size of the foreground object corresponding to the 3D model based on the perspective and proportion of the background image includes: analyzing the perspective relationship of the background image, identifying reference objects in the background image, calculating the ratio of the actual size of the object in the background image to the image size, and adjusting the size of the foreground object corresponding to the 3D model according to the ratio so that the proportion of the foreground object is consistent with the proportion of the background scene corresponding to the background image.

[0143] In some embodiments, obtaining the corresponding camera extrinsic parameters by calculating the vanishing lines of the background image includes: performing edge detection on the background image and extracting straight lines using methods such as Hough transform; finding parallel line pairs among the detected straight lines using clustering algorithms; calculating the corresponding intersection point as the vanishing point for each line pair; connecting vanishing points in different directions to form a vanishing line; and constructing an equation using the properties of the vanishing line to solve for the camera extrinsic parameters in combination with known camera intrinsic parameters.

[0144] In some embodiments, adjusting the angle of the foreground object using camera extrinsic parameters to match the perspective relationship between the foreground object and the background includes: performing rotation and translation transformations on the 3D model of the foreground object based on the position and orientation angle of the 3D degrees of freedom in the acquired camera extrinsic parameters, and projecting the transformed foreground object onto the background image plane to match the perspective relationship between the foreground object and the background image.

[0145] In some embodiments, the step of simulating the lighting conditions corresponding to the background image using a preset lighting model and applying the simulated lighting conditions to the foreground object to obtain a composite image includes: analyzing the lighting characteristics of the background image, such as the lighting direction and color temperature; using the physical lighting model in the rendering engine to simulate the lighting effects under different time and weather conditions corresponding to the background image; applying the simulated lighting effects to the foreground object; adjusting the material, shadow, and other parameters of the foreground object to make the lighting effects of the foreground object consistent with the background image; and obtaining a composite image.

[0146] In some embodiments, the step of using detail enhancement algorithms to enhance the texture details of foreground objects and performing color correction to make the color distribution consistent with the background image includes: using detail enhancement algorithms such as image sharpening to enhance the corresponding texture details of foreground objects in the composite image; and using histogram matching algorithms or color transfer algorithms to perform color correction on the composite image to make the color distribution of the composite image consistent with the color distribution of the background image, thereby enhancing the overall harmony.

[0147] In some embodiments, the step of integrating known foreground object annotation information to generate corresponding virtual data for each synthesized image includes: extracting foreground object category information from the metadata of the 3D model to calculate foreground object annotation information for the position, bounding box coordinates, or contour coordinates of the foreground object in the synthesized image; and integrating the corresponding image identifier and foreground object annotation information to generate structured data in a preset format, wherein the foreground object annotation information includes object category, position, bounding box, or contour.

[0148] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the virtual data generation system and its modules that combine 3D foreground objects with real background images described above can be found in the corresponding contents of the various embodiments of the virtual data generation method that combines 3D foreground objects with real background images, and will not be repeated here.

[0149] The aforementioned method for generating virtual data by combining 3D foreground objects with real background images can be implemented as a computer program, which can be used in various ways, such as... Figure 3 It runs on the device shown.

[0150] Please see Figure 4 , Figure 4 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application. The computer device includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and internal memory.

[0151] The storage medium may store operating devices and computer programs. The computer program includes program instructions that, when executed, cause the processor to perform any method of virtual data generation that combines 3D foreground objects with real-world background images.

[0152] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0153] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any virtual data generation method that combines 3D foreground objects with real background images.

[0154] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0155] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0156] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0157] A 3D model constructed using lightweight 3D modeling technology is obtained. The 3D model is then simplified in polygons and optimized in texture mapping to reduce the amount of rendering computation. The model categories corresponding to the 3D model include at least pedestrians, vehicles, and animals. Background images are obtained by filtering from multiple preset real-world scene images based on preset diversity requirements. The multiple real-world scene images cover different geographical, climatic, and temporal conditions.

[0158] The size of the foreground object in the 3D model is adjusted according to the perspective and proportion of the background image. The corresponding camera extrinsic parameters are obtained by calculating the vanishing line of the background image, and the angle of the foreground object is adjusted using the camera extrinsic parameters to match the perspective relationship between the foreground object and the background. After matching, the size and angle of the foreground object are saved so that the corresponding size and angle of the foreground object can be loaded when using the background image to randomly place the virtual foreground object. The camera extrinsic parameters include the position and the orientation angle of the 3D degrees of freedom.

[0159] The lighting conditions corresponding to the background image are simulated using a preset lighting model, and the simulated lighting conditions are applied to the foreground object to obtain a composite image; the lighting conditions include the lighting effects under different times and weather conditions.

[0160] The synthetic image is enhanced with a detail enhancement algorithm to improve the texture details of the foreground objects, and color correction is performed to make the color distribution consistent with the background image. Noise is simulated in real shooting, including lens flare and noise. For each synthetic image, the known foreground object annotation information is integrated to generate corresponding virtual data. The foreground object annotation information includes at least one or more of the following: object category, position, bounding box, and outline.

[0161] In some embodiments, obtaining a 3D model constructed based on lightweight 3D modeling technology includes: selecting the 3D model from a preset lightweight 3D model resource library, or training to generate the 3D model.

[0162] In some embodiments, the simplification of polygons and optimization of texture mapping of the 3D model includes: simplifying the polygons of the 3D model using a mesh simplification algorithm to reduce the number of polygons; and optimizing the texture mapping of the 3D model by adjusting the texture resolution and compressing the texture mapping format to reduce the amount of computation during rendering while maintaining the texture and details of the 3D model.

[0163] In some embodiments, the step of filtering from a plurality of preset real-world scene images to obtain a background image based on preset diversity requirements includes: classifying and filtering from the plurality of real-world scene images according to different geographical regions, climate conditions, and time conditions, and selecting a background image that meets the requirements.

[0164] In some embodiments, adjusting the size of the foreground object corresponding to the 3D model based on the perspective and proportion of the background image includes: analyzing the perspective relationship of the background image, identifying reference objects in the background image, calculating the ratio of the actual size of the object in the background image to the image size, and adjusting the size of the foreground object corresponding to the 3D model according to the ratio so that the proportion of the foreground object is consistent with the proportion of the background scene corresponding to the background image.

[0165] In some embodiments, obtaining the corresponding camera extrinsic parameters by calculating the vanishing lines of the background image includes: performing edge detection on the background image and extracting straight lines using methods such as Hough transform; finding parallel line pairs among the detected straight lines using clustering algorithms; calculating the corresponding intersection point as the vanishing point for each line pair; connecting vanishing points in different directions to form a vanishing line; and constructing an equation using the properties of the vanishing line to solve for the camera extrinsic parameters in combination with known camera intrinsic parameters.

[0166] In some embodiments, adjusting the angle of the foreground object using camera extrinsic parameters to match the perspective relationship between the foreground object and the background includes: performing rotation and translation transformations on the 3D model of the foreground object based on the position and orientation angle of the 3D degrees of freedom in the acquired camera extrinsic parameters, and projecting the transformed foreground object onto the background image plane to match the perspective relationship between the foreground object and the background image.

[0167] In some embodiments, the step of simulating the lighting conditions corresponding to the background image using a preset lighting model and applying the simulated lighting conditions to the foreground object to obtain a composite image includes: analyzing the lighting characteristics of the background image, such as the lighting direction and color temperature; using the physical lighting model in the rendering engine to simulate the lighting effects under different time and weather conditions corresponding to the background image; applying the simulated lighting effects to the foreground object; adjusting the material, shadow, and other parameters of the foreground object to make the lighting effects of the foreground object consistent with the background image; and obtaining a composite image.

[0168] In some embodiments, the step of using detail enhancement algorithms to enhance the texture details of foreground objects and performing color correction to make the color distribution consistent with the background image includes: using detail enhancement algorithms such as image sharpening to enhance the corresponding texture details of foreground objects in the composite image; and using histogram matching algorithms or color transfer algorithms to perform color correction on the composite image to make the color distribution of the composite image consistent with the color distribution of the background image, thereby enhancing the overall harmony.

[0169] In some embodiments, the step of integrating known foreground object annotation information to generate corresponding virtual data for each synthesized image includes: extracting foreground object category information from the metadata of the 3D model to calculate foreground object annotation information for the position, bounding box coordinates, or contour coordinates of the foreground object in the synthesized image; and integrating the corresponding image identifier and foreground object annotation information to generate structured data in a preset format, wherein the foreground object annotation information includes object category, position, bounding box, or contour.

[0170] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the virtual data generation method combining 3D foreground objects and real background images as provided in any embodiment of this application.

[0171] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0172] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating virtual data by combining 3D foreground objects with real background images, characterized in that, include: A 3D model constructed based on lightweight 3D modeling technology is obtained. The 3D model is simplified in polygons and optimized in texture mapping to reduce the amount of rendering computation. The model category corresponding to the 3D model includes at least pedestrians, vehicles and animals. Based on the preset diverse needs, the background image is selected from multiple preset real-world scene images. Multiple real-world scene images cover different geographical, climatic, and temporal conditions; The size of the foreground object in the 3D model is adjusted according to the perspective and proportion of the background image. The corresponding camera extrinsic parameters are obtained by calculating the vanishing lines of the background image, including: edge detection of the background image and extraction of straight lines using Hough transform; finding parallel line pairs among the detected lines using a clustering algorithm; calculating the intersection point as the vanishing point for each line pair; connecting vanishing points in different directions to form a vanishing line; constructing an equation using the properties of the vanishing line to solve for the camera extrinsic parameters in conjunction with known camera intrinsic parameters; and adjusting the angle of the foreground object using the camera extrinsic parameters to match the perspective relationship between the foreground object and the background, including: rotating and translating the 3D model of the foreground object based on the position and orientation angle of the 3D degrees of freedom in the obtained camera extrinsic parameters; projecting the transformed foreground object onto the background image plane to match the perspective relationship between the foreground object and the background image; saving the corresponding foreground object size and angle after matching so that the corresponding foreground object size and angle can be imported when using the background image to randomly position the virtual foreground object; the camera extrinsic parameters include position and orientation angle of the 3D degrees of freedom. The process involves simulating the lighting conditions corresponding to a background image using a preset lighting model, and then applying the simulated lighting conditions to foreground objects to obtain a composite image. This includes: analyzing the lighting direction and color temperature of the background image; using the physically based lighting model in the rendering engine to simulate the lighting effects corresponding to the background image under different time and weather conditions; applying the simulated lighting effects to the foreground objects; adjusting the materials and shadows of the foreground objects to make their lighting effects consistent with the background image; and obtaining a composite image. The lighting conditions include lighting effects under different time and weather conditions. The synthetic image is enhanced with a detail enhancement algorithm to improve the texture details of the foreground objects, and color correction is performed to make the color distribution consistent with the background image. Noise is simulated in real shooting, including lens flare and noise. For each synthetic image, the known foreground object annotation information is integrated to generate corresponding virtual data. The foreground object annotation information includes at least one or more of the following: object category, position, bounding box, and outline.

2. The method according to claim 1, characterized in that, The process of obtaining a 3D model constructed based on lightweight 3D modeling technology includes: Select the 3D model from the preset lightweight 3D model resource library, or train and generate the 3D model.

3. The method according to claim 1, characterized in that, The process of simplifying polygons and optimizing texture mapping in the 3D model includes: A mesh simplification algorithm is used to simplify the polygons in the 3D model, reducing the number of polygons. The texture maps of 3D models are optimized by adjusting the texture resolution and compressing the texture map format to reduce the amount of computation during rendering while maintaining the texture and details of the 3D model.

4. The method according to claim 1, characterized in that, The step of selecting a background image from multiple preset real-world scene images based on preset diversity requirements includes: Based on the diverse needs of different scenarios, background images that meet the requirements are selected from multiple real-world scene images, categorized and filtered according to different geographical regions, climate conditions, and time conditions.

5. The method according to claim 1, characterized in that, The step of adjusting the size of the 3D model corresponding to the foreground object based on the perspective and proportion of the background image includes: The perspective relationship of the background image is analyzed, and reference objects in the background image are identified. The ratio of the actual size of the objects in the background image to the image size is calculated. Based on the ratio, the size of the foreground objects corresponding to the 3D model is adjusted so that the ratio of the foreground objects to the background scene corresponding to the background image is consistent.

6. The method according to claim 1, characterized in that, The process of enhancing the texture details of foreground objects using a detail enhancement algorithm and performing color correction to make the color distribution consistent with the background image includes: Image sharpening is applied to foreground objects in the synthesized image to enhance the corresponding texture details; Histogram matching or color transfer algorithms are used to perform color correction on the synthesized image, so that the color distribution of the synthesized image is consistent with the color distribution of the background image, thereby enhancing the overall harmony.

7. The method according to claim 1, characterized in that, For each synthesized image, the known foreground object annotation information is integrated to generate corresponding virtual data, including: Extract the category information of foreground objects from the metadata of the 3D model, and use it to calculate the foreground object annotation information such as the position, bounding box coordinates or contour coordinates of the foreground object in the synthesized image. The corresponding image identifiers and foreground object annotation information are integrated to generate structured data in a preset format, where the foreground object annotation information includes object category, location, bounding box or outline.

Images