Random data enhancement method and device based on three-dimensional space structure modeling and storage medium

Abstract

The invention relates to the technical field of computer vision and three-dimensional modeling, and discloses a random data enhancement method and device based on three-dimensional space structure modeling and a storage medium, and the method comprises the steps: carrying out the three-dimensional geometry and appearance modeling of a target object, building a scene space model, and defining an internal reference matrix of a virtual camera; generating a camera position and a camera orientation meeting a view cone constraint condition; projecting the target object and the scene space model to a two-dimensional image plane based on the internal reference matrix of the virtual camera and the constructed external reference matrix to generate a two-dimensional basic image; and on the basis of the two-dimensional basic image, generating a final training sample. According to the method, three-dimensional geometry and appearance modeling are performed on the target object, the target object is projected to the two-dimensional image plane based on the internal parameter matrix and the external parameter matrix, and the two-dimensional basic image is generated by combining illumination model rendering, so that the problem that a traditional two-dimensional data enhancement method cannot simulate view angle change and illumination interaction is solved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D modeling technology, specifically to a random data augmentation method, device, and storage medium based on 3D spatial structure modeling. Background Technology

[0002] With the application of deep learning technology in computer vision, the training effect of deep neural network models depends on the quantity and quality of training samples. In practical applications, acquiring labeled data covering diverse object shapes, different lighting conditions, and complex background environments often faces problems such as high acquisition costs and long labeling cycles. To expand the training dataset and improve the model's generalization ability, data augmentation methods are widely used in the preprocessing stage of model training, aiming to supplement the deficiencies of real-world data with algorithmically generated samples.

[0003] Existing data augmentation techniques primarily focus on processing data in the two-dimensional image domain. Conventional image processing methods typically involve performing geometric transformations on the original 2D image, such as rotation, translation, scaling, and flipping; adjusting the image's brightness, contrast, and saturation to simulate changes in illumination; and adding noise or blurring to simulate image quality degradation. Furthermore, cut-and-paste-based augmentation methods increase the diversity of object-background combinations by segmenting 2D image patches of the target object from the original image and pasting them onto different background images. This approach enriches the semantic content of the training samples to some extent.

[0004] However, the aforementioned enhancement methods based on two-dimensional images struggle to simulate realistic three-dimensional spatial transformations and physical lighting interactions. First, simple geometric transformations cannot generate perspective distortions of the target object from different viewpoints, and simple brightness adjustments fail to reflect the true impact of changes in light source position on the surface's light and shadow distribution, resulting in a lack of geometric and lighting consistency in the generated samples. Second, cut-and-paste methods, lacking depth information and physical constraints, easily lead to disproportionate ratios, spatial displacement, or physically incompatible overlaps between the synthesized target object and background, failing to properly handle occlusion relationships. Furthermore, traditional background blending methods struggle to preserve high-frequency texture details and parallax effects, and simple noise addition methods cannot simulate the complex sensor degradation and environmental interference in a real imaging process. Therefore, ensuring the correctness of geometric perspective relationships, lighting distribution, and physical spatial logic during data augmentation, and generating training samples with realistic textures and imaging features, are technical problems that need to be solved in this field. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a random data augmentation method, device, and storage medium based on three-dimensional spatial structure modeling. This solves the problems that existing two-dimensional data augmentation techniques cannot simulate the geometric perspective changes and lighting interactions of target objects in three-dimensional space, and it is difficult to guarantee the correctness of the physical spatial logic and occlusion relationship between the target object and the background environment, resulting in a lack of realism and physical consistency in the generated training samples.

[0006] To address the above problems, the present invention provides the following technical solution: This invention provides a random data augmentation method based on three-dimensional spatial structure modeling, employing the following technical solution: A stochastic data augmentation method based on three-dimensional spatial structure modeling includes the following steps: S100. Perform 3D geometric and appearance modeling on the target object, establish a scene space model, and define the intrinsic parameter matrix of the virtual camera. S200. Randomly set the position and orientation of the target object in the world coordinate system in the scene space model, configure the virtual light source parameters, and generate the camera position and camera orientation that satisfy the view frustum constraint. S300: Based on the intrinsic parameter matrix of the virtual camera and the extrinsic parameter matrix constructed according to the camera position and the camera orientation, the target object and the scene space model after the position and posture are set are projected onto a two-dimensional image plane. The spatial occlusion relationship between the target object and the scene space model is processed using depth buffering technology, and rendering is performed in combination with the virtual light source parameters and lighting model to generate a physically consistent two-dimensional basic image. S400: Based on the two-dimensional base image, sensor noise, two-dimensional region occlusion, illumination changes and imaging artifact simulation are superimposed to generate the final training sample.

[0007] By adopting the above technical solution, which combines 3D scene construction with 2D image post-processing, a 2D base image conforming to geometric perspective and lighting distribution is first generated in 3D space through a physical rendering pipeline. Then, physical defects of the imaging sensor and environmental interference are simulated in the 2D plane. Therefore, the obtained data samples possess geometric diversity and physical rationality in 3D space, and cover signal degradation characteristics in the imaging process. This solves the problem that 2D data augmentation methods struggle to simulate viewpoint changes, lighting interactions, and occlusion relationships between objects, reduces the domain difference between synthetic data and real-world scene data, and improves the physical consistency of training samples and the model's generalization ability.

[0008] Furthermore, the establishment of the scene space model includes: acquiring three-dimensional data of the target category to be enhanced, and representing the three-dimensional data as a mesh model containing a set of vertices, a set of faces, and a texture map; The texture map is used to represent the color distribution and reflective properties of the target object's surface; A virtual background environment model for placing the target object is established. The virtual background environment model can be implemented using various three-dimensional representation methods, including but not limited to geometric mesh models or three-dimensional Gaussian sputtering sets. When the three-dimensional Gaussian sputtering set is used for characterization, the Gaussian elements in the three-dimensional Gaussian sputtering set include the position mean, covariance matrix, opacity, and spherical harmonic coefficients.

[0009] By employing the above technical solution, a mesh model is used to represent the geometric structure of the target object, and it supports the reconstruction of complex realistic background environments using 3D Gaussian sputtering technology. Compared to geometric mesh backgrounds, 3D Gaussian sputtering preserves the high-frequency texture details and parallax effects of the scene, making the background part of the synthesized image closer to the real shooting effect, thus improving the environmental fidelity of the training samples.

[0010] Furthermore, the step of randomly setting the position and orientation of the target object in the world coordinate system in the scene space model includes: randomly sampling translation vectors and randomly sampling rotation matrices within a preset three-dimensional scene bounding box. Using the rotation matrix and the translation vector, a rigid body transformation is performed on the vertex coordinates of the target object in the local coordinate system, and the vertex coordinates of the target object in the world coordinate system are calculated. Collision detection is performed using the axis-aligned bounding box or oriented bounding box of the target object. If the target object is found to have volume overlap with the scene space model, the translation vector and the rotation matrix are resampled until the non-penetrating physical constraint is met.

[0011] By adopting the above technical solution, a collision detection mechanism based on physical volume is introduced during pose randomization, forcibly constraining the target object and the background environment to meet the non-penetration condition. This avoids the phenomenon of the target object embedding inside walls, the ground, or other obstacles, ensuring the geometrical correctness of the synthesized scene and preventing the model from learning incorrect spatial context features.

[0012] Furthermore, generating the camera position and camera orientation that satisfy the view frustum constraint includes: randomly sampling the camera position within a preset space centered on the target object, and calculating the camera orientation according to the gaze point strategy to obtain the extrinsic parameter matrix of the virtual camera; The key points of the target object are transformed from the world coordinate system to the camera coordinate system and projected onto the two-dimensional image plane to calculate pixel coordinates and depth values; If the pixel coordinates are within the range of the image width and image height of the two-dimensional image plane, and the depth value is between the distances of the near and far clipping planes of the virtual camera, then the current camera position and camera orientation are determined to be valid.

[0013] By employing the above technical solution, the validity of randomly generated camera parameters is verified through a view frustum culling mechanism, ensuring that the target object is within the imaging field of view of the virtual camera and within the set depth of field. This avoids generating invalid empty samples or samples where the target object accounts for too small a proportion, thus improving the computational efficiency and sample effectiveness of the data augmentation process.

[0014] Furthermore, the step of projecting the target object and the scene space model after the position and posture settings onto the two-dimensional image plane includes: using the extrinsic parameter matrix of the virtual camera to transform the vertex coordinates in the world coordinate system to the camera coordinate system to obtain camera coordinate system coordinates containing a depth value, wherein the depth value is the vertical distance along the camera optical axis relative to the camera plane. Using the intrinsic parameter matrix of the virtual camera, perspective projection and perspective division operations are performed on the camera coordinate system coordinates to generate pixel coordinates on the two-dimensional image plane.

[0015] By adopting the above technical solution and following the perspective projection transformation process of computer graphics, the imaging geometry process of a real camera is simulated to ensure that the size and shape of the target object in the synthesized image conform to the perspective law as distance and viewing angle change.

[0016] Furthermore, the process of using depth buffering technology to handle the spatial occlusion relationship between the target object and the scene space model includes: rasterizing the geometric primitives projected onto the two-dimensional image plane to generate pixel fragments, and calculating the fragment depth value of the pixel fragments using centroid coordinate interpolation. Initialize the depth buffer and set the depth value of all pixel positions in the depth buffer to the preset maximum depth value; Perform a depth test on each pixel fragment: compare the current fragment depth value of the pixel fragment with the depth values ​​already stored in the depth buffer; If the fragment depth value is less than the depth value already stored in the depth buffer, then the pixel fragment is determined to be visible, and the depth buffer and color buffer are updated. If the fragment depth value is greater than or equal to the depth value already stored in the depth buffer, then the pixel fragment is discarded.

[0017] By employing the above technical solution, the occlusion problem between objects in a 3D scene is resolved at the pixel level using a depth testing mechanism. When the target object is occluded by the background or other objects, the pixels of the occluded part are discarded, ensuring that the generated image accurately reflects the visible area and eliminating depth order errors.

[0018] Furthermore, the step of rendering by combining the virtual light source parameters and the lighting model to generate a physically consistent two-dimensional base image includes: sampling the texture color of the target object surface for the pixel fragments that have passed the depth test, and interpolating to calculate the unit normal vector. Based on the virtual light source parameters, a viewing direction vector, a light incident direction vector, and a light reflection direction vector are constructed for the pixel fragment. Based on the local illumination model, the ambient light component, diffuse reflection component, and specular reflection component are calculated using the texture color, the unit normal vector, the viewing direction vector, the light incident direction vector, and the light reflection direction vector, respectively. The ambient light component, the diffuse reflection component, and the specular reflection component are summed to synthesize the final color value of the pixel in the two-dimensional base image.

[0019] By employing the above technical solution, the physical interaction process between light and the object surface is simulated based on a local illumination model. By calculating the ambient light, diffuse reflection, and specular reflection components, the synthesized target object surface exhibits varying brightness and highlight effects that change with the illumination angle, thus blending it with the ambient lighting conditions and enhancing the physical realism of the image.

[0020] Furthermore, the step of superimposing sensor noise, two-dimensional region occlusion, illumination changes and imaging artifact simulation on the basis of the two-dimensional base image includes: using an additive white Gaussian noise model to inject random noise variables that follow a Gaussian distribution into the pixel values ​​of the two-dimensional base image, and performing numerical truncation to generate noisy image data. A rectangular mask region is randomly generated on the two-dimensional image plane of the noisy image data. The pixel values ​​falling within the rectangular mask region are replaced with preset fill values ​​or random noise fill to generate occluded image data. A convolution kernel is applied to the occluded image data using a convolution operation to generate a blurred image. The blurred image is then subjected to a nonlinear transformation using gamma correction coefficients and linear gain coefficients to complete the simulation of imaging artifacts and the superposition of illumination changes.

[0021] By employing the above technical solution, a cascaded two-dimensional image degradation chain is constructed. First, sensor thermal noise is simulated; second, occlusion of the lens by nearby objects is simulated; and finally, lens defocusing blur, motion blur, and color distortion during image signal processing are simulated through convolution and nonlinear transformation. This layered overlay strategy simulates various complex interferences that may occur in real imaging systems, allowing training samples to cover a wider image domain and enabling the model to learn robust feature representations.

[0022] The present invention also provides a computer device, which adopts the following technical solution: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the steps of the method described above.

[0023] The present invention also provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium storing computer instructions that, when executed by a processor, cause a computer device to perform any of the steps described above.

[0024] This invention provides a method, device, and storage medium for random data augmentation based on three-dimensional spatial structure modeling. It offers the following advantages: 1. This invention employs a 3D modeling method to structurally model the target object and its scene space. Based on virtual camera parameters, the 3D space is projected onto a 2D image plane, and rendered using a physically consistent lighting model. Sensor noise and imaging artifacts are then simulated. This method accurately preserves 3D geometric perspective relationships and lighting distribution characteristics in 2D images, and covers signal degradation characteristics of the real imaging process, solving the technical problem that traditional 2D data augmentation methods struggle to simulate viewpoint changes and lighting interactions.

[0025] 2. This invention introduces a collision detection mechanism when randomly posing the target object in three-dimensional space, and uses depth buffering technology to handle the occlusion relationship between the object and the scene. This ensures both the non-penetrating physical constraints between objects in the synthesized scene and avoids the irrationality of image logic caused by incorrect depth order, thereby ensuring the physical consistency of the synthesized image in terms of spatial structure and occlusion relationship.

[0026] 3. This invention supports the construction of virtual background environment models using various 3D representation methods, such as geometric meshes and 3D Gaussian sputtering sets, which can be flexibly selected according to scene complexity and requirements. Based on this, by superimposing lighting changes, 2D occlusion, and imaging artifact simulations onto the rendered 2D images, it can simulate complex environmental interference and imaging defects in real-world shooting, significantly improving the richness of training samples in terms of texture details, lighting diversity, and image domain distribution, thereby enhancing the realism of the synthetic data and the model's generalization ability. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the method flow in an embodiment of the present invention; Figure 2 This is a schematic diagram of the rendering and projection process based on physical consistency in an embodiment of the present invention; Figure 3 This is a schematic diagram of the imaging link simulation and data augmentation process in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] It should be noted that the random data augmentation method based on three-dimensional spatial structure modeling provided in this embodiment of the invention is implemented using a computer device and a computer-readable storage medium. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method steps in the following embodiments. Furthermore, the computer-readable storage medium stores computer instructions, which, when executed by the processor, cause the computer device to execute the method steps in the following embodiments. Those skilled in the art will understand that all or part of the processes in the methods of the following embodiments can be implemented using hardware related to the computer program instructions.

[0030] See attached document Figure 1 This invention provides a random data augmentation method based on three-dimensional spatial structure modeling, mainly comprising: S100. Perform 3D geometric and appearance modeling on the target object, establish a scene space model, and define the intrinsic parameter matrix of the virtual camera. S200. Randomly set the position and orientation of the target object in the 3D scene, configure the virtual light source parameters, and generate the camera position and orientation that satisfy the view frustum constraint. S300, based on the camera's intrinsic and extrinsic parameter matrices, projects the 3D scene model onto the 2D image plane, uses depth buffering technology to handle spatial occlusion relationships, and combines lighting model rendering to generate a physically consistent 2D base image. S400: Based on the generated two-dimensional base image, sensor noise, two-dimensional region occlusion, illumination changes, and imaging artifact simulations are superimposed to generate the final training sample.

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical principles and implementation methods of each step of this invention will be further described in detail below with reference to specific embodiments.

[0032] First, step S100 is executed. This step covers the construction of digital representations of the target object and the scene environment, as well as the parameter initialization of the virtual imaging system. In order to achieve the mapping from three-dimensional geometric space to the two-dimensional image domain, this step establishes a mathematical model that includes geometric structure, appearance texture, and camera intrinsic parameters.

[0033] It should be noted that, in this application, a virtual camera refers to a camera installed at any location in space.

[0034] It should be noted that, in this invention, the construction of the three-dimensional scene space model can employ various three-dimensional representation methods, including but not limited to geometric mesh models, three-dimensional Gaussian sputtering sets, and neural radiation fields. For ease of explanation, the following description uses geometric mesh models and three-dimensional Gaussian sputtering as examples; however, those skilled in the art should understand that other representation methods applicable to three-dimensional modeling and rendering also fall within the scope of protection of this invention.

[0035] Specifically, in step S101, the 3D data of the target category to be enhanced is acquired, and a suitable 3D representation method is used to model it. As a typical implementation, it can be represented as a mesh model containing geometric topology and surface texture. Let the... The target object model is .in, Defined as the set of vertices of the model, each vertex Includes its three-dimensional spatial coordinates in the object's local coordinate system . Defined as a set of facets, typically triangular facets are used to describe the connections between vertices to determine the surface topology of an object. For the storage and processing of facet structures, those skilled in the art can use common OBJ, PLY, or STL file formats for parsing; their specific data structures are well-known techniques in the field. To generate images with color information in subsequent rendering, the model also includes texture maps or material property sets. . The color distribution and reflectivity of the object's surface are defined, and a two-dimensional image is mapped to a three-dimensional mesh surface using texture coordinates, i.e., UV coordinates. In practical implementation, It can be a diffuse map, or it can include a normal map or a roughness map to support more complex lighting calculations. The 3D geometry and appearance modeling described in the specification covers the combined representation methods of the above-mentioned vertices, patches, and texture maps. It should be understood that other 3D representation methods existing in the art (such as 3D Gaussian sputtering, neural radiation fields, etc.) are also applicable to this invention.

[0036] Next, in step S102, a virtual background environment model for placing the target object is established. Depending on the complexity of the scene and the rendering requirements, the background scene model can adopt a variety of 3D digital representation methods, including but not limited to geometric meshes, 3D Gaussian sputtering sets, neural radiation fields, etc.

[0037] It should be noted that a virtual background environment model for placing the target object is established. The virtual background environment model can adopt various three-dimensional representation methods, such as geometric mesh representation or three-dimensional Gaussian sputtering set representation. When the three-dimensional Gaussian sputtering set is used for representation, the Gaussian elements in the three-dimensional Gaussian sputtering set include position mean, covariance matrix, opacity and spherical harmonic coefficients.

[0038] It should be noted that, in this embodiment, a geometric mesh model is used as an exemplary representation of a three-dimensional scene for clear illustration of the technical solution. It should be understood that the present invention is not limited to this; other three-dimensional representation methods such as voxel meshes, point clouds, and implicit neural fields are also applicable to the method described in this invention.

[0039] As one implementation method, an explicit geometric mesh can be used for characterization, i.e. ,in , , These correspond to the vertex coordinate set, triangle facet set, and texture mapping information of the scene, respectively, and are suitable for rigid scenes with clear structures.

[0040] As another implementation, implicit representations based on continuous fields or discrete primitives, such as neural radiation fields (NeRF) or three-dimensional Gaussian sputtering (3DGS), can be used to achieve realistic reconstruction of complex lighting and fine structures. When specifically using a set of three-dimensional Gaussian sputtering for representation, the scene model is represented as follows: Each of the Gaussian elements Includes the following parameters: the average position of the Gaussian sphere. , representing the center position of the Gaussian sphere in the scene's world coordinate system; covariance matrix The shape and rotation direction of the Gaussian sphere are determined and are typically stored as a scaling factor and a rotation quaternion; opacity The Gaussian sphere's ability to block light is represented by , and the spherical harmonic coefficients are used to fit color information related to the viewing direction. When specifically using neural radiation field representation, a multilayer perceptron network is constructed to encode the volume density and color radiation field of spatial points. Scene spatial representation encompasses the aforementioned mesh geometry representation, implicit radiation field, and discrete primitive representation. It should be understood that other representation methods suitable for 3D scene modeling can also be used to construct the aforementioned virtual background environment model.

[0041] Subsequently, in step S103, in order to simulate the real imaging process, a pinhole camera model is established, and the camera's intrinsic parameter matrix is ​​defined. The intrinsic parameter matrix determines the scale and positional relationship of points in three-dimensional space when projected onto the image plane, and its mathematical expression is: ; In the formula, and Indicates the camera is in shaft and Normalized focal length along the axial direction, in pixels. This parameter is determined by both the physical focal length and the pixel density of the image sensor. Represents the principal point coordinates of the image plane, typically located at the geometric center of the image, in pixels; the zero elements in the matrix indicate that the camera's sensor pixels are assumed to be tilt-free. shaft and The axes are orthogonal. Through the above steps, the digital preparation of the 3D assets and imaging parameters required for data augmentation has been completed.

[0042] Proceed to step S200, which is mainly used to construct a physically meaningful scene layout in a virtual three-dimensional space. Through a randomized parameter generation strategy, it simulates the diversity of object positions, lighting conditions, and shooting angles in a real environment.

[0043] In step S201, object space layout generation based on rigid body transformation is performed. In the defined world coordinate system, the target object model constructed in step S100 is... Instantiate and place the object. To simulate the uncertainty of the object's position and angle in a real scene, apply a random rigid body transformation to the object model. Let the coordinates of any vertex in the object's local coordinate system be... Its corresponding coordinates in the world coordinate system The following results were obtained through rotation and translation transformations: ; in, This represents the vertex coordinates in the transformed world coordinate system; The rotation matrix of an object relative to the world coordinate system belongs to... group; This represents the coordinates of any vertex of the object model in the local coordinate system. This represents the translation vector of an object in the world coordinate system, i.e., the three-dimensional coordinates of the object's center.

[0044] Translation vector It is obtained by random sampling within a preset 3D scene bounding box to ensure that the object is located within the effective space of the scene. This is the rotation matrix of an object relative to the world coordinate system, used to describe the object's pose. Rotation Matrix The system simulates objects and their diverse orientations by randomly sampling Euler angles (yaw, pitch, roll) or quaternions and converting them. Furthermore, to ensure the rationality of the physical layout, during the generation process... and Subsequently, collision detection can be performed using the axis-aligned or oriented bounding box of the object model. If the current object is detected to have volume overlap with the scene background or other placed objects, the position and orientation parameters are resampled until the non-penetrating physical constraint is met. For the specific algorithms of bounding box calculation and collision detection, those skilled in the art can use conventional methods such as the Separating Axis Theorem (SAT) to implement them.

[0045] In step S202, a set of virtual light sources is constructed and its parameters are configured. This is done to calculate lighting and shadows in subsequent rendering steps, by configuring the set of virtual light sources in the scene. The selection of light sources is directly related to the simulation of environmental characteristics and lighting changes. For each light source in the set... Define its attribute parameters, including the position of the light source. (For point light sources) or the direction of light rays (For parallel light), light source color and light intensity In practice, to simulate sunlight in nature or directional indoor lighting, a parallel light source is set, with its direction vector... Random sampling is performed within the upper hemisphere; to simulate localized lighting or artificial light sources, point light sources are defined, and their positions are determined. Randomly distributed within the scene space. Light intensity. With light source color Random values ​​are selected within a preset dynamic range to cover various lighting environments, from dim lighting and normal exposure to strong light.

[0046] In step S203, random generation of camera extrinsic parameters and verification of view frustum constraints are performed. The pose of the virtual camera in the world coordinate system is determined, i.e., the camera extrinsic parameters. The camera extrinsic parameters are determined by the rotation matrix. With translation vector Composition. Among them... The camera's shooting position was determined. This determines the camera's shooting orientation.

[0047] To ensure that the generated image contains valid target features, the camera pose generation must satisfy the frustum culling constraint. The specific generation process is as follows: First, the camera translation vector is randomly sampled within a spherical shell of a preset radius centered on the target object, or within a specific hemispherical space. Secondly, calculate the camera's rotation matrix. A typical approach is to use a gaze-point strategy, where the camera's optical axis is set to point towards the center of the target object or a random point nearby. The corresponding rotation matrix is ​​then calculated to ensure the object is roughly within the camera's field of view. Finally, a view frustum constraint check is performed. This is done using the generated... and To capture the key points of the target object (e.g., the geometric center) is transformed from the world coordinate system to the camera coordinate system and projected onto the image plane to calculate its pixel coordinates. and depth value If the following inequality conditions are satisfied: ; in, and For image width and height, and This is the distance between the near and far clipping planes of the camera. If the distance is between these planes, the current camera pose is considered valid; otherwise, the pose results in the object being outside the field of view or being clipped, requiring resampling. and This continues until the above constraints are met. The camera pose that satisfies the frustum constraint, as described in the instruction manual, refers to the camera extrinsic parameters determined through the above verification process.

[0048] See attached document Figure 2 Step S300 is executed, which details the mathematical transformation process of mapping geometric information in a 3D scene to a 2D image plane. It should be noted that for different types of scene space models established in step S100, the system uses a corresponding rendering pipeline for image generation. For example, rasterization rendering is used for mesh models, while volume rendering or a snowball algorithm is used for neural radiation field or Gaussian sputtering models. To clearly illustrate the data processing flow of this invention, the following describes the rasterization rendering flow based on a geometric mesh in detail: In step S301, a transformation from the model coordinate system to the world coordinate system is performed. For each target object in the scene, the target object model is extracted. Vertex data in the dataset. Let... For the first in the object model The position of a vertex in the object's local coordinate system is represented as... Based on the object pose parameters rotation matrix generated in step S200. With translation vector Transform the vertex to the world coordinate system to obtain the world coordinate point. : ; This transformation process is commonly referred to as model transformation in computer graphics. If a scene contains multiple objects, the above transformation is performed independently and in parallel on all vertices of each object.

[0049] In step S302, a transformation from the world coordinate system to the camera coordinate system is performed. Points in the world coordinate system are... Transform to the virtual camera's viewing coordinate system. Assume the camera extrinsic parameters generated in step S200 include a rotation matrix. With translation vector This parameter defines the rigid body transformation relationship from the world coordinate system to the camera coordinate system. The transformed camera coordinate system vertex coordinates... The calculation is as follows: ; in, The transformed vertex coordinates of the camera coordinate system are denoted as . ; The rotation matrix of the virtual camera determines the camera's viewing orientation; Represents the vertex coordinates in the world coordinate system; The translation vector of the virtual camera determines its spatial position.

[0050] In this coordinate system, the origin is located at the camera's optical center, and the Z-axis typically points forward along the camera's optical axis. The calculated... The component represents the perpendicular distance of the point relative to the camera plane along the camera's optical axis, i.e., the depth value. This depth value differs from the Euclidean distance from the point to the camera's optical center, ensuring a linear relationship in subsequent perspective projection transformations. These are key parameters for subsequent projection division and depth buffer occlusion determination. The coordinate transformation technology features described in the manual specifically refer to the cascaded transformation process from the local coordinate system through the world coordinate system to the camera coordinate system.

[0051] In step S303, perspective projection and pixel coordinate generation are performed. This is done using the pinhole camera intrinsic parameter matrix defined in step S100. The camera coordinate system vertex coordinates in the camera coordinate system Projecting onto a two-dimensional image plane yields pixel coordinates. To handle perspective relationships, this step is performed in a homogeneous coordinate system: ; in, As a scaling factor, in the pinhole model Expand the matrix multiplication above and eliminate the scale factor. That is, perform perspective division to obtain the final pixel coordinates. : ; ; in, , Represents the pixel coordinates projected onto the two-dimensional image plane; , This represents the camera's normalized focal length. , , Represents the coordinate components of a point in the camera coordinate system, where For depth; , Represents the coordinates of the principal point on the image plane.

[0052] Using the above formula, geometric points in three-dimensional space are precisely mapped to sampling points on a two-dimensional image plane. For points outside the image boundary... or The points are marked or cropped at this stage.

[0053] To ensure that the generated image conforms to the physical occlusion rules of the real world, step S3 then uses depth buffering technology to determine pixel-level visibility.

[0054] It should be noted that the rendering method described in this invention is not limited to rasterization rendering; other rendering methods such as volumetric rendering and neural rendering can also be used, as long as physically consistent projection from a 3D scene to a 2D image can be achieved. For ease of explanation, the following description uses rasterization rendering based on a geometric mesh model as an example.

[0055] In step S304, rasterization and fragment depth interpolation are performed. Geometric primitives projected onto the two-dimensional plane, such as triangular facets, are discretized to generate a series of pixel fragments covering the primitive region. For each fragment, the depth value of the primitive vertex is used... The planar projection depth of any point inside a primitive is calculated by interpolating the centroid coordinates. Let the three vertices of the triangular facet be... , , The corresponding camera coordinate system depth is , , For the pixel coordinates covered by this patch Its corresponding depth value The calculation is as follows: ; in, , , Let be the centroid coordinate coefficients of a pixel relative to its three vertices, satisfying .

[0056] In step S305, the depth buffer is constructed and initialized. A two-dimensional matrix space with the same resolution as the output image is allocated in memory, i.e., the depth buffer. Before rendering each frame, the depth buffer is initialized by setting the depth value of all pixel locations to the maximum possible depth (such as infinity or far from the clipping plane). ; in Indicates the depth buffer at pixel coordinates The stored depth value, This indicates the initial maximum depth.

[0057] In step S306, depth testing and visibility update are performed. For each fragment of all objects in the scene, a depth test is performed to determine its visibility. The depth value of the current fragment is then updated. The values ​​already stored in the depth buffer Compare them. If the conditions are met... If the current fragment is located in front of the rendered object, it is determined to be within the visible portion. At this point, an update operation is performed: the depth buffer is updated to... Simultaneously, update the color buffer, updating the color value of the pixel to the color value calculated for the current fragment. If the above conditions are not met, i.e. If the current fragment is occluded, its data is discarded. Through this pixel-by-pixel depth comparison mechanism, occluded geometric surfaces are automatically hidden, ensuring that the generated 2D image has strict physical consistency in its geometric structure.

[0058] In order to make the generated image have a realistic sense of light and shadow and material details, step S3 also includes using a lighting model to perform color calculations on the fragments that have passed the test.

[0059] In step S307, surface property sampling and normal interpolation are performed. For each pixel coordinate that passes the depth test... Based on its corresponding centroid coordinate coefficients, the vertex attributes in the object model are interpolated to obtain the precise position of the pixel in three-dimensional space. And the corresponding texture coordinates. Based on texture coordinates The texture map constructed in step S100 Sampling is performed to obtain the base albedo color at that point. Simultaneously, calculate the unit normal vector of that pixel. If the model includes a normal map, it needs to be transformed from tangent space to camera coordinates or world coordinates; if only vertex normals are used, the model can be obtained by interpolating and normalizing the vertex normals. This is to ensure the accuracy of the angle calculation for illumination.

[0060] In step S308, illumination vector construction and reflection calculation are performed. This is based on the set of virtual light sources configured in step S200. For the current pixel, construct the geometric vectors required for lighting calculations. First, calculate the view direction vector. That is, from the three-dimensional position of the pixel The unit vector pointing to the optical center of the camera. Secondly, for each light source in the set... Calculate the incident direction vector of the light ray. If the light source is parallel light, If it is a constant vector; if it is a point light source, then it is from... A unit vector pointing to the location of the light source. Further calculation of the light reflection direction vector. According to the law of reflection, the direction vector of reflection is... Determined by the direction of the incident light and the surface normal, the calculation formula is: All the above vector calculations are performed in the same coordinate system, usually the camera coordinate system, and are ensured to be unit vectors.

[0061] In step S309, pixel color synthesis is performed based on the rendering equation. Using the Von Leigh lighting model or a similar local lighting model, the final color value of the pixel is calculated by integrating ambient light, diffuse reflection, and specular reflection components. The specific rendering calculation formula is defined as follows: ; in, This represents the ambient light component, used to simulate scattered light from all directions in a scene; The set of valid light sources participating in the calculation; This indicates that all light sources within the set are summed. The diffuse reflectance coefficient is... This represents the Lambert cosine term, reflecting the effect of the angle of incidence on brightness; For the first The intensity and color of diffuse reflected light from a single light source; The specular reflection coefficient is used to control the intensity of the highlight area; Represents the unit normal vector; Indicates the highlight item, where Gloss index The larger the value, the smaller and sharper the highlight spot, simulating the reflective properties of a smooth surface; For the first The intensity and color of the specular reflection of a light source; The color of the object surface texture sampled in step S307; The element-wise multiplication of vectors, known as the Hadamard product, involves weighted calculations of the R, G, and B color channels. Through these calculations, the system generates a two-dimensional base image containing accurate occlusion relationships, texture details, and lighting effects.

[0062] See attached document Figure 3 Finally, step S400 is executed, which introduces various perturbations and defects from the real imaging process at the pixel level. This step aims to compensate for the distribution differences between the idealized rendered image and the image acquired by the real sensor.

[0063] In step S401, random noise from the sensor is superimposed. This is to simulate the electronic thermal noise or shot noise generated by the camera's photosensitive element during photoelectric conversion and signal transmission, and to add random noise to the two-dimensional base image. Noise injection is performed. This embodiment uses an additive white Gaussian noise model to characterize the sensor noise. For each pixel coordinate in the image... The pixel values ​​of the noisy image after adding noise The calculation is as follows: ; in, This represents the pixel value of the noisy image after adding noise; Represents the pixel values ​​of a two-dimensional basic image; For random noise variables that follow a Gaussian distribution, i.e. . This represents the standard deviation of the noise level, used to control the noise intensity. In practice, to increase sample diversity, The value is within the preset range Random sampling is performed within the color space. After adding noise, the pixel values ​​need to be truncated to ensure that their values ​​are within the valid color space range.

[0064] In step S402, random occlusion in a two-dimensional region is generated. Unlike the physical occlusion based on three-dimensional geometry in step S300, this step simulates unstructured random occlusion in the two-dimensional image domain to enhance the model's robustness to missing local features. This occlusion simulates dirt on a lens surface or sudden objects not modeled in the three-dimensional scene obstructing the view. One or more rectangular mask regions are randomly generated on the image plane. For a pixel in an image, if its coordinates... Falling into the mask area Inside, the pixel values ​​are replaced to generate the occluded image. : ; in, This represents the pixel values ​​of the partially occluded image after processing. This represents a randomly generated mask area; This represents the pixel values ​​retained in the unobstructed area; This represents the fill value for the occluded area. It can be set to a constant such as black, gray, the average pixel value of the image, or random noise fill within the area. Mask area The location, aspect ratio, and area size are all determined through random sampling, and an upper limit for the area threshold is set to prevent the key features of the target object from being completely lost due to an excessively large occlusion area.

[0065] In step S403, optical defects and ambient lighting effects are simulated. To reproduce motion blur, focus deviation, and nonlinear photometric response commonly seen in real-world shooting, optical and photometric transformations are applied to the image. For the blurring effect, a convolution operation is used to simulate the point spread function of the optical system. The resulting blurred image... Represented as: ; in, Represents a blurred image; Indicates a partially occluded image; This represents the convolution operation. The kernel is used for convolution. When simulating motion blur caused by relative camera motion... Designed as a linear motion kernel in a specific direction; when simulating Gaussian blur caused by focus misalignment, Designed with an isotropic Gaussian kernel. For photometric response, it simulates the nonlinear mapping of light intensity by the display device or sensor, i.e., Gamma correction and brightness / contrast fluctuations. The final output is an enhanced image. The calculation formula is as follows: ; In the formula, This represents the final generated training sample image; Indicates the pixel value of a blurred image; The gamma correction factor is used to adjust the overall tonal distribution of an image and simulate different exposure curve characteristics. This is the linear gain coefficient, used to simulate the overall fluctuation of ambient brightness. and All samples are randomly sampled within a preset neighborhood centered at 1. Through the above steps, this invention transforms ideal rendering data into training samples containing rich noise patterns and environmental interference, realizing the specific technical implementation of sensor noise, two-dimensional region occlusion, illumination changes, and imaging artifact simulation, ensuring that the computer instructions recorded in the storage medium can completely reproduce the above data augmentation process when executed by the processor.

Claims

1. A random data augmentation method based on three-dimensional spatial structure modeling, characterized in that, Includes the following steps: S100. Perform 3D geometric and appearance modeling on the target object, establish a scene space model, and define the intrinsic parameter matrix of the virtual camera. S200. Randomly set the position and orientation of the target object in the world coordinate system in the scene space model, configure the virtual light source parameters, and generate the camera position and camera orientation that satisfy the view frustum constraint. S300: Based on the intrinsic parameter matrix of the virtual camera and the extrinsic parameter matrix constructed according to the camera position and the camera orientation, the target object and the scene space model after the position and posture are set are projected onto a two-dimensional image plane. The spatial occlusion relationship between the target object and the scene space model is processed using depth buffering technology, and rendering is performed in combination with the virtual light source parameters and lighting model to generate a physically consistent two-dimensional basic image. S400: Based on the two-dimensional base image, sensor noise, two-dimensional region occlusion, illumination changes and imaging artifact simulation are superimposed to generate the final training sample.

2. The random data augmentation method based on three-dimensional spatial structure modeling according to claim 1, characterized in that, In step S100, establishing the scene space model includes: Obtain the 3D data of the target category to be enhanced, and represent the 3D data as a mesh model containing a set of vertices, a set of faces, and a texture map; the texture map is used to represent the color distribution and reflection characteristics of the target object's surface; A virtual background environment model for placing the target object is established. The virtual background environment model can be implemented using various three-dimensional representation methods, including but not limited to geometric mesh models or three-dimensional Gaussian sputtering sets. When the three-dimensional Gaussian sputtering set is used for characterization, the Gaussian elements in the three-dimensional Gaussian sputtering set include the position mean, covariance matrix, opacity, and spherical harmonic coefficients.

3. The stochastic data augmentation method based on three-dimensional spatial structure modeling according to claim 1, characterized in that, In step S200, randomly setting the position and orientation of the target object in the world coordinate system within the scene space model includes: Randomly sample translation vectors and rotation matrices within the preset 3D scene bounding box; Using the rotation matrix and the translation vector, a rigid body transformation is performed on the vertex coordinates of the target object in the local coordinate system, and the vertex coordinates of the target object in the world coordinate system are calculated. Collision detection is performed using the axis-aligned bounding box or oriented bounding box of the target object. If the target object is found to have volume overlap with the scene space model, the translation vector and the rotation matrix are resampled until the non-penetrating physical constraint is met.

4. The stochastic data augmentation method based on three-dimensional spatial structure modeling according to claim 1, characterized in that, In step S200, generating the camera position and camera orientation that satisfy the view frustum constraint includes: The camera position is randomly sampled within a preset space centered on the target object, and the camera orientation is calculated according to the gaze point strategy to obtain the extrinsic parameter matrix of the virtual camera. The key points of the target object are transformed from the world coordinate system to the camera coordinate system and projected onto the two-dimensional image plane to calculate pixel coordinates and depth values; If the pixel coordinates are within the range of the image width and image height of the two-dimensional image plane, and the depth value is between the distances of the near and far clipping planes of the virtual camera, then the current camera position and camera orientation are determined to be valid.

5. The stochastic data augmentation method based on three-dimensional spatial structure modeling according to claim 1, characterized in that, In step S300, projecting the target object and the scene space model, after the position and orientation settings are defined, onto the two-dimensional image plane includes: Using the extrinsic parameter matrix of the virtual camera, the vertex coordinates in the world coordinate system are transformed to the camera coordinate system to obtain camera coordinates containing depth values, where the depth values ​​are the vertical distance along the camera optical axis relative to the camera plane. Using the intrinsic parameter matrix of the virtual camera, perspective projection and perspective division operations are performed on the camera coordinate system coordinates to generate pixel coordinates on the two-dimensional image plane.

6. The stochastic data augmentation method based on three-dimensional spatial structure modeling according to claim 1, characterized in that, In step S300, processing the spatial occlusion relationship between the target object and the scene spatial model using depth buffering technology includes: The geometric primitives projected onto the two-dimensional image plane are rasterized to generate pixel fragments, and the fragment depth value of the pixel fragments is calculated using centroid coordinate interpolation. Initialize the depth buffer and set the depth value of all pixel positions in the depth buffer to the preset maximum depth value; Perform a depth test on each pixel fragment: compare the current fragment depth value of the pixel fragment with the depth values ​​already stored in the depth buffer; If the fragment depth value is less than the depth value already stored in the depth buffer, the pixel fragment is determined to be visible, and the depth buffer and color buffer are updated; if the fragment depth value is greater than or equal to the depth value already stored in the depth buffer, the pixel fragment is discarded.

7. The stochastic data augmentation method based on three-dimensional spatial structure modeling according to claim 6, characterized in that, The rendering process, which combines the virtual light source parameters and lighting model to generate a physically consistent two-dimensional base image, includes: For the pixel fragments that pass the depth test, sample the texture color of the target object surface and interpolate to calculate the unit normal vector; Based on the virtual light source parameters, a viewing direction vector, a light incident direction vector, and a light reflection direction vector are constructed for the pixel fragment. Based on the local illumination model, the ambient light component, diffuse reflection component, and specular reflection component are calculated using the texture color, the unit normal vector, the viewing direction vector, the light incident direction vector, and the light reflection direction vector, respectively. The ambient light component, the diffuse reflection component, and the specular reflection component are summed to synthesize the final color value of the pixel in the two-dimensional base image.

8. The stochastic data augmentation method based on three-dimensional spatial structure modeling according to claim 1, characterized in that, In step S400, the step of superimposing sensor noise, two-dimensional region occlusion, illumination changes, and imaging artifact simulation onto the two-dimensional base image includes: An additive white Gaussian noise model is used to inject random noise variables that follow a Gaussian distribution into the pixel values ​​of the two-dimensional base image, and numerical truncation is performed to generate noisy image data. A rectangular mask region is randomly generated on the two-dimensional image plane of the noisy image data. The pixel values ​​falling within the rectangular mask region are replaced with preset fill values ​​or random noise fill to generate occluded image data. A convolution kernel is applied to the occluded image data using a convolution operation to generate a blurred image. The blurred image is then subjected to a nonlinear transformation using gamma correction coefficients and linear gain coefficients to complete the simulation of imaging artifacts and the superposition of illumination changes.

9. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method steps of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions, which, when executed by a processor, cause a computer device to perform the method steps of any one of claims 1 to 8.

Images