Training data generation method, system and equipment based on Gaussian splashing and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a training data generation method, system and device based on Gaussian splashing and a medium, and the method comprises the steps: obtaining a two-dimensional multi-view picture of a to-be-detected target, carrying out the motion structure recovery of the two-dimensional multi-view picture, and obtaining a three-dimensional initialization point cloud of the to-be-detected target; performing three-dimensional reconstruction according to the three-dimensional initialized point cloud to obtain a three-dimensional target model; performing Gaussian splash multi-view rendering on the three-dimensional target model by using a preset virtual camera to obtain a target rendered image; and performing target labeling and background synthesis on the target rendering image to obtain a training image set of the to-be-detected target. Through the implementation of the invention, the to-be-detected target can be fused with the real background of the real background image in the real background image library, various training image sets are efficiently generated, and efficient and accurate generation of target detection model image training data is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device and medium for generating training data based on Gaussian splashing. Background Technology

[0002] With the widespread application of deep learning in the field of computer vision, the demand for high-quality labeled data for object detection models is increasing. Traditional training dataset production methods mainly rely on manual labeling. In order to overcome the human resource and time costs of manual labeling, existing technologies are mainly based on three-dimensional reconstruction and simulation technologies to automatically generate two-dimensional training data.

[0003] Three-dimensional reconstruction and simulation technologies generate two-dimensional data by constructing complete three-dimensional models of objects and environments in a virtual digital space through procedural or data-driven methods, and simulating their physical appearance and dynamics to generate image data. In recent years, neural radiation field technology has achieved high-quality synthesis of new perspectives through implicit neural representations. It can construct implicit 3D models of scenes from images of known locations and render 2D images from unseen perspectives. Furthermore, 3D Gaussian Splatting (3DGS), as an emerging explicit rendering technology, uses anisotropic Gaussian ellipsoids to represent three-dimensional scenes, achieving real-time rendering and high-fidelity image data generation.

[0004] However, in existing technologies, neural radiation fields and 3D Gaussian splashing are mainly used for the 3D scene reconstruction and new perspective synthesis of the target to be detected, and have not been effectively converted into the ability to automatically generate 2D detection data. They cannot automatically generate accurate bounding boxes or segmentation mask annotations. At the same time, 3D reconstruction technology can only provide sparse reconstruction results, making it difficult to generate high-fidelity images. This results in the inability to automatically generate accurate bounding boxes or segmentation mask annotations, still requiring significant manual intervention to convert the rendering results into 2D training data.

[0005] Therefore, how to efficiently and accurately generate image training data for object detection models has become an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a method, system, device, and medium for generating training data based on Gaussian splashing, the main purpose of which is to solve the problems of low efficiency and accuracy when generating image training data for target detection models.

[0007] To achieve the above objectives, this invention provides a training data generation method based on Gaussian splashing, comprising: acquiring two-dimensional multi-view images of a target to be detected; performing motion structure recovery on the two-dimensional multi-view images to obtain a three-dimensional initial point cloud of the target to be detected; performing three-dimensional reconstruction of the target to be detected based on the three-dimensional initial point cloud to obtain a three-dimensional target model; performing Gaussian splashing multi-view rendering on the three-dimensional target model using a preset virtual camera to obtain a target rendering image; and performing target annotation and background synthesis on the target rendering image to obtain a training image set of the target to be detected.

[0008] This invention also provides a training data generation system based on Gaussian splashing. The system includes: a motion structure recovery module for acquiring two-dimensional multi-view images of the target to be detected, performing motion structure recovery on the two-dimensional multi-view images to obtain a three-dimensional initial point cloud of the target to be detected; a three-dimensional reconstruction model for performing three-dimensional reconstruction of the target to be detected based on the three-dimensional initial point cloud to obtain a three-dimensional target model; a Gaussian rendering module for performing Gaussian splashing multi-view rendering on the three-dimensional target model using a preset virtual camera to obtain a target rendered image; and a training image set assembly module for performing target annotation and background synthesis on the target rendered image to obtain a training image set of the target to be detected.

[0009] The present invention also provides an electronic device, comprising: a memory communicatively connected to at least one processor; wherein the processor is configured to execute a computer program stored in the memory; the memory stores a computer program executable by at least one processor, the computer program being executed by at least one processor to enable at least one processor to execute the above-described method for generating training data based on Gaussian splashing.

[0010] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the training data generation method based on Gaussian splashing described above.

[0011] This invention reconstructs the motion structure of a target object from two-dimensional multi-view images to obtain a three-dimensional initial point cloud, providing a foundation for subsequent three-dimensional reconstruction to obtain a three-dimensional target model. Three-dimensional reconstruction based on the initial point cloud accurately represents the geometric and appearance information of the original point cloud, resulting in a precise three-dimensional target model. A pre-set virtual camera is used to perform Gaussian splash multi-view rendering on the three-dimensional target model to obtain a target rendering image. This simulates the appearance of the target object in a complex real-world environment, avoiding visual horror and unrealistic lighting issues, effectively improving the accuracy of training image set generation. Target annotation and background synthesis are performed on the target rendering image, fusing the target object with the real background of real background images in a real background image library. This efficiently generates a training image set covering various lighting, weather, seasons, and geographical environments, achieving efficient and accurate generation of image training data for the target detection model. Attached Figure Description Figure 1 This is a flowchart illustrating a method for generating training data based on Gaussian splashing, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the process for three-dimensional reconstruction of the target to be detected according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the process of performing Gaussian splash multi-view rendering on a three-dimensional target model according to an embodiment of the present invention; Figure 4 A functional block diagram of a training data generation system based on Gaussian splashing provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements a training data generation method based on Gaussian splashing, according to an embodiment of the present invention.

[0012] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0014] This application provides a training data generation method based on Gaussian splashing. The execution entity of the training data generation method based on Gaussian splashing includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the training data generation method based on Gaussian splashing can be executed by software or hardware installed on a terminal device or a server device; the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0015] Reference Figure 1 The diagram shown is a flowchart illustrating a training data generation method based on Gaussian splashing according to an embodiment of the present invention. In this embodiment, the training data generation method based on Gaussian splashing includes: S1. Obtain two-dimensional multi-view images of the target to be detected, perform motion structure restoration on the two-dimensional multi-view images, and obtain the three-dimensional initial point cloud of the target to be detected.

[0016] In this embodiment of the invention, the target to be detected is a target that needs to be automatically detected in fields such as drone monitoring, autonomous driving, intelligent robots, virtual reality and augmented reality (VR / AR), digital protection of cultural heritage and industrial quality inspection, such as pedestrians, vehicles, cars, trucks, drones, traffic signs, virtual actions, etc.

[0017] Furthermore, multiple images of the target to be detected can be captured from different angles and distances using drones or fixed camera groups to ensure coverage of the target from all angles, thus obtaining two-dimensional multi-view images of the target.

[0018] Specifically, motion structure recovery is performed on two-dimensional multi-view images to obtain a three-dimensional initial point cloud of the target to be detected, including: uniformly sampling the two-dimensional multi-view images to obtain a target image set; extracting image features of each image in the target image set, performing image matching based on the image features to obtain image matching pairs; calculating the camera pose parameter sequence of the target image set based on the image matching pairs, and calculating the three-dimensional initial point cloud based on the camera pose parameter sequence.

[0019] In this embodiment of the invention, uniform sampling is to sample two-dimensional multi-view images at uniform time intervals. For example, images are filtered from multiple two-dimensional multi-view images at time intervals of 1 second, 0.5 seconds, etc., in order to balance computational efficiency and scene coverage of the target to be detected.

[0020] In detail, the image features of each image are unique and repeatable local features (such as corner points and edge textures) extracted from each target image. Scale-invariant feature transformation can be used to extract key points. For each key point, a 16×16 neighborhood aligned with the main direction is taken and divided into 4×4 units. Gradient histograms in 8 directions are calculated within each unit to generate an 8-dimensional vector. The vectors of the 4×4 units are concatenated to form a 128-dimensional (or 256-dimensional) binary descriptor. Then, using each descriptor, all descriptors of other target images in the target image set are traversed, and Hamming distance or Euclidean distance is calculated. The two target images with the smallest distance are selected as the image matching pair.

[0021] Furthermore, the camera pose parameter sequence of the target image set is calculated by using the matched pixels (interior points) in each image matching pair to solve for the camera extrinsic parameters (rotation matrix R, translation vector t) and intrinsic parameters (focal length f, principal point coordinates cx / cy, distortion coefficients) of the target image. Then, for the matched pixels, i.e. the projection points of the same 3D point cloud in the image matching pair, linear triangulation is used to generate an initial sparse point cloud.

[0022] Linear triangulation involves projecting the same 3D point cloud onto the image matching pair, constructing the camera projection model equation based on the corresponding image matching pair, and solving the simultaneous equations to obtain the 3D initial point cloud.

[0023] In this embodiment of the invention, by using multiple image matching pairs and the camera pose parameter sequence of the target image set, the point cloud data of the target to be detected can be fully covered from multiple perspectives, thereby obtaining the three-dimensional initial point cloud corresponding to the target to be detected, providing a foundation for subsequent three-dimensional reconstruction to obtain a three-dimensional target model.

[0024] S2. Based on the 3D initial point cloud, perform 3D reconstruction of the target to be detected to obtain a 3D target model.

[0025] In this embodiment of the invention, the three-dimensional reconstruction utilizes 3D Gaussian Splatting (3DGS) to perform high-precision three-dimensional reconstruction of the target to be detected, and efficiently renders a realistic three-dimensional target model of the target to be detected.

[0026] Specifically, participate Figure 2 As shown, a 3D target model is obtained by reconstructing the target from the 3D initialized point cloud, including: S21. Convert the 3D initialized point cloud into 3D Gaussian elements; S22. Calculate the rendering loss value between the 3D Gaussian unit and the preset real image; S23. Optimize the parameters of the 3D Gaussian unit based on the rendering loss value to obtain the target 3D reconstruction model.

[0027] In this embodiment of the invention, the 3D Gaussian unit is composed of Gaussian parameters such as the position information, color, and normal direction of each three-dimensional initial point cloud, and its shape and orientation are defined by the covariance matrix. The three-dimensional initial point cloud can be clustered first, such as by K-means, mean shift or voxel grid filtering, and the centroid of each cluster is taken as the mean of the Gaussian distribution of the point cloud data within the cluster, that is, the Gaussian position parameter.

[0028] Furthermore, color parameters can be obtained from the color information of corresponding points in the target image corresponding to the point cloud data, and normal direction parameters can be calculated through covariance analysis of the neighborhood point cloud or geometry-based normal estimation methods. The covariance matrix is ​​used to describe the uncertainty and distribution around the point, and can be calculated based on the local geometry and density of the point cloud. The initial covariance matrix is ​​usually set as a diagonal matrix, and its diagonal elements represent the variance of each axis direction. The initial value can be set to a value related to the point cloud density. 3D Gaussian units are formed by the Gaussian parameters of each 3D initial point cloud.

[0029] In detail, the real image is the image from which a 3D initial point cloud is generated, which can be a set of target images obtained by uniformly sampling 2D multi-view images. The rendering loss value is the difference between the rendering result (projecting 3D Gaussian units onto a 2D image plane for efficient rendering) and the real image. By comparing the rendering result with the real image, the quality of the rendering result can be evaluated, and the parameters of the Gaussian distribution can be adjusted according to the difference to improve the accuracy of 3D scene reconstruction.

[0030] Specifically, the rendering loss value between the 3D Gaussian unit and the preset real image is calculated, including: spatially partitioning the 3D Gaussian unit to obtain a spatial data set; performing Gaussian projection and depth sorting on the spatial data set to obtain projected Gaussian units; rendering the projected Gaussian units with color transparency to obtain a rendered image; and calculating the loss value between the rendered image and the real image to obtain the rendering loss value.

[0031] In detail, spatial partitioning divides the data space containing the 3D Gaussian cell into spheres, ellipsoids, cubes, AABBs (axis-aligned bounding boxes), octrees, and voxel primitives to obtain a spatial data set. Spatial partitioning can significantly improve the processing efficiency and quality of the 3D Gaussian cell.

[0032] Furthermore, Gaussian projection projects each 3D Gaussian unit in the spatial data set onto the image plane based on the camera extrinsic and intrinsic parameters calculated above, obtaining its 2D elliptical projection on the image. Then, the 2D elliptical projection is sorted according to the distance (depth) of the 3D Gaussian unit from the camera optical center. This ensures that Gaussian units that are far away are rendered first and those that are close are rendered later, effectively handling occlusion relationships and improving the accuracy of the generated rendered image.

[0033] In detail, color transparency rendering calculates the color contribution of all Gaussian units projected onto each 2D pixel, and then superimposes them according to the transparency α to complete image composition and obtain the rendered image. The alpha value is a scalar between 0 and 1, representing the transparency (or opacity) of an object. It is a key parameter used to control color blending and transparency, and its core function is to measure the visibility of a primitive (such as a Gaussian unit, pixel, etc.) when superimposed onto the target image.

[0034] Specifically, a weighted loss of L1 loss and differentiable SSIM loss can be used as the rendering loss value. The L1 loss is calculated by averaging the absolute error of the rendered image and the real image pixel by pixel; the SSIM loss is an indicator that measures the structural similarity between the rendered image and the real image, calculated from three dimensions: brightness (L), contrast (C), and structure (S), which is closer to human visual judgment of structural similarity.

[0035] In this embodiment of the invention, parameter optimization of the 3D Gaussian unit involves optimizing the Gaussian parameters in the 3D Gaussian unit, such as position information, color, normal direction, and transparency, until the rendering loss value is less than a preset threshold, thereby obtaining the target 3D reconstruction model.

[0036] In detail, the three-dimensional target model is a set of Gaussian parameters of 3D Gaussian units after parameter optimization, namely the optimized Gaussian parameters such as position, covariance, color, and opacity, which can accurately represent the geometric and appearance information of the original point cloud and obtain an accurate target three-dimensional model.

[0037] S3. Use a preset virtual camera to perform Gaussian splash multi-view rendering on the 3D target model to obtain the target rendering image.

[0038] In this embodiment of the invention, the virtual camera defines the parameters (position, pose, intrinsic parameters) of the virtual camera in the world coordinate system of the three-dimensional target model, and then projects the three-dimensional Gaussian units in the three-dimensional target model onto the image plane of the virtual camera through virtual rendering, and outputs a rendered image that conforms to the imaging law of a real camera.

[0039] In detail, the virtual cameras can be generated using Python with random positions and poses to produce virtual sampled images that surround a 3D target model. Each virtual camera is oriented towards the center of the 3D target model.

[0040] In this embodiment of the invention, Gaussian splash multi-view rendering outputs target rendering images of two different types of targets in parallel from the same virtual camera viewpoint, including a real scene rendering image and a segmented rendering image. The real scene rendering image and the segmented rendering image share the virtual camera's intrinsic parameters (focal length, resolution, etc.) and extrinsic parameters (position, pose).

[0041] The realistic scene rendering image is a target rendering image that integrates realistic lighting and texture, simulating the appearance of the target to be detected in a complex real environment. The segmentation rendering image is a target segmentation mask (green screen image) with a uniform high-contrast solid color (such as green) as the background. The segmentation rendering image shields the interference of complex backgrounds during rendering, and can obtain accurate pixel-level segmentation information of the target to be detected.

[0042] Specifically, participate Figure 3 As shown, a pre-set virtual camera is used to perform Gaussian splash multi-view rendering on the 3D target model to obtain the target rendering image, including: S31. Obtain the camera parameters of each virtual camera, and perform two-dimensional elliptical projection on the three-dimensional target model according to the camera parameters to obtain two-dimensional Gaussian elements. S32. Divide the three-dimensional Gaussian unit corresponding to the three-dimensional target model into background Gaussian unit and segmentation Gaussian unit; S33. Calculate the pixel coverage of the segmented Gaussian unit in the two-dimensional Gaussian unit, and perform scene rendering on the segmented Gaussian unit based on the pixel coverage to obtain the segmented rendering image; S34. Calculate the background pixel coverage of the background Gaussian unit in the two-dimensional Gaussian unit, and perform real scene rendering on the background Gaussian unit based on the segmented pixel coverage to obtain a real scene rendering image. S35. Combine real scene rendering images and segmented rendering images to obtain the target rendering image.

[0043] In this embodiment of the invention, the camera parameters are the preset camera extrinsic parameters (rotation matrix R, translation vector t) and camera intrinsic parameters (focal length f, principal point coordinates cx / cy, distortion coefficient) for each virtual camera. Then, with the geometric center of the three-dimensional target model as the center of the sphere, the virtual camera positions are uniformly sampled on the sphere of radius R, with the orientation of all cameras facing the center of the sphere (suitable for a surround view, such as a 360° panoramic view of an object), ensuring that the view of the obtained rendered image covers the three-dimensional target model.

[0044] Furthermore, based on the camera parameters, a two-dimensional elliptical projection is performed on the three-dimensional target model. This is achieved by using extrinsic parameters R and t to convert the center of the three-dimensional Gaussian unit in the three-dimensional target model from the world coordinate system to the camera coordinate system where the virtual camera is located. Then, using intrinsic parameter K, the point at the center in the camera coordinate system is projected onto the image plane to obtain two-dimensional pixel coordinates. The three-dimensional covariance matrix Σ (describing the spatial shape) of the three-dimensional Gaussian unit is converted into the covariance matrix Σ_2D of the three-dimensional ellipse (describing the projection shape of the Gaussian on the image, such as a circle or an ellipse) through the camera projection matrix, thus obtaining the two-dimensional Gaussian unit.

[0045] In this embodiment of the invention, the segmented (foreground) Gaussian unit refers to the Gaussian unit covered by the target to be detected in the 3D target model, and the background Gaussian unit refers to the Gaussian unit covered by the background region. The segmented Gaussian unit and the background Gaussian unit have different appearance attribute codes, so they can be modeled separately to obtain a real scene rendering image and a segmented rendering image.

[0046] Furthermore, supervised classification and automatic segmentation methods can be used to divide the 3D Gaussian unit into background Gaussian unit and segmentation Gaussian unit.

[0047] In supervised classification, information from the 2D semantic annotation map is mapped to 3D Gaussian units. By calculating the overlap rate between the coverage area of ​​each 3D Gaussian unit projected onto the 2D segmentation map and the foreground label, semantic labels are finally assigned to the 3D Gaussian units. Automatic segmentation and classification (such as FlashSplat) solves for the label of each 3D Gaussian unit (1 for the foreground and 0 for the background) through linear programming. Specifically, the 2D rendering result is regarded as a linear function of the 3D Gaussian unit label. By minimizing the error between the rendering mask and the ideal foreground region, the contribution of the 3D Gaussian unit to the foreground is inferred, resulting in the background Gaussian unit and the segmentation Gaussian unit.

[0048] In detail, the pixel coverage of the segmented Gaussian unit in the two-dimensional Gaussian unit is calculated as the two-dimensional elliptical coverage of the segmented Gaussian unit in the image.

[0049] In this embodiment of the invention, scene rendering is performed on the segmented Gaussian unit based on the pixel coverage area to obtain a segmented rendering image, including: initializing a blank image; determining the appearance attribute code according to the segmented Gaussian unit; calculating the pixel value of each pixel in the pixel coverage area based on the appearance attribute code; and rendering the blank image based on the pixel value to obtain the segmented rendering image.

[0050] In this embodiment of the invention, the blank image is a blank RGB image with a resolution of W×H (initial pixel value is 0), and a corresponding cumulative transparency buffer is created (initial value is 0, used to record the total transparency of each pixel that has been blended).

[0051] Furthermore, the appearance attribute encoding is the color c_mask and transparency α_mask of the segmentation mask channel corresponding to the segmentation Gaussian unit. For each pixel covered by the segmentation Gaussian unit in the pixel coverage area, the pixel color is updated according to the Alpha mixing formula. The updated pixel value is filled into the blank image, and a segmented rendering image in which the foreground pixels completely cover the background pixels can be obtained. For example, a segmented rendering image (also known as a green screen image) in which, except for the foreground of the target to be detected, all others are high-contrast solid colors (such as green) as the background.

[0052] In detail, appearance attribute encoding can extract the RGB value of the corresponding pixel by projecting the coordinates of each Gaussian cell into a two-dimensional Gaussian cell, removing outliers from the occluded viewpoint, and taking the average value as the initial color. The initial value is usually set to 1 (completely opaque).

[0053] Similarly, for rendering background Gaussian units, a realistic scene rendering image that integrates realistic lighting and texture can be generated based on the appearance attribute encoding of any two-dimensional real background image, simulating the appearance of the target to be detected in a complex real environment.

[0054] For example, for the background Gaussian unit, the transparency is set to 0 within the background pixel coverage area of ​​the two-dimensional Gaussian unit, and the transparency of other pixels and the pixels where the target to be detected is located is set to 1. The pixel color is updated according to the Alpha mixing formula of any preset two-dimensional real background image to obtain the real scene rendering image rendered under different real backgrounds.

[0055] Specifically, the pixel value in a real-world scene rendered image = foreground pixel × Alpha value + background pixel × (1 Alpha value).

[0056] For example, if the target to be detected is a car, the pixels of the car body covering the background are foreground pixels, completely opaque (Alpha=1); the car glass, displaying both glass texture and background trees, are semi-transparent pixels (Alpha=0.5): the blended pixel = 50% foreground + 50% background; the sky area not covered by the foreground is completely transparent pixels (Alpha=0): the blended pixel = background pixels, only displaying the background. Through a pixel-by-pixel alpha blending formula, the foreground and background layers are superimposed, achieving pixel-level fusion between the target to be detected and different real-world background images.

[0057] In this embodiment of the invention, the target rendering image is rendered through dual-channel rendering of the foreground and the real scene (background), which can combine the precise control of the target itself in 3D reconstruction with the unparalleled realism of the 2D real background in terms of light and shadow and texture complexity. This fundamentally avoids the visual horror and unrealistic lighting problems that are difficult to avoid in full 3D simulation, and effectively improves the accuracy of training image set generation.

[0058] S4. Perform target annotation and background synthesis on the target rendered image to obtain the training image set of the target to be detected.

[0059] In this embodiment of the invention, target annotation involves adding semantic labels and spatial location information to the target to be detected in the rendered image, transforming image pixels into structured data that can be understood by machine learning, and providing training data for deep learning models (such as object detection, semantic segmentation, and image generation).

[0060] Furthermore, background synthesis involves automatically synthesizing the target to be detected with a large and diverse library of real background images. The background images in the library do not semantically contain the target to be detected, ensuring that the model learns the features of the target itself, rather than its association with a specific background.

[0061] Specifically, target annotation and background synthesis are performed on the target rendered image to obtain a training image set of the target to be detected. This includes: comparing feature points of the real scene rendered image and the segmented rendered image in the target rendered image to obtain the target region of the target to be detected; constructing an annotated rendered image of the target to be detected in the target rendered image based on the target region; replacing the background with a preset real background image library using the segmented rendered image to obtain a replacement background image library; annotating each replacement background image in the replacement background image library to obtain an annotated background image library corresponding to the replacement background image; and constructing a training image set of the target to be detected based on the annotated rendered image and the annotated background image library.

[0062] In this embodiment of the invention, edge detection can be performed using edge detection and other methods to obtain the real scene rendering image and the target region where the segmented rendering image is located in the target rendering image. In the target region, annotations including bounding boxes, polygon segmentation annotations and instance segmentation masks are generated in the format of mainstream object detection datasets such as YOLO (You Only Look Once) and COCO (Common Objects in Context) to obtain an annotated rendering image.

[0063] Furthermore, by using spatial alignment and pixel-level blending, the target to be detected in the segmented rendered image can be fused with the real background of the real background image in the real background image library to obtain a replacement background image library that includes the target image.

[0064] In detail, spatial alignment involves adjusting the virtual camera intrinsics of the target to be detected based on the camera intrinsics (such as focal length) of the real background image to ensure that the perspective effect of the target is consistent with the background (e.g., if distant objects in the background are small in scale, the target also needs to be reduced accordingly); at the same time, position matching and scale matching of the target to be detected are performed.

[0065] Furthermore, pixel-level blending achieves a natural overlay of the target image and the real background image through the Alpha blending algorithm. For example, the entity pixels of the target image (Alpha=1) completely cover the background, while the shadow pixels of the target image (Alpha=0.3) are partially blended with the background pixels, which can simulate the "transparency effect" of real shadows.

[0066] In detail, the target to be detected in the replacement background image can be determined by segmenting the pixel region where the target to be detected is located in the rendered image. Then, the target to be detected can be labeled in the replacement background image. For example, the bounding box of the target to be detected and polygon segmentation labels can be added to the replacement background image. This efficiently generates a labeled background image library, which is beneficial for subsequent training of the target detection model.

[0067] Preferably, a large number of differentiated samples can be generated by randomly adjusting the scene domain parameters of each image in the training images, thus avoiding overfitting of the training image set during model training. The scene domain parameters include adjusting brightness, contrast, the pose of the target to be detected, image noise, and texture details.

[0068] In this embodiment of the invention, by performing target annotation and background synthesis on the target rendered image, a training image set is obtained, which can generate a nearly infinite number of training samples covering various lighting, weather, seasons and geographical environments. This greatly enhances the diversity of the training image set, avoids overfitting during subsequent target detection model training, and achieves efficient and accurate generation of image training data for the target detection model.

[0069] like Figure 4 The diagram shown is a functional block diagram of a training data generation system based on Gaussian splashing provided in an embodiment of the present invention.

[0070] The training data generation system 400 based on Gaussian splashing of the present invention can be installed in an electronic device. Depending on the functions implemented, the training data generation system 400 based on Gaussian splashing may include a motion structure recovery module 401, a three-dimensional reconstruction model 402, a Gaussian rendering module 403, and a training image assembly module 404. A module can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0071] In this embodiment, the functions of each module / unit are as follows: The motion structure recovery module 401 is used to acquire two-dimensional multi-view images of the target to be detected, perform motion structure recovery on the two-dimensional multi-view images, and obtain the three-dimensional initial point cloud of the target to be detected. The 3D reconstruction model 402 is used to perform 3D reconstruction of the target to be detected based on the 3D initialized point cloud to obtain a 3D target model. The Gaussian rendering module 403 is used to perform Gaussian splash multi-view rendering on the 3D target model using a preset virtual camera to obtain the target rendering image; The training image set is grouped into module 404, which is used to perform target annotation and background synthesis on the target rendering image to obtain the training image set of the target to be detected.

[0072] In detail, each module in the Gaussian splash-based training data generation system 400 in this embodiment of the invention adopts the same approach as described above. Figures 1 to 3 The method uses the same techniques as the Gaussian splashing-based training data generation method and can produce the same technical effects, so it will not be elaborated here.

[0073] like Figure 5 The diagram shown is a schematic representation of an electronic device that implements a training data generation method based on Gaussian splashing, according to an embodiment of the present invention.

[0074] The electronic device 500 may include a processor 501, a memory 502, a communication bus 503, and a communication interface 504. It may also include a computer program stored in the memory and capable of running on the processor, such as a training data generation method program based on Gaussian splashing.

[0075] In some embodiments, the processor 501 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0076] The memory 502 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 502 may be an internal storage unit of an electronic device, such as a portable hard drive of the electronic device.

[0077] The communication bus 503 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 502 and at least one processor 501.

[0078] The communication interface 504 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface.

[0079] Figure 5 The image only shows electronic devices with components; it will be understood by those skilled in the art that... Figure 5 The structure shown does not constitute a limitation on the electronic device 500, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0080] Specifically, the processor's implementation method for the above instructions can be found in the description of the relevant steps in the corresponding embodiments in the accompanying drawings, and will not be repeated here.

[0081] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0082] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0084] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0085] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0086] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0087] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim may also be implemented by a single unit or system through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating training data based on Gaussian splashing, characterized in that, The method includes: Two-dimensional multi-view images of the target to be detected are acquired, and motion structure recovery is performed on the two-dimensional multi-view images to obtain the three-dimensional initial point cloud of the target to be detected. Based on the three-dimensional initial point cloud, the target to be detected is reconstructed in three dimensions to obtain a three-dimensional target model; The three-dimensional target model is rendered using a preset virtual camera with Gaussian splash multi-view rendering to obtain a target rendering image; The target rendered image is labeled and the background is synthesized to obtain the training image set of the target to be detected.

2. The training data generation method based on Gaussian splashing as described in claim 1, characterized in that, The process of restoring the motion structure of the two-dimensional multi-view image to obtain the three-dimensional initial point cloud of the target to be detected includes: The target image set is obtained by uniformly sampling the two-dimensional multi-view images; Extract the image features of each image in the target image set, and perform image matching based on the image features to obtain image matching pairs; Calculate the camera pose parameter sequence of the target image set based on the image matching pairs, and calculate the 3D initial point cloud based on the camera pose parameter sequence.

3. The training data generation method based on Gaussian splashing as described in claim 1, characterized in that, The step of reconstructing the target to be detected in three dimensions based on the three-dimensional initial point cloud to obtain a three-dimensional target model includes: The three-dimensional initial point cloud is converted into 3D Gaussian units; Calculate the rendering loss value between the 3D Gaussian unit and the preset real image; The parameters of the 3D Gaussian unit are optimized based on the rendering loss value to obtain the target 3D reconstruction model.

4. The training data generation method based on Gaussian splashing as described in claim 3, characterized in that, The calculation of the rendering loss value between the 3D Gaussian unit and the preset real image includes: The 3D Gaussian unit is spatially partitioned to obtain a spatial data set; The spatial data set is subjected to Gaussian projection and depth sorting to obtain projected Gaussian cells; Color transparency rendering is performed on the projected Gaussian unit to obtain a rendered image; Calculate the loss value between the rendered image and the real image to obtain the rendering loss value.

5. The training data generation method based on Gaussian splashing as described in claim 1, characterized in that, The step of using a preset virtual camera to perform Gaussian splash multi-view rendering on the 3D target model to obtain a target rendering image includes: Obtain the camera parameters of each virtual camera, and perform a two-dimensional elliptical projection on the three-dimensional target model based on the camera parameters to obtain two-dimensional Gaussian units; The three-dimensional Gaussian unit corresponding to the three-dimensional target model is divided into background Gaussian unit and segmentation Gaussian unit; Calculate the pixel coverage area of ​​the segmentation Gaussian unit in the two-dimensional Gaussian unit, and perform scene rendering on the segmentation Gaussian unit based on the pixel coverage area to obtain a segmented rendering image; Calculate the background pixel coverage area of ​​the background Gaussian unit in the two-dimensional Gaussian unit, and perform real scene rendering on the background Gaussian unit based on the segmented pixel coverage area to obtain a real scene rendering image; The target rendered image is obtained by combining the real scene rendered image and the segmented rendered image.

6. The training data generation method based on Gaussian splashing as described in claim 5, characterized in that, The step of performing scene rendering on the segmented Gaussian unit based on the pixel coverage range to obtain a segmented rendered image includes: Initialize a blank image and determine the appearance attribute encoding based on the segmented Gaussian unit; The pixel value of each pixel in the pixel coverage area is calculated based on the appearance attribute encoding; The blank image is rendered based on the pixel values ​​to obtain a segmented rendered image.

7. The training data generation method based on Gaussian splashing as described in claim 1, characterized in that, The step of performing target annotation and background synthesis on the target rendered image to obtain the training image set of the target to be detected includes: By comparing feature points of the real scene rendered image and the segmented rendered image in the target rendered image, the target region of the target to be detected is obtained; Construct a labeled rendering image of the target to be detected in the target rendering image based on the target region; The segmented and rendered image is compared with a preset real background image library to perform background replacement, resulting in a replacement background image library; Each replacement background image in the replacement background image library is labeled to obtain the labeled background image library corresponding to the replacement background image. A training image set for the target to be detected is constructed based on the annotated rendered image and the annotated background image library.

8. A training data generation system based on Gaussian splashing, characterized in that, The system includes: The motion structure recovery module is used to acquire two-dimensional multi-view images of the target to be detected, perform motion structure recovery on the two-dimensional multi-view images, and obtain the three-dimensional initial point cloud of the target to be detected. A 3D reconstruction model is used to reconstruct the target to be detected in 3D based on the 3D initialized point cloud, thereby obtaining a 3D target model. The Gaussian rendering module is used to perform Gaussian splash multi-view rendering on the three-dimensional target model using a preset virtual camera to obtain a target rendering image; The training image set is assembled into a module, which is used to perform target annotation and background synthesis on the target rendered image to obtain the training image set of the target to be detected.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The processor is used to execute computer programs stored in the memory; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training data generation method based on Gaussian splashing as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training data generation method based on Gaussian splashing as described in any one of claims 1 to 7.

Images