Virtual camera simulation data generation method and system based on gaussian point cloud model

Abstract

The application discloses a virtual camera simulation data generation method and system based on a Gaussian point cloud model, relates to the technical field of three-dimensional point cloud processing and modeling, and comprises the following steps: after completing multi-source data interaction, a point cloud dataset is established according to the data interaction result; after completing space-time registration on the dataset, a fusion space dataset is constructed, and a Gaussian point cloud model is obtained through modeling; then, virtual camera parameters are configured, a frustum is constructed to identify invalid points, and point labels are established; center perception fusion is carried out by using the point labels to obtain a perspective simulation image; and after target perception feedback correction, simulation data is established. The application solves the technical problem that the prior art cannot accurately fuse multi-source information, the virtual camera simulation data generated has a large deviation from a real scene, and the data processing result is inaccurate, and achieves the technical effect that the virtual camera simulation data accurately fits the real scene and the accuracy of the data processing result is improved.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud processing and modeling technology, and in particular to a method and system for generating virtual camera simulation data based on Gaussian point cloud models. Background Technology

[0002] In fields such as autonomous driving and robot vision, virtual camera simulation data is crucial for algorithm training and testing, and its quality directly affects the effectiveness of related technologies. Existing technologies largely rely on simple modeling or discrete point cloud stitching to generate simulation data, which has significant shortcomings: a lack of accurate modeling capabilities based on Gaussian point cloud models, difficulty in handling the heterogeneous characteristics of multi-source data, inability to achieve deep alignment of multimodal features through Gaussian clustering and covariance fitting, and low accuracy in multi-source information fusion; furthermore, discrete point clouds cannot accurately capture the details of complex scenes and the changing trends of dynamic targets. This results in large deviations between the generated simulation data and real-world scenes, inaccurate data processing results, and an inability to meet the needs of high-precision virtual testing and algorithm training, thus hindering technological upgrades in related fields. Summary of the Invention

[0003] This application provides a method and system for generating virtual camera simulation data based on a Gaussian point cloud model, which solves the technical problem that existing technologies are unable to accurately integrate multi-source information, resulting in large deviations between the generated virtual camera simulation data and the real scene, leading to inaccurate data processing results.

[0004] The first aspect of this application provides a method for generating virtual camera simulation data based on a Gaussian point cloud model. The method includes: after data interaction with multi-source data, establishing a point cloud dataset based on the data interaction results; after spatiotemporal registration of the point cloud dataset, constructing a unified temporal fusion spatial dataset; performing modal consistency modeling of multimodal features on the fusion spatial dataset to construct a Gaussian point cloud model; configuring camera parameters for a virtual camera, including pose matrix, focal length, photosensitive area, resolution, and illumination angle; constructing a view frustum for the virtual camera based on the camera parameters; identifying invalid points in the Gaussian point cloud model based on angle matching between the view frustum and the principal covariance axis, and establishing point labels, including directly used point labels and associated point labels; performing center perception fusion under the virtual camera's viewpoint using the point labels, and establishing a viewpoint simulation image based on the center perception fusion result; after performing target perception feedback on the viewpoint simulation image, correcting the viewpoint simulation image using the target perception feedback result, and establishing simulation data.

[0005] A second aspect of this application provides a virtual camera simulation data generation system based on a Gaussian point cloud model. The system includes: a point cloud dataset construction module, used to build a point cloud dataset based on the data interaction results after multi-source data interaction; a Gaussian point cloud model construction module, used to perform spatiotemporal registration on the point cloud dataset, construct a unified temporal fusion spatial dataset, perform modal consistency modeling of multimodal features on the fusion spatial dataset, and construct a Gaussian point cloud model; and a camera parameter configuration module, used to configure the camera parameters of the virtual camera, including pose matrix, focal length, photosensitive area, and resolution. The system includes: a point labeling module for constructing a virtual camera's view frustum based on the camera parameters; an invalid point identification module for the Gaussian point cloud model based on angle matching between the view frustum and the covariance principal axis direction; and a point labeling module for constructing a view simulation image using the point labels and center perception fusion under the virtual camera's viewpoint. The system also includes a simulation data construction module for correcting the view simulation image using target perception feedback and establishing simulation data.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0007] This application establishes a point cloud dataset through multi-source data interaction, constructs a fused spatial dataset through spatiotemporal registration, and performs modal consistency modeling to form a Gaussian point cloud model. It configures virtual camera parameters to construct a view frustum and identifies invalid points to establish point labels. Using the point labels, it performs center perception fusion to generate a view simulation image. Combined with target perception feedback, it corrects the image to establish simulation data. Through the above data processing, it accurately generates virtual camera simulation data, making the simulation data more consistent with the real scene and meeting the needs of high-precision applications. It achieves the technical effect of accurately matching the virtual camera simulation data with the real scene and improving the accuracy of data processing results. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the method for generating virtual camera simulation data based on a Gaussian point cloud model provided in this application embodiment.

[0010] Figure 2This is a schematic diagram of the structure of the virtual camera simulation data generation system based on the Gaussian point cloud model provided in the embodiments of this application.

[0011] Figure labeling: Point cloud dataset construction module 1, Gaussian point cloud model construction module 2, camera parameter configuration module 3, point label construction module 4, viewpoint simulation image construction module 5, simulation data construction module 6. Detailed Implementation

[0012] This application provides a method and system for generating virtual camera simulation data based on a Gaussian point cloud model, which solves the technical problem that existing technologies are unable to accurately integrate multi-source information, resulting in large deviations between the generated virtual camera simulation data and the real scene, leading to inaccurate data processing results.

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0015] Example 1, as Figure 1 As shown, a method for generating virtual camera simulation data based on a Gaussian point cloud model is described, wherein the method includes:

[0016] Step A100: After data interaction with multiple sources, establish a point cloud dataset based on the data interaction results.

[0017] Specifically, in practical applications, multi-source data typically includes 3D point coordinate data acquired by LiDAR, RGB image data acquired by vision cameras, and depth map data generated by depth sensors. Due to differences in the principles of the acquisition devices, these data suffer from heterogeneous formats and spatiotemporal reference differences. The interaction of multi-source data first requires establishing a data communication protocol to achieve real-time transmission and format conversion of data from different devices. For example, LiDAR point cloud data, stored in (x,y,z) 3D coordinates, and camera image data stored in pixel matrix format, can be converted into a unified data interface format.

[0018] Subsequently, based on the spatiotemporal association rules determined during data interaction, preliminary association was performed on the multi-source data. First, timestamp matching was used to associate laser point clouds collected at the same time with camera images. Assuming the lidar frame rate is 10Hz and the camera frame rate is 30Hz, then every 3 frames of camera images are associated with 1 frame of laser point cloud data. Then, through spatial coordinate calibration, the pixel coordinates of the camera images are transformed into the three-dimensional coordinate system of the lidar using calibration parameters to achieve spatial location correspondence.

[0019] Based on the above interaction results, the associated multi-source data is integrated, and invalid or duplicate information is removed. For example, isolated points caused by noise in the LiDAR and points whose distance exceeds a preset threshold are filtered out, and invalid pixel areas caused by motion blur in the camera image are removed. Finally, the effective data is aggregated to form a point cloud dataset containing multi-dimensional features such as three-dimensional coordinates, color information, and depth values.

[0020] By interactively processing and integrating multi-source data, the problem of data heterogeneity was solved, and a point cloud dataset containing rich features was established, providing basic data support for subsequent spatiotemporal registration and Gaussian point cloud model construction.

[0021] Step A200: After performing spatiotemporal registration on the point cloud dataset, construct a unified temporal fusion spatial dataset, perform modal consistency modeling on the fusion spatial dataset for multimodal features, and construct a Gaussian point cloud model.

[0022] In this embodiment of the application, the Gaussian point cloud model is constructed by performing modal consistency modeling of multimodal features on a unified temporal fusion spatial dataset constructed through spatiotemporal registration.

[0023] Optionally, after aligning the point cloud dataset in time, a standard modal representation is established using a feature extraction network. After adversarial projection loss and distribution matching loss analysis and structural alignment, a fused spatial dataset is constructed, and a Gaussian point cloud model is then constructed based on this dataset. The specific steps are explained in detail in A210-A230.

[0024] Step A300: Configure the camera parameters of the virtual camera, including pose matrix, focal length, photosensitive area, resolution and illumination angle.

[0025] In this embodiment, the virtual camera is a virtual imaging system constructed by configuring parameters such as pose matrix, focal length, photosensitive area, resolution, and illumination angle.

[0026] In one embodiment of this application, when configuring the camera parameters of a virtual camera, the pose matrix must first be determined. This matrix contains rotation and translation information to describe the camera's position and orientation in three-dimensional space. It can be obtained through preset coordinates or external calibration tools. For example, the rotation matrix can be set to the Euler angle transformation result around the x-axis (30°), y-axis (15°), and z-axis (0°), and the translation vector can be (5,2,1) meters, combined to form a 4×4 pose matrix, accurately locating the camera's spatial pose in the world coordinate system.

[0027] The focal length parameter is configured according to the observation range of the simulated scene. If simulating close-up shooting, it can be set to 50mm, corresponding to a horizontal field of view of approximately 46°; if wide-angle coverage is required, it can be set to 24mm, extending the horizontal field of view to 84°. At the same time, the physical focal length is converted into pixel focal length. For example, a 50mm focal length at a resolution of 1920×1080 corresponds to a pixel focal length of approximately 1000 pixels after conversion based on the sensor size, ensuring accurate projection ratio from spatial point to pixel.

[0028] The photosensitive area is determined based on the sensor model. For example, if a 1 / 2.3-inch chip is selected, its photosensitive area is 6.17×4.55mm². Combined with a resolution of 1280×720, the size of a single pixel can be calculated to be 4.8μm×4.8μm, so that the pixel distribution matches the physical photosensitive area and avoids spatial distortion during imaging.

[0029] The resolution is set according to the application requirements. Simulation images used for algorithm training can use 1280×720, while high-precision test scenarios use 3840×2160. The resolution must be adapted to the photosensitive area to ensure consistent pixel density in both the horizontal and vertical directions. For example, when a resolution of 1280×720 corresponds to a photosensitive area of ​​6.17×4.55mm², the aspect ratio and the ratio of the photosensitive area are both 4:3 to maintain a coordinated imaging ratio.

[0030] The lighting angle is configured through the light source direction vector. For example, setting the angle between the lighting and the camera optical axis to 45° and the azimuth angle to 30° simulates the side-above lighting effect. This parameter will affect the brightness rendering of each point in the Gaussian point cloud model and provide a lighting reference for subsequent image generation.

[0031] By precisely configuring parameters such as pose matrix, focal length, photosensitive area, resolution, and illumination angle, the imaging characteristics of the virtual camera were determined, laying the foundation for constructing the view frustum and generating simulated images that conform to the real scene.

[0032] Step A400: Construct the view frustum of the virtual camera based on the camera parameters, identify invalid points of the Gaussian point cloud model based on the angle matching between the view frustum and the covariance principal axis direction, and establish point labels, including direct point labels and associated point labels.

[0033] In this embodiment, the view frustum is a visible frustum space established based on the viewpoint of the virtual camera and the near and far clipping planes and horizontal and vertical viewing angles calculated according to camera parameters. This space is used to define the effective observation range of the virtual camera. The principal covariance axis is a direction vector calculated after extracting the covariance matrix for each Gaussian point, reflecting the main extension directions of the Gaussian points in space.

[0034] Specifically, a view frustum is constructed based on the virtual camera parameters. The view frustum is used to perform spatial cross recognition and occlusion removal to obtain the first recognition result. The covariance matrix of the Gaussian points is extracted to calculate the principal axis direction vector. The second recognition result is obtained by filtering based on the angle between the vector vector and the camera view direction. The two are combined to establish point labels. The specific steps are explained in detail in A410-A450.

[0035] Step A500: Use the point labels to perform center perception fusion under the virtual camera's perspective, and establish a perspective simulation image based on the center perception fusion result.

[0036] Specifically, center perception fusion under the virtual camera's perspective is performed using point labels to establish a viewpoint simulation image. This involves first identifying targets in a Gaussian point cloud model to determine the spatial distribution of targets and their feature weights. Then, the Gaussian points corresponding to the point labels are directly used as center points. The perception fusion space is configured by combining the target feature weights. Next, the space is compensated by calculating the perception complexity to obtain the real perception fusion space. Finally, center perception fusion is performed based on the real perception fusion space and the point labels to establish the center perception fusion result. The specific steps are explained in detail in pages A510-A540.

[0037] Next, a viewpoint simulation image is created based on the center perception fusion result, and the 3D fusion information is projected onto a 2D image plane by combining the parameters of the virtual camera. First, key information such as spatial coordinates, color features, and depth values ​​contained in the center perception fusion result is extracted. This information integrates the weighted features of candidate points in the real perception space and can accurately reflect the comprehensive features of the center point and the surrounding area.

[0038] Subsequently, the projection transformation relationship is determined using the pose matrix of the virtual camera, converting the three-dimensional spatial coordinates into two-dimensional image coordinates. The focal length and photosensitive area determine the scaling ratio of the image, while the resolution limits the pixel size range of the image. For example, the three-dimensional coordinates of a point in the fusion result, after being transformed by the pose matrix, correspond to the (u, v) pixel position in the image; points closer to the camera occupy a larger pixel area in the image.

[0039] Meanwhile, based on the color features in the fusion result, the corresponding pixels are assigned values. If a certain area is dominated by vehicle blue after fusion, then the RGB values ​​of the pixels in that area are based on the blue series. The depth value is used to assist in processing occlusion relationships and ensure that the imaging logic of near objects occluding distant objects conforms to visual rules.

[0040] Finally, the pixel brightness is adjusted in conjunction with the camera's illumination angle to match the image's light and dark distribution with the virtual lighting conditions. After the above steps, a two-dimensional perspective simulation image that conforms to the virtual camera's viewpoint is generated.

[0041] Step A600: After performing target perception feedback on the simulated viewpoint image, the simulated viewpoint image is corrected using the target perception feedback results to establish simulation data.

[0042] Specifically, after providing target perception feedback to the simulated viewpoint image, the feedback results are first analyzed to clarify the specific manifestations and distribution of errors. Feedback results typically indicate target positioning deviations in the image, feature consistency issues (such as discrepancies between color and texture and the real scene), and deviations in depth information. For example, feedback might show that the bounding box of a vehicle target does not completely cover the actual area, or that the color of a pedestrian's clothing differs significantly from the calibration information; this information provides clear guidance for subsequent corrections.

[0043] When using feedback results for correction, corresponding technical means must be adopted for different types of errors. For positioning deviations, coordinate transformation algorithms can be used to adjust the pixel position of the target in the image. By aligning key feature points of the target, such as the corners of vehicles and the joints of pedestrians, with the calibrated position, the offset of the bounding box can be corrected. For feature inconsistency problems, image enhancement or feature transfer techniques can be used to adjust color channel parameters or texture details, making the visual features of the simulated image closer to the real scene. For depth information errors, depth estimation models can be combined to compensate for pixel depth based on the feedback depth deviation value, ensuring that the distance relationship of the target conforms to the actual perception law.

[0044] The calibration process requires multiple rounds of verification and fine-tuning. The initially calibrated image is compared again with the calibration information. If errors remain, the calibration parameters are adjusted based on the new feedback until all features meet the preset consistency standard. This iterative optimization mechanism can gradually narrow the gap between the simulated image and the real scene, avoiding potential oversights in a single calibration.

[0045] After completing all calibration steps, the verified viewpoint simulation images, along with corresponding 3D coordinates, target labels, depth information, and other auxiliary data, are integrated to form complete simulation data. This data retains the flexibility of virtual camera simulation while improving consistency with real-world perception through calibration.

[0046] By analyzing the perception feedback to identify errors, making targeted corrections, and iteratively optimizing, the final simulation data can more accurately simulate real-world scenarios, providing a reliable data source for the training and verification of virtual camera-related algorithms.

[0047] Furthermore, step A500 in the method provided in this application embodiment includes:

[0048] A510: Perform target recognition on the Gaussian point cloud model to perceive the scene, establish the spatial target distribution, and configure the target feature weights of the spatial target distribution.

[0049] A520: The Gaussian point corresponding to the directly used point label is used as the center point, and the calibration perception fusion space of the center point is configured using the target feature weight.

[0050] A530: Perform the calculation of the perception complexity of the execution center point, use the calculation result to perform spatial compensation of the calibrated perception fusion space, and establish a real perception fusion space.

[0051] A540: Perform center perception fusion based on the real perception fusion space and the point labels to establish the center perception fusion result.

[0052] In this embodiment, Gaussian points are the basic building blocks of the Gaussian point cloud model. They are constructed from the initial Gaussian center, the covariance tensor, and the spatial confidence factor, and can characterize the distribution characteristics of feature points in three-dimensional space.

[0053] Specifically, when performing target recognition in a scene using a Gaussian point cloud model, existing deep learning models based on point cloud features, such as PointNet++, can be used to classify and locate the Gaussian points in the model. For example, by extracting the covariance and spatial location features of each Gaussian point, the model can be input to identify targets such as vehicles, pedestrians, and buildings. After recognition, the spatial extent of each target is marked with a 3D bounding box, establishing a spatial target distribution. For example, the bounding box coordinates of a vehicle target are... to This covers all relevant Gaussian points of the vehicle.

[0054] When configuring the target feature weights for spatial target distribution, the weights are set based on the semantic importance and feature recognizability of the targets. Taking a traffic scenario as an example, the weight of vehicle targets is set to 0.7, as they are the core focus in a traffic scenario; pedestrians are set to 0.6; and buildings are set to 0.3, serving as background references. The higher the weight value, the more dominant the target will be in the subsequent fusion process.

[0055] Then, the Gaussian points corresponding to the point labels are used directly as the center points. These points are valid points determined after frustum filtering and angle matching. For example, the coordinates of the Gaussian points marked as directly used points in a vehicle target are used as the center points of that vehicle target.

[0056] The calibration perception fusion space of the center point is configured using target feature weights. The center point of a target with a higher weight has a larger fusion space range. For example, the calibration perception fusion space of the vehicle target center point (weight 0.7) is set as a sphere with a radius of 2 meters, covering the vehicle and surrounding related feature points; the pedestrian target center point (weight 0.6) is set as a sphere with a radius of 1.5 meters; and the building center point (weight 0.3) is set as a sphere with a radius of 1 meter. This ensures that targets of different importance can be included in the fusion range that matches their feature size.

[0057] Next, the calculation of the perceptual complexity of the center point includes configuring its neighborhood space, obtaining four types of factors in the space: visual saliency, depth variability, semantic boundary, and structural complexity, and then calculating the perceptual complexity based on these factors. This is explained in detail in steps A531-A536.

[0058] Next, the result of the perceptual complexity calculation is a quantitative indicator that reflects the richness of the spatial features of the neighborhood of the center point. Its value range is usually between 0 and 1. The higher the value, the more drastic the changes in visual, depth, semantic, and structural features in the neighborhood, and more related points need to be included to fully capture the information. The lower the value, the more gradual the features are, and the fusion range can be appropriately reduced to reduce redundancy.

[0059] When using this result for spatial compensation, a mapping relationship between perceptual complexity and compensation coefficient must first be established. For example, when the perceptual complexity is set to high complexity (0.8), the compensation coefficient is 1.3, meaning the original calibrated perceptual fusion space is expanded by a factor of 1.3; when the perceptual complexity is set to low complexity (0.3), the compensation coefficient is 0.8, meaning the original space is shrunk by a factor of 0.8. This mapping can be achieved using a linear function, such as compensation coefficient = 0.5 + perceptual complexity, ensuring that the coefficient changes linearly with the complexity.

[0060] The calibration perception fusion space for different targets is adjusted according to the above coefficients. Taking a vehicle target as an example, its original radius of calibration perception fusion space is 2 meters. If the perception complexity is 0.7 and the corresponding compensation coefficient is 1.2, then the adjusted radius is 2 × 1.2 = 2.4 meters, expanding the space to include more vehicle edges and detailed feature points. Taking a building target as an example, with a calibration radius of 1 meter, perception complexity of 0.2, and compensation coefficient of 0.7, the adjusted radius is 0.7 meters, shrinking the space to avoid including too many irrelevant background points.

[0061] Simultaneously, upper and lower limits for spatial compensation need to be set to prevent excessive expansion or contraction. For example, it should be stipulated that the maximum compensated spatial radius should not exceed 1.2 times the diagonal length of the target 3D bounding box, and the minimum should not be less than 0.5 times the diagonal length of the bounding box. If the diagonal of a vehicle's bounding box is 3 meters, its maximum compensated radius should not exceed 3.6 meters, and the minimum should not be less than 1.5 meters, ensuring that the spatial range always matches the actual scale of the target.

[0062] By integrating and adjusting spatial parameters such as radius and center coordinates, the final real-world perception fusion space is determined. This space retains the basic range of the calibration space while dynamically adjusting according to feature complexity, enabling it to accurately adapt to the information needs of different regions.

[0063] Finally, center perception fusion is performed based on the real perception fusion space and point labels to establish the center perception fusion result. Gaussian points that fall into the real perception fusion space are selected from the Gaussian point cloud model to establish a candidate point set. The candidate point weights are configured using a multi-dimensional evaluation channel with evaluation layers such as spatial distance layer. Then, the center point perception reconstruction of the candidate point set is performed using these weights. The steps A541-A543 are explained in detail.

[0064] By identifying targets and configuring weights, the key targets in the scene and their fusion priorities were identified, and an initial fusion space was defined for each center point, providing a targeted basic framework for subsequent perceptual complexity calculation and spatial compensation.

[0065] Furthermore, step A530 in the method provided in this application embodiment includes:

[0066] A531: Configure the neighborhood space of the center point.

[0067] A532: Perform image gradient recognition within the neighborhood space, and establish a visual saliency factor based on the image gradient recognition results.

[0068] A533: Perform depth gradient identification within the neighborhood space and establish a depth variation factor.

[0069] A534: Perform semantic boundary identification within the neighborhood space and establish semantic boundary factors.

[0070] A535: Calculate local curvature, normal vector consistency, and covariance anisotropy within the neighborhood space to construct the structural complexity factor.

[0071] A536: Perform perceptual complexity calculation based on the visual saliency factor, the depth variability factor, the semantic boundary factor, and the structural complexity factor.

[0072] Optionally, when configuring the neighborhood space of the center point, the range should be set according to the scale of the target to which the center point belongs. For example, the neighborhood of the center point of a vehicle target is set as a sphere with a radius of 1.5 meters, covering about 200 Gaussian points; the neighborhood of the center point of a pedestrian target is set as a sphere with a radius of 1 meter, covering about 100 Gaussian points, ensuring that the neighborhood space can contain sufficient related features without exceeding the reasonable range of the target itself.

[0073] Image gradient recognition is performed within the neighborhood space. Color features of Gaussian points, such as RGB values, are extracted, and the grayscale change rate of neighboring points in the horizontal and vertical directions is calculated. For example, in the neighborhood of a vehicle's center point, the image gradient value at the vehicle's edge reaches 200 (the grayscale value of an image pixel ranges from 0 to 255), while the gradient value in smooth regions is only 30. After normalizing the gradient values ​​and mapping them to the 0-1 range, a visual saliency factor is obtained. The factor for edge regions is 200 / 255≈0.8, and for smooth regions it is 30 / 255≈0.12. This factor reflects the saliency of visual changes within the neighborhood.

[0074] Depth gradient recognition requires calculating the depth values ​​of Gaussian points in the neighborhood, such as the rate of change of the z-coordinate along the camera's optical axis. For example, assuming a maximum threshold of 10 (m / m) for the depth gradient, and a depth difference of 3 meters between the front of the vehicle and the background, the gradient value within a 0.5-meter distance is 6 (m / m), which, after normalization, yields a depth variation factor of 6 / 10 = 0.6. The depth gradient value of the flat area on top of the vehicle is 1, corresponding to a factor of 1 / 10 = 0.1. This factor reflects the drastic change in spatial distance within the neighborhood, where (m / m) is the unit representing the depth gradient, i.e., the amount of depth change per unit distance. The numerator is the depth change in meters, and the denominator is the distance in meters along a certain direction, such as the horizontal direction. Essentially dimensionless, it is used to intuitively represent the rate of change of depth with distance.

[0075] When performing semantic boundary recognition within the neighborhood space of the central point, the semantic labels of all Gaussian points in the space are first obtained, such as vehicle, pedestrian, and building. Then, the boundary regions between Gaussian points with different semantic labels are found by label comparison, i.e., semantic boundaries. The number of Gaussian points at these boundaries is counted, and their proportion to the total number of Gaussian points in the neighborhood is calculated, which serves as the basis for setting the semantic boundary factor. For example, if there are 100 Gaussian points in the neighborhood, of which 40 are vehicle labels, 30 are pedestrian labels, and the remaining 30 are background labels, and there are 15 points in the boundary region between vehicle and pedestrian labels, accounting for 15% of the total number of points, then the semantic boundary factor is set to 0.7, indicating the existence of a relatively obvious boundary between different targets; if all 100 Gaussian points in the neighborhood are vehicle labels, with no other semantic labels, there is no boundary between different targets, and the proportion of points at the boundary is 0, then the semantic boundary factor is set to 0.2, indicating that there is no significant semantic boundary in this neighborhood.

[0076] Local curvature calculation employs the least squares method to fit the surface within the neighborhood, obtaining the curvature value at each point, and taking the mean as the curvature index. Normal vector consistency is calculated by averaging the angles between normal vectors within the neighborhood; the smaller the angle, the higher the consistency. Covariance anisotropy is determined by the ratio of the largest to the smallest eigenvalue of the covariance matrix; the larger the ratio, the stronger the anisotropy. These three indices, each accounting for 1 / 3 of the weight, are combined to obtain the structural complexity factor.

[0077] When calculating perceptual complexity based on the visual saliency factor, depth variation factor, semantic boundary factor, and structural complexity factor, equal weights are used, i.e., each factor is summed with a weight of 0.25 to obtain the final perceptual complexity. For example, if the four factors of a certain center point are 0.8, 0.6, 0.7, and 0.85, its perceptual complexity is (0.8 + 0.6 + 0.7 + 0.85) × 0.25 = 0.7375. A higher value indicates a higher perceptual complexity in the neighborhood space of that center point.

[0078] By extracting and comprehensively calculating multi-dimensional complexity factors within the neighborhood space, the perceptual complexity around the center point is quantified, providing a quantitative basis for accurate compensation of the calibrated perceptual fusion space and improving the rationality of the subsequent real perceptual fusion space.

[0079] Furthermore, step A540 in the method provided in this application embodiment includes:

[0080] A541: Select Gaussian points from the Gaussian point cloud model that fall within the real-perception fusion space and establish a candidate point set.

[0081] A542: The candidate point weights of the candidate point set are configured using a multi-dimensional evaluation channel. The evaluation layer of the multi-dimensional evaluation channel includes a spatial distance layer, a modal similarity layer, a label consistency layer, and a view frustum orientation consistency layer.

[0082] A543: Use the candidate point weights to perform center point perception reconstruction based on the candidate point set, and establish a center perception fusion result.

[0083] In this embodiment, the real perception fusion space is established by performing a perception complexity calculation on the center point and then using the calculation result to perform spatial compensation on the calibrated perception fusion space of the center point.

[0084] Specifically, firstly, to select Gaussian points from the Gaussian point cloud model that fall within the real-perception fusion space, it is necessary to first define the boundary parameters of the real-perception fusion space. For example, the real-perception fusion space of a vehicle target is a sphere with a radius of 2.4 meters, and the center of the sphere is the coordinate of the center point. The process iterates through all Gaussian points in the Gaussian point cloud model, calculating their spatial coordinates to determine if each point satisfies the condition that its distance to the center of the sphere is ≤2.4 meters. If so, it is included in the candidate point set. Assuming there are 1200 Gaussian points around the target, and 950 points fall into the real-perception fusion space after screening, a candidate point set containing 950 points is established.

[0085] Then, when configuring candidate point weights using multi-dimensional evaluation channels, it is necessary to calculate the individual weights through the four evaluation layers and then synthesize them. The spatial distance layer uses the center point as a reference to calculate the Euclidean distance between the candidate point and the center point. The closer the distance, the higher the weight. The formula is: Spatial weight = 1 - distance / maximum distance. In the example above, the maximum distance is 2.4 meters. If the distance to a point is 0.6 meters, the spatial weight = 1 - 0.6 / 2.4 = 0.75. The modal similarity layer compares the candidate point with the center point in terms of modal features such as color and texture, and calculates the similarity using cosine similarity. A similarity of 0.9 corresponds to a weight of 0.9, and a similarity of 0.4 corresponds to a weight of 0.4. In the label consistency layer, if the candidate point label is a directly used point label, it is consistent with the center point and the weight is set to 1.0. If it is an associated point label, the weight is set to 0.6. The view frustum direction consistency layer calculates the cosine value of the angle between the principal axis direction of the candidate point and the view frustum direction. An angle of 30° corresponds to a cosine value of 0.866 and a weight of 0.866, and an angle of 60° corresponds to a cosine value of 0.5 and a weight of 0.5. The total weight of a candidate point is obtained by summing the four individual weights at a weighting ratio of 25%. For example, if the four weights of a certain point are 0.75, 0.9, 1.0, and 0.866, the total weight is approximately 0.879 (0.75 + 0.9 + 1.0 + 0.866) × 0.25.

[0086] Subsequently, during center point perception reconstruction using candidate point weights, a weighted fusion algorithm is used to integrate the spatial coordinates, color, and depth features of the candidate points. For spatial coordinates, the sum of weights multiplied by the coordinates is divided by the total weights to obtain the fused center coordinate offset. For color features, the RGB values ​​are weighted and averaged to highlight color information with high weights. For depth information, the depth value accuracy is also optimized using a weighted average. After reconstruction, the output includes a comprehensive feature set containing fused coordinates, optimized color, and depth, which is the center perception fusion result.

[0087] By screening effective candidate points, assessing the importance of multidimensional weighted quantization points, and integrating weighted reconstruction features, accurate fusion of Gaussian points within the real-perception fusion space was achieved. The established center-perception fusion result can more accurately reflect the central region features under the virtual camera's perspective, providing a reliable basis for the generation of perspective simulation images.

[0088] Furthermore, step A200 in the method provided in this application embodiment includes:

[0089] A210: After aligning the point cloud dataset in time, feature extraction is performed using a feature extraction network to establish a standard modal representation.

[0090] A220: Perform adversarial projection loss and distribution matching loss analysis on the standard modal representation, and establish a fused spatial dataset after performing structural alignment.

[0091] A230: Construct a Gaussian point cloud model based on the fused spatial dataset.

[0092] In this embodiment, the feature extraction network is a multi-branch fusion network designed for different data sources.

[0093] Specifically, point cloud datasets contain information collected by multiple devices at different times, resulting in time reference differences that require time alignment. For example, a LiDAR system collects data at a frequency of 10Hz, generating a point cloud frame every 100ms, while a visual camera collects data at 30Hz, generating an image frame every 33ms. By interpolating timestamps, the camera data is matched to the LiDAR time series, unifying all data into a 100ms interval time frame and ensuring consistency in the time dimension.

[0094] After time alignment, the data is processed using a feature extraction network. In multimodal feature extraction scenarios, the feature extraction network is typically a multi-branch fusion network designed for different data sources. During construction, corresponding sub-networks need to be set according to the modal types contained in the point cloud dataset, such as laser point clouds, RGB images, depth maps, etc., as shown in Table 1. Each sub-network is connected to a dimension transformation layer to uniformly map the feature vectors of different modalities to a 64-dimensional space, forming the overall architecture of the network.

[0095] Table 1: Subnetworks Corresponding to Modal Types in Point Cloud Datasets

[0096]

[0097] During feature extraction, the time-aligned point cloud data is first split by modality: laser point cloud data, such as 10,000 (x,y,z) points per frame, is input into the PointNet sub-network. After layer-by-layer convolution and non-linear activation, 512-dimensional intermediate features describing the spatial distribution of the point cloud are extracted, and then compressed to 64 dimensions through a fully connected layer; RGB image data, such as a 3-channel image of 256×256 pixels, is input into the CNN sub-network. After 16 layers of convolution, 2048-dimensional texture features are extracted, and then reduced to 64 dimensions through a dimension transformation layer; depth map data, such as a 256×256 single-channel image, is input into the depth sub-network. After extracting 512-dimensional distance features, it is converted to 64 dimensions. Finally, the 64-dimensional feature vectors of each modality together constitute the standard modality representation, achieving the unification of heterogeneous features in terms of dimension and semantics. This process uses a multi-branch network to selectively extract the core features of each modality and unify the dimensions, providing a structurally consistent feature foundation for subsequent modality consistency modeling.

[0098] For the aforementioned standard modal representations, adversarial projection loss and distribution matching loss analysis are performed: adversarial projection loss analysis relies on an adversarial training framework. First, the standard modal representations of different modalities, such as the 64-dimensional feature vectors of laser point clouds and camera images, are mapped to the same feature space through their respective projection matrices, obtaining modal projection features. Then, a discriminator network is introduced, whose input is the projection feature and output is the probability of the feature belonging to that modality, such as the probability of belonging to the laser point cloud modality or the probability of belonging to the camera image modality.

[0099] In the initial stage, the projection features of different modalities differ significantly, and the discriminator can easily distinguish them. For example, the discrimination accuracy for laser point cloud projection features and camera image projection features can reach over 85%. Then, adversarial training is initiated: the discriminator optimizes its parameters through cross-entropy loss to improve its modal discrimination ability for projection features; while the projection matrix parameters of the feature extraction network are updated through backpropagation to minimize the discriminator's discrimination accuracy. For example, when the discriminator misclassifies a laser point cloud projection feature as a camera image modality, the projection matrix is ​​adjusted according to the error, making subsequent projections more difficult to distinguish. After multiple rounds of iterative training, the discriminator's discrimination accuracy for each modal projection feature gradually decreases, eventually stabilizing at around 50%. At this point, the projection features of different modalities are difficult for the discriminator to distinguish in this space, achieving adversarial consistency in modal projection.

[0100] Distribution matching loss analysis focuses on aligning the overall probability distributions of different modal features. First, for each modality's standard modal representation, its probability distribution is fitted using kernel density estimation or a Gaussian mixture model. For example, laser point cloud modal features follow a multidimensional Gaussian distribution with mean μ1 and covariance Σ1, while camera image modal features follow a multidimensional Gaussian distribution with mean μ2 and covariance Σ2. Then, KL divergence is used to calculate the difference between the two distributions. A larger KL divergence value indicates a more significant distribution difference; initially, due to differences in modal characteristics, the KL divergence may reach 3.8.

[0101] Furthermore, to reduce distribution discrepancies, KL divergence can be incorporated as a loss term into the optimization objective. During model training, gradient descent is used to adjust the parameters of the feature extraction network, gradually bringing the probability distribution parameters (mean, covariance) of different modal features closer together. For example, μ1 of the laser point cloud modality converges to μ2 of the camera image modality, reducing the difference between Σ1 and Σ2, and consequently lowering the KL divergence. After multiple rounds of iterative optimization, the KL divergence can be reduced to below 0.6, at which point the probability distributions of the two modal features highly overlap, achieving statistical distribution consistency. Specifically, the adversarial projection loss eliminates the distinguishability of modal projections through adversarial training, while the distribution matching loss achieves overall distribution alignment through KL divergence optimization. The two work synergistically to promote dual consistency in structure and distribution of the multimodal standard modal representation in the feature space.

[0102] Next, based on the loss analysis results, structural alignment is performed to adjust the spatial correlation of features, ensuring that multimodal features at the same physical location remain corresponding in the fusion space, and finally forming a unified temporal fusion space dataset, which contains aligned multimodal features and corresponding spatiotemporal information.

[0103] Finally, based on local clustering, the fused spatial dataset is clustered to establish initial Gaussian centers and cluster mappings. The covariance matrix is ​​fitted to the feature points within the cluster to establish a covariance tensor. The spatial confidence factor is calculated based on the principal axis distribution of the covariance matrix, the point density gradient, and the feature aggregation degree. The Gaussian point cloud model is then constructed by combining the three. The specific steps are explained in detail in A231-A234.

[0104] By performing temporal alignment, feature extraction, loss analysis, and structural alignment on multi-source point cloud data, a unified fusion spatial dataset was constructed, providing a consistent foundation of data for the subsequent construction of Gaussian point cloud models.

[0105] Furthermore, step A230 in the method provided in this application embodiment includes:

[0106] A231: Perform data clustering of the fused spatial dataset based on local clustering, establish initial Gaussian centers, and configure the cluster mapping of the initial Gaussian centers.

[0107] A232: Fit the covariance matrix to the spatial feature variation trend of feature points within a cluster and establish the covariance tensor.

[0108] A233: Calculate the spatial confidence factor based on the principal axis distribution of the covariance matrix, the point density gradient, and the feature aggregation degree.

[0109] A234: Construct a Gaussian point cloud model based on the spatial confidence factor, the covariance tensor, and the initial Gaussian center.

[0110] Specifically, firstly, the fused spatial dataset contains aligned multimodal feature points, which exhibit different distribution densities and feature associations in space. When grouping them based on local clustering, a density clustering algorithm can be used: When performing density clustering on the feature points in the fused spatial dataset, a spatial neighborhood with a radius of 0.3 meters is first defined centered on each point, and the number of feature points contained in each neighborhood, including the point itself, is counted. If a point has at least 5 points in its neighborhood, it is marked as a core point; if it has fewer than 5 points in its neighborhood, it is marked as a non-core point.

[0111] Subsequently, starting from the first unclassified core point, all points within its neighborhood are included in the same temporary cluster. This process is then repeated for other core points within the temporary cluster, adding their neighborhoods to the same temporary cluster. This process continues until no new points can be added, forming a cluster. Afterward, a new core point is selected from the unprocessed core points, and a new cluster is generated in the same way, until all core points have been processed. For non-core points not included in any cluster, if a core point belonging to a certain cluster exists within its neighborhood, it is assigned to that cluster; otherwise, it is considered noise and discarded. Finally, all feature points are aggregated into multiple clusters through this process. The mean of the three-dimensional coordinates of all points within each cluster is the initial Gaussian center. The cluster number corresponding to each feature point is recorded, forming a cluster mapping.

[0112] Then, for each cluster, a covariance matrix needs to be fitted to describe the spatial distribution characteristics of the feature points within the cluster. Taking a cluster containing 30 feature points as an example, the three-dimensional coordinates (x, y, z) of each point are extracted, and the mean of each point in the x, y, and z dimensions is calculated. Then, the variance and covariance are calculated based on the deviation of each point from the mean, forming a 3×3 covariance matrix. This matrix reflects the degree of dispersion of points within the cluster in different directions. The covariance matrices of all clusters are arranged in order to construct a covariance tensor. For example, 10 clusters correspond to 10 3×3 matrices, forming a covariance tensor with a dimension of 10×3×3.

[0113] Next, the calculation of the spatial confidence factor requires a comprehensive consideration of multiple dimensions. The principal axis distribution of the covariance matrix is ​​obtained through eigenvalue decomposition. The more concentrated the principal axis direction, i.e., the larger the ratio of the largest to the smallest eigenvalue, the stronger the directionality of the distribution of points within the cluster, corresponding to a weight of 0.3. The point density gradient is calculated by the rate of change of the number of points per unit volume. Regions with high density and small gradient (e.g., gradient value < 0.1 points / cubic meter) have a weight of 0.3. The feature aggregation degree is calculated by the mean cosine similarity of multimodal features within the cluster. Clusters with a mean similarity > 0.85 have a weight of 0.4. The spatial confidence factor in the range of 0-1 is obtained by weighting the three indicators. For example, the calculation result for a certain cluster is 0.72, indicating that the spatial distribution characteristics of the cluster have high reliability.

[0114] Finally, the three-dimensional coordinates of the initial Gaussian center are used as the basic position parameter, the covariance tensor as the parameter describing the spatial distribution range, and the spatial confidence factor as the parameter characterizing the reliability of the distribution. These three are then fused to construct a Gaussian point cloud model. For example, each Gaussian point in the model contains coordinates... The corresponding 3×3 covariance matrix and the corresponding confidence factor together accurately characterize the spatial features and distribution patterns of the 3D scene.

[0115] By employing local clustering, covariance fitting, confidence calculation, and multi-parameter fusion, a Gaussian point cloud model that accurately characterizes the spatial distribution and feature reliability of a scene is constructed, providing a precise 3D data foundation for virtual camera simulation.

[0116] Furthermore, step A400 in the method provided in this application embodiment includes:

[0117] A410: Centered on the viewpoint of the virtual camera, calculate the near and far clipping planes and horizontal and vertical viewing angle ranges of the view frustum according to the camera parameters, establish the view frustum space, and use the view frustum space as the view frustum.

[0118] A420: Using the aforementioned view frustum, spatial intersection recognition of the Gaussian point cloud model is performed. After occlusion removal is applied to the spatial intersection recognition results, a first recognition result is established.

[0119] A430: Extract the covariance matrix for each Gaussian point and calculate the principal axis direction vector.

[0120] A440: Perform angle threshold filtering based on the angle between the main axis direction vector and the virtual camera's viewpoint direction to establish a second recognition result.

[0121] A450: Establish point labels based on the first recognition result and the second recognition result.

[0122] In one embodiment, firstly, using the viewpoint of the virtual camera as the spatial origin, the key parameters of the viewing cone are calculated. First, the horizontal and vertical viewing angles are calculated: For example, given a horizontal dimension of 6.17 mm, a vertical dimension of 4.55 mm, and a focal length of 50 mm, the horizontal viewing angle is determined by both the horizontal dimension and the focal length according to the viewing angle calculation formula. The formula is: Horizontal viewing angle = 2 × arctan(horizontal dimension / (2 × focal length)). Substituting the values, we get 2 × arctan(6.17 / (2 × 50)), which approximates the horizontal viewing angle to be approximately 46°. Similarly, the vertical viewing angle is calculated using the formula: Vertical viewing angle = 2 × arctan(vertical dimension / (2 × focal length)). Substituting the values ​​of 4.55 mm and 50 mm, we get a vertical viewing angle of approximately 26°. These two viewing angles determine the opening angle of the viewing cone in the horizontal and vertical directions.

[0123] Next, the near and far clipping plane parameters were set. Based on the scene requirements, the near clipping plane distance was set to 0.5m, meaning that the area within 0.5m in front of the virtual camera was not included in the observation range; the far clipping plane distance was set to 50m, meaning that the area beyond 50m was also not included in the observation range. Finally, using the virtual camera's viewpoint as the vertex, the side boundary of the cone was determined by combining the horizontal and vertical viewing angles. Then, the cone was truncated using the near and far clipping planes to form a frustum-shaped visible cone space, i.e., the view cone, thus defining the spatial range that the virtual camera can observe.

[0124] Then, the spatial positions of all Gaussian points in the Gaussian point cloud model are compared with the view frustum to determine whether a point falls within the view frustum, thus initially filtering out points that may be observed by the camera. Subsequently, the Z-buffering algorithm is used to perform occlusion removal, calculating the depth value of the point along the camera's line of sight, retaining the point with the smallest depth value, i.e., the point closest to the camera, and removing subsequent points occluded by it, forming the first recognition result.

[0125] Then, for each Gaussian point in the first identification result, its covariance matrix is ​​extracted, such as a 3×3 matrix. The principal axis direction vector of the matrix is ​​calculated by eigenvalue decomposition. This vector reflects the main extension direction of the Gaussian point in space. For example, the principal axis direction vector of a certain Gaussian point is (0.7, 0.2, 0.68).

[0126] Next, the angle threshold is set to 35°. The angle between the principal axis direction vector of each Gaussian point and the virtual camera's viewpoint direction, such as the camera's optical axis direction vector (0,0,1), is calculated. Points with an angle less than 35° are retained, and points with angles exceeding the threshold are removed to form the second recognition result.

[0127] Finally, points that exist in both the first and second identification results are marked as directly usable point labels, and points that exist in only one result but have a spatial relationship with directly usable points, such as a distance of less than 0.2m, are marked as associated point labels, thus establishing a point labeling system that includes two types of labels.

[0128] By constructing a view frustum for spatial filtering and occlusion removal, and combining it with angle filtering along the principal axis of covariance, effective points are accurately identified and point labels are established, providing a reliable basis for point selection for center perception fusion under the virtual camera's perspective.

[0129] Furthermore, step A600 in the method provided in this application embodiment includes:

[0130] A610: Establish target calibration and labeling data.

[0131] A620: Based on the calibration and annotation data, perform error identification on the simulated viewpoint image, and establish target perception feedback results based on the error identification results.

[0132] Optionally, firstly, establish target calibration data, constructing a benchmark based on high-precision reference information. For example, obtain the target's 3D coordinates by scanning the real scene with LiDAR, and manually annotate the images captured by a high-resolution camera to determine the target's 2D bounding box, such as the top-left and bottom-right pixel coordinates of the vehicle target in the image, its 3D dimensions (length, width, height), category label, and key feature points, such as the wheel center coordinates. The positioning error of this data needs to be controlled within 1 pixel to ensure the accuracy of the calibration.

[0133] When identifying errors in simulated images based on calibration data, comparisons are performed from multiple dimensions. For spatial positioning, the Intersection over Union (IoU) ratio between the target bounding box in the simulated image and the calibration bounding box is calculated. If the IoU of a vehicle target's simulated bounding box to its calibration bounding box is 0.65, which is below the threshold of 0.8, a positioning deviation is identified. For feature consistency, the color mean values ​​of the simulated image and calibration data are compared. If the difference between the simulated vehicle's RGB mean and the calibration value exceeds 30 (within the range of 0-255), a color error is identified. For depth information, the absolute error between the simulated target's depth value and the calibration depth is calculated. If the simulated depth of a point is 5.2 meters and the calibration depth is 4.5 meters, with an error of 0.7 meters exceeding the threshold of 0.5 meters, a depth error is identified.

[0134] Finally, based on the error identification results, target perception feedback results are established, summarizing the quantitative indicators and location information of various errors. For example, a feedback report is generated: the vehicle target (coordinate area (55,105)-(205,305)) has a positioning error (IoU=0.65), a color error (RGB difference 35), and a depth error (mean 0.6 meters), clearly defining the error type, degree, and location, providing a specific basis for subsequent viewpoint simulation image correction.

[0135] By establishing a high-precision calibration benchmark and identifying the errors between the simulation image and the benchmark in multiple dimensions, the resulting target perception feedback can accurately guide the correction direction of the simulated image, thereby improving the consistency between the simulation data and the real scene.

[0136] Furthermore, step A600 in the method provided in this application embodiment includes:

[0137] A630: Set the data iteration update cycle.

[0138] A640: When the time node meets the iterative update cycle, execute the data update instruction.

[0139] A650: Perform incremental data updates of the simulation data according to the data update instruction.

[0140] In one embodiment, when setting the data iteration update cycle, the cycle length needs to be determined based on the application scenario and data change frequency of the simulation data. For example, for simulation data of dynamic traffic scenarios, since targets such as vehicles and pedestrians move frequently, the iteration update cycle can be set to 1 hour to quickly capture scene changes; for static building scenarios, where data changes slowly, the cycle can be set to 24 hours to reduce unnecessary update consumption. Simultaneously, the cycle setting needs to be stored in the system configuration file, allowing for later adjustment according to actual needs, such as adjusting the traffic scenario cycle from 1 hour to 30 minutes through the parameter modification interface.

[0141] When the time node meets the iteration update cycle, the system will trigger a data update command through a built-in timer or timestamp comparison mechanism. For example, if the cycle is 1 hour, when the system time reaches the hourly times such as 8:00 or 9:00 every day, the timer will generate an update command. The command contains the time range and target type of the data to be updated, such as key information like newly added vehicles or disappeared pedestrians, ensuring that the update range is accurate and controllable.

[0142] When incrementally updating simulation data according to data update instructions, it is necessary to first filter out the data that has changed within the period, rather than replacing the entire dataset. For example, by comparing the Gaussian point cloud coordinates and target labels of the old and new data, 200 newly added Gaussian points, 50 vehicle targets with changed positions, and 30 missing pedestrian targets can be identified, and only these changed data are processed. During the update process, the integrity of unchanged data must be preserved. For example, the Gaussian point cloud of building targets remains unchanged, and only the feature data of newly added vehicles is integrated into the existing simulation data. This reduces the amount of data after the update by more than 80% compared to a full update, improving efficiency while ensuring data continuity.

[0143] By setting a reasonable iteration cycle, accurately triggering update commands, and executing incremental updates, dynamic maintenance of simulation data is achieved. This significantly reduces update costs while ensuring data timeliness, enabling simulation data to continuously adapt to changes in the scenario.

[0144] In summary, the virtual camera simulation data generation method based on Gaussian point cloud model provided in this application has the following technical effects:

[0145] This application establishes a point cloud dataset through interactive processing of multi-source data. After spatiotemporal registration and multimodal feature modality consistency modeling, a Gaussian point cloud model is constructed. Virtual camera parameters are configured and a view frustum is constructed to identify invalid points and establish point labels. The point labels are used to perform center perception fusion to establish a viewpoint simulation image. The image is corrected by combining the target perception feedback results, thereby generating virtual camera simulation data. This makes the generation of virtual camera simulation data more accurate and reliable, achieving the technical effect of accurately matching the virtual camera simulation data with the real scene and improving the accuracy of data processing results.

[0146] Example 2, as Figure 2 As shown, based on the same inventive concept as in Embodiment 1 above, this application provides a virtual camera simulation data generation system based on a Gaussian point cloud model, the system comprising:

[0147] Point cloud dataset construction module 1 is used to build a point cloud dataset based on the data interaction results after data interaction with multiple sources.

[0148] Gaussian point cloud model construction module 2 is used to construct a unified temporal fusion spatial dataset after spatiotemporal registration of the point cloud dataset, perform modal consistency modeling of multimodal features on the fusion spatial dataset, and construct a Gaussian point cloud model.

[0149] The camera parameter configuration module 3 is used to configure the camera parameters of the virtual camera, including the pose matrix, focal length, photosensitive area, resolution, and illumination angle.

[0150] Point label construction module 4 is used to construct the view frustum of the virtual camera according to the camera parameters, identify invalid points of the Gaussian point cloud model based on the angle matching between the view frustum and the covariance principal axis direction, and establish point labels. The point labels include directly used point labels and associated point labels.

[0151] The viewpoint simulation image construction module 5 uses the point labels to perform center perception fusion under the virtual camera viewpoint, and builds a viewpoint simulation image based on the center perception fusion result.

[0152] The simulation data construction module 6 is used to correct the view simulation image by applying target perception feedback to the view simulation image and establish simulation data.

[0153] Furthermore, the viewpoint simulation image construction module 5 is used to perform the following steps:

[0154] The Gaussian point cloud model is used to identify targets in a perceived scene, establish a spatial target distribution, and configure target feature weights for the spatial target distribution. The Gaussian points corresponding to the directly used point labels are used as center points, and the calibration perception fusion space of the center points is configured using the target feature weights. The perception complexity of the center points is calculated, and the spatial compensation of the calibration perception fusion space is performed using the perception complexity calculation results to establish a real perception fusion space. Center perception fusion is performed based on the real perception fusion space and the point labels to establish a center perception fusion result.

[0155] Furthermore, the viewpoint simulation image construction module 5 is used to perform the following steps:

[0156] Configure the neighborhood space of the center point; perform image gradient recognition within the neighborhood space, and establish a visual saliency factor based on the image gradient recognition results; perform depth gradient recognition within the neighborhood space, and establish a depth variation factor; perform semantic boundary recognition within the neighborhood space, and establish a semantic boundary factor; calculate local curvature, normal vector consistency, and covariance anisotropy within the neighborhood space, and construct a structural complexity factor; perform perceptual complexity calculation based on the visual saliency factor, the depth variation factor, the semantic boundary factor, and the structural complexity factor.

[0157] Furthermore, the viewpoint simulation image construction module 5 is used to perform the following steps:

[0158] From the Gaussian point cloud model, Gaussian points that fall within the real perception fusion space are selected to establish a candidate point set; the candidate point weights of the candidate point set are configured using a multi-dimensional evaluation channel, wherein the evaluation layer of the multi-dimensional evaluation channel includes a spatial distance layer, a modal similarity layer, a label consistency layer, and a view frustum orientation consistency layer; the center point perception reconstruction based on the candidate point set is performed using the candidate point weights to establish a center perception fusion result.

[0159] Furthermore, the Gaussian point cloud model construction module 2 is used to perform the following steps:

[0160] After temporally aligning the point cloud dataset, feature extraction is performed using a feature extraction network to establish a standard modal representation. Adversarial projection loss and distribution matching loss analysis are performed on the standard modal representation, and after structural alignment, a fused spatial dataset is established. A Gaussian point cloud model is constructed based on the fused spatial dataset.

[0161] Furthermore, the Gaussian point cloud model construction module 2 is used to perform the following steps:

[0162] Data clustering of the fused spatial dataset is performed based on local clustering, initial Gaussian centers are established, and cluster mapping of the initial Gaussian centers is configured; the spatial feature variation trend of feature points within the clusters is fitted with a covariance matrix to establish a covariance tensor; spatial confidence factor is calculated based on the principal axis distribution of the covariance matrix, point density gradient, and feature aggregation degree; a Gaussian point cloud model is constructed based on the spatial confidence factor, the covariance tensor, and the initial Gaussian centers.

[0163] Furthermore, the dot label construction module 4 is used to perform the following steps:

[0164] Centered on the viewpoint of the virtual camera, the near and far clipping planes and horizontal and vertical viewing angle ranges of the view frustum are calculated based on the camera parameters to establish a visible frustum space, which is then used as the view frustum. Spatial cross-identification of the Gaussian point cloud model is performed using the view frustum. After occlusion culling of the spatial cross-identification results, a first identification result is established. The covariance matrix is ​​extracted for each Gaussian point, and the principal axis direction vector is calculated. An angle threshold filtering is performed based on the angle between the principal axis direction vector and the virtual camera's viewing angle direction to establish a second identification result. Point labels are established based on the first and second identification results.

[0165] Furthermore, the simulation data construction module 6 is used to perform the following steps:

[0166] Establish target calibration data; perform error identification on the simulated image of the viewpoint based on the calibration data; and establish target perception feedback results based on the error identification results.

[0167] Furthermore, the simulation data construction module 6 is used to perform the following steps:

[0168] Set the data iteration update cycle; when the time node meets the iteration update cycle, execute the data update instruction; perform incremental data updates of the simulation data according to the data update instruction.

[0169] The virtual camera simulation data generation system based on Gaussian point cloud model provided in this embodiment of the invention can execute the virtual camera simulation data generation method based on Gaussian point cloud model provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0170] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0171] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for generating virtual camera simulation data based on a Gaussian point cloud model, characterized in that, The method comprises: After data interaction of multi-source data is performed, a point cloud dataset is established according to a data interaction result; After the point cloud dataset is spatio-temporally registered, a unified time-series fusion spatial dataset is constructed, a modal consistency modeling of multi-modal features is performed on the fusion spatial dataset, and a Gaussian point cloud model is constructed; Camera parameters of a virtual camera are configured, the camera parameters comprising a pose matrix, a focal length, a photosensitive area, a resolution, and an illumination angle; A view cone of the virtual camera is constructed according to the camera parameters, invalid point identification of the Gaussian point cloud model is performed based on angle matching of the view cone and a covariance principal axis direction, a point label is established, and the point label comprises a direct use point label and an associated point label; Central perception fusion under a virtual camera view is performed using the point label, a view simulation image is established according to a central perception fusion result; After target perception feedback is performed on the view simulation image, the view simulation image is corrected using a target perception feedback result, and simulation data is established.

2. The method of claim 1, wherein the Gaussian point cloud model-based virtual camera simulation data generation method is characterized by, The central perception fusion under the virtual camera view using the point label, and the establishment of the view simulation image according to the central perception fusion result, comprise: Target identification of a perception scene is performed on the Gaussian point cloud model, a spatial target distribution is established, and target feature weights of the spatial target distribution are configured; A Gaussian point corresponding to the direct use point label is taken as a center point, a calibration perception fusion space of the center point is configured using the target feature weights; Perception complexity calculation of the center point is performed, spatial compensation of the calibration perception fusion space is performed using a perception complexity calculation result, and a real perception fusion space is established; Central perception fusion is performed according to the real perception fusion space and the point label, and a central perception fusion result is established.

3. The method of claim 2, wherein the Gaussian point cloud model-based virtual camera simulation data generation method is characterized by, The perception complexity calculation of the center point comprises: A neighborhood space of the center point is configured; Image gradient identification is performed in the neighborhood space, and a visual saliency factor is established according to an image gradient identification result; Depth gradient identification is performed in the neighborhood space, and a depth change degree factor is established; Semantic boundary identification is performed in the neighborhood space, and a semantic boundary factor is established; Local curvature, normal vector consistency, and covariance anisotropy calculation are performed in the neighborhood space, and a structure complexity factor is constructed; Perception complexity calculation is performed according to the visual saliency factor, the depth change degree factor, the semantic boundary factor, and the structure complexity factor.

4. The method of claim 2, wherein the Gaussian point cloud model-based virtual camera simulation data generation method is characterized by, The central perception fusion according to the real perception fusion space and the point label, and the establishment of the central perception fusion result, comprise: Gaussian points falling into the real perception fusion space are screened from the Gaussian point cloud model, and a candidate point set is established; Candidate point weights of the candidate point set are configured using a multi-dimensional evaluation channel, an evaluation layer of the multi-dimensional evaluation channel comprising a spatial distance layer, a modal similarity layer, a label consistency layer, and a view cone direction consistency layer; Central point perception reconstruction based on the candidate point set is performed using the candidate point weights, and a central perception fusion result is established.

5. The method of claim 1, wherein, After the spatiotemporal registration of the point cloud dataset, a unified time sequence fusion spatial dataset is constructed, modal consistency modeling of multi-modal features is performed on the fusion spatial dataset, and a Gaussian point cloud model is constructed, including: After time alignment of the point cloud dataset, feature extraction is performed using a feature extraction network to establish a standard modal representation; After analyzing the standard modal representation for adversarial projection loss and distribution matching loss, a fusion spatial dataset is established after structure alignment; The Gaussian point cloud model is constructed according to the fusion spatial dataset.

6. The method of claim 5, wherein the Gaussian point cloud model-based virtual camera simulation data generation method is characterized by, The Gaussian point cloud model is constructed according to the fusion spatial dataset, including: Data clustering of the fusion spatial dataset is performed based on local clustering to establish an initial Gaussian center and configure a cluster mapping of the initial Gaussian center; The spatial feature variation trend of the feature points in the cluster is fitted by a covariance matrix to establish a covariance tensor; The spatial confidence factor is calculated according to the principal axis distribution of the covariance matrix, the point density gradient, and the feature aggregation degree; The Gaussian point cloud model is constructed according to the spatial confidence factor, the covariance tensor, and the initial Gaussian center.

7. The method of claim 1, wherein the method further comprises: The view frustum of the virtual camera is constructed according to the camera parameters, the angle matching between the view frustum and the covariance principal axis direction is performed for invalid point recognition of the Gaussian point cloud model, and a point label is established, including: The near-far clipping plane, horizontal and vertical angle range of the view frustum are calculated according to the camera parameters with the viewpoint of the virtual camera as the center to establish a view frustum space, and the view frustum space is taken as the view frustum; The spatial intersection recognition of the Gaussian point cloud model is performed using the view frustum, and a first recognition result is established after performing occlusion removal on the spatial intersection recognition result; The covariance matrix of each Gaussian point is extracted, and the principal axis direction vector is calculated; The angle threshold filtering is performed according to the angle between the principal axis direction vector and the viewing angle direction of the virtual camera to establish a second recognition result; The point label is established according to the first recognition result and the second recognition result.

8. The method of claim 1, wherein, The target perception feedback of the view simulation image includes: Calibration and labeling data of the target are established; Error recognition of the view simulation image is performed according to the calibration and labeling data, and a target perception feedback result is established according to the error recognition result.

9. The method of claim 1, wherein, After the simulation data is established, including: An iterative update period of the data is set; When the time node meets the iterative update period, a data update instruction is executed; The data increment update of the simulation data is performed according to the data update instruction.

10. A virtual camera simulation data generation system based on a Gaussian point cloud model, characterized by, The system for implementing the Gaussian point cloud model-based virtual camera simulation data generation method of any one of claims 1-9, the system comprising: A point cloud dataset construction module for establishing a point cloud dataset according to the data interaction result after data interaction of multi-source data; A Gaussian point cloud model construction module for constructing a unified time sequence fusion spatial dataset after spatiotemporal registration of the point cloud dataset, performing modal consistency modeling of multi-modal features on the fusion spatial dataset, and constructing a Gaussian point cloud model; A camera parameter configuration module for configuring camera parameters of a virtual camera, the camera parameters including a pose matrix, a focal length, a photosensitive area, a resolution, and an illumination angle; A point label construction module is configured to construct a view cone of a virtual camera according to the camera parameters, identify invalid points of a Gaussian point cloud model based on angle matching between the view cone and a covariance principal axis direction, and establish point labels including direct use point labels and associated point labels; A view angle simulation image construction module is configured to perform central perception fusion under a view angle of a virtual camera using the point labels, and establish a view angle simulation image according to a central perception fusion result; A simulation data construction module is configured to perform target perception feedback on the view angle simulation image, correct the view angle simulation image using a target perception feedback result, and establish simulation data.

Images