A simulation dataset generation method of a micro-renderable and clustering method

Abstract

The application discloses a simulation data set generation method of a differentiable rendering and clustering method, and comprises the following steps: generating an original image data set according to a multi-sensor simulation environment; based on the original image data set, calculating a patch soft coverage cumulative value of each triangular patch by using a differentiable rendering framework, and generating a synthetic rendering image data set; performing difference analysis on the synthetic rendering image data set and the original image data set, calculating a visibility contribution value of a corresponding triangular patch according to a difference analysis result and the patch soft coverage cumulative value, and constructing a position identifier of all uncovered triangular patches according to the visibility contribution value; extracting a corresponding unit normal vector according to the position identifier of the uncovered triangular patch, and dividing a normal space into multiple normal clustering clusters; generating a pose parameter set of a supplementary sensor according to all unit normal vectors of each normal clustering cluster; generating a new image data set based on the pose parameter set of the supplementary sensor, and fusing the new image data set with the original image data set as a simulation data set.

Description

Technical Field

[0001] This invention belongs to the field of differentiable rendering and sensor optimization technology, specifically relating to a method for generating simulation datasets using differentiable rendering and clustering methods. Background Technology

[0002] With the rapid development of 3D perception technology, the importance of digital models in fields such as industrial quality inspection and autonomous driving is becoming increasingly prominent. As a core component of the 3D digitization process, the completeness assessment of model datasets faces severe challenges. Traditional methods rely on exhaustive observation using dense sensor arrays, which is seriously inadequate in assessing the completeness of 3D model datasets. This reliance on exhaustive observation with dense sensor arrays results in a large number of invalid observation poses due to viewpoint redundancy, leading to low data acquisition efficiency and soaring storage costs, becoming a key bottleneck for the implementation of 3D digitization. Taking automotive component inspection as an example, completely covering 30,000 triangular facets of a car requires deploying over 300 sensor poses, generating over 100GB of raw data, of which over 60% is redundant observation. This invention addresses this pain point by proposing an intelligent optimization framework based on differentiable rendering and geometric clustering. By accurately locating and covering blind spots, it only needs to supplement less than 5% of the key poses to achieve a facet coverage rate of over 95%, reducing data acquisition costs by 80% while maintaining detection accuracy.

[0003] Traditional methods for evaluating the completeness of 3D model datasets typically consider visibility detection and dataset size. However, this approach has several limitations: First, it can only determine whether patches are occluded, failing to quantify observation quality, making it difficult to assess core quality indicators such as geometric accuracy, texture fidelity, and semantic consistency. Second, relying solely on dataset size can lead to data redundancy, increased storage costs, and an inability to ensure coverage quality in critical areas. Existing improvements, such as ray casting statistics, can locate uncovered areas but cannot generate optimal patching strategies, resulting in high data redundancy and significant variations in patch detection distribution.

[0004] To overcome the limitations of traditional 3D evaluation methods, the academic community has introduced differentiable rendering technology to quantify observation quality. However, mainstream solutions such as SoftRAS still have significant drawbacks: while its soft rasterization visibility response can mark uncovered areas, it cannot derive the optimal patching strategy; and statistical methods based on ray density ignore the correlation between patch orientations, resulting in low patching efficiency. Taking the inspection of high-end automotive parts as an example, existing solutions require the addition of 50+ poses to achieve 95% coverage, of which 35% of poses suffer from insufficient observation efficiency due to discrete normal distribution. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a simulation dataset generation method based on differentiable rendering and clustering. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a method for generating simulation datasets using differentiable rendering and clustering methods. The simulation dataset generation method includes: Construct a multi-sensor simulation environment and generate an original image dataset based on the multi-sensor simulation environment; Based on the original image dataset, the cumulative soft overlay value of each triangular facet in the 3D model at a specified viewpoint is calculated using a differentiable rendering framework, and a synthetic rendering image dataset at the specified viewpoint is generated. Perform a difference analysis on the synthetic rendering image dataset and the original image dataset. Based on the difference analysis results and the cumulative soft coverage value of each triangular facet in the specified viewpoint, calculate the visibility contribution value of the corresponding triangular facet in the 3D model in the specified viewpoint. Based on the visibility contribution value, construct a location identifier describing all uncovered triangular facets in the 3D model. Extract the unit normal vector of the uncovered triangle based on the position identifier of the uncovered triangle, divide the normal space formed by the unit normal vectors of all uncovered triangles into multiple normal clusters, and generate a set of pose parameters for the supplementary sensor based on all the unit normal vectors corresponding to each normal cluster. A new image dataset is generated based on the supplementary sensor pose parameter set. The new image dataset is then merged with the original image dataset to form an enhanced image dataset, which serves as the final simulation dataset.

[0006] In one embodiment of the present invention, a multi-sensor simulation environment is constructed, and an original image dataset is generated based on the multi-sensor simulation environment, including: Configure a semantic segmentation sensor and its corresponding ordinary optical sensor, use the semantic segmentation sensor to collect a semantic segmentation image dataset, and use the ordinary optical sensor to collect an initial optical image dataset. By utilizing the intrinsic parameter matrix of the semantic segmentation sensor and the rigid body transformation matrix from the world coordinate system to the sensor coordinate system, the sampling points of the semantic segmentation image dataset in the world coordinate system are projected onto the sensor coordinate system to obtain the semantic transformation image dataset. The nearest neighbor interpolation method is then used to resample the semantic transformation image dataset to achieve pixel-level alignment. By utilizing the intrinsic parameter matrix of a common optical sensor and the rigid body transformation matrix from the world coordinate system to the sensor coordinate system, the sampling points of the initial optical image dataset in the world coordinate system are projected onto the sensor coordinate system to obtain the optically transformed image dataset. Then, the optically transformed image dataset is resampled using cubic interpolation to achieve pixel-level alignment, and the optically transformed image dataset is used as the original image dataset.

[0007] In one embodiment of the present invention, based on the original image dataset, a differentiable rendering framework is used to calculate the cumulative soft overlay value of each triangular facet in the 3D model at a specified viewpoint, and a synthetic rendered image dataset at the specified viewpoint is generated, including: Configure the SoftRAS differentiable rendering framework, load the pose parameters of a common optical sensor, establish a soft rasterization rendering pipeline, perform soft rasterization processing on the 3D model at a specified viewpoint, calculate the cumulative soft cover value of each triangular facet in the 3D model at a specified viewpoint based on the soft rasterization processing results, and perform differentiable shading processing on the soft rasterization processing results to generate a target object rendering image dataset at a specified viewpoint. A binary mask for the target region is generated based on the semantically transformed image data. Under the constraint of the binary mask for the target region, the difference analysis and image fusion of the target object rendered image set and the original image dataset are performed to generate a synthetic rendered image dataset.

[0008] In one embodiment of the present invention, the visibility contribution value of the corresponding triangular facet in the 3D model at a specified viewpoint is calculated based on the difference analysis results and the cumulative soft coverage value of each triangular facet at a specified viewpoint, including: Based on the difference analysis results, determine whether the parameters within the SoftRAS differentiable rendering framework are reasonable. If reasonable, calculate the visibility contribution value of the corresponding triangular facet in the 3D model at the specified viewpoint based on the cumulative soft cover value of each triangular facet at the specified viewpoint. If unreasonable, modify the parameters within the SoftRAS differentiable rendering framework and regenerate the synthetic rendering image dataset.

[0009] In one embodiment of the present invention, before constructing location identifiers describing all uncovered triangular faces in the 3D model based on visibility contribution values, the method further includes: The visibility contribution value is normalized.

[0010] In one embodiment of the present invention, extracting the unit normal vector corresponding to the uncovered triangular facet based on the position identifier of the uncovered triangular facet includes: Based on the position identifier of the uncovered triangle, the unit normal vector of the corresponding uncovered triangle is calculated using the vertex coordinates of the uncovered triangle; the calculation of the unit normal vector is expressed as: ; in, Indicates the first uncovered triangular face among all uncovered triangular faces. The unit normal vector of a triangular facet. The value can be 1 to N, where N represents the number of uncovered triangular faces. For the first The vertex coordinates of a triangular facet. This indicates the calculation of the L2 norm.

[0011] In one embodiment of the present invention, the normal space formed by the unit normal vectors of all uncovered triangular facets is divided into multiple normal clusters, including: The K-means++ clustering algorithm is used to divide the normal space formed by the unit normal vectors of all uncovered triangular facets into multiple normal clusters.

[0012] In one embodiment of the present invention, the K-means++ clustering algorithm is used to divide the normal space formed by the unit normal vectors of all uncovered triangular facets into multiple normal clusters, including: The elbow rule is used to determine the optimal number of clusters, k. Based on the K-means++ clustering algorithm, the k most dispersed unit normal vectors are selected from all uncovered triangular facets as the initial centroids of the normal clusters. Vector assignment and centroid updates are performed alternately through expectation maximization iteration until the position change of the centroids of the normal clusters is less than a preset radian. Multiple normal clusters are output, and the centroid of each normal cluster represents the principal component of a normal direction distribution.

[0013] In one embodiment of the present invention, generating a set of pose parameters for a supplementary sensor based on all unit normal vectors corresponding to each normal cluster includes: For each normal cluster, the process includes: taking the opposite direction of the unit normal vector at the centroid of the normal cluster as the optimal observation direction vector; estimating the minimum observation distance based on the relative spatial relationship between the convex hull vertex set formed by the vertices of all uncovered triangular faces corresponding to the normal cluster and the center point of the target region; and generating supplementary sensor pose parameters for the normal cluster based on the convex hull vertex set, the optimal observation direction vector, and the minimum observation distance. The pose parameter set of the supplementary sensor is formed by all the pose parameters of the supplementary sensor.

[0014] The beneficial effects of this invention are: The simulation dataset generation method proposed in this invention, based on differentiable rendering and clustering, significantly reduces the size and acquisition cost of the simulation dataset while ensuring target detection accuracy. It is applicable to fields such as industrial 3D quality inspection, autonomous driving, and adversarial robustness testing. Specifically: First, this invention accurately locates uncovered areas on the surface of a 3D model through normal clustering, and intelligently generates a small number of supplementary observation points only for key blind areas, which greatly reduces the size of the optimized simulation dataset. At the same time, it avoids the accumulation of redundant data caused by traditional uniform sampling, effectively reducing storage pressure and hardware resource consumption.

[0015] Second, this invention achieves autonomous optimization based on a differentiable rendering framework, automatically completes the identification of coverage blind spots and supplements sensor pose derivation, significantly shortens the simulation dataset construction cycle, and is adaptable to various complex 3D model scenarios.

[0016] Third, by accurately locating uncovered areas and supplementing key observation points, this invention can effectively identify and resist adversarial camouflage, such as disguised obstacles or vehicles, thereby improving the anti-deception capability of detection systems in scenarios such as autonomous driving and enhancing their reliability and safety in complex environments.

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a simulation dataset generation method for a differentiable rendering and clustering method provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the distribution effect of the number of triangular patch coverage (hit) provided in an embodiment of the present invention; Figure 3 This invention is based on Figure 1 A schematic diagram illustrating the effect of evaluating the enhanced image dataset generated by the method shown in the adversarial training; Figures 4(a) to 4(b) The traditional exhaustive search dataset and the dataset based on this invention Figure 1 The diagram illustrates the change in confidence level of the enhanced image dataset generated by the method shown before and after an adversarial attack. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0020] Please see Figure 1 This invention provides a method for generating simulation datasets using differentiable rendering and clustering methods. The method includes: S10. Construct a multi-sensor simulation environment and generate the original image dataset based on the multi-sensor simulation environment.

[0021] This invention constructs a multi-sensor simulation environment and generates an original image dataset based on this environment. The process includes: configuring a semantic segmentation sensor and its corresponding ordinary optical sensor; acquiring a semantic segmentation image dataset using the semantic segmentation sensor; acquiring an initial optical image dataset using the ordinary optical sensor; projecting the sampling points of the semantic segmentation image dataset in the world coordinate system to the sensor coordinate system using the intrinsic parameter matrix of the semantic segmentation sensor and the rigid body transformation matrix from the world coordinate system to the sensor coordinate system to obtain a semantically transformed image dataset; resampling the semantically transformed image dataset using nearest neighbor interpolation to achieve pixel-level alignment; and projecting the sampling points of the initial optical image dataset in the world coordinate system to the sensor coordinate system using the intrinsic parameter matrix of the ordinary optical sensor and the rigid body transformation matrix from the world coordinate system to the sensor coordinate system to obtain an optically transformed image dataset; resampling the optically transformed image dataset using cubic interpolation to achieve pixel-level alignment; and using the interpolated and aligned optically transformed image dataset as the original image dataset. More specifically: This invention constructs a multi-sensor simulation environment by configuring a semantic segmentation sensor and its corresponding ordinary optical sensor. N initial observation points are preset within the three-dimensional model space, and each observation point records its spatial coordinates. and Euler angle posture It drives the semantic sensor to acquire a semantic segmentation image dataset and drives the ordinary optical sensor to acquire an initial optical image dataset.

[0022] Next, the intrinsic parameter matrix of the sensor is calculated based on the sensor's resolution and horizontal field of view. This intrinsic parameter matrix includes focal length parameters and image principal point coordinates, which are used for the projection of three-dimensional points to two-dimensional pixel coordinates. The specific process is as follows: construct the intrinsic parameter matrix of the sensor, calculate the focal length parameters based on the sensor's resolution and horizontal field of view, and use the calculated focal length parameters as the focal length item in the intrinsic parameter matrix to participate in the projection calculation of three-dimensional points to two-dimensional pixel coordinates. Based on the sensor's pose parameters, a rigid body transformation matrix is ​​constructed from the world coordinate system to the camera coordinate system. This rigid body transformation matrix is ​​a 4×4 homogeneous transformation matrix containing rotation and translation components, used for the transformation from the world coordinate system to the camera coordinate system. The specific transformation process is as follows: points in the world coordinate system are converted to homogeneous coordinate form, i.e., homogeneous components are added to the original 3D coordinates to allow for unified calculation with the homogeneous transformation matrix from the world coordinate system to the sensor coordinate system. Then, the rigid body transformation matrix from the world coordinate system to the sensor coordinate system is used to transform it back to the sensor coordinate system, realizing the projection of 3D points in the world coordinate system onto the 2D image plane of the sensor coordinate system. Perspective division is then performed to obtain the final 2D image plane in the sensor coordinate system. In this embodiment of the invention, the semantic segmentation sensor and the ordinary optical sensor respectively process the semantic segmentation image dataset and the initial optical image dataset to form corresponding semantically transformed image datasets and optically transformed image datasets. The semantic segmentation sensor and the ordinary optical sensor are distributed in the same location, meaning they can use the same sensor coordinate system.

[0023] Finally, nearest neighbor interpolation was used to resample the semantically transformed image data to achieve pixel-level alignment, and cubic interpolation was used to resample the optically transformed image data to achieve pixel-level alignment. The interpolated and aligned optically transformed image dataset was then used as the original image dataset, while the semantically transformed image dataset was used for generating subsequent rendered images.

[0024] S20. Based on the original image dataset, use a differentiable rendering framework to calculate the cumulative soft overlay value of each triangular facet in the 3D model at a specified viewpoint, and generate a synthetic rendering image dataset at the specified viewpoint.

[0025] This invention, based on an original image dataset, utilizes a differentiable rendering framework to calculate the cumulative soft cover value of each triangular facet in a 3D model from a specified viewpoint, and generates a synthetic rendered image dataset from that viewpoint. The process includes: generating a binary mask for the target region based on semantically transformed image data; configuring the SoftRAS differentiable rendering framework, loading pose parameters from a common optical sensor, establishing a soft rasterization rendering pipeline, performing soft rasterization processing on the 3D model from the specified viewpoint, calculating the cumulative soft cover value of each triangular facet in the 3D model from the specified viewpoint based on the soft rasterization processing results, and performing differentiable shading processing on the soft rasterization results to generate a target object rendered image dataset from the specified viewpoint; and, under the constraint of the target region binary mask, performing difference analysis and image fusion between the target object rendered image dataset and the original image dataset to generate a synthetic rendered image dataset. More specifically: This invention first configures a SoftRAS (Soft Rasterizer) differentiable rendering framework for view composition. SoftRAS is a differentiable rendering technique based on probability density functions. By softening the hard decision boundaries in the traditional rasterization process, it makes the rendering process continuously differentiable, making it particularly suitable for 3D scene analysis tasks. During the rendering process, the pose parameters of a common optical sensor are loaded. These pose parameters include complete spatial position and orientation information, which determine the viewpoint of the common optical sensor. A soft rasterization rendering pipeline is established to perform soft rasterization processing on the 3D model at a specified viewpoint. Based on the soft rasterization processing results, the cumulative soft coverage value of each triangular facet in the 3D model at the specified viewpoint is calculated. This cumulative soft coverage value reflects the contribution of the triangular facet to the visibility of the composite rendered image at the specified viewpoint. The soft rasterization processing results are then further processed with differentiable shading, including simulating real optical characteristics, such as material surface reflection, ambient lighting, and shadow effects, to generate a dataset of rendered images of the target object at the specified viewpoint.

[0026] Next, a binary mask of the target region is generated using the semantic transformation image dataset. Then, using this binary mask as a constraint, the effective region for difference analysis and image fusion between the target object rendering image dataset and the original image dataset is used as the synthetic rendering image dataset.

[0027] This invention also allows for rigorous pixel-level alignment verification of the synthesized rendered image dataset, ensuring consistency between the rendered result and the original data. This process is achieved through multi-dimensional similarity evaluation, including consistency in color distribution, matching degree of geometric shape, and similarity of structural features. When a significant deviation is detected, an iterative optimization process is automatically initiated, gradually improving alignment accuracy by fine-tuning rendering parameters.

[0028] S30. Perform a difference analysis on the synthetic rendering image dataset and the original image dataset. Based on the difference analysis results and the cumulative soft coverage value of each triangular facet at a specified viewpoint, calculate the visibility contribution value of the corresponding triangular facet in the 3D model at the specified viewpoint. Based on the visibility contribution value, construct a location identifier describing all uncovered triangular facets in the 3D model.

[0029] This invention, in its embodiments, calculates the visibility contribution value of the corresponding triangular facet in a 3D model from a specified viewpoint based on the difference analysis results and the cumulative soft coverage value of each triangular facet at a specified viewpoint. This includes: determining whether the parameters within the SoftRAS differentiable rendering framework are reasonable based on the difference analysis results; if reasonable, calculating the visibility contribution value of the corresponding triangular facet in the 3D model from the specified viewpoint based on the cumulative soft coverage value of each triangular facet; if unreasonable, modifying the parameters within the SoftRAS differentiable rendering framework and regenerating the synthetic rendering image dataset. This invention, in its embodiments, constructs location markers describing all uncovered triangular facets in the 3D model based on the visibility contribution values, and further includes: normalizing the visibility contribution values. More specifically: In this embodiment of the invention, the parameters within the SoftRAS differentiable rendering framework are determined based on the difference analysis results. If reasonable, the visibility contribution value of the corresponding triangular facet in the 3D model at the specified viewpoint is calculated based on the cumulative soft coverage value of each triangular facet at the specified viewpoint. If unreasonable, to ensure the correctness of the cumulative soft coverage value of each triangular facet output by the SoftRAS differentiable rendering framework at the specified viewpoint, the parameters within the SoftRAS differentiable rendering framework are modified, and the process returns to step S20 to regenerate the synthetic rendering image dataset, until the parameters within the SoftRAS differentiable rendering framework are determined to be reasonable based on the difference analysis results. Furthermore, the visibility contribution value of the corresponding triangular facet in the 3D model at the specified viewpoint is calculated based on the cumulative soft coverage value of each triangular facet at the specified viewpoint. The visibility contribution value is normalized to the [0,1] interval using the Sigmoid function. The normalized visibility contribution value is compared with a preset contribution threshold, and the comparison results are used to construct position markers describing all uncovered triangular facets in the 3D model. When the normalized visibility contribution value is lower than the preset contribution threshold, it indicates that the triangular patch is not effectively observed from the current specified viewpoint. The system marks it as an uncovered triangular patch and records the location identifier of the uncovered triangular patch.

[0030] This invention enables multi-dimensional in-depth analysis of coverage data, performing five-dimensional statistical analysis on selected active faces: basic statistical analysis counts the total number of covered triangles, reflecting the coverage strength of the dataset; distribution statistical analysis reveals the average number of covered triangles and their standard deviation; state classification statistical analysis classifies triangles into covered and uncovered triangles; extreme value statistical analysis captures the highest coverage record of a triangle and marks uncovered triangles; finally, the system automatically generates a structured report. For example... Figure 2 The diagram illustrates the distribution of the number of hits by the triangular facets.

[0031] This invention can also visualize hit data by generating heatmaps and designing a dynamic color mapping system using visual encoding: First, the dynamic range of texture values ​​is scanned, and the original values ​​are mapped to the [0,1] interval using a linear normalization conversion formula. A continuous gradient color level encoding scheme is created, with the low-hit interval, medium-hit interval, and high-hit interval displayed using a color effect from blue to red in sequence.

[0032] S40. Extract the unit normal vector of the corresponding uncovered triangular facet based on the position identifier of the uncovered triangular facet. Divide the normal space formed by the unit normal vectors of all uncovered triangular facets into multiple normal clusters. Generate a set of pose parameters for the supplementary sensor based on all the unit normal vectors corresponding to each normal cluster.

[0033] This invention embodiment extracts the unit normal vector of the corresponding uncovered triangle based on the position identifier of the uncovered triangle, including: calculating the unit normal vector of the corresponding uncovered triangle based on the position identifier of the uncovered triangle and the vertex coordinates of the uncovered triangle; the calculation of the unit normal vector is expressed as: ; in, Indicates the first uncovered triangular face among all uncovered triangular faces. The unit normal vector of a triangular facet. The value can be 1 to N, where N represents the number of uncovered triangular faces. For the first The vertex coordinates of a triangular facet. This indicates the calculation of the L2 norm. This calculation is performed on all uncovered triangular faces, generating a unit normal vector matrix, which provides geometric feature input for subsequent cluster analysis.

[0034] This invention, in its embodiments, divides the normal space formed by the unit normal vectors of all uncovered triangular facets into multiple normal clusters. This includes: using the K-means++ clustering algorithm to divide the normal space into multiple normal clusters. Specifically, this invention uses the K-means++ clustering algorithm to divide the normal space into multiple normal clusters, including: determining the optimal number of clusters k using the elbow rule; specifically calculating the curve of the sum of squares (WSS) within each cluster changing with the value of k; selecting the inflection point where the rate of change of WSS is less than 5% as the value of k; based on the K-means++ clustering algorithm, selecting the k most dispersed unit normal vectors from all uncovered triangular facets as the initial centroids of the normal clusters; iteratively performing vector allocation and centroid updates of the normal clusters through expectation maximization until the change in the position of the centroid of the normal cluster is less than a preset radian, for example, the change in the position of the centroid of the normal cluster is less than... The output is in radians, and multiple normal clusters are generated. The centroid of each normal cluster represents a principal component of a normal direction distribution.

[0035] This invention generates a set of pose parameters for a supplementary sensor based on all unit normal vectors corresponding to each normal cluster. The process includes: for each normal cluster, the steps include: using the opposite direction of the unit normal vector at the centroid of the normal cluster as the optimal observation direction vector; estimating the minimum observation distance based on the relative spatial relationship between the vertex set of the convex hull formed by the vertices of all uncovered triangular faces corresponding to the normal cluster and the center point of the target region; generating supplementary sensor pose parameters for the normal cluster based on the convex hull vertex set, the optimal observation direction vector, and the minimum observation distance; and forming a set of pose parameters for the supplementary sensor from all the supplementary sensor pose parameters. More specifically: This invention embodiment targets each normal cluster, for example, the centroid of the normal cluster is... , Indicates the first The centroid of a normal cluster, The value ranges from 1 to k, where k represents the number of normal clusters, and the corresponding unit normal vector is denoted as . The unit normal vector The opposite direction is taken as the optimal observation direction vector: ; Combined with the The set of convex hull vertices of all uncovered triangular faces corresponding to each normal cluster. Based on its relative spatial relationship with the center point of the target area, the minimum observation distance is estimated. ; in, Indicates the first The minimum observation distance corresponding to each normal cluster Indicates the first The set of convex hull vertices formed by all the vertices of the uncovered triangular faces corresponding to each normal cluster. Indicates the center point of the target area.

[0036] Finally, the pose parameters corresponding to the k normal clusters are generated: ; in, Indicates the first Pose parameters of the supplementary sensor corresponding to each normal cluster Indicates the first The optimal observation direction vector corresponding to each normal cluster. These represent the components of the optimal observation direction vector along the x-axis, y-axis, and z-axis, respectively. Used to calculate the corresponding plane azimuth angle. This is used to calculate the included angle related to the z-axis direction. The rotation parameters employ Euler angle representation to ensure coverage of over 90% of the triangular facets in the corresponding cluster. The unit normal vectors of all uncovered facets are analyzed using the K-means++ clustering algorithm. The sensor pose parameter set generated from the pose parameters of the supplementary sensors corresponding to each normal cluster can more efficiently cover the model surface, while simultaneously addressing the problem of uneven distribution in triangular facet detection.

[0037] In this embodiment of the invention, a supplementary sensor pose parameter set can be input into the differentiable rendering framework of S20 to regenerate the synthetic rendering image and perform visibility analysis in S30. The rate of elimination of uncovered triangular patches is calculated. :

[0038] in, This represents the number of uncovered triangular faces, as counted based on the supplementary sensor pose parameter set. This represents the total number of triangular faces counted based on the supplementary sensor pose parameter set, including both uncovered and covered triangular faces. If the set of supplementary sensor pose parameters is deemed valid and can be used as simulation data, then the process returns to step S40 to adjust the clustering parameters. The final output is the supplementary sensor pose parameter set.

[0039] This invention can also generate a sensor deployment scheme analysis report. The report includes core optimization indicators: the number of supplementary poses and the coverage improvement; it also provides a heatmap of pose distribution in three-dimensional space, and marks the observation cone region of k groups of sensors on the convex hull surface of the model.

[0040] S50. Generate a new image dataset based on the supplementary sensor pose parameter set, and merge the new image dataset with the original image dataset to form an enhanced image dataset as the final simulation dataset.

[0041] In this embodiment of the invention, a new image dataset is first generated based on the supplementary sensor pose parameter set. The supplementary sensor pose parameter set generated by normal clustering is then loaded, driving the semantic segmentation sensor and its corresponding ordinary optical sensor in the multi-sensor simulation environment to perform the acquisition of new image data. The process is similar to the generation of the original image dataset, maintaining the imaging specifications of the original image dataset.

[0042] Then, the newly added image dataset is fused with the original image dataset: a joint pose index is established to achieve spatiotemporal alignment, a unified YOLO format annotation standard is used to ensure label continuity, and pixel-level verification is performed to eliminate metadata conflicts. The fused enhanced image dataset, even after capacity expansion, still maintains an order of magnitude with the original small-scale image dataset, maximizing coverage gain while maintaining storage efficiency, and achieving full coverage of visible triangular patches.

[0043] To verify the effectiveness of the simulation dataset generation method of the differentiable rendering and clustering method provided in this embodiment of the invention, the following experiments were conducted.

[0044] 1. Experimental conditions The CPU is an AMD Ryzen 5 7600X 6-Core Processor, and the GPU is an NVIDIA GeForce RTX 4090. 4. The operating system used is Ubuntu 22.04, the deep learning framework used is PyTorch 2.4.0, and the NVIDIA computing package used is CUDA 12.4 + cuDNN 8.9.7.

[0045] Carla was used as the simulation platform to generate training and test data.

[0046] 2. Experiment Content Experiment 1 compares the performance differences between two simulation dataset construction methods: one is an exhaustive dataset based on 324 uniformly distributed pose points on a hemispherical shell, and the other uses the enhanced image dataset generation algorithm proposed in this invention. The experiment uses the official YOLOv7 architecture model to perform adversarial camouflage training on both generated datasets, and systematically records key performance indicators such as dataset generation time, confidence decline rate during model training, average training time, and average evaluation time. The comparison data of dataset generation efficiency and model computational performance are shown in Table 1.

[0047] Table 1 shows the comparison of training time and testing time between the two methods.

[0048] As shown in Table 1, compared with the simulation dataset constructed by exhaustively enumerating poses, the simulation dataset generation and optimization method proposed in this invention has significant advantages in terms of data efficiency and quality. Specifically, the data generation time is significantly reduced from 162 seconds to 24 seconds, an improvement of approximately 85.2%; the required data volume is reduced from 324 images to 102 images, saving approximately 68.5% of storage resources; and the target triangle coverage rate increases from 94.51% to 97.32%, an increase of 2.81 percentage points. Particularly noteworthy is that the average number of hits per triangle patch decreased from 118.5742 to 32.2808, a reduction of approximately 72.8%, indicating the effective elimination of a large number of redundant observations; the standard deviation of the hit count decreased from 42.4525 to 19.7116, a decrease of approximately 53.6%, proving that the method of this invention can make the sampling distribution more uniform and reasonable. These data fully demonstrate that the method of this invention can significantly improve data generation efficiency and optimize data quality, improving the integrity and uniformity of target coverage while reducing computational resource consumption.

[0049] Experiment 2: For the YOLOv7 architecture model, adversarial camouflage training was performed on the exhaustive image dataset and the augmented image dataset constructed in this invention using the official YOLOv7x-e6e weights. The average training time, average evaluation time, and confidence reduction rate at different distances were recorded. The average training time and average evaluation time are shown in Table 2, and the confidence reduction effect is shown in Table 3. Figure 3 , Figures 4(a) to 4(b) As shown, where, Figure 3 To illustrate the specific implementation effect of adversarial camouflage training using the simulation dataset constructed in this invention, Figure 4(a) shows the specific confidence performance of adversarial camouflage training using the simulation dataset constructed with exhaustive poses, and Figure 4(b) shows the specific confidence performance of adversarial camouflage training using the simulation dataset constructed in this invention.

[0050] Table 2 shows the comparison of training time and testing time for the two methods.

[0051] As shown in Table 2, the simulation dataset constructed in this invention reduces training time from 251 seconds per generation to 50 seconds, a reduction of approximately 80.1%; evaluation time is reduced from 102 seconds per generation to 27 seconds, saving approximately 73.5% of computational resources. A comparison of Figures 4(a) and 4(b) shows that, under the same parameter settings and uniform distance evaluation conditions, the simulation dataset constructed using this invention improves confidence reduction by approximately 20% compared to the simulation dataset constructed using exhaustive pose estimation, indicating a more significant adversarial camouflage effect. These results fully demonstrate that the enhanced image dataset constructed in this invention has significant advantages as a simulation dataset in improving training efficiency and adversarial effectiveness.

[0052] In summary, the simulation dataset generation method based on differentiable rendering and clustering proposed in this invention significantly reduces the size and acquisition cost of the simulation dataset while ensuring target detection accuracy. It is applicable to fields such as industrial 3D quality inspection, autonomous driving, and adversarial robustness testing. Specifically: First, this invention accurately locates uncovered areas on the surface of a 3D model through normal clustering, and intelligently generates a small number of supplementary observation points only for key blind areas, which greatly reduces the size of the optimized simulation dataset. At the same time, it avoids the accumulation of redundant data caused by traditional uniform sampling, effectively reducing storage pressure and hardware resource consumption.

[0053] Second, this invention achieves autonomous optimization based on a differentiable rendering framework, automatically completes the identification of coverage blind spots and supplements sensor pose derivation, significantly shortens the simulation dataset construction cycle, and is adaptable to various complex 3D model scenarios.

[0054] Third, by accurately locating uncovered areas and supplementing key observation points, this invention can effectively identify and resist adversarial camouflage, such as disguised obstacles or vehicles, thereby improving the anti-deception capability of detection systems in scenarios such as autonomous driving and enhancing their reliability and safety in complex environments.

[0055] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0056] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the specification and accompanying drawings, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0057] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for generating simulation datasets using differentiable rendering and clustering methods, characterized in that, The method for generating the simulation dataset includes: Construct a multi-sensor simulation environment and generate an original image dataset based on the multi-sensor simulation environment; Based on the original image dataset, the cumulative soft overlay value of each triangular facet in the 3D model at a specified viewpoint is calculated using a differentiable rendering framework, and a synthetic rendering image dataset at the specified viewpoint is generated. Perform a difference analysis on the synthetic rendering image dataset and the original image dataset. Based on the difference analysis results and the cumulative soft coverage value of each triangular facet in the specified viewpoint, calculate the visibility contribution value of the corresponding triangular facet in the 3D model in the specified viewpoint. Based on the visibility contribution value, construct a location identifier describing all uncovered triangular facets in the 3D model. Extract the unit normal vector of the uncovered triangle based on the position identifier of the uncovered triangle, divide the normal space formed by the unit normal vectors of all uncovered triangles into multiple normal clusters, and generate a set of pose parameters for the supplementary sensor based on all the unit normal vectors corresponding to each normal cluster. A new image dataset is generated based on the supplementary sensor pose parameter set. The new image dataset is then merged with the original image dataset to form an enhanced image dataset, which serves as the final simulation dataset.

2. The simulation dataset generation method of the differentiable rendering and clustering method according to claim 1, characterized in that, Construct a multi-sensor simulation environment and generate a raw image dataset based on the multi-sensor simulation environment, including: Configure a semantic segmentation sensor and its corresponding ordinary optical sensor, use the semantic segmentation sensor to collect a semantic segmentation image dataset, and use the ordinary optical sensor to collect an initial optical image dataset. By utilizing the intrinsic parameter matrix of the semantic segmentation sensor and the rigid body transformation matrix from the world coordinate system to the sensor coordinate system, the sampling points of the semantic segmentation image dataset in the world coordinate system are projected onto the sensor coordinate system to obtain the semantic transformation image dataset. The nearest neighbor interpolation method is then used to resample the semantic transformation image dataset to achieve pixel-level alignment. Using the intrinsic parameter matrix of a common optical sensor and the rigid body transformation matrix from the world coordinate system to the sensor coordinate system, the sampling points of the initial optical image dataset in the world coordinate system are projected onto the sensor coordinate system to obtain the optical transformed image dataset. Then, the optical transformed image dataset is resampled using cubic interpolation to achieve pixel-level alignment. The interpolated and aligned optical transformed image dataset is used as the original image dataset.

3. The simulation dataset generation method of the differentiable rendering and clustering method according to claim 2, characterized in that, Based on the original image dataset, a differentiable rendering framework is used to calculate the cumulative soft overlay value of each triangular facet in the 3D model at a specified viewpoint, and to generate a synthetic rendered image dataset at the specified viewpoint, including: Configure the SoftRAS differentiable rendering framework, load the pose parameters of a common optical sensor, establish a soft rasterization rendering pipeline, perform soft rasterization processing on the 3D model at a specified viewpoint, calculate the cumulative soft cover value of each triangular facet in the 3D model at a specified viewpoint based on the soft rasterization processing results, and perform differentiable shading processing on the soft rasterization processing results to generate a target object rendering image dataset at a specified viewpoint. A binary mask for the target region is generated based on the semantically transformed image data. Under the constraint of the binary mask for the target region, the difference analysis and image fusion of the target object rendered image set and the original image dataset are performed to generate a synthetic rendered image dataset.

4. The simulation dataset generation method of the differentiable rendering and clustering method according to claim 3, characterized in that, Based on the difference analysis results and the cumulative soft overlay value of each triangular facet at a specified viewpoint, the visibility contribution value of the corresponding triangular facet in the 3D model at the specified viewpoint is calculated, including: Based on the difference analysis results, determine whether the parameters within the SoftRAS differentiable rendering framework are reasonable. If reasonable, calculate the visibility contribution value of the corresponding triangular facet in the 3D model at the specified viewpoint based on the cumulative soft cover value of each triangular facet at the specified viewpoint. If unreasonable, modify the parameters within the SoftRAS differentiable rendering framework and regenerate the synthetic rendering image dataset.

5. The simulation dataset generation method of the differentiable rendering and clustering method according to claim 1, characterized in that, Before constructing location identifiers for all uncovered triangular faces in the 3D model based on visibility contribution values, the following steps are also included: The visibility contribution value is normalized.

6. The simulation dataset generation method of the differentiable rendering and clustering method according to claim 1, characterized in that, Extract the unit normal vector of the uncovered triangle based on its position identifier, including: Based on the position identifier of the uncovered triangle, the unit normal vector of the corresponding uncovered triangle is calculated using the vertex coordinates of the uncovered triangle; the calculation of the unit normal vector is expressed as: ； in, Indicates the first uncovered triangular face among all uncovered triangular faces. The unit normal vector of a triangular facet. The value can be 1 to N, where N represents the number of uncovered triangular faces. For the first The vertex coordinates of a triangular facet. This indicates the calculation of the L2 norm.

7. The simulation dataset generation method of the differentiable rendering and clustering method according to claim 1, characterized in that, The normal space formed by the unit normal vectors of all uncovered triangular faces is divided into multiple normal clusters, including: The K-means++ clustering algorithm is used to divide the normal space formed by the unit normal vectors of all uncovered triangular facets into multiple normal clusters.

8. The simulation dataset generation method of the differentiable rendering and clustering method according to claim 7, characterized in that, The K-means++ clustering algorithm is used to divide the normal space formed by the unit normal vectors of all uncovered triangular faces into multiple normal clusters, including: The elbow rule is used to determine the optimal number of clusters, k. Based on the K-means++ clustering algorithm, the k most dispersed unit normal vectors are selected from all uncovered triangular facets as the initial centroids of the normal clusters. Vector assignment and centroid updates are performed alternately through expectation maximization iteration until the position change of the centroids of the normal clusters is less than a preset radian. Multiple normal clusters are output, and the centroid of each normal cluster represents the principal component of a normal direction distribution.

9. The simulation dataset generation method of the differentiable rendering and clustering method according to claim 1, characterized in that, A set of pose parameters for the supplementary sensor is generated based on all unit normal vectors corresponding to each normal cluster, including: For each normal cluster, the process includes: taking the opposite direction of the unit normal vector at the centroid of the normal cluster as the optimal observation direction vector; estimating the minimum observation distance based on the relative spatial relationship between the convex hull vertex set formed by the vertices of all uncovered triangular faces corresponding to the normal cluster and the center point of the target region; and generating supplementary sensor pose parameters for the normal cluster based on the convex hull vertex set, the optimal observation direction vector, and the minimum observation distance. The pose parameter set of the supplementary sensor is formed by all the pose parameters of the supplementary sensor.

Images