An image sensor simulation model performance detection apparatus

Abstract

The application provides an image sensor simulation model performance detection device, which comprises the following steps: acquiring a real image, and generating a simulation scene based on the real image; acquiring a simulation image obtained by detecting the simulation scene by a to-be-tested image sensor simulation model; processing the real image and the simulation image based on a pre-trained simulation image authenticity evaluation model, to obtain distribution probabilities of image features of the real image and the simulation image on a Gaussian mixture model; determining authenticity of the simulation image according to the distribution probabilities of the image features of the real image and the simulation image on the Gaussian mixture model; and obtaining a performance detection result of the to-be-tested image sensor simulation model according to the authenticity of the simulation image. Through the application, the performance of the image sensor simulation model can be detected, and the restoration degree of the image sensor simulation model is improved.

Description

Technical Field

[0001] This application relates to the field of sensor simulation technology, specifically to a device for testing the performance of an image sensor simulation model. Background Technology

[0002] Intelligent connected vehicles are a recent trend in the automotive industry. These vehicles use various types of sensors to perceive their surroundings, and the different environmental information detected by these sensors affects the decision-making and control module responses of the autonomous driving system. Sensor modeling is a crucial step in intelligent connected vehicle simulation testing, as its quality determines the realism of the perceived environmental targets in the simulation. To ensure higher reliability of simulation results, the sensor models established in the simulation platform need to achieve higher fidelity. Therefore, a device / mechanism is required to test the performance of image sensor simulation models. Summary of the Invention

[0003] The purpose of this application is to propose a performance testing device for image sensor simulation models, so as to test the performance of image sensor simulation models and improve the fidelity of image sensor simulation models.

[0004] To achieve the above objectives, embodiments of this application provide an image sensor simulation model performance testing device, the device comprising:

[0005] A simulated scene generation module is used to acquire real images and generate simulated scenes based on the real images;

[0006] The simulation image acquisition module is used to acquire the simulation image obtained by the simulation model of the image sensor under test detecting the simulation scene;

[0007] The authenticity evaluation module is used to process the real image and the simulation image based on a pre-trained simulation image authenticity evaluation model, obtain the distribution probability of the image features of the real image and the simulation image on the Gaussian mixture model, and determine the authenticity of the simulation image based on the distribution probability of the image features of the real image and the simulation image on the Gaussian mixture model.

[0008] The performance diagnostic module is used to obtain the performance test results of the simulation model of the image sensor under test based on the realism of the simulation image.

[0009] The image sensor simulation model performance testing device proposed in this application has the following beneficial effects:

[0010] By calculating and comparing the distribution probabilities of image features of real and simulated images on a Gaussian mixture model using a simulation image authenticity evaluation model, the authenticity of the simulated image can be determined based on the comparison results. This achieves the quantification of the authenticity of the simulated image, accurately evaluates the authenticity of the simulated images generated by the image sensor simulation model, and provides an objective and quantifiable standard for evaluating the performance of the image sensor simulation model. By detecting the performance of the image sensor simulation model, developers can identify and improve the shortcomings in the image sensor modeling process, optimize image sensor modeling, improve the fidelity of the image sensor simulation model, and thus further improve the quality of intelligent connected vehicle simulation testing. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a structural diagram of an image sensor simulation model performance testing device according to an embodiment of this application.

[0013] Figure 2 This is a schematic diagram of the spatial distribution of data features in an embodiment of this application.

[0014] Figure 3 This is a schematic diagram illustrating the training process of the simulation image authenticity evaluation model in the embodiments of this application.

[0015] Figure 4 This is a schematic diagram illustrating the analysis process of the confidence level of the simulated image in the embodiments of this application. Detailed Implementation

[0016] The detailed description of the accompanying drawings is intended to illustrate the present embodiments of this application and is not intended to represent only the forms in which this application can be implemented. It should be understood that the same or equivalent functions can be accomplished by different embodiments intended to be included within the spirit and scope of this application.

[0017] See Figure 1 This application provides an image sensor simulation model performance testing device, the device comprising:

[0018] The simulated scene generation module 1 is used to acquire real images and generate simulated scenes based on the real images;

[0019] Specifically, real images can be captured in a real environment using actual image sensors (such as cameras). These real images should cover as many different scenes and conditions as possible, such as different lighting conditions, different weather conditions, and different traffic conditions, to ensure that the simulated scenes generated later have sufficient realism and representativeness. Furthermore, computer graphics techniques are used to generate simulated scenes based on the acquired real images. The purpose of generating simulated scenes is to simulate various situations in the real environment so that simulated images can be generated based on these scenes and compared with real images later.

[0020] Simulation image acquisition module 2 is used to acquire simulation images obtained by the simulation model of the image sensor under test detecting the simulated scene;

[0021] Specifically, the simulation model of the image sensor under test simulates the real image sensor, including the optical characteristics, noise characteristics, resolution and other factors of the simulated sensor. The simulation scene generated by the simulation scene generation module 1 is subjected to environmental detection to obtain the simulation image. The simulation image is the result of the image sensor simulation model's "perception" of the simulation scene, reflecting the performance of the image sensor simulation model under a given scene.

[0022] The authenticity evaluation module 3 is used to process the real image and the simulation image based on a pre-trained simulation image authenticity evaluation model, obtain the distribution probability of the image features of the real image and the simulation image on the Gaussian mixture model, and determine the authenticity of the simulation image based on the distribution probability of the image features of the real image and the simulation image on the Gaussian mixture model.

[0023] Specifically, the pre-trained simulation image authenticity evaluation model refers to a model that has been trained with a large number of real images and corresponding simulation images, learning how to acquire and determine the authenticity of the simulation image based on the distribution probability of the image features of the real and simulation images on a Gaussian mixture model. The acquired real and simulation images are input into the pre-trained simulation image authenticity evaluation model, which processes these images and extracts their image features. A Gaussian mixture model (GMM) is a probabilistic model used to describe the complexity of data distribution. In this embodiment, GMM is used to describe the distribution of image features in real and simulation images. Specifically, GMM assigns a probability distribution to each image feature, representing the probability distribution of that feature on the Gaussian mixture model. The Gaussian distribution of the real image can be understood as the truth value. By comparing the distribution probabilities of the real and simulation images on the Gaussian mixture model, the authenticity of the simulation image can be determined. If the distribution probability of the image features of the simulation image on the Gaussian mixture model is very close to the distribution probability of the real image, then the simulation image can be considered to have high authenticity. Conversely, if the distribution probability differs significantly, it indicates that the authenticity of the simulation image is low, and there may be insufficient modeling or errors.

[0024] Performance diagnostic module 4 is used to obtain the performance test results of the simulation model of the image sensor under test based on the realism of the simulation image;

[0025] Specifically, the realism evaluation module 3 has obtained the realism quantification result of the simulated image through the pre-trained simulated image realism evaluation model and Gaussian mixture model (GMM). This result can be a numerical value or a score, representing the degree of similarity or realism level between the simulated image and the real image. Based on this, a threshold or standard can be set. For example, if the realism score of the simulated image is higher than a certain threshold, the performance of the simulated image sensor under test is considered to be up to standard or excellent. If the realism score of the simulated image is lower than a certain threshold, the performance of the simulated image sensor under test is considered to be substandard and needs further optimization.

[0026] As described in the above embodiments, the device in this embodiment calculates and compares the distribution probabilities of image features of real and simulated images on a Gaussian mixture model using a simulation image authenticity evaluation model. Based on the comparison results, the authenticity of the simulated image can be determined, realizing the quantification of the authenticity of the simulated image. This accurately evaluates the authenticity of the simulated image generated by the image sensor simulation model, providing an objective and quantifiable standard for evaluating the performance of the image sensor simulation model. By detecting the performance of the image sensor simulation model, developers can identify and improve the shortcomings in the image sensor modeling process, optimize image sensor modeling, and improve the fidelity of the image sensor simulation model, thereby further improving the quality of intelligent connected vehicle simulation testing.

[0027] In some embodiments, the authenticity evaluation module 3 includes:

[0028] The feature extraction unit extracts image features from the real image and the simulated image, and performs index quantization on the image features of the real image and the simulated image to obtain image feature index values ​​of the real image and the simulated image; specifically, it quantizes the extracted image features of each type to obtain corresponding feature index values ​​so as to map them to the same space; for example, the feature vector can be normalized or transformed in other ways so that they can be compared on the same scale.

[0029] The feature mapping unit is used to map the image feature index values ​​of the real image and the simulated image to the same space to obtain a data feature space; mapping the image feature index values ​​of the real image and the simulated image to the same space allows for comparison and calculation on the same benchmark; for example... Figure 2 The image shown is an example of the distribution of structural similarity features between real and simulated images;

[0030] The Gaussian distribution calculation unit is used to calculate the distribution probability of image features of the real image and the simulated image on the Gaussian mixture model based on the pre-trained simulation image realism evaluation model and the data feature space. The Gaussian mixture model is a statistical model used to describe data distribution. It assumes that the data is composed of a mixture of multiple Gaussian distributions. In this embodiment, the Gaussian mixture model is used to describe the feature distribution of the real image and the simulated image in the data feature space. The image features are multimodal, and each feature is calculated using the Gaussian distribution to obtain the corresponding distribution probability.

[0031] Specifically, assume that the Gaussian distribution probability of any image feature is calculated using the following formula:

[0032]

[0033] Where p(x) represents the Gaussian distribution probability, X represents the feature index data point, K represents the number of Gaussian distributions (a pre-set parameter), and π k The mixture weights are the k-th Gaussian distribution, N(X|μ) k ,∑ k ) is the k-th Gaussian distribution with mean μ. k The sum of the covariance matrix ∑ k .

[0034] In some embodiments, the image features of the real image include at least one of the following: spatial domain features, frequency domain features, structural similarity features, peak signal-to-noise ratio features, and intrinsic parameter similarity features of the real image;

[0035] The image features of the simulated image include at least one of the following: spatial domain features, frequency domain features, structural similarity features, peak signal-to-noise ratio features, and intrinsic parameter similarity features.

[0036] The structural similarity features of the real image and the simulated image are both structural similarity between the simulated image and the real image, and the intrinsic parameter similarity features of the real image and the simulated image are both camera intrinsic parameter similarity between the simulated image and the real image;

[0037] Specifically, spatial domain features refer to the pixel-level features of an image, such as pixel values, grayscale distribution, and color histograms; frequency domain features refer to the features of an image in the frequency domain, which can be obtained through Fourier transform, such as frequency components and power spectra; structural similarity features refer to the degree of structural similarity between two images; peak signal-to-noise ratio features refer to the degree of image distortion; and intrinsic parameter similarity features refer to the degree of similarity between two images in terms of camera intrinsic parameters (such as focal length, principal point coordinates, etc.).

[0038] In some embodiments, the simulation image realism evaluation model is trained based on a model training module, which includes:

[0039] The image sample acquisition unit is used to acquire real image samples and generate simulation image samples based on the real image samples. Specifically, the test vehicle scene data is used as the real image sample, and the Logsim scene conversion is performed based on the test vehicle scene data to obtain the simulation scene. Further, the sensor model and the simulation scene are used to generate simulation image samples. It should be noted that the simulation image samples generated here must be qualified. Here, "qualified" usually means that the performance of the simulation image sample in the perception algorithm model should be as close as possible to the performance of the real image sample. This is to ensure that the simulation image authenticity evaluation model can learn how to correctly identify the quality of the simulation image.

[0040] The realism evaluation unit is used to obtain the distribution probability of image features of the real image samples and the simulated image samples on a Gaussian mixture model based on the trained simulation image realism evaluation model, and to obtain the confidence level of the simulated image samples based on the distribution probability of image features of the real image samples and the simulated image samples on the Gaussian mixture model. Specifically, the calculation of the distribution probability of image features on the Gaussian mixture model is described in the foregoing content of the device in this embodiment, and will not be repeated here. Based on the distribution probability of image features of the real image samples and the simulated image samples on the Gaussian mixture model, the confidence level of the simulated image samples can be determined. The confidence level of the simulated image samples reflects the degree of similarity between the simulated image and the real image. The confidence level analysis process is as follows: Figure 4 As shown;

[0041] The perception algorithm unit is used to perform perception recognition on the real image sample based on a preset perception algorithm model to obtain a first recognition result; it is also used to perform perception recognition on the simulated image sample based on the preset perception algorithm model to obtain a second recognition result, and to correct the second recognition result according to the confidence level of the simulated image sample; specifically, a correction coefficient can be determined according to the confidence level of the simulated image sample, and the correction coefficient can be a linear or nonlinear function, used to correct the second recognition result;

[0042] The training result recognition unit is used to determine whether to adjust the model parameters of the simulation image realism evaluation model based on the first recognition result and the corrected second recognition result. Specifically, it compares the first recognition result with the corrected second recognition result, and determines whether the performance of the current simulation image realism evaluation model meets the requirements based on the comparison result. If it does not meet the requirements, it adjusts the model parameters and performs the next round of training. If it meets the requirements, it ends the training or continues to use other samples for training.

[0043] In some embodiments, the training result recognition unit is used to obtain the error between the first recognition result and the corrected second recognition result; if the error is not within a preset range, the model parameters of the simulation image authenticity evaluation model are adjusted.

[0044] Specifically, the preset range is a pre-defined threshold range used to determine whether the error is acceptable. The calculated error is compared with the preset range. If the error is not within the preset range, it indicates that there is a significant difference between the output of the simulation image authenticity evaluation model and the expected output of the perception algorithm model. The model parameters need to be adjusted to reduce this difference (because the previous modules have ensured the quality of the simulation image samples, theoretically the error should be within the preset range. If there is a significant difference in the error, it means that there is a problem / error in the confidence analysis calculation of the simulation image samples by the model). The adjustment method can use various optimization algorithms in machine learning to adjust the model parameters, such as gradient descent, stochastic gradient descent, etc.

[0045] In some embodiments, such as Figure 4 As shown, the first recognition result includes a first semantic segmentation result and a first target detection result, and the corrected second recognition result includes a corrected second semantic segmentation result and a second target detection result;

[0046] The training result recognition unit is used for:

[0047] Obtain the semantic segmentation error between the first semantic segmentation result and the corrected second semantic segmentation result;

[0048] Obtain the target detection error between the first target detection result and the corrected second target detection result;

[0049] If the semantic segmentation error is not within the first error range or the target detection error is not within the second error range, then the model parameters of the simulation image authenticity evaluation model are adjusted.

[0050] Specifically, semantic segmentation error is the difference between the first and second semantic segmentation results. It can be calculated using metrics such as Intersection over Union (IoU) and pixel accuracy. The error is calculated by comparing the first and second semantic segmentation results with their corrected confidence levels. Object detection error is the difference between the first and second object detection results. It can be calculated using metrics such as accuracy and recall. The error is calculated by comparing the first and second object detection results with their corrected confidence levels. First and second error ranges are set for semantic segmentation error and object detection error, respectively. The calculated semantic segmentation error is compared with the first error range, and the object detection error is compared with the second error range. If the semantic segmentation error is not within the first error range, or the object detection error is not within the second error range, it indicates that the simulated image sample is unqualified and the model parameters need to be adjusted.

[0051] In some embodiments, the image sample acquisition unit is configured to:

[0052] A simulated scene is generated based on the real image samples, and a simulated image sample is generated by detecting the simulated scene based on a preset image sensor simulation model; the preset image sensor simulation model is an image sensor simulation model that has passed performance testing.

[0053] Specifically, such as Figure 4 As shown, the test vehicle scene data is used as a real image sample. Based on the test vehicle scene data, Logsim scene conversion is performed to obtain a simulated scene. Furthermore, a preset image sensor model is used to perform scene detection on the simulated scene to generate simulated image samples.

[0054] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A performance testing device for an image sensor simulation model, characterized in that, The device includes: A simulated scene generation module is used to acquire real images and generate simulated scenes based on the real images; The simulation image acquisition module is used to acquire the simulation image obtained by the simulation model of the image sensor under test detecting the simulation scene; The authenticity evaluation module is used to process the real image and the simulation image based on a pre-trained simulation image authenticity evaluation model, obtain the distribution probability of the image features of the real image and the simulation image on the Gaussian mixture model, and determine the authenticity of the simulation image based on the distribution probability of the image features of the real image and the simulation image on the Gaussian mixture model. The performance diagnostic module is used to obtain the performance test results of the simulation model of the image sensor under test based on the realism of the simulation image.

2. The apparatus according to claim 1, characterized in that, The authenticity evaluation module includes: The feature extraction unit extracts image features from the real image and the simulated image, and performs index quantization on the image features of the real image and the simulated image to obtain image feature index values ​​of the real image and the simulated image; The feature mapping unit is used to map the image feature index values ​​of the real image and the simulated image to the same space to obtain a data feature space; The Gaussian distribution calculation unit is used to calculate the distribution probability of image features of the real image and the simulated image on the Gaussian mixture model based on the pre-trained simulation image authenticity evaluation model and the data feature space.

3. The apparatus according to claim 2, characterized in that, The image features of the real image include at least one of the following: spatial domain features, frequency domain features, structural similarity features, peak signal-to-noise ratio features, and intrinsic parameter similarity features. The image features of the simulated image include at least one of the following: spatial domain features, frequency domain features, structural similarity features, peak signal-to-noise ratio features, and intrinsic parameter similarity features. The structural similarity features of the real image and the simulated image are both structural similarities between the simulated image and the real image, and the intrinsic parameter similarity features of the real image and the simulated image are both camera intrinsic parameter similarities between the simulated image and the real image.

4. The apparatus according to claim 1, characterized in that, The simulation image authenticity evaluation model is trained based on a model training module, which includes: An image sample acquisition unit is used to acquire real image samples and generate simulated image samples based on the real image samples; The authenticity evaluation unit is used to obtain the distribution probability of the image features of the real image sample and the simulated image sample on the Gaussian mixture model based on the trained simulation image authenticity evaluation model, and to obtain the confidence level of the simulated image sample based on the distribution probability of the image features of the real image sample and the simulated image sample on the Gaussian mixture model. The perception algorithm unit is used to perform perception recognition on the real image sample based on a preset perception algorithm model to obtain a first recognition result; it is also used to perform perception recognition on the simulated image sample based on the preset perception algorithm model to obtain a second recognition result, and to correct the second recognition result according to the confidence level of the simulated image sample. The training result recognition unit is used to determine whether to adjust the model parameters of the simulation image authenticity evaluation model based on the first recognition result and the corrected second recognition result.

5. The apparatus according to claim 4, characterized in that, The training result recognition unit is used to obtain the error between the first recognition result and the corrected second recognition result. If the error is not within a preset range, the model parameters of the simulation image authenticity evaluation model are adjusted.

6. The apparatus according to claim 5, characterized in that, The first identification result includes a first semantic segmentation result and a first target detection result, and the corrected second identification result includes a corrected second semantic segmentation result and a second target detection result; The training result recognition unit is used for: Obtain the semantic segmentation error between the first semantic segmentation result and the corrected second semantic segmentation result; Obtain the target detection error between the first target detection result and the corrected second target detection result; If the semantic segmentation error is not within the first error range or the target detection error is not within the second error range, then the model parameters of the simulation image authenticity evaluation model are adjusted.

7. The apparatus according to claim 4, characterized in that, The image sample acquisition unit is used for: A simulated scene is generated based on the real image samples, and a simulated image sample is generated by detecting the simulated scene based on a preset image sensor simulation model; the preset image sensor simulation model is an image sensor simulation model that has passed performance testing.

Images