Method, system and device for evaluating domain difference between synthetic data and real data
By extracting features in multiple dimensions and evaluating domain differences in multiple levels, this method solves the problems of multi-level and interpretability in domain difference evaluation in existing technologies, and achieves accurate evaluation of the domain differences between synthetic data and real data and prediction of the impact on model performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG YUEYING TECH CO LTD
- Filing Date
- 2026-05-20
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391748A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method, system and device for evaluating the domain difference between synthetic data and real data. Background Technology
[0002] Since training large-scale visual models relies on massive amounts of high-quality data, synthetic data techniques based on simulation environments and generative models have emerged. However, there are often significant domain differences between synthetic and real data, which severely impacts the model's generalization performance in real-world scenarios.
[0003] Existing methods for evaluating domain dissimilarity mainly fall into two categories: one is based on statistical distribution, which measures domain dissimilarity by calculating the statistical distribution differences between synthetic and real data at the pixel or feature level; the other is based on deep feature analysis, which uses pre-trained deep neural networks to extract features and calculate domain dissimilarity in the feature space. However, these methods have the following shortcomings: First, they have a single evaluation dimension, often focusing only on differences at a certain level and lacking a comprehensive evaluation system with multiple levels and dimensions; second, they lack interpretability, usually only providing an overall dissimilarity score without indicating the specific source and location of the differences; and third, they are not sufficiently correlated with model performance, making it difficult to predict the degree of impact of domain dissimilarity on the performance of specific tasks, thus hindering the optimization and improvement of synthetic data. Summary of the Invention
[0004] Therefore, it is necessary to provide a method, system, and device for evaluating the domain differences between synthetic and real data to address the aforementioned technical issues. This method and device can comprehensively evaluate the domain differences between synthetic and real data from multiple dimensions and levels, thereby accurately determining the sources of domain differences and their impact on model performance, and providing guidance for data generation, screening, and domain adaptation strategies.
[0005] Firstly, a method for assessing the domain difference between synthetic data and real data is provided, including: Obtain the synthetic data and real data to be evaluated; Based on the synthetic data and the real data, multi-dimensional features of the synthetic data and the real data are extracted. The multi-dimensional features include texture features, geometric features, semantic features and temporal features. Based on the statistical distribution differences between the synthetic data and the real data, and the multi-dimensional features, the multi-level domain differences between the synthetic data and the real data are obtained. Based on the multi-level domain differences, assess the main sources of the domain differences and evaluate the impact of the multi-level domain differences on the performance of the target model.
[0006] In some examples, obtaining the multi-level domain differences between the synthetic data and the real data based on the statistical distribution differences between the synthetic data and the real data, and the multi-dimensional features, includes: Based on the statistical distribution differences between the synthesized data and the real data, the pixel-level domain differences between the synthesized data and the real data are obtained; Based on the texture features and the geometric features, the feature-level domain differences between the synthetic data and the real data are obtained; Based on the semantic features, the semantic-level domain differences between the synthetic data and the real data are obtained; Based on the synthetic data and the real data, the task-level domain differences between the synthetic data and the real data are obtained through a pre-built proxy model.
[0007] In some examples, obtaining the task-level domain difference between the synthetic data and the real data using a pre-built proxy model includes: The proxy model is trained and tested on the real data to obtain the benchmark performance of the proxy model; The proxy model is trained on the synthetic data and tested on the real data to obtain the actual performance of the proxy model; Calculate the performance degradation rate of the proxy model based on the baseline performance and the actual performance; The synthetic data and real data are classified using a domain classifier, and the classification accuracy is calculated. The task-level domain difference is obtained based on the performance degradation rate and / or the classification accuracy.
[0008] In some examples, the assessment of the primary sources of domain differences based on the multi-level domain differences includes: Based on the reference values of the differences between the multi-level domains and the differences between each level domain, the deviation value of the differences between each level domain is calculated. Based on the deviation values of the domain differences at each level, the main sources of the domain differences are determined.
[0009] In some examples, before calculating the deviation of each level-domain difference based on the benchmark value of the multi-level domain difference and the level-domain difference, the method further includes: Based on the real data, two disjoint subsets of real data are obtained. Calculate the differences between the hierarchical domains of the real data subsets, and use these differences as the baseline values for the differences between the hierarchical domains.
[0010] In some examples, evaluating the impact of the multi-level domain differences on the target model performance includes: The multi-level domain differences are input into the pre-trained prediction model to obtain the performance prediction value of the target model.
[0011] In some examples, before inputting the multi-level domain differences into a pre-trained prediction model to obtain the performance prediction value of the target model, the method further includes: The synthetic data is divided into multiple sets of synthetic data subsets, and combined with the real data, multiple sets of multi-level domain differences corresponding to the multiple sets of synthetic data subsets are obtained. The target model is trained on the synthetic dataset and tested on the real data to obtain the target model performance corresponding to the differences between the multiple sets of multi-level domains; Based on the multiple sets of multi-level domain differences and the target model performance corresponding to the multiple sets of multi-level domain differences, a prediction model from the multi-level domain differences to the target model performance is constructed.
[0012] In some examples, it also includes: The evaluation results are visualized and an evaluation report is generated.
[0013] Secondly, a domain difference assessment system for synthetic data and real data is provided, including: The acquisition module is used to acquire the synthetic data and real data to be evaluated; The feature extraction module is used to extract multi-dimensional features of the synthetic data and real data based on the synthetic data and real data. The multi-dimensional features include texture features, geometric features, semantic features and temporal features. The calculation module is used to obtain the multi-level domain differences between the synthetic data and the real data based on the statistical distribution differences between the synthetic data and the real data, as well as the multi-dimensional features. The evaluation module is used to evaluate the main sources of the domain differences based on the multi-level domain differences, and to evaluate the impact of the multi-level domain differences on the performance of the target model.
[0014] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method for evaluating the domain difference between synthetic data and real data as described in the first aspect and any possible implementation of the first aspect.
[0015] The embodiments of this application first obtain synthetic data and real data to be evaluated; then, based on the synthetic and real data, multi-dimensional features of the synthetic and real data are extracted, including texture features, geometric features, semantic features, and temporal features; next, based on the statistical distribution differences between the synthetic and real data, and the multi-dimensional features, multi-level domain differences between the synthetic and real data are obtained; finally, based on the multi-level domain differences, the main sources of domain differences are evaluated, and the impact of multi-level domain differences on the performance of the target model is assessed. Therefore, a comprehensive evaluation of the domain differences between synthetic and real data can be performed from multiple dimensions and levels, thereby accurately determining the sources of domain differences and their impact on model performance, providing guidance for data generation, screening, and domain adaptation strategies. Attached Figure Description
[0016] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating the domain difference assessment method between synthetic data and real data provided in this application embodiment; Figure 2 A schematic diagram illustrating the execution of the domain difference assessment method between synthetic data and real data provided in the embodiments of this application; Figure 3 Data flow diagram provided for embodiments of this application; Figure 4 A flowchart of multidimensional feature extraction provided for embodiments of this application; Figure 5 An overall architecture diagram provided for embodiments of this application; Figure 6 This is a structural block diagram of the domain difference assessment system between synthetic data and real data provided in the embodiments of this application; Figure 7 A structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation
[0017] The present application will now be described in further detail with reference to the embodiments and accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the application are shown in the accompanying drawings.
[0018] It should be noted that, unless otherwise specified, the embodiments and features of the embodiments in this application can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments.
[0019] The following describes in detail, with reference to the accompanying drawings, a method, system, and apparatus for evaluating the domain difference between synthetic data and real data according to embodiments of this application.
[0020] Figure 1 This is a flowchart of a method for evaluating the domain difference between synthetic data and real data according to an embodiment of this application. Figure 1 As shown, and in combination Figure 2 and Figure 3 The domain difference evaluation method between synthetic data and real data according to embodiments of this application includes the following steps: S101: Obtain the synthetic data and real data to be evaluated.
[0021] Specifically, the data to be evaluated includes image data and / or video data. Synthetic data refers to data generated through simulation environments or generative models, while real data refers to data actually collected by sensors. In a specific example, 50,000 frames of synthetic in-vehicle video data and 10,000 frames of real in-vehicle video data are collected. The synthetic in-vehicle video data is generated through a 3D simulation environment, including driving scenarios with different road types, weather conditions, and lighting conditions; the real in-vehicle video data is collected by a data acquisition vehicle in a real indoor environment.
[0022] S102: Based on the synthetic data and the real data, extract the multi-dimensional features of the synthetic data and the real data. The multi-dimensional features include texture features, geometric features, semantic features and temporal features.
[0023] Specifically, such as Figure 4 As shown, the aforementioned multi-dimensional features can be calculated using neural network models. Specifically, texture features can be calculated using deep convolutional neural networks, such as Residual Network (ResNet) and Visual Geometry Group Network (VGG); geometric features can be calculated using depth estimation neural network models, such as Monocular Depth Estimation Network (MiDaS) and DepthNet; semantic features can be calculated using neural network models for object classification and detection, such as Vision Transformer (ViT) and YOLO; and temporal features can be calculated using optical flow estimation neural networks, such as Recurrent All-Pairs FieldTransforms (RAFT) and PWC-Net.
[0024] In a specific example, the VGG-19 network is used to extract texture features, with a feature dimension of 512; the MiDaS network is used to extract geometric features, with a feature dimension of 256; the ViT network is used to extract semantic features, with a feature dimension of 768; and the RAFT network is used to extract temporal features, with a feature dimension of 256. Combining these features, the final multi-dimensional feature has a dimension of 1792.
[0025] S103: Based on the statistical distribution differences between the synthetic data and the real data, and the multi-dimensional features, the multi-level domain differences between the synthetic data and the real data are obtained.
[0026] In one embodiment of this application, obtaining multi-level domain differences between the synthetic data and real data based on the statistical distribution differences between the synthetic data and real data, and the multi-dimensional features, includes: obtaining pixel-level domain differences between the synthetic data and real data based on the statistical distribution differences between the synthetic data and real data; obtaining feature-level domain differences between the synthetic data and real data based on the texture features and the geometric features; obtaining semantic-level domain differences between the synthetic data and real data based on the semantic features; and obtaining task-level domain differences between the synthetic data and real data based on the synthetic data and real data through a pre-built proxy model.
[0027] For the multi-dimensional features obtained in the above steps, targeted measurement methods are used to calculate the domain differences between synthetic and real data at different levels. Specifically, pixel-level domain difference refers to the statistical distribution difference between synthetic and real data in the pixel space, which can be evaluated through color histogram distribution differences, including KL divergence, spectral difference, and the mean of the Structural Similarity Index Measure (SSIM); feature-level domain difference refers to the distribution difference between synthetic and real data in the neural network feature space, which is obtained by calculating indicators such as Maximum Mean Discrepancy (MMD) and Fréchet Inception Distance (FID) based on the extracted texture and geometric features; semantic-level domain difference refers to the distribution difference between synthetic and real data in the semantic feature space, which is obtained by calculating indicators such as MMD and FID based on the extracted semantic features; task-level domain difference refers to the performance difference between synthetic and real data in specific task execution, which is calculated through a pre-built surrogate model.
[0028] In one embodiment of this application, obtaining the task-level domain difference based on the synthetic data and real data using a pre-built proxy model includes: training and testing the proxy model on the real data to obtain the baseline performance of the proxy model; training the proxy model on the synthetic data and testing it on the real data to obtain the actual performance of the proxy model; calculating the performance degradation ratio of the proxy model based on the baseline performance and the actual performance; classifying the synthetic data and real data using a domain classifier and calculating the classification accuracy; and obtaining the task-level domain difference based on the performance degradation ratio and / or the classification accuracy.
[0029] Among them, the baseline performance represents the model's performance on real data. The greater the decrease in actual performance relative to real performance, or the higher the classification accuracy of the domain classifier, the greater the damage to task performance, that is, the greater the task-level domain difference.
[0030] S104: Based on the multi-level domain differences, assess the main sources of the domain differences and evaluate the impact of the multi-level domain differences on the performance of the target model.
[0031] In one embodiment of this application, assessing the primary source of domain differences based on the multi-level domain differences includes: calculating the deviation value of each level of domain difference based on the benchmark value of the multi-level domain differences and the domain differences at each level; and determining the primary source of the domain differences based on the deviation value of each level of domain differences.
[0032] In one embodiment of this application, before calculating the deviation value of each level domain difference based on the multi-level domain difference and the benchmark value of each level domain difference, the method further includes: dividing the real data into two non-overlapping real data subsets based on the real data; calculating the level domain difference between the real data subsets as the benchmark value of each level domain difference.
[0033] Specifically, to assess the significance of domain differences at each level, two non-overlapping subsets of real data are first drawn from the actual data, and the domain differences at each level between these two subsets are used as benchmark values. Then, the domain differences at each level calculated in step S103 are compared with the corresponding benchmark values to obtain the deviation values of each level's domain differences. The larger the deviation value, the more significant the domain difference at that level; the domain difference corresponding to the largest deviation value is the main source of the domain difference. This multi-level domain difference assessment method can identify differences missed by single assessment methods, and in practical applications, the assessment coverage can be improved by more than 40%.
[0034] In one embodiment of this application, evaluating the impact of the multi-level domain differences on the performance of the target model includes: inputting the multi-level domain differences into a pre-trained prediction model to obtain the performance prediction value of the target model.
[0035] In one embodiment of this application, before inputting the multi-level domain differences into a pre-trained prediction model to obtain the performance prediction value of the target model, the method further includes: dividing the synthetic data into multiple sets of synthetic data subsets, and combining them with the real data to obtain multiple sets of multi-level domain differences corresponding to the multiple sets of synthetic data subsets; training the target model on the synthetic dataset and testing it on the real data to obtain the target model performance corresponding to the multiple sets of multi-level domain differences; and constructing a prediction model from the multi-level domain differences to the target model performance based on the multiple sets of multi-level domain differences and the target model performance corresponding to the multiple sets of multi-level domain differences.
[0036] Specifically, to establish the mapping relationship between multi-level domain differences and the performance of the target model, the synthetic data is first divided into multiple subsets, and the multi-level domain differences between each subset and the real data are calculated. Next, the target model is trained on each subset and its performance is tested on the real data to obtain the corresponding model performance metrics. Finally, the multiple sets of multi-level domain differences and their corresponding model performances are used as training samples to construct a predictive model from multi-level domain differences to the target model performance. This predictive model can be a machine learning model, such as a Multilayer Perceptron (MLP) model or a multinomial regression model. In practical applications, the multi-level domain differences between synthetic and real data can be input into this predictive model to dynamically evaluate their impact on the target model performance.
[0037] In a specific example, the synthetic data is randomly divided into 200 subsets, each containing 2000 randomly selected synthetic data points. The multi-level domain differences between each subset and the real data are calculated. Then, the target model is trained or fine-tuned using each subset, and its performance on the real data is tested, resulting in 200 data pairs relating multi-level domain differences to the target model's performance. Next, the MLP model is trained using these 200 data pairs, thus constructing a predictive model from multi-level domain differences to the target model's performance. Experimental results show that the prediction correlation coefficient obtained through the above steps can reach above 0.85, and the prediction error is controlled within 5%.
[0038] In one embodiment of this application, the method further includes: visualizing the evaluation results and generating an evaluation report.
[0039] Specifically, such as Figure 3As shown, the distribution of domain differences obtained in the above steps at each level is visualized, and a color heatmap is generated according to the level. At the same time, an evaluation report is automatically generated, including: overall domain difference score, detailed analysis of domain differences at each level, results of the sources of domain differences, predictive impact on the performance of the target model, and targeted optimization suggestions.
[0040] like Figure 5 As shown, the overall system architecture is divided into a multi-dimensional feature extraction module, a multi-level domain difference calculation module, a domain difference evaluation module, and an evaluation report generation module. In a specific example, the implementation methods of each module are as follows: The multi-dimensional feature extraction module is built on the PyTorch framework and supports GPU acceleration. The ViT-L / 16 model loads pre-trained weights, and its input image resolution supports 224×224 to 512×512. The VGG-19 model is used to extract texture features and calculate texture statistics using the Gram matrix. The MiDaS model performs monocular depth estimation and outputs a depth map with the same resolution as the input. The RAFT model calculates temporal features, and its output resolution is also the same as the input. In the multi-level domain difference measurement module, pixel-level domain difference measurement uses OpenCV to perform color histogram calculation and spectral analysis; feature-level domain difference measurement uses PyTorch to implement MMD and FID calculations and supports batch processing; semantic-level domain difference measurement uses a pre-trained detection and segmentation model; and task-level domain difference measurement uses a domain classifier built with a three-layer MLP. The performance impact prediction module also employs a three-layer MLP architecture. The input is a 16-dimensional domain-discrepancy feature vector, and the output is the predicted performance value of the target model. The model is trained on a collected domain-discrepancy-performance dataset, which contains evaluation results from 200 different synthetic datasets. In the visualization and report generation module, heatmaps are drawn using Matplotlib, supporting multiple color schemes. Evaluation reports are automatically generated by a template engine and support PDF or HTML output.
[0041] According to the domain difference assessment method for synthetic and real data according to embodiments of this application, the synthetic and real data to be assessed are first obtained; then, multi-dimensional features of the synthetic and real data are extracted, including texture features, geometric features, semantic features, and temporal features; next, multi-level domain differences between the synthetic and real data are obtained based on the statistical distribution differences and the multi-dimensional features; finally, the main sources of domain differences are assessed based on the multi-level domain differences, and the impact of multi-level domain differences on the performance of the target model is evaluated. Therefore, a comprehensive assessment of the domain differences between synthetic and real data can be performed from multiple dimensions and levels, thereby accurately determining the sources of domain differences and their impact on model performance, providing guidance for data generation, screening, and domain adaptation strategies.
[0042] Figure 6 This is a structural block diagram of a domain difference assessment system for synthetic data and real data according to an embodiment of this application. Figure 6 As shown, the domain difference evaluation system for synthetic data and real data according to an embodiment of this application includes: an acquisition module 610, a feature extraction module 620, a calculation module 630, and an evaluation module 640, wherein: The acquisition module 610 is used to acquire the synthetic data and real data to be evaluated; The feature extraction module 620 is used to extract multi-dimensional features of the synthetic data and real data based on the synthetic data and real data. The multi-dimensional features include texture features, geometric features, semantic features and temporal features. The calculation module 630 is used to obtain the multi-level domain difference between the synthetic data and the real data based on the statistical distribution difference between the synthetic data and the real data, as well as the multi-dimensional features. Evaluation module 640 is used to evaluate the main sources of the domain differences based on the multi-level domain differences, and to evaluate the impact of the multi-level domain differences on the performance of the target model.
[0043] According to the domain difference assessment method for synthetic and real data according to embodiments of this application, the synthetic and real data to be assessed are first obtained; then, multi-dimensional features of the synthetic and real data are extracted, including texture features, geometric features, semantic features, and temporal features; next, multi-level domain differences between the synthetic and real data are obtained based on the statistical distribution differences and the multi-dimensional features; finally, the main sources of domain differences are assessed based on the multi-level domain differences, and the impact of multi-level domain differences on the performance of the target model is evaluated. Therefore, a comprehensive assessment of the domain differences between synthetic and real data can be performed from multiple dimensions and levels, thereby accurately determining the sources of domain differences and their impact on model performance, providing guidance for data generation, screening, and domain adaptation strategies.
[0044] Specific limitations regarding the domain difference assessment system for synthetic and real data can be found in the limitations of the domain difference assessment method for synthetic and real data described above, and will not be repeated here. Each module of the aforementioned domain difference assessment system for synthetic and real data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory in software form, so that the processor can call and execute the corresponding operations of each module.
[0045] In one embodiment, a computer device is provided. Figure 7 This is a structural block diagram of the computer device provided in the embodiments of this application, with reference to... Figure 7 The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned embodiment of the domain difference evaluation method for synthetic data and real data. For example, it performs the following steps: acquiring synthetic data and real data to be evaluated; extracting multi-dimensional features of the synthetic data and real data based on the synthetic data and real data, the multi-dimensional features including texture features, geometric features, semantic features, and temporal features; obtaining multi-level domain differences between the synthetic data and real data based on the statistical distribution differences between the synthetic data and real data, and the multi-dimensional features; evaluating the main sources of domain differences based on the multi-level domain differences, and evaluating the impact of the multi-level domain differences on the performance of the target model.
[0046] This application also provides a computer-readable storage medium storing a computer program. When a processor executes the computer program, it implements the aforementioned embodiment of the domain difference evaluation method for synthetic data and real data. For example, it executes the following: acquiring synthetic data and real data to be evaluated; extracting multi-dimensional features of the synthetic data and real data based on the synthetic data and real data, the multi-dimensional features including texture features, geometric features, semantic features, and temporal features; obtaining multi-level domain differences between the synthetic data and real data based on the statistical distribution differences between the synthetic data and real data, and the multi-dimensional features; evaluating the main sources of the domain differences based on the multi-level domain differences, and evaluating the impact of the multi-level domain differences on the performance of the target model.
[0047] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0048] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0049] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for evaluating the domain difference between synthetic data and real data, characterized in that, include: Obtain the synthetic data and real data to be evaluated; Based on the synthetic data and the real data, multi-dimensional features of the synthetic data and the real data are extracted. The multi-dimensional features include texture features, geometric features, semantic features and temporal features. Based on the statistical distribution differences between the synthetic data and the real data, and the multi-dimensional features, the multi-level domain differences between the synthetic data and the real data are obtained. Based on the multi-level domain differences, assess the main sources of the domain differences and evaluate the impact of the multi-level domain differences on the performance of the target model.
2. The method for evaluating the domain difference between synthetic data and real data according to claim 1, characterized in that, The step of obtaining multi-level domain differences between the synthetic data and the real data based on the statistical distribution differences between the synthetic data and the real data, and the multi-dimensional features, includes: Based on the statistical distribution differences between the synthesized data and the real data, the pixel-level domain differences between the synthesized data and the real data are obtained; Based on the texture features and the geometric features, the feature-level domain differences between the synthetic data and the real data are obtained; Based on the semantic features, the semantic-level domain differences between the synthetic data and the real data are obtained; Based on the synthetic data and the real data, the task-level domain differences between the synthetic data and the real data are obtained through a pre-built proxy model.
3. The method for evaluating the domain difference between synthetic data and real data according to claim 2, characterized in that, The step of obtaining the task-level domain difference between the synthetic data and the real data using a pre-built proxy model includes: The proxy model is trained and tested on the real data to obtain the benchmark performance of the proxy model; The proxy model is trained on the synthetic data and tested on the real data to obtain the actual performance of the proxy model; Calculate the performance degradation rate of the proxy model based on the baseline performance and the actual performance; The synthetic data and real data are classified using a domain classifier, and the classification accuracy is calculated. The task-level domain difference is obtained based on the performance degradation rate and / or the classification accuracy.
4. The method for evaluating the domain difference between synthetic data and real data according to claim 1, characterized in that, The assessment of the primary sources of domain differences based on the multi-level domain differences includes: Based on the reference values of the differences between the multi-level domains and the differences between each level domain, the deviation value of the differences between each level domain is calculated. Based on the deviation values of the domain differences at each level, the main sources of the domain differences are determined.
5. The method for evaluating the domain difference between synthetic data and real data according to claim 4, characterized in that, Before calculating the deviation value of each level-domain difference based on the benchmark values of the multi-level domain differences and the differences of each level-domain, the method further includes: Based on the real data, two disjoint subsets of real data are obtained. Calculate the differences between the hierarchical domains of the real data subsets, and use these differences as the baseline values for the differences between the hierarchical domains.
6. The method for evaluating the domain difference between synthetic data and real data according to claim 1, characterized in that, The assessment of the impact of the multi-level domain differences on the performance of the target model includes: The multi-level domain differences are input into the pre-trained prediction model to obtain the performance prediction value of the target model.
7. The method for evaluating the domain difference between synthetic data and real data according to claim 6, characterized in that, Before inputting the multi-level domain differences into the pre-trained prediction model to obtain the performance prediction value of the target model, the method further includes: The synthetic data is divided into multiple sets of synthetic data subsets, and combined with the real data, multiple sets of multi-level domain differences corresponding to the multiple sets of synthetic data subsets are obtained. The target model is trained on the synthetic dataset and tested on the real data to obtain the target model performance corresponding to the differences between the multiple sets of multi-level domains; Based on the multiple sets of multi-level domain differences and the target model performance corresponding to the multiple sets of multi-level domain differences, a prediction model from the multi-level domain differences to the target model performance is constructed.
8. The method for evaluating the domain difference between synthetic data and real data according to any one of claims 1-7, characterized in that, Also includes: The evaluation results are visualized and an evaluation report is generated.
9. A domain difference assessment system for synthetic data and real data, characterized in that, include: The acquisition module is used to acquire the synthetic data and real data to be evaluated; The feature extraction module is used to extract multi-dimensional features of the synthetic data and real data based on the synthetic data and real data. The multi-dimensional features include texture features, geometric features, semantic features and temporal features. The calculation module is used to obtain the multi-level domain differences between the synthetic data and the real data based on the statistical distribution differences between the synthetic data and the real data, as well as the multi-dimensional features. The evaluation module is used to evaluate the main sources of the domain differences based on the multi-level domain differences, and to evaluate the impact of the multi-level domain differences on the performance of the target model.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the domain difference evaluation method between synthetic data and real data according to any one of claims 1-8.