Method, device and equipment for testing synthetic image data set and storage medium

Abstract

The invention relates to the technical field of composite image data set testing, and discloses a composite image data set testing method, device and equipment and a storage medium, and the method comprises the steps: obtaining a composite image data set, a real image data set and a deep learning model; training the deep learning model to convergence based on the synthetic image training set to obtain a first convergence model, and training the deep learning model to convergence based on each real image training set to obtain a corresponding second convergence model; testing a first convergence model based on one of the synthetic image test set and each real image data set to obtain corresponding first test data; testing each second convergence model based on the synthesized image data set to obtain corresponding second test data; testing each second convergence model based on the real image data set to obtain corresponding third test data; and performing weighted average based on the first test data, the second test data and the third test data to obtain a test result of the composite image data set.

Description

Technical Field

[0001] The present invention relates to the technical field of synthetic image data set testing, and in particular to a method, an apparatus, a device and a storage medium for synthetic image data set testing. Background Art

[0002] Synthetic image datasets are computer-generated datasets that can be tailored to specific needs and can be created quickly and cheaply compared to collecting and labeling real-world data.

[0003] The quality testing technology of synthetic image datasets is of great significance in the development of deep learning object detection perception algorithms. Due to the substantial increase in hardware computing power, the demand for the number of datasets in deep learning has increased. Therefore, synthetic image datasets have become a common means of training models or improving model performance based on generating and supplementing data.

[0004] However, there is a lack of research on quality testing of synthetic image datasets in the related art. Summary of the invention

[0005] In view of this, the present invention provides a method, an apparatus, a device and a storage medium for testing a synthetic image dataset to solve the problem that there is a lack of a method for testing a synthetic image dataset in the related art.

[0006] In a first aspect, the present invention provides a method for testing a synthetic image dataset, the method comprising: obtaining a synthetic image dataset, a real image dataset and a deep learning model, the synthetic image dataset comprising: a synthetic image training set and a synthetic image test set, the real image dataset comprising: a real image training set, and the number of the real image datasets is at least one; training the deep learning model to convergence based on the synthetic image training set to obtain a first converged model, and training the deep learning model to convergence based on each of the real image training sets to obtain a corresponding second converged model; testing the first converged model based on the synthetic image test set and one of the real image datasets to obtain corresponding first test data; testing each of the second converged models based on the synthetic image dataset to obtain corresponding second test data; testing each of the second converged models based on the real image dataset to obtain corresponding third test data; and performing weighted averaging based on the first test data, the second test data and the third test data to obtain a test result of the synthetic image dataset.

[0007] In an optional embodiment, the real image dataset also includes: a real image test set; testing each of the second convergence models based on the real image dataset to obtain corresponding third test data, including: testing the target second convergence model based on the target real image test set in the target real image dataset to obtain target third test data, wherein the target second convergence model is obtained by training the deep learning model to convergence using the target real image training set in the target real image dataset; testing the target second convergence model based on other real image datasets except the target real image dataset to obtain initial third test data.

[0008] In an optional embodiment, the first convergence model is tested based on the synthetic image test set and one of the real image data sets to obtain corresponding first test data, including: testing the first convergence model based on the synthetic image test set to obtain target first test data; testing the first convergence model based on each of the real image data sets to obtain corresponding initial first test data.

[0009] In an optional embodiment, a test result of the synthetic image dataset is obtained by weighted averaging the first test data, the second test data, and the third test data, including: obtaining a corresponding first similarity based on a ratio of each of the initial first test data to the target first test data, obtaining a corresponding second similarity based on a ratio of each of the second test data to the target third test data, and determining a first weight coefficient based on each of the second similarities; and obtaining a first test result of the synthetic image dataset by weighted summing the first similarities based on the first weight coefficient.

[0010] In an optional embodiment, a test result of the synthetic image data set is obtained by taking a weighted average based on the first test data, the second test data and the third test data, and further includes: obtaining a corresponding third similarity based on a ratio of each of the initial third test data to the target third test data, and determining a second weight coefficient based on each of the third similarities; and taking a weighted sum of the first similarities based on the second weight coefficient to obtain a second test result of the synthetic image data set.

[0011] In an optional implementation, performing weighted averaging based on the first test data, the second test data, and the third test data to obtain a test result of the synthetic image data set further includes: obtaining the first test result A0 by calculating using the following formula:

[0012]

[0013] Where i = 1, 2, ..., N, wi is the first weight coefficient, R oi For the first similarity, the first weight coefficient is determined based on the similarity of the test data obtained by testing the converged deep learning model based on the synthetic image dataset and each of the real image datasets.

[0014] In an optional implementation, the test result of the synthetic image data set is obtained by weighted averaging the first test data, the second test data, and the third test data, further comprising: the second test result S o Calculated by the following formula:

[0015]

[0016] Among them, v i is the second weight coefficient, R oi is the first similarity, and N is the number of the real image data sets.

[0017] In a second aspect, the present invention provides a device for testing a synthetic image dataset, the device comprising: an acquisition module, used to acquire a synthetic image dataset, a real image dataset and a deep learning model, the synthetic image dataset comprising: a synthetic image training set and a synthetic image test set, the real image dataset comprising: a real image training set, and the number of the real image datasets is at least one; a training module, used to train the deep learning model to convergence based on the synthetic image training set to obtain a first converged model, and to train the deep learning model to convergence based on each of the real image training sets to obtain a corresponding second converged model; a test data acquisition module, used to test the first converged model based on the synthetic image test set and one of the real image datasets to obtain corresponding first test data; test each of the second converged models based on the synthetic image dataset to obtain corresponding second test data; test each of the second converged models based on the real image dataset to obtain corresponding third test data; a testing module, used to perform weighted averaging based on the first test data, the second test data and the third test data to obtain a test result of the synthetic image dataset.

[0018] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for testing a synthetic image data set according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for testing a synthetic image data set according to the first aspect or any corresponding embodiment thereof.

[0020] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for testing a synthetic image data set according to the first aspect or any corresponding embodiment thereof.

[0021] The embodiment of the present invention proposes a method for testing a synthetic image dataset based on a generalized cross mutual validation method. The method is based on the inverse process of deep learning model transfer learning, and the test data is obtained by testing the same deep learning model on different target domain datasets. Cross-validation is performed based on the test data to obtain test results for the synthetic image dataset. The method does not rely on expert experience and has objectivity and universality. The test results can characterize the quality of the synthetic image dataset and reduce, to a certain extent, the deviations and errors of the deep learning models trained with the synthetic image dataset in real physical world applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic flow chart of a method for testing a synthetic image data set provided by an embodiment of the present invention is shown;

[0024] Figure 2 A schematic diagram of another method flow for testing a synthetic image data set provided by an embodiment of the present invention is shown;

[0025] Figure 3 is a schematic diagram of the structure of a device for testing a synthetic image data set provided by an embodiment of the present invention;

[0026] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0028] The quality of synthetic image datasets affects the performance and expression of deep learning models. The biases and errors in synthetic image datasets may also cause deviations in the application of deep learning models in the real physical world, which in turn may lead to some erroneous consequences.

[0029] For the research on image dataset testing methods, an earlier and more popular method is to use the analytic hierarchy process. For the dataset to be tested, the analytic hierarchy process or its derivative fuzzy analytic hierarchy process is used to set artificial weights and consistency tests for different indicators of interest according to the tester, and then test the dataset. However, this method relies on expert experience to determine the indicator selection and weight setting, and is still highly subjective. The universality of testing different datasets needs to be improved.

[0030] In addition, in the process of generating some synthetic image datasets, there are also some experiments that characterize the effectiveness of the datasets by training the model with synthetic image datasets. For the effectiveness and credibility of the datasets, some scholars have tested the data integrity, label richness, and scene integrity.

[0031] Based on the current lack of a universally recognized quantitative testing method for synthetic image datasets, the present invention provides a method embodiment for testing synthetic image datasets. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0032] In this embodiment, a method for testing a synthetic image dataset is provided, which can be used in a terminal with an operating system, such as a mobile phone, a desktop computer, a notebook, or a tablet computer. Figure 1 FIG. 4 is a flow chart of a method for testing a synthetic image dataset provided by an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0033] Step S101, obtaining a synthetic image dataset, a real image dataset and a deep learning model, the synthetic image dataset includes: a synthetic image training set and a synthetic image test set, the real image dataset includes: a real image training set, and the number of the real image dataset is at least one.

[0034] In this step, a deep learning model of the task concerned by the synthetic image dataset D0 is selected. The selected deep learning model is a classic deep learning model with advanced performance in the training task concerned by the synthetic image dataset or a more common deep learning model in the current training task. For example, it is a target detection deep learning model in the target detection task.

[0035] Based on the deep learning model for the target training task, at least one real image dataset is determined. The real image dataset and the synthetic image dataset have the same category annotations or have category intersections. The real image dataset has good representativeness in the deep learning task of interest and can be effectively trained in the training sets of the synthetic image dataset and the real image dataset.

[0036] For example, select N real image datasets D that have the same domain and detection target category as the synthetic image dataset (if the synthetic image dataset is a synthetic image dataset for target detection of an autonomous driving vehicle, an image dataset containing an intersection target category of vehicles or pedestrians may be selected) or have an intersection with the calibration target category of the synthetic image dataset. i , i=1,2,……,N.

[0037] The following is an exemplary explanation of this step. For example, for a deep learning model for a target detection task, several real target detection image datasets that have the same target detection category flag as the target detection synthetic image dataset, or that have a target detection category intersection with the synthetic image dataset, can be selected as real image datasets. The intersection of the target detection annotation types of the synthetic image dataset and the real image dataset is obtained, and the data and labels of the synthetic image dataset are screened and cleaned based on the annotation intersection, and image data without intersection labels are eliminated, and image data with intersection category detection annotations and the labels corresponding to the aforementioned image data are retained. The data and labels of each filtered dataset are divided into training sets and test sets. When dividing, it should be noted that the data size of each training set should be the same, and it should be ensured that the remaining test set data has corresponding labels.

[0038] Step S103: Train the deep learning model to convergence based on the synthetic image training set to obtain a first converged model, and train the deep learning model to convergence based on each real image training set to obtain a corresponding second converged model.

[0039] In this step, the number of real image data sets is at least one, the number of obtained second convergence models is also at least one, and the number of real image data sets is the same as the number of second convergence models. Based on the training set of the synthetic image data set D0, the selected deep learning model is set according to the intersection category and initialized for training until the model converges to obtain a first convergence model M0.

[0040] Reuse the real image dataset D i The selected deep learning model is trained according to the intersection category and initialized until the model converges to obtain the corresponding second convergence model M i .

[0041] Step S105, test the first convergence model based on the synthetic image test set and one of the real image data sets to obtain corresponding first test data; test each second convergence model based on the synthetic image data set to obtain corresponding second test data; test each second convergence model based on the real image data set to obtain corresponding third test data.

[0042] In this step, the number of first test data is multiple, and the first test data includes: testing the first convergence model based on the synthetic image test set to obtain target first test data; testing the first convergence model based on each real image data set to obtain corresponding initial first test data. The first test data includes one target first test data and multiple initial first test data, and the number of first test data is the sum of one target first test data and multiple initial first test data.

[0043] The number of the second test data is the same as that of the real image dataset.

[0044] A general test index P of the deep learning model can be selected, and the performance of the convergence model can be tested based on the relevant data set and the general test index to obtain corresponding test data. The general test index P of the deep learning model can be, but is not limited to, one or more of accuracy, precision, recall, harmonic mean of precision and recall, ROC curve and AUC value, and average accuracy of target detection. The performance of each second convergence model M0 is tested based on the synthetic image data set D0 to obtain the corresponding second test data P i0 .

[0045] Step S107 , performing weighted averaging based on the first test data, the second test data, and the third test data to obtain a test result of the synthetic image data set.

[0046] In this step, based on the calculation method of the first test data, the second test data and the third test data, the number of the first test data, the second test data and the third test data is (N+1) 2Based on the first test data, the second test data and the third test data, a test data matrix can be obtained, and the test data matrix is ​​as follows:

[0047] Table 1 Test data matrix

[0048]

[0049] The first test data, the second test data, and the third test data are converted into data having the same scale. For example, the data in the test data matrix may be normalized, standardized, or regularized so that the test data have the same scale, making different test data comparable and facilitating comprehensive evaluation or analysis.

[0050] From Table 1, it can be seen that the first test data, the second test data, and the third test data are calculated in the following way: the first test data, the second test data, or the third test data, the row where it is located represents the training set represented by the row where it is located, and the column where it is located represents the test set represented by the column where it is located. The row where the first test data is located is the normalized test data of the first converged model on the synthetic image data set and the remaining N real image data sets.

[0051] As can be seen from Table 1, each row of test data in the matrix comes from the same training set model, and each column of test data comes from the same test set test. The row where the first test data is located and the column where the second test data is located are the cross-validation results of the synthetic image dataset in the real image dataset.

[0052] The embodiment of the present invention proposes a method for testing a synthetic image dataset based on a generalized cross mutual validation method. The method is based on the inverse process of deep learning model transfer learning, and the test data is obtained by testing the same deep learning model on different target domain datasets. Cross-validation is performed based on the test data to obtain test results for the synthetic image dataset. The method does not rely on expert experience and has objectivity and universality. The test results can characterize the quality of the synthetic image dataset and reduce, to a certain extent, the deviations and errors of the deep learning models trained with the synthetic image dataset in real physical world applications.

[0053] In an optional embodiment, the real image dataset also includes: a real image test set; testing each second convergence model based on the real image dataset to obtain corresponding third test data, including: testing the target second convergence model based on the target real image test set in the target real image dataset to obtain target third test data, wherein the target second convergence model is obtained by training the deep learning model to convergence with the target real image training set in the target real image dataset; testing the target second convergence model based on other real image datasets except the target real image dataset to obtain initial third test data.

[0054] In this embodiment, the target real image dataset can be D i For each of the above, the target third test data is P ii For example, the deep learning model is trained to convergence by the target real image training set in the target real image data set D3 to obtain the target second convergence model M3, and the target second convergence model M3 is tested based on the target real image test set in the target real image data set to obtain the target third test data P 33 .

[0055] Cross-test each second convergence model on each real image dataset, and the initial third test data obtained is P ij , j = 1, 2, ..., N. Based on other real image datasets other than the target real image dataset, for example, D2 tests the target second convergence model M3, and the initial third test data obtained is P 32 , or D4 test target second convergence model M3, the initial third test data obtained is P 34 The obtained test data matrix is ​​shown in Table 2 below:

[0056] Table 2 Test data matrix

[0057]

[0058]

[0059] In an optional embodiment, the first convergence model is tested based on a synthetic image test set and one of the real image data sets to obtain corresponding first test data, including: testing the first convergence model based on the synthetic image test set to obtain target first test data; testing the first convergence model based on each real image data set to obtain corresponding initial first test data.

[0060] In this embodiment, a general test index P is selected, and the performance of the first convergence model M0 is tested based on the synthetic image data set D0 to obtain the target first test data P 00 Based on the remaining N real data sets, the performance of the first convergence model M0 is tested respectively to obtain the corresponding initial first test data P 0i , i = 1, 2, ..., N. The obtained test data matrix is ​​shown in Table 3 below:

[0061] Table 3 Test data matrix

[0062]

[0063]

[0064] In an optional embodiment, a weighted average is performed based on the first test data, the second test data, and the third test data to obtain a test result of the composite image data set, including: obtaining a corresponding first similarity based on a ratio of each initial first test data to the target first test data, obtaining a corresponding second similarity based on a ratio of each second test data to the target third test data, and determining a first weight coefficient based on each second similarity; and obtaining a first test result of the composite image data set by weighted summation of the first similarities based on the first weight coefficient.

[0065] In this embodiment, the first similarity R oi The calculation method of is as follows:

[0066]

[0067] Among them, P 0i is the initial first test data, P 00 is the target first test data. The second similarity R io It can be calculated by the following formula:

[0068]

[0069] Among them, P io is the second test data, P ii The third test data for the target.

[0070] Since the absolute performance of different deep learning models is related to the number of model parameters and structural design, the embodiments of the present invention focus more on the impact of different image datasets on the performance of deep learning models, and use this as an indicator to reflect the quality of the synthetic image dataset.

[0071] Therefore, for the algorithm performance of migrating the synthetic image data set to the real image data set in the test data matrix, the test data is normalized to obtain the normalized migration performance of the deep learning model, that is, the first similarity R oi The first similarity can reflect the similarity and substitutability between the synthetic image dataset and the real image dataset through the migration effect of the synthetic image dataset in the corresponding real image dataset.

[0072] In most cases, the first similarity is less than 1, that is, due to the domain deviation of the training set and the test set, the test effect of the deep learning model on different non-training image datasets will decrease to varying degrees. The closer the synthetic image dataset is to the real image dataset, and the better the data diversity and quality of the synthetic image dataset, the larger the value. If the first similarity is greater than 1, it means that the data in the synthetic image dataset has good coverage of the real image dataset, and the data diversity, difficulty and quality of the synthetic image dataset are higher than those of the real image dataset, and it has good substitutability for the real image dataset.

[0073] Based on the elements in the test data matrix in Table 3, the normalized migration performance of the deep learning model is calculated to obtain a generalized cross validation matrix, where each element in the generalized cross validation matrix is ​​the migration performance of the deep learning model normalized according to the training set of the data set, and each element in its row represents the migration from the training set represented by the row, and its column represents the migration to the test set represented by the column. The diagonal elements of the generalized cross validation matrix are 1, and the first row is the normalized migration performance of the deep learning model on the synthetic image data set and each real image data set. The generalized cross validation matrix is ​​shown in Table 4 below:

[0074] Table 4 Generalized cross validation matrix

[0075]

[0076] The migration performance of the deep learning model can be converted into the migration performance between the synthetic image dataset and each real image dataset through the generalized cross validation matrix. The model migration performance is converted into the similarity and migration evaluation between datasets, generating universal, quantifiable and comparable synthetic image dataset test results.

[0077] In an optional embodiment, a test result of a synthetic image data set is obtained by weighted averaging the first test data, the second test data, and the third test data, and the method also includes: obtaining a corresponding third similarity based on a ratio of each initial third test data to the target third test data, and determining a second weight coefficient based on each third similarity; and obtaining a second test result of the synthetic image data set by weighted summation of the first similarities based on the second weight coefficient.

[0078] In this embodiment, the third similarity R ij Characterize the real image dataset D i and the real image data D j For the migration relationship of the same convergent deep learning model, the third similarity R ij The larger the value, the higher the D i Right D j It has better similarity and coverage.

[0079] In an optional implementation, the weighted average is performed based on the first test data, the second test data, and the third test data to obtain the test result of the synthetic image data set, further comprising: the first test result A0 is calculated by the following formula:

[0080]

[0081] Where i = 1, 2, ..., N, w i is the first weight coefficient, R oi is the first similarity, the first weight coefficient, determined based on the similarity of the test data obtained by testing the converged deep learning model based on the synthetic image dataset and each real image dataset. The first weight coefficient w i , can be determined based on the similarity of the deep learning model performance test of the synthetic image dataset and each real image dataset. The normalized migration performance of the deep learning model of different real image datasets is weighted and summed according to the migration weight of the synthetic image dataset to obtain the simulation performance test result of the synthetic image dataset. That is, the first similarity is weighted and summed based on the first weight coefficient to obtain the first test result A0.

[0082] Since the scenes and data in the real world are very diverse, the domain distribution, focus and difficulty of different real image datasets for the same task are also different. Therefore, for a synthetic image dataset, if it is close enough to any real image dataset, it can be said that the image and label authenticity of the synthetic image dataset has good performance, and it is not necessary to perform well in all real image datasets.

[0083] The first weight coefficient w can also be calculated by the following formula: i :

[0084]

[0085] Among them, R io is the second similarity, ∑ j R jo is the sum of the second similarity. The second similarity R io , reflects the migration similarity of the real image dataset to the deep learning model performance test of the synthetic image dataset. The larger the second similarity, the smaller the change of the domain migration of the image data on the performance of the deep learning model, and the more similar the real image dataset and the synthetic image dataset are for the same deep learning model. Therefore, the first weight coefficient w i It can be determined based on the normalization of the migration performance of each real image dataset to the synthetic image dataset.

[0086] The better the migration performance of the real image dataset to the synthetic image dataset, the smaller the domain deviation of the synthetic image dataset in the synthetic image dataset, and the better the substitutability for the synthetic image dataset. Therefore, the more important the imitation and migration performance of the synthetic image dataset to the real image dataset is in reflecting the authenticity of the synthetic image dataset, the larger the corresponding first weight coefficient is.

[0087] The calculation formula of the first test result A0 can quantify the data similarity between the synthetic image dataset and the real image dataset. Based on the first test result, it can be determined whether the image data in the synthetic image dataset is sufficiently real or close to a certain real image dataset.

[0088] In an optional implementation, the test result of the synthetic image data set is obtained by weighted averaging the first test data, the second test data, and the third test data, and further includes: the second test result S o Calculated by the following formula:

[0089]

[0090] Among them, v i is the second weight coefficient, R oi is the first similarity, and N is the number of real image datasets.

[0091] Based on the second weight coefficient v i , calculate the sum of the quality coefficients of each real image dataset as the source domain migration, and normalize them to obtain the second test result. The larger the second weight coefficient is, the more the migration performance of the synthetic image dataset on the real image dataset can reflect the migration ability of the synthetic image dataset on different datasets.

[0092] The test of the synthetic image dataset also includes the diversity and coverage of its synthetic image data, that is, whether the synthetic image dataset can have good training complementarity for different real image datasets with deep learning task training data intersection, or whether the synthetic image dataset can be well migrated to various real image dataset task scenarios. The second weight coefficient can reflect the diversity and migration of the synthetic image dataset.

[0093] The second weight coefficient v i The second weight coefficient v is determined based on the migration relationship between each real image data set. i The calculation formula is as follows:

[0094]

[0095] Among them, v i is the second weight coefficient, R oiis the first similarity, R ij is the third similarity, P ij is the initial third test data, P ii The third test data is the target, C ij is the intermediate coefficient.

[0096] Intermediate coefficient C ij Characterizing the importance of dataset migration in a pair of real image datasets can be based on the intermediate coefficient C ij Determine the real image dataset D i and the real image data D j If a real image dataset has better coverage and migration performance as the source domain data for another real image dataset, the test data of the real image dataset will have a greater weight in the test results. ij The symmetry between different real image datasets can reflect the cross-relationship between multiple real image datasets.

[0097] This embodiment also provides another method for testing a synthetic image dataset. Figure 2 A schematic flow chart of another method for testing a synthetic image dataset provided by an embodiment of the present invention is shown.

[0098] like Figure 2 As shown, the method used for testing the synthetic image dataset includes:

[0099] Step S201, selecting N initial target detection real image datasets with intersections according to the target detection task annotation categories of the initial synthetic image dataset;

[0100] Step S202, screening and cleaning the initial synthetic image dataset and the initial target detection real image dataset based on the intersection of the target detection task annotation categories to obtain a synthetic image dataset and a real image dataset, wherein the synthetic image dataset includes: a synthetic image training set and a synthetic image test set, and the real image dataset includes: a real image training set and a real image test set;

[0101] Step S203, selecting an image object detection model based on deep learning for the object detection task concerned by the synthetic image dataset;

[0102] Step S204, performing a cross validation test based on the synthetic image data set, that is, training the image target detection model based on the synthetic image data set until convergence, and testing the performance index P of the converged image target detection model on the synthetic image test set and the real image test set, respectively, to obtain first test data;

[0103] Step S205, for each real image data set, a generalized cross mutual validation test is performed, that is, for each real image data set, the image object detection model is retrained until convergence, and the performance index P of the converged image object detection model is tested on the synthetic image data set and the real image data set, respectively, to obtain second test data and third test data;

[0104] Step S206, based on the first test data, the second test data and the third test data, calculate the first similarity, the second similarity and the third similarity; calculate the first weight coefficient based on the second similarity, and weighted sum the first similarity based on the first weight coefficient to obtain a first test result characterizing the simulation quality of the synthetic image data set; calculate the second weight coefficient based on the first similarity and the third similarity, and weighted sum the first similarity based on the second weight coefficient to obtain a second test result characterizing the migration quality of the synthetic image data set.

[0105] A synthetic image dataset, at least one real image dataset and a deep learning model for the same deep learning task are selected, and the cross-performance indicators of the deep learning model on the synthetic image dataset and each real image dataset are tested based on the migration test. The similarity of the test data is calculated using the established normalized generalized cross mutual validation matrix, which realizes the test of the quality of the synthetic image dataset and quantifies the test results of the synthetic image dataset, reflecting the authenticity of the synthetic image dataset and the migration between the synthetic image dataset and the real image dataset.

[0106] The following is an exemplary description of an embodiment of the present invention: a synthetic image dataset Virtual KITTI is obtained, and a deep learning model for image target detection in an autonomous driving scenario based on the synthetic image dataset is selected, and real image datasets are selected: autonomous driving image target detection dataset KITTI and BDD100K.

[0107] The intersection target detection category is selected as vehicle, and the image data and labels of the three data sets are screened and cleaned to remove irrelevant labels and image data. According to the scale of the screened synthetic image data set, the training set size is selected as 7400, and each real image data set is divided.

[0108] According to the target detection category of the synthetic image dataset, the deep learning model selects the YOLOv5s model of the classic image target detection algorithm YOLO series, and the performance indicator P is selected as the target detection accuracy AP50. The YOLOv5 model is trained using the selected synthetic image dataset and the training set of the real image dataset to obtain three converged models, and the performance tests are performed on each dataset. The cross performance matrix is ​​shown in Table 5:

[0109] Table 5 Cross-performance matrix

[0110]

[0111] Based on the calculation method of the first similarity, the second similarity and the third similarity, the generalized cross mutual confirmation matrix is ​​obtained as shown in Table 6:

[0112] Table 6 Generalized cross-validation matrix

[0113]

[0114]

[0115] Based on the calculation method of the first test result and the second test result, it is calculated that

[0116]

[0117] In this way, the first test result of the synthetic image dataset Virtual KITTI, which is 0.49 and the second test result of the synthetic image dataset migration quality, is 0.45, which is 0.49. The quality test of the synthetic image dataset is realized, and the test results of the synthetic image dataset are quantified, which reflects the authenticity of the synthetic image dataset and the migration between the synthetic image dataset and the real image dataset.

[0118] In this embodiment, a device for synthetic image data set testing is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0119] This embodiment provides a device for testing a synthetic image dataset. Figure 3 FIG. 1 is a schematic diagram of the structure of a device for testing a synthetic image data set provided by an embodiment of the present invention. Figure 3 As shown, including:

[0120] The acquisition module 301 is used to acquire a synthetic image dataset, a real image dataset and a deep learning model. The synthetic image dataset includes: a synthetic image training set and a synthetic image test set. The real image dataset includes: a real image training set. The number of real image datasets is at least one.

[0121] The training module 302 is used to train the deep learning model to convergence based on the synthetic image training set to obtain a first converged model, and to train the deep learning model to convergence based on each real image training set to obtain a corresponding second converged model.

[0122] The test data acquisition module 303 is used to test the first convergence model based on the synthetic image test set and one of the real image data sets to obtain the corresponding first test data; test each second convergence model based on the synthetic image data set to obtain the corresponding second test data; test each second convergence model based on the real image data set to obtain the corresponding third test data.

[0123] The testing module 304 is used to perform weighted averaging based on the first test data, the second test data and the third test data to obtain a test result of the synthetic image data set.

[0124] In some optional implementations, the test data acquisition module 303 includes:

[0125] The first test data acquisition unit is used to test the target second convergence model based on the target real image test set in the target real image data set to obtain target third test data, wherein the target second convergence model is obtained by training the deep learning model to convergence with the target real image training set in the target real image data set; and test the target second convergence model based on other real image data sets except the target real image data set to obtain initial third test data.

[0126] In some optional implementations, the test data acquisition module 303 further includes:

[0127] The second test data acquisition unit is used to test the first convergence model based on the synthetic image test set to obtain target first test data; and test the first convergence model based on each real image data set to obtain corresponding initial first test data.

[0128] In some optional implementations, the testing module 304 includes:

[0129] The first test unit is used to obtain the corresponding first similarity based on the ratio of each initial first test data to the target first test data, obtain the corresponding second similarity based on the ratio of each second test data to the target third test data, determine the first weight coefficient based on each second similarity; and perform weighted summation of the first similarities based on the first weight coefficient to obtain the first test result of the synthetic image data set.

[0130] In some optional implementations, the testing module 304 further includes:

[0131] The second test unit is used to obtain the corresponding third similarities based on the ratio of each initial third test data to the target third test data, determine the second weight coefficient based on each third similarity; and perform weighted summation of the first similarities based on the second weight coefficient to obtain the second test result of the synthetic image data set.

[0132] In some optional implementations, the testing module 304 further includes:

[0133] The third test unit is used to calculate the first test result A0 by the following formula:

[0134]

[0135] Where i = 1, 2, ..., N, w i is the first weight coefficient, R oi is a first similarity, a first weight coefficient, and is determined based on the similarity of the test data obtained by testing the converged deep learning model based on the synthetic image dataset and each real image dataset.

[0136] In some optional implementations, the testing module 304 further includes:

[0137] The fourth test unit is used to calculate the second test result S by the following formula: o :

[0138]

[0139] Among them, v i is the second weight coefficient, R oi is the first similarity, and N is the number of real image datasets.

[0140] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0141] The embodiment of the present invention proposes a device for testing a synthetic image dataset based on a generalized cross mutual validation method. The device is based on the inverse process of deep learning model transfer learning, and obtains test data by testing the same deep learning model on different target domain datasets. Cross-validation is performed based on the test data to obtain test results for the synthetic image dataset. The device does not rely on expert experience and has objectivity and universality. The test results can characterize the quality of the synthetic image dataset and reduce, to a certain extent, the deviations and errors of the deep learning models trained with the synthetic image dataset in real physical world applications.

[0142] The device for synthetic image data set testing in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0143] The embodiment of the present invention also provides a computer device having the above Figure 3 The apparatus shown is used for testing on a synthetic image dataset.

[0144] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0145] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0146] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.

[0147] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0148] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0149] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0150] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0151] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0152] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0153] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for testing a synthetic image dataset, characterized in that: The method comprises: Obtain a synthetic image dataset, a real image dataset, and a deep learning model, wherein the synthetic image dataset includes: a synthetic image training set and a synthetic image test set, and the real image dataset includes: a real image training set, and the number of the real image dataset is at least one; Training the deep learning model to convergence based on the synthetic image training set to obtain a first converged model, and training the deep learning model to convergence based on each of the real image training sets to obtain a corresponding second converged model; Testing the first convergence model based on the synthetic image test set and one of the real image data sets to obtain corresponding first test data; testing each of the second convergence models based on the synthetic image data sets to obtain corresponding second test data; and testing each of the second convergence models based on the real image data sets to obtain corresponding third test data; A test result of the synthetic image dataset is obtained by performing weighted averaging based on the first test data, the second test data, and the third test data.

2. The method according to claim 1, characterized in that The real image dataset also includes: a real image test set; Testing each of the second convergence models based on the real image dataset to obtain corresponding third test data includes: Testing a target second converged model based on a target real image test set in a target real image dataset to obtain target third test data, wherein the target second converged model is obtained by training the deep learning model to convergence using a target real image training set in the target real image dataset; The target second convergence model is tested based on other real image datasets except the target real image dataset to obtain initial third test data.

3. The method according to claim 2, characterized in that Testing the first convergence model based on the synthetic image test set and one of the real image data sets to obtain corresponding first test data includes: Testing the first convergence model based on the synthetic image test set to obtain target first test data; The first convergence model is tested based on each of the real image data sets to obtain corresponding initial first test data.

4. The method according to claim 3, characterized in that Obtaining a test result of the synthetic image dataset by performing weighted averaging based on the first test data, the second test data, and the third test data, including: Obtaining a corresponding first similarity based on a ratio of each of the initial first test data to the target first test data, obtaining a corresponding second similarity based on a ratio of each of the second test data to the target third test data, and determining a first weight coefficient based on each of the second similarities; A first test result of the synthetic image data set is obtained by weighted summing the first similarities based on the first weight coefficient.

5. The method according to claim 4, characterized in that The method further comprises: performing weighted averaging based on the first test data, the second test data, and the third test data to obtain a test result of the synthetic image dataset; obtaining a corresponding third similarity based on a ratio of each of the initial third test data to the target third test data, and determining a second weight coefficient based on each of the third similarities; A second test result of the synthetic image data set is obtained by weighted summing the first similarities based on the second weight coefficient.

6. The method according to claim 4, characterized in that The method further comprises: performing weighted averaging based on the first test data, the second test data, and the third test data to obtain a test result of the synthetic image dataset; The first test result A0 is calculated by the following formula: Where i = 1, 2, ..., N, w i is the first weight coefficient, R oi For the first similarity, the first weight coefficient is determined based on the similarity of the test data obtained by testing the converged deep learning model on the synthetic image dataset and each of the real image datasets.

7. The method according to claim 5, characterized in that The method further comprises: performing weighted averaging based on the first test data, the second test data, and the third test data to obtain a test result of the synthetic image dataset; The second test result S o Calculated by the following formula: Among them, v i is the second weight coefficient, R oi is the first similarity, and N is the number of the real image datasets.

8. A device for testing a synthetic image dataset, characterized in that: The device comprises: An acquisition module is configured to acquire a synthetic image dataset, a real image dataset, and a deep learning model, wherein the synthetic image dataset includes a synthetic image training set and a synthetic image test set, and the real image dataset includes a real image training set. The number of the real image dataset is at least one; A training module, configured to train the deep learning model to convergence based on the synthetic image training set to obtain a first converged model, and to train the deep learning model to convergence based on each of the real image training sets to obtain a corresponding second converged model; a test data acquisition module, configured to test the first convergence model based on one of the synthetic image test set and each of the real image data sets to obtain corresponding first test data; test each of the second convergence models based on the synthetic image data sets to obtain corresponding second test data; and test each of the second convergence models based on the real image data sets to obtain corresponding third test data; A testing module is configured to perform weighted averaging based on the first test data, the second test data, and the third test data to obtain a test result of the synthetic image data set.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for testing a synthetic image data set according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for synthetic image dataset testing according to any one of claims 1 to 7.

Images