Method for making data sets more realistic in case of data cascading

Abstract

Data set is the basis of deep learning, and building a data set close to the real scene has an important influence on the deep learning model, but there is a data cascade situation, which makes the trained model not well applied to the real scene. The invention wants to solve how to provide a method for making the data set closer to the real scene in the case of data cascade. The data set used for training the model is taken as the training set, and the real scene sample is taken as the test set. The features of the training set and the test set are extracted respectively, the inter-domain similarity between the training set and the test set is calculated, and the intra-domain uncertainty of the training set is calculated, and the comprehensive similarity is obtained by comprehensively calculating the inter-domain similarity and the intra-domain uncertainty. The comprehensive similarity can be used to select good samples in the training set, and the model trained by the good samples has better performance and reduces the influence of data cascade. The method is verified through experiments.

Description

Technical Field

[0001] This invention relates to a dataset construction method, specifically to a method for making a dataset more closely resemble a real-world scenario when data cascading occurs. Background Technology

[0002] Deep learning is a crucial research area in artificial intelligence. With the development of AI, how to perform deep learning efficiently and accurately has become a vital research topic. Data sets are the foundation of deep learning and directly impact the problem-solving capabilities of deep learning models. Constructing datasets that closely resemble real-world scenarios has a significant impact on deep learning models; however, data cascading can occur, preventing trained models from being well-suited for real-world applications. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention aims to provide a method for making datasets more closely resemble real-world scenarios in the event of data cascading. To solve the aforementioned technical problem, this invention adopts the following technical solution:

[0004] A method for making a dataset more closely resemble a real-world scenario in the event of data cascading includes the following steps:

[0005] (1) Establish a feature extractor model based on Python ;

[0006] (2) Establish training and testing sets by using the dataset used to train the model as the training set and real-world scene samples as the test set, where the training set is... The training set samples are The test set is The test set samples are ;

[0007] (3) Using the training set Training Feature Extractor ;

[0008] (4) Use the trained feature extractor Extract training set samples Features Test set samples Features ;

[0009]

[0010] (5) Calculate the different categories in the training set samples Features The average value is used to obtain the feature prototypes of each class in the training set. ; Calculate the different categories of the test set samples Features The average value is used to obtain the feature prototypes of each class in the test set. :

[0011]

[0012] (6) Calculate the values ​​of each sample in the training set. Similarity to the test set :

[0013] Edge distance:

[0014] Calculate similarity :

[0015]

[0016] (7) Determine each sample in the training set Uncertainty:

[0017] European distance:

[0018]

[0019] (7.1) Calculate the sample Euclidean distance from various feature prototypes in the training set :

[0020]

[0021] (7.2) Calculate the probability distribution of Euclidean distance:

[0022]

[0023] (7.3) Calculate the uncertainty:

[0024]

[0025] (8) Set the weight parameter α and calculate the overall similarity. :

[0026]

[0027] (9) Based on the overall similarity The training set samples are sorted by size, where Small samples are considered good samples; select the required number of samples based on the needs.

[0028] The advantages and positive effects of this invention are as follows:

[0029] (1) This invention highlights the importance of datasets and makes full use of data to improve model performance.

[0030] (2) The proposed method of the present invention can solve the problem of data cascading, make the model more adaptable to real-world scenarios, and improve the accuracy of the model.

[0031] (3) This method can be applied to various models and does not depend on the structure of the model.

[0032] (4) This method has been tested and verified through a large number of experiments, which has effectively improved the reliability of the method. Attached Figure Description

[0033] Figure 1 This is the overall process for obtaining comprehensive similarity in a specific embodiment of the present invention;

[0034] Figure 2 This is the dataset used in a specific embodiment of the present invention; Detailed Implementation

[0035] This example uses a dataset as the research object to describe in detail the implementation of the present invention. To address the data cascading phenomenon, suitable samples are selected by calculating comprehensive similarity to form a new training set, making the trained model more consistent with real-world scenarios, thus solving the data cascading problem.

[0036] To make the objectives and technical solutions of this invention clearer, the specific implementation steps of this invention will be described in detail below with reference to the accompanying drawings.

[0037] See Figure 1 The flowchart illustrating the implementation of the dataset construction method proposed in this invention is described in detail below:

[0038] 1. Set the dataset to NICO-Animal, see [link / reference] Figure 2 This involves selecting animal-related samples from the NICO dataset, which includes ten categories. Each category contains 9-10 different contexts, encompassing aspects such as posture, action, and background information. Each context contains 83-215 images, treating images from different contexts as data with varying distributions. Images from different contexts are selected as the training and test sets, representing training samples and real-world scenarios, respectively. The differences in context represent the differences between the training samples and real-world scenarios, thus simulating data cascading.

[0039] 2. A ResNet-18 network was selected as the feature extractor. The training batch size was set to 50, the learning rate to 0.01, and the training epochs to 100. The stochastic gradient descent (SGD) method was used as the model optimizer, and the momentum factor was set to 0.9. After training on the training set, features were extracted from the training set. Features of the test set .

[0040] 3. Calculate the average value of the features extracted from samples of different categories in the training and test sets respectively, to obtain the feature prototypes of each category in the training and test sets. , .

[0041] 4. Calculate inter-domain similarity using the edge distance formula, based on the feature prototypes of the training set samples and the test set. .

[0042] 5. First, calculate the Euclidean distance between the training set samples and the prototypes of each feature in the training set. :

[0043] Then calculate the probability distribution of the distance. :

[0044] Finally, the in-domain uncertainty of the training set is derived from the definition of entropy. :

[0045] 6. Set the weight parameter α to 0.9 and calculate the overall similarity. :

[0046]

[0047] 7. Sort the training set samples according to their overall similarity; samples with lower similarity are considered better. The samples were sorted from largest to smallest to obtain combinations of samples at three levels: farthest, middle, and nearest. These samples were then used to train the model, resulting in Table 1. The overall similarity can be seen in... Smaller samples are better samples, and datasets composed of such selected samples can make the dataset closer to the real-world scenario when data cascading occurs.

[0048] Table 1. Accuracy of training models using samples at different comprehensive similarity levels.

[0049] Farthest middle recent 60.89% 64.43% 70.41%

[0050] The above-described embodiments are merely preferred embodiments. The description of specific implementations in this paper is intended to help understand the method and core ideas of the present invention. Modifications to the technical solutions described in the foregoing embodiments or equivalent substitutions of some indicators should be included within the protection scope of the present invention.

Claims

1. A method for making a dataset more closely resemble a real-world scenario in the event of data cascading, characterized in that, The dataset construction method includes: (1) Establish a feature extractor model based on Python ; (2) Obtain samples and use the dataset used to train the model as the training set. Real-world scenario samples were used as the test set. Construct training and testing sets, where the training set samples are... The test set samples are Images from different contexts were selected as the training set and the test set, representing training samples and real-world scenarios, respectively. (3) Using the training set Training Feature Extractor ; (4) Use the trained feature extractor Extract training set samples Features Test set samples Features ,in: (5) Calculate the different categories of the training set samples Features The average value is used to obtain the feature prototypes of each class in the training set. ; Calculate the different categories of the test set samples Features The average value is used to obtain the feature prototypes of each class in the test set. : Calculate training set sample features using the edge distance metric. With various feature prototypes in the test set The distance between the training and test sets is used to derive the inter-domain similarity between them. : (6) Calculate the features of the training set samples using the Euclidean distance metric. With various feature prototypes in the training set Distance between : Obtain the distance vector : Calculate the probability distribution of distance : Uncertainty within the computational domain : (7) Set weight parameters 0 < α < 1, determined by inter-domain similarity and intra-domain uncertainty Calculate the overall similarity : (8) Sort the training set samples according to the magnitude of the comprehensive similarity. Samples with low comprehensive similarity are good samples. The dataset composed of good samples is closer to the real scene and reduces the impact of data cascading.

2. The method for making a dataset closer to a real-world scenario in the event of data concatenation, as described in claim 1, is characterized in that: Weight parameters The value is 0.9.

Images