Model training data processing method, device, computer equipment and storage medium

Abstract

The present application relates to a model training data processing method, device, computer equipment and storage medium. The method comprises: obtaining model training data; extracting data features of the model training data; obtaining the distance relationship between each of the data features; and screening and processing the model training data according to the distance relationship between each of the data features. The method can remove problematic model training data, thereby ensuring the accuracy of the remaining model training data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training data processing method, device, computer equipment and storage medium. Background Art

[0002] Convolutional Neural Network (CNN) is inspired by biological natural visual cognition and has excellent performance in large-scale image processing. It is widely used in image classification, target recognition, target positioning, and semantic segmentation.

[0003] In actual production applications, convolutional neural networks need to use different network models to achieve the best prediction results according to product types, production conditions, defect types, etc. In addition, neural network models need to be trained with certain data before they can be deployed in a production environment for use. Therefore, image classification tasks are often divided into two categories: model training and model prediction. The model training task requires inputting modeling data into the neural network, and adjusting the neural network parameters by backpropagation based on the difference between the output and the actual value. After repeated adjustments, an AI model that can be used on the production line can be generated; the model prediction task assists in completing the quality inspection process on the production line, and its effect depends on the effect of the previous model training.

[0004] The accuracy of modeling data is an important factor affecting the quality of AI models. Improving the reliability of modeling data can effectively improve the results of model training.

[0005] Currently, modeling data are mainly screened manually, and the number of modeling data can easily reach thousands to tens of thousands. Complete screening requires a lot of manpower and time, and the screening is done in a subjective way, with vague and inconsistent standards. Summary of the invention

[0006] Based on this, it is necessary to provide a model training data processing method, device, computer equipment and storage medium that can improve the accuracy of model training data in response to the above technical problems.

[0007] A model training data processing method, the method comprising:

[0008] Get model training data;

[0009] Extracting data features of the model training data;

[0010] Obtaining the distance relationship between each of the data features;

[0011] The model training data is screened according to the distance relationship between each of the data features.

[0012] In one embodiment, extracting data features of the model training data includes:

[0013] Extracting a multi-dimensional vector of each of the model training data;

[0014] The obtaining of the distance relationship between each of the data features includes:

[0015] Mapping each of the multi-dimensional vectors to a dimensional space;

[0016] The distance relationship between each of the multi-dimensional vectors in the dimensional space is obtained as the distance relationship between the data features.

[0017] In one embodiment, the screening process of the model training data according to the distance relationship between each of the data features includes:

[0018] Calculating outlier data in the model training data according to the distances between the respective data features, and deleting the outlier data; and / or

[0019] The model training data is calculated based on the distance between each of the data features to perform similar data, and the number of the similar data is reduced to a preset number.

[0020] In one embodiment, the calculating the outlier data in the model training data according to the distances between the data features includes:

[0021] Calculate the center position according to the distance relationship between each of the data features;

[0022] Calculating a first distance between each of the data features and the center position;

[0023] Obtaining a first statistical value of the first distance;

[0024] Obtaining a first preset value, and calculating a first threshold value according to the first statistical value and the first preset value;

[0025] The model training data is screened according to the first threshold to obtain outlier data.

[0026] In one embodiment, the step of calculating similarity data of the model training data according to the distance between each of the data features includes:

[0027] Calculate the center position according to the distance relationship between each of the data features;

[0028] Calculating a first distance between each of the data features and the center position;

[0029] Obtaining a distribution interval of the model training data according to each of the first distances;

[0030] Calculate a second threshold value according to the distribution interval;

[0031] Preliminarily screening each of the model training data according to the first distance to obtain preliminary training data whose distance meets a preset condition;

[0032] Calculating a second distance between each of the preliminary training data;

[0033] Similar data is calculated based on the preliminary training data, the second distance and the second threshold.

[0034] In one embodiment, the calculating the second threshold according to the distribution interval includes:

[0035] Obtain the maximum difference distance of the model training data contained in each distribution interval;

[0036] A second threshold is determined according to the maximum difference distance.

[0037] In one embodiment, after the model training data is screened according to the distance relationship between each of the data features, the following steps are performed:

[0038] Output the filtered model training data;

[0039] An inspection instruction for the filtered model training data is received, and the filtered model training data is secondary screened according to the inspection instruction.

[0040] A model training data processing device, the device comprising:

[0041] A model training data acquisition module is used to acquire model training data;

[0042] A feature extraction module, used to extract data features of the model training data;

[0043] A distance relationship acquisition module, used to acquire the distance relationship between each of the data features;

[0044] The screening module is used to screen the model training data according to the distance relationship between each of the data features.

[0045] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0046] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method in any one of the above embodiments.

[0047] After collecting the model training data, the above-mentioned model training data processing method, device, computer equipment and storage medium first calculate the data features of each model training data, and screen the model training data according to the distance relationship between each data feature, so as to remove the model training data with problems, thereby ensuring the accuracy of the remaining model training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 An application environment diagram of a model training data processing method in one embodiment;

[0049] Figure 2 A schematic diagram of a flow chart of a method for processing model training data in one embodiment;

[0050] Figure 3 is a flowchart of processing model training data in one embodiment;

[0051] Figure 4 is a schematic diagram of a dimensional space in an embodiment;

[0052] Figure 5 A schematic diagram of a flow chart of a method for processing model training data in another embodiment;

[0053] Figure 6 It is a structural block diagram of a model training data processing device in one embodiment;

[0054] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The model training data processing method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the database 104 through the network, and the terminal 102 can obtain the model training data from the database 104, and then extract the data features of the model training data, and calculate the distance relationship between each data feature, so as to screen the model training data according to the distance relationship, so as to remove the model training data with problems, so as to ensure the accuracy of the remaining model training data. Among them, the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the database 104 can be implemented with an independent database or a database cluster composed of multiple databases.

[0057] In one embodiment, Figure 2 As shown, a model training data processing method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0058] S202: Obtain model training data.

[0059] Specifically, the model training data may be data used for model training, such as labeled data, etc. The model training data may be classified and stored according to different models, so that the terminal can obtain the corresponding model training data from the database according to the type of model and continue subsequent processing.

[0060] In other embodiments, when model training data of multiple models exists in the database, the model training data can be read from the database in parallel by multiple terminals for subsequent processing.

[0061] S204: Extracting data features of the model training data.

[0062] Specifically, data features refer to features of model training data, wherein the data features can be extracted through a pre-trained model. Specifically, the terminal can pre-acquire a small amount of model training data, and then perform feature extraction on the small amount of model training data, and determine the processing method for each training data based on the extracted data features, and compare them with the marked actual processing method to correct the model extraction part, so as to make the extracted data features accurate.

[0063] In one of the embodiments, the terminal extracts multi-dimensional vector features of each model training data.

[0064] S206: Obtain the distance relationship between each of the data features.

[0065] Specifically, the distance relationship is the relationship between various data features, for example, it can be Euclidean distance (actual distance), cosine distance (angle relationship), etc., which is a distance calculation method for any multi-dimensional vector.

[0066] S208: Screening the model training data according to the distance relationship between each of the data features.

[0067] Specifically, the terminal may filter and process the model training data according to the distance relationship between data features.

[0068] Optionally, the model training data is screened according to the distance relationship between each of the data features, including: calculating outlier data in the model training data according to the distance between each of the data features, and deleting the outlier data; and / or calculating similar data in the model training data according to the distance between each of the data features, and reducing the number of similar data to a preset number.

[0069] Specifically, outlier data refers to data that deviates from the center of each model training data obtained according to the distance relationship. Similar data refers to model training data whose distance between each model training data determined according to the distance relationship is less than a certain value.

[0070] For outlier data, the terminal directly deletes it, and for similar data, in order to avoid the model repeatedly learning the same features and over-relying on specific features, the terminal can delete the number of similar features to a preset number, for example, to one.

[0071] The above-mentioned model training data processing method, after collecting the model training data, first calculates the data features of each model training data, and screens the model training data according to the distance relationship between each data feature, so as to remove the model training data with problems, thereby ensuring the accuracy of the remaining model training data.

[0072] In one embodiment, the extracting data features of the model training data includes: extracting multi-dimensional vectors of each model training data; the obtaining the distance relationship between each data feature includes: mapping each multi-dimensional vector to a dimensional space; obtaining the distance relationship between each multi-dimensional vector in the dimensional space as the distance relationship between the data features.

[0073] Specifically, combined Figure 3 and Figure 4 As shown, Figure 3 is a flowchart of processing model training data in one embodiment, Figure 4is a schematic diagram of a dimensional space in an embodiment. In this embodiment, the terminal first extracts multi-dimensional features of each model training data through a pre-trained model, and then maps the multi-dimensional features to the dimensional space. Figure 4 Take the two-dimensional space in as an example, Figure 4 Each point in represents a model training data. The terminal calculates the distance relationship between each model training data in the dimensional space, and then automatically sets the threshold according to the distribution of each model training data in the dimensional space to filter the model training data to obtain outlier data and similar data, and processes the model training data according to the outlier data and similar data.

[0074] In one embodiment, the method of calculating the outlier data in the model training data based on the distance between each of the data features includes: calculating a center position based on the distance relationship between each of the data features; calculating a first distance between each of the data features and the center position; obtaining a first statistical value of the first distance; obtaining a first preset value, and calculating a first threshold value based on the first statistical value and the first preset value; and screening the model training data based on the first threshold value to obtain the outlier data.

[0075] Specifically, when calculating outlier data, the terminal first obtains the center position based on the distance relationship between each data feature. For example, the terminal can calculate the average value based on each data feature to obtain the center position, and then the terminal calculates the first distance between each data feature and the center position, so as to obtain the distribution of the model training data. The terminal can obtain a first statistical value of the first distance, for example, calculate the average value of the first distance, and then obtain a first threshold value based on the average value and the first statistical value to automatically select a suitable threshold, so as to filter the model training data according to the first threshold to obtain outlier data.

[0076] Specifically, the terminal can calculate the first distance between each data feature and the center position, and then calculate the average distance of the first distance, and calculate the first threshold value based on the average distance and the first preset value. For example, the terminal can use the average distance as the center to segment the first preset value, so as to obtain the number of model training data in each segment, and then obtain the first threshold value according to the number of model training data. For example, with the average distance as the center, the maximum and minimum values ​​of the first preset value as the boundary, and then average segmentation is performed to obtain the number of model training data in each segment, and then the segment value corresponding to the segment with a large difference in the number of model training data in the segment and other segments (that is, the boundary value between the segment and other segments) is obtained as the first threshold, so as to screen the model training data to obtain outlier data. In his embodiment, the terminal can use the average distance as the center, and then move to both ends at a certain distance to determine the selection range, so as to determine the number of model training data outside the range. If the number of model training data outside the range is less than a certain value, the model training data outside the range is determined to be outlier data.

[0077] In one embodiment, the similarity data of the model training data is calculated based on the distance between each of the data features, including: calculating the center position based on the distance relationship between each of the data features; calculating the first distance between each of the data features and the center position; obtaining the distribution interval of the model training data based on each of the first distances, and calculating the second threshold based on the distribution interval; performing a preliminary screening on each of the model training data based on the first distance to obtain preliminary training data whose distance meets a preset condition; calculating the second distance between each of the preliminary training data; and calculating similarity data based on the preliminary training data, the second distance and the second threshold.

[0078] In one of the embodiments, calculating the second threshold value according to the distribution interval includes: obtaining a maximum difference distance of the model training data contained in each distribution interval; and determining the second threshold value according to the maximum difference distance.

[0079] In this embodiment, the center position of the data feature is still calculated, and then the terminal calculates the first distance between each data feature and the center position, so that the distribution interval of the model training data can be obtained according to the first clustering. For example, the first distance can be clustered to obtain multiple distribution intervals, and then the terminal performs similarity judgment on the model training data in the same distribution interval. For example, the model training data in the same distribution interval is used as preliminary training data, and the second distance between each preliminary training data in the same distribution interval is calculated, and then similar data is calculated according to the second distance and the corresponding second threshold.

[0080] The second threshold can be determined based on the maximum difference distance of the model training data contained in the distribution interval. For example, the maximum difference distance of the model training data contained in the distribution interval is first obtained, and then a certain proportion of the maximum difference distance is taken as the second threshold to screen data with higher similarity.

[0081] In this embodiment, the model training data is first screened by the distribution interval, and then screened by the second threshold, so that the screening of the similarity data is more accurate.

[0082] Specifically, see Figure 5 As shown, the terminal first calculates the center position of the model training data and then calculates the distance between each model training data and the center position.

[0083] Calculations for outlier data, i.e. data that may not be part of the model’s training data, include:

[0084] The terminal calculates the average distance between all model training data and the center position, and takes a certain ratio as the first threshold; the terminal regards the model training data greater than the first threshold as outlier data.

[0085] For similar data, that is, the calculation of pairwise similar data in the model training data includes:

[0086] The terminal calculates the distribution interval of the model training data based on the distance between each model training data and the central position, and takes a certain proportion as the second threshold.

[0087] The terminal performs preliminary screening based on the distance between each model training data and the center position. Only two similar model training data are likely to be similar. The distance between the model training data is further calculated. If the distance is less than the second threshold, it is similar data.

[0088] In this way, the terminal filters out outlier data from the model training data as a check result for classification errors to improve accuracy; filtering out similar data can prevent the model from repeatedly learning the same features and over-relying on specific features.

[0089] In the above embodiments, the reliability of the training source is effectively improved; important features are used for screening to avoid the ambiguity and inconsistency of manual screening standards; and manpower and time costs are saved.

[0090] In one of the embodiments, after the model training data is screened according to the distance relationship between each of the data features, it includes: outputting the screened model training data; receiving an inspection instruction for the screened model training data, and performing a second screening on the screened model training data according to the inspection instruction.

[0091] Specifically, in this embodiment, after the terminal screens out outlier data and similar data, it performs model training after manual inspection, that is, the remaining model training data and the deleted model training data are output to the terminal, and then manually screened for the second time, thereby ensuring the accuracy of the data for the final model training.

[0092] It should be understood that although Figure 2-5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0093] In one embodiment, Figure 6 As shown, a model training data processing device is provided, comprising: a model training data acquisition module 100, a feature extraction module 200, a distance relationship acquisition module 300 and a screening module 400, wherein:

[0094] Model training data acquisition module 100, used to acquire model training data;

[0095] A feature extraction module 200, used to extract data features of the model training data;

[0096] A distance relationship acquisition module 300 is used to acquire the distance relationship between each of the data features;

[0097] The screening module 400 is used to screen the model training data according to the distance relationship between each of the data features.

[0098] In one embodiment, the feature extraction module 200 is used to extract a multi-dimensional vector of each model training data;

[0099] The distance relationship acquisition module 300 includes:

[0100] A mapping unit, used for mapping each of the multi-dimensional vectors to a dimensional space;

[0101] The distance relationship acquisition unit is used to acquire the distance relationship between each of the multi-dimensional vectors in the dimensional space as the distance relationship between the data features.

[0102] In one embodiment, the screening module 400 is used to calculate the outlier data in the model training data according to the distance between each of the data features, and delete the outlier data; and / or calculate the similar data in the model training data according to the distance between each of the data features, and reduce the number of similar data to a preset number.

[0103] In one embodiment, the screening module 400 includes:

[0104] A first center position acquisition unit, configured to calculate the center position according to the distance relationship between each of the data features;

[0105] A first distance calculation unit, used for calculating a first distance between each of the data features and the center position;

[0106] A first statistical value obtaining unit, configured to obtain a first statistical value of the first distance;

[0107] a first threshold value calculating unit, configured to obtain a first preset value, and calculate a first threshold value according to the first statistical value and the first preset value;

[0108] The first screening unit is used to screen the model training data according to the first threshold to obtain outlier data.

[0109] In one embodiment, the screening module 400 includes:

[0110] A second center position acquisition unit, configured to calculate the center position according to the distance relationship between each of the data features;

[0111] A second distance calculation unit, used for calculating a first distance between each of the data features and the center position;

[0112] A distribution interval obtaining unit, used for obtaining a distribution interval of the model training data according to each of the first distances;

[0113] A second threshold calculation unit, configured to calculate a second threshold according to the distribution interval;

[0114] A second screening unit, configured to preliminarily screen each of the model training data according to the first distance to obtain preliminary training data whose distance meets a preset condition;

[0115] A third distance calculation unit, used for calculating a second distance between each of the preliminary training data;

[0116] A third screening unit is used to calculate similar data according to the preliminary training data, the second distance and the second threshold.

[0117] In one of the embodiments, the second threshold calculation unit is used to obtain the maximum difference distance of the model training data contained in each distribution interval; and determine the second threshold according to the maximum difference distance.

[0118] In one embodiment, the model training data processing device further includes:

[0119] Output module, used to output the filtered model training data;

[0120] The secondary screening module is used to receive an inspection instruction for the filtered model training data, and perform a secondary screening on the filtered model training data according to the inspection instruction.

[0121] For the specific definition of the model training data processing device, please refer to the definition of the model training data processing method above, which will not be repeated here. Each module in the above-mentioned model training data processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0122] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store XXX data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model training data processing method is implemented.

[0123] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining model training data; extracting data features of the model training data; obtaining the distance relationship between each of the data features; and screening and processing the model training data according to the distance relationship between each of the data features.

[0125] In one embodiment, the extracting of data features of the model training data implemented when the processor executes a computer program includes: extracting multi-dimensional vectors of each model training data; obtaining the distance relationship between each data feature includes: mapping each multi-dimensional vector to a dimensional space; obtaining the distance relationship between each multi-dimensional vector in the dimensional space as the distance relationship between the data features.

[0126] In one embodiment, the screening and processing of the model training data according to the distance relationship between each of the data features implemented when the processor executes a computer program includes: calculating outlier data in the model training data according to the distance between each of the data features, and deleting the outlier data; and / or calculating similar data in the model training data according to the distance between each of the data features, and reducing the number of similar data to a preset number.

[0127] In one embodiment, the calculation of outlier data in the model training data based on the distance between each of the data features implemented when the processor executes a computer program includes: calculating a center position based on the distance relationship between each of the data features; calculating a first distance between each of the data features and the center position; obtaining a first statistical value of the first distance; obtaining a first preset value, and calculating a first threshold value based on the first statistical value and the first preset value; and screening the model training data according to the first threshold value to obtain outlier data.

[0128] In one embodiment, the method of calculating similar data of the model training data according to the distance between each of the data features implemented when the processor executes a computer program includes: calculating a center position according to the distance relationship between each of the data features; calculating a first distance between each of the data features and the center position; obtaining a distribution interval of the model training data according to each of the first distances; calculating a second threshold value according to the distribution interval; performing a preliminary screening of each of the model training data according to the first distance to obtain preliminary training data whose distance meets a preset condition; calculating a second distance between each of the preliminary training data; and calculating similar data according to the preliminary training data, the second distance, and the second threshold value.

[0129] In one embodiment, the step of calculating the second threshold value based on the distribution interval implemented when the processor executes a computer program includes: obtaining the maximum difference distance of the model training data contained in each distribution interval; and determining the second threshold value based on the maximum difference distance.

[0130] In one embodiment, after the screening and processing of the model training data according to the distance relationship between each of the data features is implemented when the processor executes the computer program, it includes: outputting the screened model training data; receiving an inspection instruction for the screened model training data, and performing a second screening on the screened model training data according to the inspection instruction.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining model training data; extracting data features of the model training data; obtaining distance relationships between each of the data features; and screening and processing the model training data according to the distance relationships between each of the data features.

[0132] In one embodiment, the extracting of data features of the model training data implemented when the computer program is executed by a processor includes: extracting multi-dimensional vectors of each model training data; obtaining the distance relationship between each data feature includes: mapping each multi-dimensional vector to a dimensional space; obtaining the distance relationship between each multi-dimensional vector in the dimensional space as the distance relationship between the data features.

[0133] In one embodiment, the screening and processing of the model training data according to the distance relationship between each of the data features implemented when the computer program is executed by the processor includes: calculating the outlier data in the model training data according to the distance between each of the data features, and deleting the outlier data; and / or calculating the similar data of the model training data according to the distance between each of the data features, and reducing the number of similar data to a preset number.

[0134] In one embodiment, the method of calculating the outlier data in the model training data based on the distance between each of the data features implemented when the computer program is executed by the processor includes: calculating the center position based on the distance relationship between each of the data features; calculating the first distance between each of the data features and the center position; obtaining a first statistical value of the first distance; obtaining a first preset value, and calculating a first threshold value based on the first statistical value and the first preset value; and screening the model training data according to the first threshold value to obtain the outlier data.

[0135] In one embodiment, the method of calculating similar data of the model training data according to the distance between each of the data features implemented when the computer program is executed by the processor includes: calculating a center position according to the distance relationship between each of the data features; calculating a first distance between each of the data features and the center position; obtaining a distribution interval of the model training data according to each of the first distances; calculating a second threshold according to the distribution interval; performing a preliminary screening of each of the model training data according to the first distance to obtain preliminary training data whose distance meets a preset condition; calculating a second distance between each of the preliminary training data; and calculating similar data according to the preliminary training data, the second distance, and the second threshold.

[0136] In one embodiment, the step of calculating the second threshold value based on the distribution interval implemented when the computer program is executed by the processor includes: obtaining the maximum difference distance of the model training data contained in each distribution interval; and determining the second threshold value based on the maximum difference distance.

[0137] In one embodiment, after the screening and processing of the model training data according to the distance relationship between each of the data features is implemented when the computer program is executed by the processor, it includes: outputting the screened model training data; receiving an inspection instruction for the screened model training data, and performing a second screening on the screened model training data according to the inspection instruction.

[0138] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0139] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A model training data processing method, characterized in that: The method comprises: Acquire model training data, wherein the model training data is modeling data of an image-type neural network model; Extracting data features of the model training data includes: Extracting a multi-dimensional vector of each of the model training data; Obtaining the distance relationship between each of the data features includes: Mapping each of the multi-dimensional vectors to a dimensional space; Acquire the distance relationship between each of the multi-dimensional vectors in the dimensional space as the distance relationship between the data features; The model training data is screened and processed according to the distance relationship between each of the data features, including: Automatically setting a threshold value according to the distribution of each of the model training data in the dimensional space to filter the model training data to obtain outlier data and similar data; Calculating outlier data in the model training data according to the distances between the data features, and deleting the outlier data; The calculating the outlier data in the model training data according to the distances between the data features comprises: Calculate the center position according to the distance relationship between each of the data features; Calculating a first distance between each of the data features and the center position; Obtaining a first statistical value of the first distance; Acquiring a first preset value, and calculating a first threshold value according to the first statistical value and the first preset value, specifically includes: Calculate the average distance of the first distance, take the average distance as the center, take the maximum value and the minimum value of the first preset value as the boundary, then perform average segmentation to obtain the number of model training data in each segment, and then obtain the segmentation value corresponding to the segment with a large difference in the number of model training data in each segment and other segments as the first threshold, and the segmentation value is the boundary value between the segment and other segments; The model training data is screened according to the first threshold to obtain outlier data.

2. The method according to claim 1, characterized in that The screening process of the model training data according to the distance relationship between each of the data features also includes: The model training data is calculated based on the distance between each of the data features to perform similar data, and the number of the similar data is reduced to a preset number.

3. The method according to claim 2, characterized in that The calculating the similarity data of the model training data according to the distance between each of the data features includes: Calculate the center position according to the distance relationship between each of the data features; Calculating a first distance between each of the data features and the center position; Obtaining a distribution interval of the model training data according to each of the first distances; Calculate a second threshold value according to the distribution interval; Preliminarily screening each of the model training data according to the first distance to obtain preliminary training data whose distance meets a preset condition; Calculating a second distance between each of the preliminary training data; Similar data is calculated based on the preliminary training data, the second distance and the second threshold.

4. The method according to claim 3, characterized in that The calculating a second threshold value according to the distribution interval includes: Obtain the maximum difference distance of the model training data contained in each distribution interval; A second threshold is determined according to the maximum difference distance.

5. The method according to claim 1, characterized in that: After the model training data is screened according to the distance relationship between each of the data features, the method includes: Output the filtered model training data; An inspection instruction for the filtered model training data is received, and the filtered model training data is secondary screened according to the inspection instruction.

6. A model training data processing device, characterized in that: The device comprises: A model training data acquisition module is used to acquire model training data; A feature extraction module is used to extract data features of the model training data, including: Extracting a multi-dimensional vector of each of the model training data; The distance relationship acquisition module is used to acquire the distance relationship between each of the data features, including: Mapping each of the multi-dimensional vectors to a dimensional space; Acquire the distance relationship between each of the multi-dimensional vectors in the dimensional space as the distance relationship between the data features; A screening module is used to screen the model training data according to the distance relationship between each of the data features, including: Automatically setting a threshold value according to the distribution of each of the model training data in the dimensional space to filter the model training data to obtain outlier data and similar data; Calculating outlier data in the model training data according to the distances between the data features, and deleting the outlier data; The calculating the outlier data in the model training data according to the distances between the data features comprises: Calculate the center position according to the distance relationship between each of the data features; Calculating a first distance between each of the data features and the center position; Obtaining a first statistical value of the first distance; Acquiring a first preset value, and calculating a first threshold value according to the first statistical value and the first preset value, specifically includes: Calculate the average distance of the first distance, take the average distance as the center, take the maximum value and the minimum value of the first preset value as the boundary, then perform average segmentation to obtain the number of model training data in each segment, and then obtain the segmentation value corresponding to the segment with a large difference in the number of model training data in each segment and other segments as the first threshold, and the segmentation value is the boundary value between the segment and other segments; The model training data is screened according to the first threshold to obtain outlier data.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images