A file adaptive parsing method and system for battery test data

Abstract

The application discloses a kind of file adaptive analysis method and system of battery test data, it is related to data processing technical field, the method includes: extracting the discriminative feature of to-be-analyzed file, discriminative feature is used for file format identification based on deep learning algorithm;To-be-analyzed file is parsed into structured table data based on file format information;Column type prediction information is obtained by carrying out column type prediction to structured table data based on multi-label classification model;Semantic mapping is carried out based on column type prediction information to convert to-be-analyzed file into standard format file;Feedback data of user to standard format file is obtained, and file format information, column type prediction information and semantic mapping relationship are revised based on feedback data, and closed-loop learning of deep learning model is carried out based on revision result.The application can be adapted to different file formats, no longer needs to design rule and maintain extraction template separately for each document format, and development cost is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a file adaptive parsing method and system for battery test data. BACKGROUND

[0002] The battery is an important component of the unmanned aerial vehicle, which affects the flight performance, task efficiency and safety of the unmanned aerial vehicle. In order to scientifically evaluate the performance of batteries from different suppliers and build an accurate battery model, it is necessary to implement multi-working-condition charge-discharge test on the sample battery, collect key charge-discharge data such as voltage and current, and form a test file. Due to the significant format difference of data files output by different test platforms, the traditional solution needs to customize parsing rules for each format and continuously maintain, resulting in high development cost. At the same time, the existing solution usually includes data optimization and integration operations when parsing the data file, but this operation may modify the original data, which does not meet the principle requirements of the integrity and accuracy of the battery test data. Moreover, the current parsing solution cannot adaptively adapt to files of different formats, and the degree of automation of data parsing is low. SUMMARY

[0003] The present application aims to overcome the shortcomings of the prior art, and provides a file adaptive parsing method and system for battery test data, which can adapt to different file formats, no longer needs to design rules and maintain extraction templates for each document format, and reduces the development cost.

[0004] In order to solve the above technical problems, the present application provides a file adaptive parsing method for battery test data, which comprises:

[0005] Obtain a to-be-parsed file of battery test data, and extract discriminative features of the to-be-parsed file, perform file format identification based on a deep learning algorithm using the discriminative features, and obtain file format information;

[0006] Parse the to-be-parsed file into structured table data based on the file format information;

[0007] Perform column type prediction on the structured table data based on a multi-label classification model, and obtain column type prediction information;

[0008] Perform semantic mapping based on the column type prediction information, obtain a semantic mapping relationship, and convert the to-be-parsed file into a standard format file based on the semantic mapping relationship;

[0009] Obtain feedback data of the user on the standard format file, and correct the file format information, the column type prediction information and the semantic mapping relationship based on the feedback data, and perform closed-loop learning of the deep learning model based on the correction result.

[0010] Optionally, the discriminative features of the file to be parsed are extracted, the file format recognition is performed based on the deep learning algorithm using the discriminative features, and file format information is obtained, including:

[0011] The file signature and file extension of the file to be parsed are extracted.

[0012] The structural complexity feature of the file to be parsed is extracted, and structural complexity feature information is obtained.

[0013] The content feature of the file to be parsed is extracted, and content feature information is obtained.

[0014] The preset mode feature of the file to be parsed is extracted based on the confidence, and preset mode feature information is obtained.

[0015] The target deep learning model is determined based on the file signature, structural complexity feature information, and content feature information in combination with a preset decision threshold.

[0016] The discriminative features are generated based on the file signature, file extension, structural complexity feature information, content feature information, and preset mode feature information, and the file format recognition is performed based on the target deep learning model using the discriminative features, and file format information is obtained.

[0017] Optionally, the structural complexity feature of the file to be parsed is extracted, and structural complexity feature information is obtained, including:

[0018] The number of columns and the number of rows of the file to be parsed are extracted.

[0019] The number of rows with consistent separators and the total number of randomly sampled rows in the file to be parsed are determined, and the separator consistency information is determined based on the number of rows with consistent separators and the total number of randomly sampled rows, and the expression of the separator consistency information is:

[0020]

[0021] Wherein, is the separator consistency information, is the number of rows with consistent separators, is the total number of randomly sampled rows;

[0022] The nesting level analysis is performed on the file to be parsed based on the numerical estimation method, and nesting level information is obtained, and the structural complexity feature information is generated based on the number of columns, the number of rows, the separator consistency information, and the nesting level information.

[0023] Optionally, the content feature of the file to be parsed is extracted, and content feature information is obtained, including:

[0024] determine a keyword quantity of the file to be parsed and a file header length, and determine a header keyword occurrence rate based on the keyword quantity and the file header length;

[0025] determine a number of numerical cells of the file to be parsed and a total number of randomly sampled cells, and determine a numerical value proportion based on the number of numerical cells and the total number of randomly sampled cells;

[0026] determine a number of null value cells of the file to be parsed, determine a null value proportion based on the number of null value cells and the total number of randomly sampled cells, and determine content feature information based on the header keyword occurrence rate, the numerical value proportion and the null value proportion.

[0027] Optionally, the parsing the file to be parsed into structured table data based on the file format information comprises:

[0028] matching a corresponding parser based on the file format information;

[0029] parsing the file to be parsed into structured table data based on the parser.

[0030] Optionally, the column type prediction based on the multi-label classification model on the structured table data comprises:

[0031] performing column name semantic analysis on the structured table data based on a word embedding model to obtain a column name semantic vector;

[0032] determining data distribution features of each column of the structured table data, and generating a column content statistical feature vector based on the data distribution features of each column;

[0033] inputting the column name semantic vector and the column content statistical feature vector into the multi-label classification model to calculate a column type prediction probability, and determining column type prediction information based on the column type prediction probability.

[0034] Optionally, the semantic mapping based on the column type prediction information comprises:

[0035] obtaining a concept description in a unified semantic model, and calculating a semantic similarity based on the column type prediction information and the concept description;

[0036] performing key entity recognition on the column type prediction information based on a named entity recognition technology to obtain key entity information;

[0037] performing semantic mapping based on the semantic similarity and the key entity information in combination with a preset mapping rule to obtain a semantic mapping relationship.

[0038] Optionally, the calculating the semantic similarity based on the column type prediction information and the concept description comprises:

[0039] vectorizing the column type prediction information and the concept description based on a word embedding model to obtain a column type semantic vector and a concept description semantic vector;

[0040] calculating the semantic similarity of the column type semantic vector and the concept description semantic vector, and an expression of the semantic similarity is:

[0041] ,

[0042] wherein, is the semantic similarity, is the column type semantic vector, is the concept description semantic vector.

[0043] Optionally, the closed-loop learning of the deep learning model based on the correction result comprises:

[0044] extracting correction data based on the correction result, and optimizing a preset mapping rule based on the correction data;

[0045] incrementally training and closed-loop optimizing the deep learning model based on the correction data.

[0046] In addition, the application also provides a file adaptive analysis system for battery test data, the system comprising:

[0047] a file format identification module: used for obtaining a to-be-analyzed file of battery test data, extracting discriminative features of the to-be-analyzed file, performing file format identification based on the discriminative features by using a deep learning algorithm, and obtaining file format information;

[0048] a file analysis module: used for analyzing the to-be-analyzed file into structured table data based on the file format information;

[0049] a column type prediction module: used for predicting the column type of the structured table data based on a multi-label classification model to obtain column type prediction information;

[0050] a file conversion module: used for performing semantic mapping based on the column type prediction information to obtain a semantic mapping relationship, and converting the to-be-analyzed file into a standard format file based on the semantic mapping relationship;

[0051] a feedback optimization module: used for obtaining feedback data of the standard format file from a user, and correcting the file format information, the column type prediction information and the semantic mapping relationship based on the feedback data, and performing closed-loop learning of a deep learning model based on a correction result.

[0052] In the present application, the discriminative features of the file to be parsed are used for file format recognition based on a deep learning algorithm, the file to be parsed is parsed into structured table data based on file format information, column type prediction is performed on the structured table data based on a multi-label classification model, semantic mapping is performed based on column type prediction information, semantic mapping relationships are obtained, and the file to be parsed is converted into a standard format file based on the semantic mapping relationships, which can adapt to different file formats, no longer needs to design rules and maintain extraction templates for each document format, greatly reduces the development cost, and at the same time, does not need to modify the original test data, meets the principle requirements of the integrity and accuracy of battery test data. Feedback data of users on the standard format file is obtained, and the file format information, column type prediction information and semantic mapping relationships are corrected based on the feedback data, the deep learning model is closed-loop learned based on the correction results, the process of training the model is automated, the manual participation is reduced, and the automation degree and adaptability are improved. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0054] Figure 1 is a flowchart of a file adaptive parsing method for battery test data in an embodiment of the present application;

[0055] Figure 2 is a flowchart of a file adaptive parsing method for battery test data in another embodiment of the present application;

[0056] Figure 3 is a structural composition diagram of a file adaptive parsing system for battery test data in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0058] Embodiment one

[0059] Please refer to Figure 1 , Figure 1is a flowchart of a file adaptive analysis method of battery test data in an embodiment of the present application, the method comprises:

[0060] S11: obtaining a to-be-analyzed file of battery test data, and extracting discriminative features of the to-be-analyzed file, performing file format identification based on the discriminative features by using a deep learning algorithm to obtain file format information;

[0061] In the specific implementation of the present application, the to-be-analyzed file of battery test data is obtained, and the file signature and file extension of the to-be-analyzed file are extracted; the structural complexity feature of the to-be-analyzed file is extracted to obtain structural complexity feature information; the content feature of the to-be-analyzed file is extracted to obtain content feature information; the to-be-analyzed file is pre-set mode feature extraction based on confidence to obtain pre-set mode feature information; determine the target deep learning model based on the file signature, structural complexity feature information and content feature information combined with the pre-set decision threshold; generate discriminative features based on the file signature, file extension, structural complexity feature information, content feature information and pre-set mode feature information, and perform file format identification based on the discriminative features by using the target deep learning model to obtain file format information, so that the obtained discriminative features are more comprehensive, and the accuracy of file format identification can be improved.

[0062] S12: analyzing the to-be-analyzed file into structured table data based on the file format information;

[0063] In the specific implementation of the present application, the corresponding parser is matched based on the file format information; the to-be-analyzed file is parsed into structured table data based on the parser, which can match the corresponding parser for different file formats to realize adaptive analysis of different file formats.

[0064] S13: performing column type prediction on the structured table data based on a multi-label classification model to obtain column type prediction information;

[0065] In the specific implementation of the present application, the column name semantic vector is obtained by performing column name semantic analysis on the structured table data based on a word embedding model; the data distribution features of each column in the structured table data are determined, and the column content statistical feature vector is generated based on the data distribution features of each column; the column name semantic vector and the column content statistical feature vector are input into the multi-label classification model to calculate the column type prediction probability, and the column type prediction information is determined based on the column type prediction probability, which can obtain more reliable column type prediction information and provide accurate data support for subsequent semantic mapping.

[0066] S14: performing semantic mapping based on the column type prediction information to obtain a semantic mapping relationship, and converting the to-be-analyzed file into a standard format file based on the semantic mapping relationship;

[0067] In the implementation of the present application, the concept description in the unified semantic model is obtained, the semantic similarity is calculated based on the column type prediction information and the concept description, the key entity information is obtained by performing key entity recognition on the column type prediction information based on the named entity recognition technology, the semantic mapping relationship is obtained by performing semantic mapping based on the semantic similarity and the key entity information in combination with the preset mapping rule, and the file to be parsed is converted into a standard format file based on the semantic mapping relationship, so that zero coding is realized to adapt to files in different formats, and the development cost can be reduced.

[0068] S15: Obtain feedback data of the user on the standard format file, and correct the file format information, the column type prediction information and the semantic mapping relationship based on the feedback data, and perform closed-loop learning of the deep learning model based on the correction result.

[0069] In the implementation of the present application, the feedback data of the user on the standard format file is obtained, and the file format information, the column type prediction information and the semantic mapping relationship are corrected based on the feedback data. The correction data is extracted based on the correction result, and the preset mapping rule is optimized based on the correction data. The incremental training and closed-loop optimization of the deep learning model are performed based on the correction data, the model is retrained according to the feedback of the user, the process automation of the training model is realized, and the manual participation is reduced.

[0070] In the embodiment of the present application, the discriminative features of the file to be parsed are used to identify the file format based on the deep learning algorithm, the file to be parsed is parsed into structured table data based on the file format information, the column type prediction is performed on the structured table data based on the multi-label classification model, the semantic mapping is performed based on the column type prediction information, the semantic mapping relationship is obtained, and the file to be parsed is converted into a standard format file based on the semantic mapping relationship. Different file formats can be adapted, and there is no need to design rules and maintain extraction templates for each document format, which greatly reduces the development cost. At the same time, the original test data does not need to be modified, which meets the principle requirements of the integrity and accuracy of the battery test data. The feedback data of the user on the standard format file is obtained, the file format information, the column type prediction information and the semantic mapping relationship are corrected based on the feedback data, the closed-loop learning of the deep learning model is performed based on the correction result, the process automation of the training model is realized, the manual participation is reduced, and the automation degree and adaptability are improved.

[0071] Embodiment two

[0072] Please refer to Figure 2 , Figure 2 is a flowchart of a file adaptive parsing method of battery test data in another embodiment of the present application, and the method comprises:

[0073] S201: Obtain a to-be-analyzed file of battery test data, and extract a file signature and a file extension of the to-be-analyzed file;

[0074] In the implementation of the present application, a user can upload a to-be-analyzed file of battery test data of a UAV through a platform, extract a file signature and a file extension of the to-be-analyzed file, check a specific byte sequence at the beginning of the to-be-analyzed file, that is, the file signature, and the file extension is, for example,.csv,.xlsx, and the file extension is a preliminary format classification basis.

[0075] S202: Extract a structural complexity feature of the to-be-analyzed file to obtain structural complexity feature information;

[0076] In the implementation of the present application, the extraction of the structural complexity feature of the to-be-analyzed file to obtain the structural complexity feature information includes: extracting the number of columns and the number of rows of the to-be-analyzed file; determining the number of rows with consistent separators and the total number of randomly sampled rows in the to-be-analyzed file, and determining separator consistency information based on the number of rows with consistent separators and the total number of randomly sampled rows, and the expression of the separator consistency information is:

[0077]

[0078] wherein, the separator consistency information, the number of rows with consistent separators, the total number of randomly sampled rows; based on a numerical estimation method, the to-be-analyzed file is analyzed in a nested level to obtain nested level information, and the structural complexity feature information is generated based on the number of columns, the number of rows, the separator consistency information and the nested level information.

[0079] Specifically, the number of columns and the number of rows of the to-be-analyzed file are extracted, the number of rows is the total number of rows contained in the to-be-analyzed file, and the number of columns is the maximum number of columns of the to-be-analyzed file.

[0080] The number of rows with consistent separators and the total number of randomly sampled rows in the to-be-analyzed file are determined, and the separator consistency information is determined based on the number of rows with consistent separators and the total number of randomly sampled rows, and the expression of the separator consistency information is:

[0081]

[0082] wherein, the separator consistency information, the number of rows with consistent separators, the total number of randomly sampled rows. The proportion feature is closer to 1, which indicates that the separator is used consistently.

[0083] The nested level analysis is performed on the file to be parsed based on a numerical estimation method, the numerical characteristics are estimated by analyzing indentations and labels, the nested level information is obtained according to the numerical characteristics, and the structural complexity characteristic information is generated based on the column number, the row number, the separator consistency information and the nested level information, that is, the structural complexity characteristic information is composed of the column number, the row number, the separator consistency information and the nested level information.

[0084] S203: Content feature extraction is performed on the file to be parsed to obtain content feature information.

[0085] In the implementation of the present application, the content feature extraction performed on the file to be parsed to obtain content feature information includes: determining the keyword number and the file header length of the file to be parsed, and determining the header keyword occurrence rate based on the keyword number and the file header length; determining the number of digital cells and the total number of randomly sampled cells of the file to be parsed, and determining the numerical proportion based on the number of digital cells and the total number of randomly sampled cells; determining the number of null cells of the file to be parsed, and determining the null value proportion based on the number of null cells and the total number of randomly sampled cells, and determining the content feature information based on the header keyword occurrence rate, the numerical proportion and the null value proportion.

[0086] Specifically, the keyword number and the file header length of the file to be parsed are determined, the keywords of the file to be parsed are keywords contained in the file header, such as Voltage, Current, Cycle, Temp, SOC, etc., the number of keywords appearing is counted, and the header keyword occurrence rate is determined based on the keyword number and the file header length, and the expression of the header keyword occurrence rate is:

[0087]

[0088] wherein, the header keyword occurrence rate is, the keyword number is, the file header length is.

[0089] The number of digital cells and the total number of randomly sampled cells of the file to be parsed are determined, the number of digital cells is the number of cells that can be parsed into numbers in the sampling row, the total number of randomly sampled cells is the total number of cells randomly sampled in the sampling row, and the numerical proportion is determined based on the number of digital cells and the total number of randomly sampled cells, and the expression of the numerical proportion is:

[0090]

[0091] wherein, the numerical proportion is, ​​is the number of digital cells, is the total number of randomly sampled cells.

[0092] The number of null cells of the file to be parsed is determined, and the null ratio is determined based on the number of null cells and the total number of randomly sampled cells. The expression of the null ratio is:

[0093]

[0094] wherein, is the null ratio, is the number of null cells, is the total number of randomly sampled cells. The content feature information is determined based on the head keyword occurrence rate, the value ratio and the null ratio. Therefore, the content feature information contains the statistical characteristics of the file to be parsed.

[0095] S204: performing preset mode feature extraction on the file to be parsed based on the confidence, to obtain preset mode feature information;

[0096] In the specific implementation process of the present application, the preset specific mode or checksum of the known instrument data block in the file to be parsed is detected by the corresponding model. Different data blocks correspond to different model IDs. The confidence of each model is matched. The vector feature is composed according to each model ID and the corresponding confidence, that is, the preset mode feature information is obtained.

[0097] S205: determining the target deep learning model based on the file signature, the structural complexity feature information and the content feature information combined with a preset decision threshold;

[0098] In the specific implementation process of the present application, the target deep learning model is determined based on the file signature, the structural complexity feature information and the content feature information combined with a preset decision threshold, that is, the file signature, the structural complexity feature information and the content feature information are compared with different preset decision thresholds. The required deep learning model is selected according to the comparison result. The deep learning model includes a long short-term memory network, a Transformer model and both the long short-term memory network and the Transformer model in parallel. The long short-term memory network is good at processing sequence data and can effectively capture the time sequence characteristics of the file format. The Transformer model has strong structural understanding ability and is better for processing file formats with complex hierarchical structure.

[0099] ​If the confidence of the file signature is greater than 0.85, it indicates that the magic number feature is significant, and the long short-term memory network is used as the target deep learning model; if the numerical proportion in the content feature information is greater than 0.8 and the nesting level information in the structural complexity feature information is less than 2, it indicates that the high numerical feature and the low nesting, and the long short-term memory network is used as the target deep learning model; if the nesting level information in the structural complexity feature information is greater than or equal to 3, it indicates that the deep nesting structure, and the Transformer model is used as the target deep learning model; if the separator consistency information in the structural complexity feature information is less than 0.6, it indicates that the separator is highly inconsistent, and the Transformer model is used as the target deep learning model; if none of the above conditions is met, the long short-term memory network and the Transformer model are used in parallel as the target deep learning model.

[0100] S206: Generate discriminative features based on the file signature, file extension, structural complexity feature information, content feature information and preset pattern feature information, and perform file format identification based on the target deep learning model using the discriminative features to obtain file format information;

[0101] In the specific implementation of the present application, discriminative features are generated based on the file signature, file extension, structural complexity feature information, content feature information and preset pattern feature information. The discriminative features are input into the selected target deep learning model for file format identification to obtain file format information.

[0102] S207: Based on the file format information, the file to be parsed is parsed into structured table data;

[0103] In the specific implementation of the present application, based on the file format information, the file to be parsed is parsed into structured table data, which includes: matching the corresponding parser based on the file format information; and parsing the file to be parsed into structured table data based on the parser.

[0104] Specifically, the corresponding parser is matched based on the file format information, such as CSV parser, Excel parser, specific instrument SDK / adapter, etc. The file to be parsed is parsed into structured table data based on the parser, i.e. the file to be parsed is parsed into structured table data through the corresponding parser. The parsing process uses differentiated technologies for different formats, such as generating a dictionary list by automatically detecting the separator for CSV / TSV files, traversing the worksheet to extract cell values for Excel files, and converting JSON files into tabular records through standardization processing.

[0105] S208: Perform column type prediction on the structured table data based on a multi-label classification model to obtain column type prediction information;

[0106] In the implementation of the present application, the column type prediction information is obtained by predicting the column types of the structured table data based on the multi-label classification model, which includes: performing semantic analysis on the column names of the structured table data based on a word embedding model to obtain semantic vectors of the column names; determining the data distribution characteristics of each column in the structured table data and generating column content statistical feature vectors based on the data distribution characteristics of each column; inputting the semantic vectors of the column names and the column content statistical feature vectors into the multi-label classification model to calculate column type prediction probabilities, and determining the column type prediction information based on the column type prediction probabilities.

[0107] Specifically, the semantic analysis on the column names of the structured table data is performed based on a word embedding model, which can use a Word2Vec model or a FastText model. The column names of the structured table data are converted into semantic vectors by the word embedding model to obtain the semantic vectors of the column names. The data distribution characteristics of each column in the structured table data are determined, such as mean, variance, number of unique values, null rate, value range, etc., and the column content statistical feature vectors are generated based on the data distribution characteristics of each column.

[0108] The semantic vectors of the column names and the column content statistical feature vectors are input into the multi-label classification model to calculate the column type prediction probabilities. The multi-label classification model can use a multi-layer perceptron model or a simple graph network combined with the relationship between columns. The column name semantic vectors and the column content statistical feature vectors can be used by the multi-label classification model to predict the probabilities that the column belongs to the basic data type and the battery semantic type, i.e., to obtain the column type prediction probabilities. Based on the column type prediction probabilities, the prediction probabilities of each column type are compared with a preset threshold value, and the column type with a prediction probability greater than the preset threshold value is selected as the final predicted column type, such as float, int, string, datetime, Voltage, Current, Temperature, etc. Thus, the column type prediction information is obtained.

[0109] S209: performing semantic mapping based on the column type prediction information to obtain a semantic mapping relationship, and converting the file to be parsed into a standard format file based on the semantic mapping relationship;

[0110] In the implementation of the present application, the semantic mapping relationship is obtained by performing semantic mapping based on the column type prediction information, which includes: obtaining concept descriptions in a unified semantic model, calculating semantic similarity based on the column type prediction information and the concept descriptions; performing key entity recognition on the column type prediction information based on a named entity recognition technology to obtain key entity information; performing semantic mapping based on the semantic similarity and the key entity information combined with a preset mapping rule to obtain the semantic mapping relationship.

[0111] Specifically, the concept description in the unified semantic model is obtained, the unified semantic model uniformly encodes the label mode and text in the shared semantic representation, and uses unified token-linking to structure and conceptualize the information in the text. The semantic similarity is calculated based on the column type prediction information and the concept description. The semantic similarity can be calculated by cosine similarity.

[0112] The column type prediction information is subjected to key entity recognition based on a named entity recognition technology, that is, the column name or key entity in the data is identified with the aid of the named entity recognition technology, such as units V, A, ℃, s; battery model 18650, NMC811, to obtain key entity information. The key entity information can provide additional mapping basis, such as confirming the voltage or current through the unit.

[0113] Based on the semantic similarity and the key entity information, semantic mapping is performed in combination with a preset mapping rule to obtain a semantic mapping relationship. The preset mapping rule is, for example, a specific column name mode forced mapping, and the semantic relationship mapping can involve complex mapping such as one-to-many or many-to-one, such as mapping multiple columns of temperature to the Temperature in the unified semantic model. The file to be parsed is converted into a standard format file based on the semantic mapping relationship. According to the established semantic mapping relationship, the original data is converted into a standardized format conforming to the definition of the unified semantic model, and is submitted to the front end for display. The front end display is presented to the user in a visual form.

[0114] Further, the calculation of the semantic similarity based on the column type prediction information and the concept description includes: vectorizing the column type prediction information and the concept description based on a word embedding model to obtain a column type semantic vector and a concept description semantic vector; and calculating the semantic similarity of the column type semantic vector and the concept description semantic vector, the expression of the semantic similarity being:

[0115]

[0116] wherein, is the semantic similarity, is the column type semantic vector, is the concept description semantic vector.

[0117] Specifically, the column type prediction information and the concept description are vectorized based on a word embedding model to obtain a column type semantic vector and a concept description semantic vector. The word embedding model can realize vectorization by converting words into numerical vectors. Common models include Word2Vec model, GloVe model and FastText model. The semantic similarity of the column type semantic vector and the concept description semantic vector is calculated. The semantic similarity can be calculated using the formula of cosine similarity. The expression of the semantic similarity is:​

[0118] ,

[0119] wherein, is semantic similarity, is column type semantic vector, is concept description semantic vector.

[0120] S210: Obtain feedback data of the user on the standard format file, and correct the file format information, column type prediction information and semantic mapping relationship based on the feedback data, and perform closed loop learning of the deep learning model based on the correction result.

[0121] In the specific implementation process of the present application, the closed loop learning of the deep learning model based on the correction result comprises: extracting correction data based on the correction result, and optimizing the preset mapping rule based on the correction data; performing incremental training and closed loop optimization of the deep learning model based on the correction data.

[0122] Specifically, the user views the standard format file in the front end, and views the analysis and mapping data of the standard format file in the front end at the same time, obtains the feedback data of the user on the standard format file, the feedback data is the inspection data of the user on the format, column semantic type label and semantic mapping relationship of the standard format file, and corrects the file format information, column type prediction information and semantic mapping relationship based on the feedback data, that is, corrects the file format information, column type prediction information and semantic mapping relationship according to the inspection data of the user on the standard format file. Extract correction data based on the correction result, that is, collect the data of the user in the correction process after the correction is completed. And optimize the preset mapping rule based on the correction data. Perform incremental training and closed loop optimization of the deep learning model based on the correction data, use the correction data to perform incremental training and fine tuning optimization of the deep learning model used for file format recognition and column type prediction, improve the model performance and the adaptability to data analysis, and form a closed loop optimization.

[0123] In the embodiment of the present application, the discriminative features of the file to be parsed are used for file format identification based on a deep learning algorithm, the file to be parsed is parsed into structured table data based on file format information, column type prediction is performed on the structured table data based on a multi-label classification model, semantic mapping is performed based on column type prediction information, semantic mapping relationships are obtained, and the file to be parsed is converted into a standard format file based on the semantic mapping relationships, which can adapt to different file formats, no longer needs to design rules and maintain extraction templates for each document format, greatly reduces the development cost, and at the same time, does not need to modify the original test data, meets the principle requirements of the integrity and accuracy of battery test data. Feedback data of users on the standard format file is obtained, and the file format information, column type prediction information and semantic mapping relationships are corrected based on the feedback data, and closed-loop learning of the deep learning model is performed based on the correction results, which automates the process of the training model, reduces manual participation, and improves the automation degree and adaptability.

[0124] Embodiment three

[0125] Please refer to Figure 3 , Figure 3 is a structural composition schematic diagram of a file adaptive parsing system of battery test data in the embodiment of the present application, the system comprises:

[0126] The file format identification module 31 is configured to obtain the file to be parsed of the battery test data, extract discriminative features of the file to be parsed, perform file format identification based on a deep learning algorithm using the discriminative features, and obtain file format information.

[0127] The file parsing module 32 is configured to parse the file to be parsed into structured table data based on the file format information.

[0128] The column type prediction module 33 is configured to perform column type prediction on the structured table data based on a multi-label classification model, and obtain column type prediction information.

[0129] The file conversion module 34 is configured to perform semantic mapping based on the column type prediction information, obtain semantic mapping relationships, and convert the file to be parsed into a standard format file based on the semantic mapping relationships.

[0130] The feedback optimization module 35 is configured to obtain feedback data of users on the standard format file, and correct the file format information, column type prediction information and semantic mapping relationships based on the feedback data, and perform closed-loop learning of the deep learning model based on the correction results.

[0131] In the specific implementation process of the present application, the specific implementation mode of the system item can refer to the implementation mode of the above-mentioned method item, which will not be described here.

[0132] In the embodiment of the present application, the discriminative features of the to-be-analyzed file are used for file format identification based on a deep learning algorithm, the to-be-analyzed file is parsed into structured table data based on file format information, column type prediction is performed on the structured table data based on a multi-label classification model, semantic mapping is performed based on column type prediction information, semantic mapping relationships are obtained, and the to-be-analyzed file is converted into a standard format file based on the semantic mapping relationships. The method can adapt to different file formats, does not need to design rules and maintain extraction templates for each document format, greatly reduces the development cost, does not need to modify the original test data, meets the principle requirements of the integrity and accuracy of battery test data, obtains feedback data of the user on the standard format file, corrects the file format information, column type prediction information and semantic mapping relationships based on the feedback data, performs closed-loop learning of the deep learning model based on the correction results, automates the process of the training model, reduces the manual participation, and improves the automation degree and adaptability.

[0133] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0134] In addition, the above describes in detail the battery test data file adaptive parsing method and system provided by the embodiment of the present application, and the principle and implementation manner of the present application are described by using specific examples in this paper. The above embodiment is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A method for file-adaptive parsing of battery test data, characterized in that, The method comprises: obtaining a to-be-resolved file of battery test data, and extracting discriminative features of the to-be-resolved file, performing file format identification on the discriminative features based on a deep learning model to obtain file format information; resolving the to-be-resolved file into structured table data based on the file format information; performing column type prediction on the structured table data based on a multi-label classification model to obtain column type prediction information; performing semantic mapping based on the column type prediction information to obtain a semantic mapping relationship, and converting the to-be-resolved file into a standard format file based on the semantic mapping relationship; obtaining feedback data of the standard format file from a user, and correcting the file format information, the column type prediction information, and the semantic mapping relationship based on the feedback data, and performing closed-loop learning of the deep learning model based on the correction result.

2. The method of claim 1, wherein, The extraction of the discriminative features of the to-be-resolved file, the file format identification based on the discriminative features by the deep learning model, and the obtaining of the file format information comprise: extracting the file signature and the file extension of the to-be-resolved file; extracting the structural complexity features of the to-be-resolved file to obtain structural complexity feature information; extracting the content features of the to-be-resolved file to obtain content feature information; extracting the preset mode features of the to-be-resolved file based on the confidence to obtain preset mode feature information; determining a target deep learning model based on the file signature, the structural complexity feature information, and the content feature information in combination with a preset decision threshold; generating discriminative features based on the file signature, the file extension, the structural complexity feature information, the content feature information, and the preset mode feature information, and performing file format identification based on the discriminative features by the target deep learning model to obtain file format information.

3. The method of claim 2, wherein, The extraction of the structural complexity features of the to-be-resolved file to obtain the structural complexity feature information comprises: extracting the number of columns and the number of rows of the to-be-resolved file; determining the number of rows with consistent separators and the total number of randomly sampled rows in the to-be-resolved file, and determining separator consistency information based on the number of rows with consistent separators and the total number of randomly sampled rows, the expression of the separator consistency information being: wherein, is separator consistency information, is the number of lines with consistent separators, is the total number of randomly sampled lines; performing nested level analysis on the to-be-resolved file based on a numerical estimation method to obtain nested level information, and generating structural complexity feature information based on the number of columns, the number of rows, the separator consistency information, and the nested level information.

4. The method of claim 2, wherein, The extraction of the content features of the to-be-resolved file to obtain the content feature information comprises: determining the number of keywords and the length of the header of the to-be-resolved file, and determining the header keyword occurrence rate based on the number of keywords and the length of the header; determining the number of numerical cells and the total number of randomly sampled cells of the to-be-resolved file, and determining the numerical value proportion based on the number of numerical cells and the total number of randomly sampled cells; determining the number of null cells of the to-be-resolved file, and determining the null value proportion based on the number of null cells and the total number of randomly sampled cells, and determining the content feature information based on the header keyword occurrence rate, the numerical value proportion, and the null value proportion.

5. The method of claim 1, wherein, The parsing of the to-be-parsed file into structured table data based on the file format information comprises: Matching a corresponding parser based on the file format information; Parsing the to-be-parsed file into structured table data based on the parser.

6. The method of claim 1, wherein, The column type prediction of the structured table data based on the multi-label classification model to obtain column type prediction information comprises: Performing semantic analysis on column names of the structured table data based on a word embedding model to obtain a column name semantic vector; Determining data distribution characteristics of each column of the structured table data, and generating a column content statistical feature vector based on the data distribution characteristics of each column; Inputting the column name semantic vector and the column content statistical feature vector into the multi-label classification model to calculate a column type prediction probability, and determining column type prediction information based on the column type prediction probability.

7. The method of claim 1, wherein, The semantic mapping based on the column type prediction information to obtain a semantic mapping relationship comprises: Obtaining a concept description in a unified semantic model, calculating a semantic similarity based on the column type prediction information and the concept description; Performing key entity recognition on the column type prediction information based on a named entity recognition technology to obtain key entity information; Performing semantic mapping based on the semantic similarity and the key entity information in combination with a preset mapping rule to obtain a semantic mapping relationship.

8. The method of file self-adaptive parsing of battery test data according to claim 7, characterized in that, The calculation of the semantic similarity based on the column type prediction information and the concept description comprises: Vectorizing the column type prediction information and the concept description based on a word embedding model to obtain a column type semantic vector and a concept description semantic vector; Calculating a semantic similarity of the column type semantic vector and the concept description semantic vector, and an expression of the semantic similarity is: ， wherein, is a semantic similarity, is a column type semantic vector, is a concept description semantic vector.

9. The method of claim 1, wherein, The closed-loop learning of the deep learning model based on the correction result comprises: Extracting correction data based on the correction result, and optimizing a preset mapping rule based on the correction data; Performing incremental training and closed-loop optimization of the deep learning model based on the correction data.

10. A file self-adapting parsing system of battery test data, characterized in that, The system comprises: A file format identification module configured to obtain a to-be-parsed file of battery test data, extract discriminative features of the to-be-parsed file, perform file format identification based on the discriminative features by using a deep learning model, and obtain file format information; A file parsing module configured to parse the to-be-parsed file into structured table data based on the file format information; A column type prediction module configured to perform column type prediction on the structured table data based on a multi-label classification model to obtain column type prediction information; A file conversion module configured to perform semantic mapping based on the column type prediction information to obtain a semantic mapping relationship, and convert the to-be-parsed file into a standard format file based on the semantic mapping relationship; A feedback optimization module configured to obtain feedback data of the standard format file from a user, and correct file format information, column type prediction information, and a semantic mapping relationship based on the feedback data, and perform closed-loop learning of a deep learning model based on a correction result.

Images