A general data file batch intelligent processing method

Abstract

The application is suitable for the field of data processing, and provides a general data file batch intelligent processing method, which comprises the following steps: reading and analyzing data files with different content formats, unifying the data file formats, and generating encapsulated data; identifying the content of the encapsulated data to determine the matching relationship between the corresponding files of the encapsulated data and the business elements; determining at least one analysis result of the encapsulated data and the corresponding analysis mode of each database field in the uploading process according to the matching relationship, so as to fill in the database field, wherein the analysis mode comprises one or more of file column analysis, file name analysis and content area analysis before the table header; and performing encapsulated data reading and warehousing operation according to the warehousing filling result, wherein the application has the beneficial effect that when a large amount of business data with different formats is imported, the complexity and time consumption of the user are significantly reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data processing, and particularly relates to a general data file batch intelligent processing method. BACKGROUND

[0002] Importing data files into a database has always been an important prerequisite for data analysis and utilization.

[0003] In the prior art, when performing file data warehousing, the following schemes are used: 1. using database commands to perform batch import of files; 2. using database third-party tools such as kettle and navicat to perform file import of Excel, csv, txt, etc.

[0004] Firstly, the above methods require that the file columns correspond to the database columns one by one, and can only complete simple column-to-column matching data import, and are difficult to support data parsing and import of complex content formats. However, in actual situations, the formats of the files obtained are complex and diverse, and the file columns may not correspond to the database columns one by one. In many cases, a field required by a database column may exist in a file name, a sheet name, a content before a table header, or even need to be inferred from other columns of the file. The above prior art cannot implement these scenarios. Secondly, in the scenario of batch file data import, the above methods require that the data formats of the files in the batch must be consistent, otherwise batch data import cannot be performed. However, in actual situations, even the same business data has various data formats due to various data sources, resulting in various file data formats obtained. For example, call record data obtained from different operators all have basic elements of call relationship, but the file content formats are different. In traditional batch import, the data formats of various sources need to be normalized manually. If the number of processed files is large, it is time-consuming and laborious. SUMMARY

[0005] The embodiment of the present application aims to provide a general data file batch intelligent processing method, and aims to solve the problems in the background art.

[0006] The embodiment of the present application is implemented in the following manner. On the one hand, a general data file batch intelligent processing method comprises the following steps:

[0007] Reading and analyzing data files of different content formats to unify the data file formats and generate encapsulated data;

[0008] Performing content recognition on the encapsulated data to determine the matching relationship between the corresponding files of the encapsulated data and business elements;

[0009] determining at least one parsing result corresponding to each database field in the uploading process according to the matching relationship, so as to fill in the database field, wherein the parsing manner includes one or more of file column parsing, file name parsing and content area parsing before the table header;

[0010] According to the filling result, the encapsulation data is read and the operation of warehousing is performed.

[0011] As a further scheme of the present application, the data files of different content formats are read and parsed, and the data file formats are unified to generate encapsulation data, specifically including:

[0012] The format of the uploaded file data is read;

[0013] The file content is read according to the organization mode of the data file of different formats;

[0014] The read file content is uniformly encapsulated into data in two-dimensional array format.

[0015] As a further scheme of the present application, the content of the encapsulation data is identified to determine the matching relationship between the corresponding file of the encapsulation data and the business element, specifically including:

[0016] The table header content of the data file is identified, and the starting position of the table header content in the file is located;

[0017] The business type of the data file is confirmed according to the table header content;

[0018] According to the table header content, the business type and the pre-maintained business element data characteristics, the matching relationship between the column in the data file and the business element is identified and maintained.

[0019] As a further scheme of the present application, the matching relationship between the column in the data file and the business element is identified and maintained, specifically including:

[0020] When the file column is parsed, whether the data file contains the caller number column is found according to the matching relationship;

[0021] If it contains, it is determined that the caller number column will be parsed from the found file column in the final parsing.

[0022] As a further scheme of the present application, the method further includes:

[0023] When the data file does not contain the caller number column, the caller number is identified and parsed from the file name;

[0024] If the identification and parsing are successful, the field value is parsed;

[0025] If the field value is parsed, the parsing result is further cleaned and verified to determine whether the parsing result meets the requirements;

[0026] If the parsing result meets the requirements, the parsing result is backfilled to the database caller number column.

[0027] As a further scheme of the present application, the method further comprises:

[0028] If no result is obtained by parsing the file column and parsing the file name, content extraction is first performed on the area before the table header in the file content;

[0029] The content extraction includes row extraction, and the specific steps of the row extraction include: extracting all data before the table header in the row dimension, that is, concatenating all columns of each row into a complete string, and multiple rows will finally generate multiple strings;

[0030] The content extraction also includes column extraction, and the specific steps of the column extraction include: extracting the content area before the table header in the column dimension, that is, concatenating the data of each column into a string, and multiple columns will finally generate multiple strings;

[0031] Based on the at least one string generated by the above extraction, a string similar to the file name is recognized.

[0032] As a further scheme of the present application, the method further comprises:

[0033] The matching relationship is updated by accepting user adjustment of the matching relationship;

[0034] The data file and the confirmed parsing method are saved in the task to be executed, and the filling into the database is waited.

[0035] As a further scheme of the present application, before the encapsulation data reading and the database operation are performed, the method further comprises:

[0036] According to the configured cleaning and verification rules, the validity of the value of each field is judged to generate legal data and illegal data;

[0037] The legal data is processed into the database;

[0038] For the illegal data, the illegal data rows are re-collected and arranged, and the reason why each row of data does not meet the requirements is clearly marked.

[0039] As a further scheme of the present application, the method further comprises:

[0040] When the task containing the warehouse filling result is submitted to the background, according to the number of tasks currently being executed, the number of tasks currently required to be executed is confirmed, and data reading and warehousing operations are performed according to the previously determined analysis mode.

[0041] The general data file batch intelligent processing method provided by the embodiment of the application shields the strict requirement for file format during import by starting from the business characteristics and the support of various analysis modes, and significantly reduces the complexity and time consumption of users when facing large amounts of business data import with different formats. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 It is a flow chart of header identification in a general data file batch intelligent processing method.

[0043] Figure 2 It is a flow chart of analysis processing in a general data file batch intelligent processing method.

[0044] Figure 3 It is a flow chart of verification and confirmation in a general data file batch intelligent processing method.

[0045] Figure 4 It is a configuration schematic diagram of fields in the first layer of a template in a general data file batch intelligent processing method.

[0046] Figure 5 It is a configuration schematic diagram of fields in the second layer of a template in a general data file batch intelligent processing method.

[0047] Figure 6 It is a configuration schematic diagram of fields in the third layer of a template in a general data file batch intelligent processing method.

[0048] Figure 7 It is a schematic diagram of format one in a general data file batch intelligent processing method.

[0049] Figure 8 It is a schematic diagram of format two in a general data file batch intelligent processing method. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the application more clear, the application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0051] The specific implementation of the application is described in detail below in combination with specific embodiments.

[0052] The application provides a general data file batch intelligent processing method, and solves the technical problems in the background art.

[0053] Before the specific steps are implemented, some key concepts in the field are first described in detail in combination with the implementation steps.

[0054] The entire identification and analysis process relies on the field characteristics and analysis characteristics and verification requirements of all elements of the specified business collected in advance, and these characteristics and requirements are maintained as an analysis template. This template is different from the file template in the traditional sense, and does not have strict requirements on the file content format, but only requires that the field characteristics and analysis characteristics of the business elements are maintained as perfect as possible.

[0055] First, several important concepts are described. First, the "field characteristics" include the field name, the alias set of the field in the business field, the data type of the field, the data format verification requirement of the field, and the cleaning requirement of the field data. By collecting the field characteristics of all elements of the business in advance, a basic understanding of the business is formed.

[0056] Second, the "field analysis method" is used to confirm the value method of the column required by the database. Traditional import is to read the data required by the database field from the file column. In addition to analyzing data from the file column, the present scheme provides other value methods, such as file name content analysis, table header front content area text recognition analysis, other related field combination, split analysis, etc. These analysis methods make the content format of the file no longer consistent.

[0057] Then, the "analysis result verification requirement" is described. For each determined data import, the column required by the database is clear. After the identification is completed, it is necessary to judge whether the identification result meets the requirements of the column required by the database. For example, the columns in the database include: caller number, called number, call initiation time, call duration, call base station location, call type, caller number home location, called number home location, etc. If the minimum requirement is to identify and analyze the caller number, called number, call initiation time, and call duration, then in the last stage of identification, it is necessary to completely verify whether the identification result meets the requirements.

[0058] Finally, the template is analyzed. The template is an integration of the above information. The template is divided into three layers. The core responsibility of the first layer is to identify each column in the file. The first layer maintains all elements of the current business and the field characteristics of the elements. In this way, we can identify the correspondence between the columns in the file and the business elements according to the field characteristics. The second layer is responsible for maintaining the parsing method of the columns in the database and the business element fields it depends on. Moreover, each database column generally maintains multiple parsing methods. This is because the field required by the database may be distributed in different positions in different files. That is, the first layer is responsible for identifying the file header. The second layer determines how to parse the required field in the database according to the identification result of the first layer. The third layer maintains the verification logic of the parsing result to finally check the identification result.

[0059] As shown in FIG. 1, a general data file batch intelligent processing method provided by an embodiment of the present application includes the following steps. Figures 1-3

[0060] Step S10: reading and analyzing data files of different content formats, unifying the data file formats, and generating encapsulated data.

[0061] The file analysis link is mainly used for reading and analyzing data of different formats (such as xslx, csv, txt, etc.), unifying the data formats, and providing data support for subsequent identification analysis. The core logic of this link is that after the user uploads the file, the file is processed by different parsers according to the file format. Each parser reads the file content according to the organization method of the different format files. After the parser reads the file data, the read content is uniformly encapsulated as two-dimensional array format data, which is handed over to the content identification link for processing.

[0062] Step S11: content identification of the encapsulated data to determine the matching relationship between the corresponding file and the business elements of the encapsulated data.

[0063] The content identification link needs to solve four problems. The first problem is to find the starting row of the file content. The second problem is to find the starting position of the table header in the file. The third problem is to confirm the business type of the data according to the table header content. The fourth problem is to automatically identify and maintain the matching relationship between the columns in the file and the business elements according to the table header content, the business type obtained in the third step, and the data characteristics of the pre-maintained business elements. In the identification of the matching relationship, name matching, alias matching, and string similarity matching are mainly used. After the matching is completed, some special fields will be further checked and corrected according to the data characteristics.

[0064] ​Step S12: According to the matching relationship, determine the encapsulation data and at least one parsing result corresponding to each database field in the uploading process, to fill in the database field, wherein the parsing method includes one or more of file column parsing, file name parsing and table header content area parsing identification;

[0065] The parsing matching link is mainly to determine how the fields required by the database are obtained from the file. This link relies on the multiple parsing methods configured in the second layer of the template, and attempts to parse one by one according to the priority, for example: the calling number field in the call data is parsed, and we maintain three parsing methods, assuming that the first method is to parse from the file column, then we will search the file according to the first layer matching result to find out whether the calling number column is included, if it is included, then the field will be parsed from the found file column in the final parsing, assuming that the second method is file name parsing, when the calling number column is not found in the file, we will try to identify and parse the calling number from the file name, and the identification process relies on label semantics, string similarity matching and other identification methods to parse the field value, if the value can be parsed, the parsed result will be further cleaned and verified to determine whether the result meets the requirements, if it meets the requirements, the result will be filled back to the calling number column of the database, assuming that the third method is to identify from the content area before the table header, then when the two methods are not identified, we will first extract the content area before the table header from the file content, the extraction is divided into row extraction and column extraction, the row extraction is to extract all the data before the table header according to the row dimension, that is, to concatenate all the columns of each row into a complete string, multiple rows will finally generate multiple strings, the column extraction is to extract the content area before the table header according to the column dimension, that is, to concatenate the data of each column into a string, multiple columns will finally generate multiple strings, then the strings generated by the above extraction are processed according to the same identification logic as the file name identification, in the actual scene, each database field will have multiple parsing methods, and the appropriate parsing method will be automatically matched according to the structure characteristics of each file during the file uploading process;

[0066] Step S13: According to the filling result, execute the encapsulation data reading and storage operation.

[0067] The logic of the task execution link is relatively simple. When the task is submitted to the background, the task scheduling module confirms how many tasks to pull up for execution according to the number of tasks that are currently being executed. Since the matching stage has clearly defined the matching relationship between the data files and the business elements contained in each task, as well as the parsing and value taking mode of each database field, this stage mainly performs data reading and warehousing operations according to the logic determined previously. Before reading the data for warehousing, the value of each field can also be judged for validity according to the configured cleaning and verification rules. Finally, a batch of legal data and a batch of data that does not meet the requirements (this part is generally due to the fact that some rows of data in the file data are incomplete, important fields are missing, or important field values are desensitized, resulting in data losing analysis significance) are obtained. The data that meet the requirements are warehoused. The data rows that do not meet the data verification requirements are reorganized into a new file, and the reason why each row of data does not meet the requirements is clearly marked.

[0068] The above steps mainly include file parsing, content identification, parsing mode matching, task submission, and task execution. The links are in a progressive relationship. The above implementation steps rely on the data characteristics of business elements and the support of various parsing modes by the present scheme. Starting from business characteristics and parsing mode, the strict requirements for file format during import are shielded, which significantly reduces the complexity and time consumption of users when facing large amounts of business data import with different formats.

[0069] As a preferred embodiment of the present application, the content identification of the packaged data to determine the matching relationship between the corresponding file of the packaged data and the business element specifically includes:

[0070] Step S101: identifying the table header content of the data file and locating the starting position of the table header content in the file;

[0071] Step S102: confirming the business type of the data file according to the table header content;

[0072] Step S103: identifying and maintaining the matching relationship between the columns in the data file and the business elements according to the table header content, the business type, and the pre-maintained data characteristics of the business elements.

[0073] As a preferred embodiment of the present application, the identification and maintenance of the matching relationship between the columns in the data file and the business elements specifically includes:

[0074] Step S201: when parsing from the file column, searching whether the data file contains the caller number column according to the matching relationship;

[0075] Step S202: If contained, determine whether the calling number column is listed in the file column found in the final analysis.

[0076] As a preferred embodiment of the present application, the method further comprises:

[0077] Step S111: When the calling number column is not contained in the data file, identify and analyze the calling number from the file name;

[0078] Step S112: If the identification and analysis is successful, perform field value analysis;

[0079] Step S113: If the field value is analyzed, further clean and verify the analysis result to determine whether the analysis result meets the requirements;

[0080] Step S114: If the analysis result meets the requirements, backfill the analysis result to the calling number column of the database.

[0081] As a preferred embodiment of the present application, the identification and analysis method comprises:

[0082] Step S121: In the identification process, at least one of the label semantics and the string similarity matching is used, and when the matching degree of the corresponding item reaches a preset matching degree, it is determined that the identification and analysis is successful.

[0083] As a preferred embodiment of the present application, the method further comprises:

[0084] Step S131: If no result is obtained through the file column analysis and the file name analysis, first perform content extraction on the area before the table header in the file content;

[0085] Step S132: The content extraction comprises row extraction, and the specific steps of the row extraction comprise: extracting all data before the table header in the row dimension, that is, concatenating all columns of each row into a complete string, and multiple rows will finally generate multiple strings;

[0086] Step S133: The content extraction further comprises column extraction, and the specific steps of the column extraction comprise: extracting the content area before the table header in the column dimension, that is, concatenating the data of each column into a string, and multiple columns will finally generate multiple strings;

[0087] Step S134: Based on at least one string generated by the above extraction, identify a string similar to the file name.

[0088] As a preferred embodiment of the present application, the method further comprises:

[0089] Step S141: Accept the user's adjustment of the matching relationship, and update the matching relationship;

[0090] Step S142: accept the user's confirmation of the update result, save the data file and the confirmed parsing method in the task to be executed, and wait for the warehouse filling.

[0091] The matching and parsing results are also pushed to the user's foreground in real time before task execution for user confirmation. If the user believes that the matching relationship is incorrect, the user can also adjust it, and then the background automatically refreshes the parsing logic. In general, no intervention is required. If the table header name in the file column is seriously deviated from the business meaning, intervention is required. After the user confirms, the user can submit it. When submitting, the background saves the file and the parsing method for the current file that has been explicitly saved as a task to be executed, which is processed by the subsequent task scheduling logic. This link is the task submission link.

[0092] As a preferred embodiment of the present application, before performing the encapsulation data reading and warehouse operation, the method further comprises:

[0093] Step S151: according to the configured cleaning and verification rules, the validity of the value of each field is judged, and legal data and illegal data are generated;

[0094] Step S152: legal data is processed into the warehouse;

[0095] Step S153: for those illegal data, according to the illegal data row, re-collection is arranged, and the reason why each row of data does not meet the requirements is clearly marked.

[0096] As shown in Figure 5 As another preferred embodiment of the present application, a general data file batch intelligent processing method, the method further comprises:

[0097] When the task containing the warehouse filling result is submitted to the background, according to the number of tasks currently being executed, the number of tasks to be executed is confirmed, and data reading and warehouse operation are performed according to the previously determined parsing method.

[0098] The related steps will be described below in conjunction with specific examples.

[0099] Example one:

[0100] This embodiment is explained and described in conjunction with the call data import function in the actual project call analysis product. It is assumed that the call data business feature template collected in advance is as follows. In order to facilitate understanding, this paper takes the parsing of the caller number as an example to show the configuration of this field in the first layer, the second layer and the third layer of the template:

[0101] The first layer configuration is as follows Figure 4 :

[0102] The second layer configuration is as follows Figure 5 :

[0103] The third layer configuration is as follows Figure 6 :

[0104] Suppose we want to import the call data in the following two formats, format one: the calling number is in the file column, format two: the calling number is in the description area before the table header.

[0105] Format one is as follows Figure 7 ;

[0106] Format two is as follows Figure 8 ;

[0107] First, the relevant English terms are explained as follows

[0108]

[0109] I. sheetHead explanation

[0110] The sheetHead item contains two parts of information, the row where the file table header is located, and the starting row information of the data in the file. When extracting the table header, the table header recognition algorithm and the string similarity matching algorithm will be used. The principle of the table header recognition algorithm is to read the data by row, and then judge whether the number of columns in the continuous rows is constant. If it is, the first row of the continuous rows is considered to be the table header. The main function of the string similarity matching algorithm is to correspond the template column and the file table header column.

[0111] II. allTemplateColumns explanation

[0112] This is the first layer of the template, which is based on the business element field, defines all the elements that will appear in the business and its characteristics, and then discovers the matching file columns from the uploaded file. Maintain the matching relationship between these business element fields and file column fields. This item is an array item, which is the collection of field characteristics of the current business element. Each object represents a business element field. Each object has the following properties: the templateColumnId property is the unique identifier of the current field in the entire template; the templateColumnName property is the name of the current business element field; the templateColumnDesc property is the explanation of the current business element field; the matchedExcelColumnIndex property represents the subscript of the column in the file that the current business element field matches; the default value of the matchedExcelColumnIndex property is -1. During the automatic recognition and matching process, if a matching relationship is recognized, the value of the matchedExcelColumnIndex property is changed to the column subscript of the matched file column; the checker property is a checker, which is an array indicating a series of data format requirements that the current field must meet, defining different verification rules, which can be defined according to the actual business scenario, such as non-empty verification, string length verification, regular matching of ID cards, mobile phone numbers, etc.; the excelProbNames property is an array that enumerates the business aliases of the current business element field, which is an important basis for matching the file column and the business element field on the template. The contentCheck property indicates whether content format needs to be checked in addition to column name matching during column matching. true indicates that content detection is performed, and false indicates that content detection is not performed. The cleanBeforeMatch property indicates whether cleaning is required before field matching. The cleaner property defines the cleaning rules, which support multiple cleaning rules.

[0113] III. allDBTableColumns Description:

[0114] This is the second layer of the template, which is based on database fields, defines the possible parsing of each database field and the dependency of each parsing. The dependency generally refers to whether the current parsing method depends on the business element field matched in the file, that is, the matching relationship of the first layer is a basis for the second layer to parse data. This item is an array item, and the attributes of each object are as follows. The displayOnWeb attribute indicates whether the current field needs to be paid attention to in the foreground. In the parsing process, some database fields are implicitly parsed according to other fields in the file, and the parsing result does not need to be confirmed by the user. This configuration needs to exclude this part of the field from the user confirmation range. False means no attention is needed. The parserGroups attribute is the core configuration of the parsing method of the current database field and is an array item. Each item is a configuration of a parsing method. The parsing method is defined by the parser attribute. The optional value of the parser attribute is the name of the built-in centralized parsing method in this scheme. The refTemplateColumn attribute specifies the id set of the business elements (templateColumnId of one or more business elements in the allTemplateColumns in the first layer template) that the current parsing method depends on.

[0115] The value attribute stores the parsing result of some constant fields. Generally, an excel file is read in a row and column to extract the required data under normal circumstances. However, some information is not displayed in the form of rows and columns, such as the values displayed in the descriptive information at the beginning of the file. In this case, the value is directly parsed and saved in the value attribute, and the value of this field is directly obtained from the value attribute when it is stored in the database.

[0116] The valueType attribute indicates the type of the value of the current field. Currently, the value of this item can be single, muilty, enum, dict, and dict_ref. Among them, single and enum indicate that the value of the current field is parsed from the file description content, and the user can modify it by input. dict indicates that the value of the current field comes from a dictionary, and the user can modify it by selection. The dict_ref attribute indicates that the content of this item is a dictionary item, but when it is displayed, the value of the current field and the dictValue of the column referenced by the current field need to be read and a corresponding relationship between the two is established.

[0117] The webShow attribute defines how to display in the foreground. Currently, the value of this item can be value or column. Value indicates that the value of the field is directly displayed in the foreground, and column indicates that the column matching relationship is displayed.

[0118] The webShowName attribute defines the selection of the name when it is displayed in the foreground. The value of this attribute can be this and reference. This means that the name of the current field is displayed when it is displayed in the foreground, and reference means that the name of the field referenced by the current field is displayed when it is displayed in the foreground. Generally, only when webShow is value, webShowName can be this. In most cases, it is reference.

[0119] The webShowValueConfig attribute defines some limited information when it is displayed in the foreground, such as single selection, multiple selection, etc. Generally, only when webShow is value, the current field will have a value. The defaultDictCode attribute defines the default value of the current field, which is the code in the dictValues of the template column referenced by the current field. The chooseType describes whether the current field is single selection or multiple selection.

[0120] Four、requiredColumnGroup explanation:

[0121] This attribute is an array, indicating which database fields of the current business are required. After the first layer of relationship matching and the second layer of data parsing, we now know which database fields can be parsed from the file. According to the requiredColumnGroup configuration information of the necessary database fields, we can determine whether the current file recognition and parsing meets the requirements, or whether the data provided by the file meets the requirements. The properties of each object are as follows: groupDesc indicates the description of the attribute field, id is the unique identifier, and refParseColumnName configures the name of the second layer of database fields.

[0122] Now we need to parse and identify the calling number column data in the two file formats according to the above template configuration. The identification method of other data columns is consistent with the main number.

[0123] Step one:

[0124] After the two files are uploaded, the data format of the files is first determined, and then the files are assigned to different file parsers for processing, such as assigning an excel file to an excel parser for processing, and assigning other format files to corresponding format file parsers. After reading the data of the above excel file, the first step is to identify the table header. The main purpose of table header identification is to confirm the starting position of the table header data in the file. The general principle of identification is to start from the first row of the file and traverse from the first non-empty row to find continuous rows of data with equal column numbers. Further table header confirmation is performed on the rows before and after the starting row of the continuous data. Some rows with less Chinese character proportion and more number proportion are excluded during confirmation. Then, according to the data characteristics of various business elements configured before, fuzzy search and comparison are performed, and finally the table header of format one is identified as the first row and the table header of format two is identified as the third row.

[0125] Step two:

[0126] After the position of the table header is determined, a new backup is generated based on the original template before each matching starts. The matching relationship of each file is performed based on the new backup to avoid mutual influence between files of different formats. The first layer of template matching is performed first. Each field in allTemplateColumns is traversed. Assuming that the matching of the caller number is started now, it can be seen that the alias of the caller number on the template will hit the "caller" field in format one, which is located in the second column of format one. At this time, the matchedExcelColumnIndex of the caller number field in allTemplateColumns in the template for parsing format one file will become 2. However, in format two, the caller number field is not found in the table header. At this time, the matchedExcelColumnIndex of the caller number in the template for parsing format two is still -1, indicating that the matching relationship is not found. Other fields are sequentially matched. Whether the matching relationship is found or not, the result will be marked in allTemplateColumns.

[0127] After the first round of matching, the next step is parsing. In the second layer of the template, we can see that the caller number is configured with two parsing methods. The parsing methods are configured in parserGroups, which are represented by two objects. The parserName attribute of the parser attribute of each object represents the name of the specific parsing method. According to the configuration, the two parsing methods for the caller number are columnMatch and contentBeforeHead. The refTemplateColumnName in the columnMatch method indicates that this method depends on matching the call_phoneno field in the first layer of matching. The contentBeforeHead method does not depend on the matching results of any field in the first layer.

[0128] Now let's start the caller number parsing logic from format one. The template starts to process each field in allDBTableColumns. When it processes the caller number, it starts to process each parsing group of the caller number. First, it judges whether the dependent field of the first parsing group meets the requirements. According to the previous matching results, the caller number of format one matches the second column, and the dependency is met. At this time, the template that parses the format one file will change the matchedGroupId of the caller number in allDBTableColumns to the id of the columnMatch parsing method, which is 1 according to the configuration. This means that when parsing this field, the columnMatch parsing method can be used to parse it.

[0129] The caller number is parsed from the format two, which is consistent with the above manner. The parsing group of the caller number is traversed. First, it is judged whether the dependency of the first parsing group is satisfied. As can be known from the matching relationship, the caller number does not exist in the format two, and therefore the dependency of the first parsing group is not satisfied. At this time, it is considered that the first parsing group cannot parse the caller number in the file. Then, the second parsing group is parsed. The second parsing group does not exist the dependency condition. At this time, the dependency is defaulted to pass. The second parsing group is contentBeforeHead. When judging whether the parsing group can be parsed, the parsing group will judge whether the target value can be parsed in the table header area in addition to judging the dependency. Taking the parsing of the caller number in the format two as an example, first, the area before the table header is extracted. It is known that the table header is in the third row, and therefore the data in the first two rows is extracted. After extraction, each row is spliced into a complete string. The string "caller number: 151****4787" can be extracted. Then, the string is compared with the alias of the caller number maintained in the first layer. It is found that the string in the string contains the string "caller number" which is equal to or has a high degree of approximation with the alias, and therefore it is considered that the value is a possible value. Then, the cleaning and checking of the characteristics of the current field are performed. Whether the original data requirements are satisfied is further judged. If the requirements are satisfied, the parsed result is filled back to the value attribute of the contentBeforeHead parsing group. This indicates that the caller number in each row of data is a fixed value when the data is stored in the database in the future. Then, the matchedGroupId of the caller number in allDBTableColumns is changed to the id of the contentBeforeHead parsing manner. As can be known from the configuration, the value is 2, which indicates that the contentBeforeHead parsing is used when the caller number is parsed. Other fields will also be parsed according to the parsing manner configured by the fields.

[0130] Step three:

[0131] After the above flow is executed, the verification link is reached, and an important basis for confirming whether the identification analysis is successful after the identification is completed is whether the set of resolvable fields obtained can meet the necessary field requirements of the database. According to the third layer configuration, the necessary fields are the caller number, the called number, the call time, and the resolvable set is compared to preliminarily determine success or failure. Finally, we will convert the identification result into structured data and display it to the user in a friendly manner. If the user needs to intervene, he can adjust the first layer matching relationship. When adjusting, we will perform steps two and three again according to the user's adjustment results and feedback to the user until the user is satisfied. If the user does not need to adjust, the import task can be directly submitted. When submitting, it will contain the file and the parsing method information of the fields in the current file. From the use experience, when the business element template is maintained well, the user almost does not need to intervene when facing files with different formats.

[0132] When the user submits the import task, the task scheduling module in the background will pull the submitted task according to the number of tasks being executed. Since the parsing method of each field has been confirmed, during task execution, each row of data in the file is directly traversed, and the value of each field is parsed according to the parsing method of each field

[0133] The above embodiment of the present application provides a general data file batch intelligent processing method, which supports diversified and complex content data file identification and analysis. Traditional data file parsing and import is for processing single structured data format files, such as an Excel file, the first line is the header, and the second line is data with the same data line format. The data format is single, and the data complexity is low. However, if a complex Excel file is encountered, for example, there are multiple table data with different formats in a sheet, the traditional data import cannot meet the requirements. The present application provides a corresponding solution to this situation, ensuring that a business data with a certain feature can be automatically identified and parsed and imported even if it exists in multiple files with inconsistent content formats.

[0134] Support batch recognition import, real-time correction, support multiple files of multiple business types uploaded and parsed at one time, for example, we have a batch of call record data with different formats and a batch of express logistics data with different formats to import, the user only needs to upload these files to the server at one time, after the user uploads the files, relying on the characteristics of the previously maintained business data background, the data type of these files (that is, which business they belong to, such as call data or express data) is automatically judged, then the automatic identification and judgment result will be returned to the user in a structured data, the user can selectively correct the identification result, then the import can be submitted, after submission, the background will create a series of import tasks according to the final identification result, then it will be executed in turn according to a certain rhythm;

[0135] Support independent optimization, the program will record the user's correction, and will refer to the user's correction history in subsequent header recognition, and will prioritize the user's corrected priority for header recognition;

[0136] Support custom data cleaning, verification, and parsing rules, using the present application, users can customize data cleaning rules, verification rules, and parsing rules, covering all aspects of data processing;

[0137] In order to be able to load the above-mentioned method and system to run smoothly, in addition to the various modules described above, the system can include more or less components than described above, or combine certain components, or different components, for example, it can include input and output devices, network access devices, buses, processors and memories, etc.

[0138] The so-called processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The above processor is the control center of the system, and is connected with various parts by various interfaces and lines.

[0139] It should be appreciated that, although the steps in the flowcharts of the embodiments of the present application are shown in a sequence following the arrows, the steps are not necessarily executed in the order of the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, at least some of the steps in the embodiments can include a plurality of sub-steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution of the sub-steps or stages is not necessarily sequential, but can be performed in rotation or alternation with at least some of the other steps or sub-steps or stages of other steps.

[0140] The technical features of the above-described embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above-described embodiments are described, but it should be understood that any combination of the technical features is within the scope of the present application as long as the combination does not result in contradictions.

[0141] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

[0142] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims. The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A general data file batch intelligent processing method, characterized in that The method comprises the following steps: reading and parsing data files in different content formats, unifying the data file formats, and generating encapsulated data; content recognition of the encapsulated data to determine the matching relationship between the corresponding files of the encapsulated data and the business elements, and to determine at least one parsing result of the encapsulated data and the corresponding parsing mode of each database field in the uploading process, relying on a pre-collected and maintained parsing template containing the field characteristics of all elements of the specified business, the multiple parsing modes of the database columns and the dependent business element fields, and the parsing result verification logic; wherein the second layer of the parsing template is responsible for maintaining the multiple parsing modes of the columns in the database and the dependent business element fields, and each database column maintains multiple parsing modes; determining at least one parsing result of the encapsulated data and the corresponding parsing mode of each database field in the uploading process according to the matching relationship, specifically including: for each database field, trying the multiple parsing modes one by one according to the priority to determine the appropriate parsing mode to fill in the database field, wherein the parsing mode includes file column parsing, file name parsing and content area parsing before the table header; performing encapsulated data reading and database filling operation according to the database filling result.

2. The universal data file batch intelligent processing method according to claim 1, characterized in that The reading and parsing of data files in different content formats, the unification of the data file formats, and the generation of encapsulated data specifically include: reading the format of the uploaded file data; reading the file content according to the organization mode of the data file in different formats; unifying the read file content into two-dimensional array format data.

3. The universal data file batch intelligent processing method according to claim 1, characterized in that The content recognition of the encapsulated data to determine the matching relationship between the corresponding files of the encapsulated data and the business elements specifically includes: identifying the table header content of the data file and locating the starting position of the table header content in the file; confirming the business type of the data file according to the table header content; identifying and maintaining the matching relationship between the columns in the data file and the business elements according to the table header content, the business type and the pre-maintained business element data characteristics.

4. The universal data file batch intelligent processing method according to claim 3, characterized in that The identification and maintenance of the matching relationship between the columns in the data file and the business elements specifically include: when parsing from the file column, checking whether the data file contains the caller number column according to the matching relationship; if it contains, determining that the caller number column will be parsed from the found file column in the final parsing.

5. The universal data file batch intelligent processing method according to claim 4, characterized in that The method further comprises: when the data file does not contain the caller number column, identifying and parsing the caller number from the file name; if the identification and parsing is successful, performing field value parsing; if the field value is parsed, further cleaning and verifying the parsing result to determine whether the parsing result meets the requirements; if the parsing result meets the requirements, filling the parsing result back to the caller number column of the database.

6. The universal data file batch intelligent processing method according to claim 5, characterized in that The identification and parsing mode includes at least one of label semantics and string similarity matching in the identification process, and when the matching degree of the corresponding item reaches the preset matching degree, it is determined that the identification and parsing is successful.

7. The universal data file batch intelligent processing method according to claim 1, characterized in that The method further comprises: If no result is obtained by file column analysis and file name analysis, content extraction is performed on the area before the table header in the file content; The content extraction includes row extraction, and the specific steps of the row extraction include: extracting all data before the table header in the row dimension, that is, concatenating all columns of each row into a complete string, and multiple rows will finally generate multiple strings; The content extraction also includes column extraction, and the specific steps of the column extraction include: extracting the content area before the table header in the column dimension, that is, concatenating the data of each column into a string, and multiple columns will finally generate multiple strings; Based on the at least one string generated by the above extraction, a string similar to the file name is recognized.

8. The universal data file batch intelligent processing method according to claim 4 or 5 or 6, characterized in that The method further includes: accepting user adjustment of the matching relationship and updating the matching relationship; accepting user confirmation of the update result, saving the data file and the confirmed parsing mode in the task to be executed, and waiting for the data filling operation.

9. The universal data file batch intelligent processing method according to claim 1, characterized in that Before performing the encapsulated data reading and the warehousing operation, the method further includes: According to the configured cleaning and verification rules, the validity of the value of each field is judged to generate legal data and illegal data; the legal data is warehoused; for those illegal data, according to the illegal data rows, the illegal data is re-collected and arranged, and the reason why each row of data does not meet the requirements is clearly marked.

10. The universal data file batch intelligent processing method according to any one of claims 1-7, characterized in that The method further includes: When the task containing the warehousing filling result is submitted to the background, according to the number of tasks currently being executed, the number of tasks currently to be executed is confirmed, and the data reading and warehousing operation are performed according to the previously determined parsing mode.

Images