General recording processing method, device, electronic device and storage medium

Abstract

The present application discloses a general record processing method, device, electronic device, and storage medium, and relates to the field of cloud computing technology. The specific implementation scheme is as follows: identifying the header style of the original record; extracting the record row from the original record based on the header style; matching the original field in the original record with the preset standard field; in the extracted record row, replacing the corresponding original field with the successfully matched standard field to generate a general record text. The embodiment of the present application can automatically parse records and perform record field standardization processing, providing record data in a unified standard format, which can greatly improve processing efficiency and save human resources.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to the field of information processing technology. Background Art

[0002] Current application platforms often rely on data from disparate data sources, resulting in poor data compatibility and difficulty in unification. For example, current commodity trading platforms typically obtain standardized supply data by specifying standard templates for different commodity categories and requiring users to fill out forms accordingly. Commodity trading platforms aim to obtain supplier data in a relatively standardized manner, but each supplier has its own unique data template, making standardization difficult. For commodity trading platform users, existing supply data is already stored in a specific format, and the cost of re-formatting or standardizing it is high, making the platform's entry point high. If the trading platform were responsible for manually converting the data format or developing data format conversion tools for each new supply data template, this would introduce a significant amount of duplication and waste manpower. In summary, the traditional approach of creating a data template comparison table for each data source places a heavy workload on business personnel, resulting in duplication and low efficiency. Summary of the Invention

[0003] The embodiments of the present application provide a universal recording processing method, device, electronic device, and storage medium to at least solve the above technical problems in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a general record processing method, including:

[0005] Identify the header style of the original record;

[0006] Extract record rows from original records based on header patterns;

[0007] Match the original fields in the original record with the preset standard fields;

[0008] In the extracted record rows, the corresponding original fields are replaced with the successfully matched standard fields to generate a universal record text.

[0009] In the embodiment of the present application, records can be automatically parsed and record fields can be standardized to provide record data in a unified standard format, which can greatly improve processing efficiency and save human resources.

[0010] In one embodiment, identifying the header style of the original record includes:

[0011] Determine the header row range of the original record;

[0012] In the header row range, match the target field in each record row with the preset header keyword;

[0013] If all target fields in the record row successfully match the corresponding header keywords, the record row is determined to be an exact match;

[0014] The record rows that are accurately matched are used as the header rows.

[0015] In the embodiment of the present application, identifying the header style through precise matching is an important step in batch locating table records, and provides a positioning basis for the subsequent step of extracting records.

[0016] In one embodiment, the method further comprises:

[0017] If the exact match of the record rows within the header row range is unsuccessful, a first matching mixed index is calculated, where the first matching mixed index is a matching mixed index between the target field in each record row and the preset header keyword;

[0018] If the first matching degree hybrid index of all target fields in the record row and the corresponding header keywords is greater than or equal to a first preset threshold, it is determined that the record row is fuzzy matched successfully;

[0019] The record rows that successfully match the fuzzy match are used as the header rows.

[0020] In the embodiment of the present application, the header style is identified through fuzzy matching, which improves the fault tolerance and achieves better recognition effect even for data with poor standardization.

[0021] In one embodiment, extracting a record row from an original record based on a header pattern includes:

[0022] The column sequence number distribution corresponding to the valid column data in the original record is used as the recording rule;

[0023] Extract record rows from original records based on record rules and header styles.

[0024] In an embodiment of the present application, batch positioning of table records is performed based on the header style and record rules, and subsequent text normalization processing is performed on this basis to ensure the validity and standardization of the data and improve processing efficiency.

[0025] In one embodiment, matching the original fields in the original record with the preset standard fields includes:

[0026] Write the historical matching records of successful matching between the original field and the standard field into the cache;

[0027] If the original field to be matched currently matches the original field in the historical matching record successfully, it is determined that the original field to be matched currently matches the standard field successfully through the cache.

[0028] In the embodiment of the present application, the processing speed can be increased and the system performance can be improved by matching the original fields with the standard fields in the cached data.

[0029] In one embodiment, the method further comprises:

[0030] If the original field to be matched currently and the standard field are not successfully matched through the cache, the original field to be matched currently and the standard field in the preset field value set are matched;

[0031] If the original field to be matched currently matches the standard field in the preset field value set successfully, it is determined that the original field to be matched currently matches the standard field successfully through the field value set.

[0032] In the embodiment of the present application, by matching the original field with the standard field through the field value set, the accuracy of the matching can be ensured, thereby improving the accuracy of the generated data.

[0033] In one embodiment, the method further comprises:

[0034] If the original field to be matched currently fails to match the standard field through the field value set, the original field to be matched currently is matched with the alias of the standard field in the preset rule base, wherein the rule base is used to store the mapping relationship between the standard field and the alias of the standard field;

[0035] If the original field to be matched currently matches the alias of the standard field in the preset rule base successfully, it is determined that the original field to be matched currently matches the standard field successfully through the rule base.

[0036] In the embodiment of the present application, the original fields and the standard fields are matched through the rule base. Since the rule base stores the mapping relationship between the original fields and the aliases of the standard fields, the data is made compatible and the processing capability of the system is improved.

[0037] In one embodiment, the method further comprises:

[0038] If the original field to be matched currently and the standard field are not successfully matched through the rule base, a second matching degree hybrid index is calculated, where the second matching degree hybrid index is a matching degree hybrid index between the original field to be matched currently and the standard field in the field value set;

[0039] When the second matching degree hybrid index is greater than or equal to the second preset threshold, it is determined that the fuzzy matching between the original field to be matched and the standard field is successful.

[0040] In the embodiment of the present application, the method of performing fuzzy matching using a mixed matching index improves fault tolerance and can achieve better data normalization effects even for data with poor standardization.

[0041] In a second aspect, an embodiment of the present application provides a general record processing device, including:

[0042] Identification unit, used to identify the header style of the original record;

[0043] The extraction unit is used to extract record rows from the original record based on the header pattern;

[0044] A matching unit, used to match original fields in original records with preset standard fields;

[0045] The generating unit is used to replace the corresponding original fields with the successfully matched standard fields in the extracted record lines to generate a universal record text.

[0046] In one embodiment, the identification unit includes a first identification subunit, and the first identification subunit is configured to:

[0047] Determine the header row range of the original record;

[0048] In the header row range, match the target field in each record row with the preset header keyword;

[0049] If all target fields in the record row successfully match the corresponding header keywords, the record row is determined to be an exact match;

[0050] The record rows that are accurately matched are used as the header rows.

[0051] In one embodiment, the identification unit further includes a second identification subunit, and the second identification subunit is configured to:

[0052] If the exact match of the record rows within the header row range is unsuccessful, a first matching mixed index is calculated, where the first matching mixed index is a matching mixed index between the target field in each record row and the preset header keyword;

[0053] If the first matching degree hybrid index of all target fields in the record row and the corresponding header keywords is greater than or equal to a first preset threshold, it is determined that the record row is fuzzy matched successfully;

[0054] The record rows that successfully match the fuzzy match are used as the header rows.

[0055] In one embodiment, the extraction unit is configured to:

[0056] The column sequence number distribution corresponding to the valid column data in the original record is used as the recording rule;

[0057] Extract record rows from original records based on record rules and header styles.

[0058] In one embodiment, the matching unit includes a first matching subunit, and the first matching subunit is configured to:

[0059] Write the historical matching records of successful matching between the original field and the standard field into the cache;

[0060] If the original field to be matched currently matches the original field in the historical matching record successfully, it is determined that the original field to be matched currently matches the standard field successfully through the cache.

[0061] In one embodiment, the matching unit further includes a second matching subunit, which is configured to:

[0062] If the original field to be matched currently and the standard field are not successfully matched through the cache, the original field to be matched currently and the standard field in the preset field value set are matched;

[0063] If the original field to be matched currently matches the standard field in the preset field value set successfully, it is determined that the original field to be matched currently matches the standard field successfully through the field value set.

[0064] In one embodiment, the matching unit further includes a third matching subunit, which is configured to:

[0065] If the original field to be matched currently fails to match the standard field through the field value set, the original field to be matched currently is matched with the alias of the standard field in the preset rule base, wherein the rule base is used to store the mapping relationship between the standard field and the alias of the standard field;

[0066] If the original field to be matched currently matches the alias of the standard field in the preset rule base successfully, it is determined that the original field to be matched currently matches the standard field successfully through the rule base.

[0067] In one embodiment, the matching unit further includes a fourth matching subunit, which is configured to:

[0068] If the original field to be matched currently and the standard field are not successfully matched through the rule base, a second matching degree hybrid index is calculated, where the second matching degree hybrid index is a matching degree hybrid index between the original field to be matched currently and the standard field in the field value set;

[0069] When the second matching degree hybrid index is greater than or equal to the second preset threshold, it is determined that the fuzzy matching between the original field to be matched and the standard field is successful.

[0070] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0071] at least one processor; and

[0072] a memory communicatively connected to at least one processor; wherein,

[0073] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the method provided by any embodiment of the present application.

[0074] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable the computer to execute the method provided by any embodiment of the present application.

[0075] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which implements the method provided by any embodiment of the present application when executed by a processor.

[0076] One embodiment of the above application has the following advantages or beneficial effects: it can automatically parse records and perform standardization of record fields, providing record data in a unified standard format, which can greatly improve processing efficiency and save human resources.

[0077] Other effects of the above optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0079] Figure 1 is a flowchart of a general record processing method according to an embodiment of the present application;

[0080] Figure 2 is a flowchart of identifying a header style according to a general record processing method of an embodiment of the present application;

[0081] Figure 3 is a flowchart of identifying a header style according to a general record processing method of an embodiment of the present application;

[0082] Figure 4 is a flowchart of extracting record rows according to a general record processing method according to an embodiment of the present application;

[0083] Figure 5is a flowchart of matching of a general record processing method according to an embodiment of the present application;

[0084] Figure 6 is a flowchart of matching of a general record processing method according to an embodiment of the present application;

[0085] Figure 7 is a flowchart of matching of a general record processing method according to an embodiment of the present application;

[0086] Figure 8 is a flowchart of matching of a general record processing method according to an embodiment of the present application;

[0087] Figure 9A is a schematic diagram of a knowledge graph of a general record processing method according to an embodiment of the present application;

[0088] Figure 9B is a schematic diagram of a knowledge graph of a general record processing method according to an embodiment of the present application;

[0089] Figure 10 It is a schematic diagram of the module design and data flow of a general record processing method according to an embodiment of the present application;

[0090] Figure 11 is a schematic structural diagram of a general record processing device according to an embodiment of the present application;

[0091] Figure 12 is a schematic structural diagram of a general record processing device according to an embodiment of the present application;

[0092] Figure 13 is a schematic structural diagram of a general record processing device according to an embodiment of the present application;

[0093] Figure 14 4 is a block diagram of an electronic device used to implement the general recording processing method of an embodiment of the present application. DETAILED DESCRIPTION

[0094] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0095] Figure 1 : is a schematic diagram of a general record processing method according to an embodiment of the present application. The general record processing method includes:

[0096] Step S110, identifying the header style of the original record;

[0097] Step S120, extracting record rows from the original record based on the header pattern;

[0098] Step S130, matching the original fields in the original record with the preset standard fields;

[0099] Step S140: In the extracted record rows, the corresponding original fields are replaced with the successfully matched standard fields to generate a universal record text.

[0100] Normally, original records may come from different data sources. For example, in a commodity trading platform, data may come from different suppliers, and each supplier has its own data template, and the data format and expression method are difficult to unify. Taking the field name as an example, the field name used to store commodity selling price information can be called "price" or "unit price". The data from different data sources have poor compatibility with each other. Therefore, it is necessary to standardize and normalize these non-standard and non-standard data to generate a universal record text to facilitate processing and interaction. The record processing method of the embodiment of the present application is applicable to scenarios where target records are extracted from electronic spreadsheets with no style restrictions, and is applicable to subsequent scenarios where non-standard record field values ​​are normalized to standard value set (Value Set) items.

[0101] Before normalizing non-standardized original records, the original record's header style must first be identified. In step S110, the header row in the original record is identified. Identifying the header style is the basis for subsequent record extraction. In step S120, record rows are extracted from the original record based on the identified header style. The search can begin at the header of the identified original record to be processed and continue downward until the search ends or another header row is found. The data in the searched record rows is extracted.

[0102] In order to standardize and normalize records, a standard field value set database can be pre-set. The field value set stores the names of standard fields. The names of standard fields can also be called "standard field values" or "value set items." For example, the name of the field used to store product selling price information in the original record can be called "price" or "unit price." The names of the standard fields after normalization of these fields can be stored in the field value set. For example, the "price," "unit price," and "selling price" in the original fields can be unified into the standard field "price" after normalization.

[0103] In step S130, the original fields in the original record are matched with the preset standard fields. That is, the names of the original fields are matched with the names of the standard fields. For example, the name of the original field "unit price" is matched one by one with the names of the standard fields stored in the field value set to identify the standard fields corresponding to the original field. For example, as shown in Tables 1 to 4 below, Tables 1 and 2 are examples of original record tables, Table 3 is an example of a normalized general record table corresponding to Table 1, and Table 4 is an example of a normalized general record table corresponding to Table 2. Among them, the name of the standard field corresponding to the original field "stainless steel composite cold-rolled coil" in Table 1 in the field value set is "cold-rolled coil"; the name of the standard field corresponding to the original field "aluminum-zinc coated fingerprint-resistant coil" in Table 2 in the field value set is also "cold-rolled coil"; the name of the standard field corresponding to the original field "unit price" in Table 2 in the field value set is "price".

[0104] Table 1 Example 1 of original record form

[0105] Record line 1 Product Name price Record line 2 Stainless steel composite cold rolled coil 180 … … …

[0106] Table 2 Example 2 of original record form

[0107] Record line 1 Product Name unit price Record line 2 Galvalume Anti-Fingerprint Coil 160 … … …

[0108] Table 3 Example 1 of General Record Form

[0109] Record line 1 Product Name price Record line 2 Cold rolled coil 180 … … …

[0110] Table 4 Example 2 of General Record Form

[0111] Record line 1 Product Name price Record line 2 Cold rolled coil 160 … … …

[0112] In step S140, the original fields in the extracted record rows are replaced with the successfully matched standard fields. As shown in Tables 1-4, "cold-rolled coil" replaces "stainless steel composite cold-rolled coil" and "galvanized anti-fingerprint coil," and "price" replaces "unit price," generating a generic record text.

[0113] In the embodiment of the present application, records can be automatically parsed and record fields can be standardized to provide record data in a unified standard format, which can greatly improve processing efficiency and save human resources.

[0114] Figure 2 This is a flowchart of the header style identification method of the general record processing method according to the embodiment of the present application. Figure 2 As shown, in one embodiment, Figure 1 Step S110 in the example, identifying the header style of the original record, includes:

[0115] Step S210, determining the header row range of the original record;

[0116] Step S220, within the range of the header row, matching the target field in each record row with the preset header keyword;

[0117] Step S230 , if all target fields in the record row successfully match the corresponding header keywords, it is determined that the record row has been accurately matched;

[0118] Step S240: Use the record rows that have been accurately matched as the header rows.

[0119] Before standardizing and normalizing the original records, the header style of the original records must first be identified, that is, the location of the header row must be located. In step S210, the header row range of the original record sheet content is first determined. In one example, the following statement can be used to calculate the header row range: max(40, 20% * total number of rows in the sheet). Here, max is a function that selects the maximum value; "20% * total number of rows in the sheet" represents 20% of the total number of rows in the sheet and can specify the location of this portion of rows in the sheet, for example, typically the top 20% of the sheet. The calculation result of the statement "max(40, 20% * total number of rows in the sheet)" is the maximum value between "20% * total number of rows in the sheet" and "40". For example, if the total number of rows in the sheet is 100, then "20% * total number of rows in the sheet" is 20. Therefore, the maximum value between "20% * total number of rows in the sheet" and "40" is 40, so the value of "max(40, 20% * total number of rows in the sheet)" is 40.

[0120] In step S220, within the header row range of the sheet content determined above, the names of the column fields in each record row in the original field are extracted. The names of the column fields within the header row range in the original field are referred to as target fields. On the other hand, a standard header keyword set database can be pre-set, and the header keyword set database stores the header keywords after normalization of the target field. For example, in the above Table 1, the target field of record row 1 in the original record is "price", and the header keyword in the header keyword set is also "price", then it is determined that the target field and the header keyword are accurately matched successfully.

[0121] In step S230, if all target fields in the record row successfully match the corresponding header keywords, then the record row is determined to have successfully matched accurately. For example, in Tables 1 and 3 above, another target field in record row 1 in the original record of Table 1 is "Product Name", and the header keyword in the header keyword set is also "Product Name", then the target field and the header keyword are also successfully matched accurately. If all target fields in record row 1 in the original record of Table 1 successfully match the corresponding header keywords, then it is determined that record row 1 in Table 1 has successfully matched accurately. In step S240, the record row that has successfully matched accurately, such as record row 1 in Table 1, is used as the header row.

[0122] In the embodiment of the present application, identifying the header style through precise matching is an important step in batch locating table records, and provides a positioning basis for the subsequent step of extracting records.

[0123] Figure 3 This is a flowchart of the header style identification method of the general record processing method according to the embodiment of the present application. Figure 3 As shown, in one embodiment, the method further includes:

[0124] Step S310: If the exact match of the record rows within the header row range is unsuccessful, a first mixed matching index is calculated. The first mixed matching index is a mixed matching index of the target field in each record row and the preset header keyword.

[0125] Step S320: If the first matching degree hybrid index of all target fields in the record row and the corresponding header keywords is greater than or equal to a first preset threshold, it is determined that the record row is fuzzy matched successfully;

[0126] Step S330: Use the record row where the fuzzy matching succeeds as the header row.

[0127] In this embodiment, if the exact match of the current record row being identified is unsuccessful, that is, the current record row does not exactly match the keywords corresponding to all target fields, the matching criteria are relaxed and the target fields in each record row are fuzzily matched with the preset header keywords. For example, in step S310, the following formula can be used to calculate the first matching hybrid index:

[0128]

[0129] Where score represents the score of the calculation result of the first matching mixed index and the second matching mixed index, LCS represents the longest common subsequence algorithm, ED represents the edit distance, the function len(LCS(x,y)) is used to calculate the length of the longest common subsequence of strings x and y, the function len(z) is used to calculate the length of string z, ω represents the weight value, Sraw Represents the original string, S target Indicates the target string.

[0130] LCS stands for Longest Common Subsequence. A sequence is considered the longest common subsequence if it is a subsequence of two or more known sequences and is the longest of all subsequences. The value of ω in the formula can be selected based on experimental results, for example, by using a hill climbing algorithm to find the local optimum.

[0131] In the first matching degree hybrid indicator, the original string can be the target field in each record row, and the target string can be a table header keyword.

[0132] In step S320, if the first hybrid matching index is determined to be greater than or equal to a first preset threshold, it is determined that all target fields have successfully fuzzy matched the corresponding header keywords. If all target fields in the current record row have successfully fuzzy matched at least one header keyword in the header keyword set, the current record row is determined to have successfully fuzzy matched. In step S330, the current record row determined to have successfully fuzzy matched is used as the header row.

[0133] For example, in Table 2 above, the target field in the original record is "unit price," and the header keyword corresponding to "unit price" in the header keyword set is "price." Therefore, "unit price" and "price" can be fuzzily matched successfully. If the matching hybrid index corresponding to all target fields in row 1 of the original record in Table 2 is greater than or equal to the first preset threshold, then row 1 in Table 2 is determined to be a successful fuzzy match. Row 1 in Table 2 is used as the header row.

[0134] In the embodiment of the present application, the header style is identified through fuzzy matching, which improves the fault tolerance and achieves better recognition effect even for data with poor standardization.

[0135] Figure 4 FIG. 1 is a flowchart of extracting record rows according to a general record processing method according to an embodiment of the present application. Figure 4 As shown, in one embodiment, Figure 1 Step S120 in the above embodiment extracts record rows from the original record based on the header pattern, including:

[0136] Step S410: using the column number distribution corresponding to the valid column data in the original record as a recording rule;

[0137] Step S420: extract record rows from the original record according to the record rule and header style.

[0138] In this embodiment, the following method is used to generate recording rules: a standard recording form is established in advance. Taking the source record as an example, the header keywords in the standard record form can be established to include: product name, price, place of origin, and production date. Compared with the standard record form, invalid column data in the original source record form is identified, which is also said to be useless information. Column data that is different from the standard record form is regarded as useless information. The field corresponding to the valid column data is called the target field, and the target field column number distribution is used as the recording rule. For example, the column data in the original source record form includes: Column 1: Product Name, Column 2: Product Category, Column 3: Price, Column 4: Appearance Color, Column 5: Place of Origin, Column 6: Product Grade, Column 7: Production Date. Compared with the standard record form, columns 1, 3, 5, and 7 in the original source record form are determined as target field column numbers.

[0139] In step S410, all headers in the original record are analyzed to determine the column number of the target field in each header, and the target field column number distribution is used as the record rule. In step S420, a record search is performed in the original record, and each record is extracted according to the header style and the record rule.

[0140] In one example, the steps for logging a search are as follows:

[0141] (1) Calculate the column range of each sub-table in the original record table according to the record rules. For example, in the example of the original source record table above, the 1st, 3rd, 5th, and 7th columns, which are determined as the target field column numbers, are the column range of the original record table to be searched.

[0142] (2) Search downward from the header row of each sub-table, collecting candidate record rows until the current row reaches the bottom of the sheet to complete the search, or, when there is a conflict with the header row or column range of other sub-tables, the search is completed.

[0143] (3) Filter the candidate record rows obtained in step (2) and retain the rows that contain valid content of the necessary target fields as record rows. For example, "Product Name" contains at least one Chinese character, "Brand Number" is not empty, and "Origin" contains at least one Chinese character.

[0144] (4) According to the record rules, the record rows are converted into a standard record format, for example, the column fields include: "(product name, price, place of origin, production date)".

[0145] In an embodiment of the present application, batch positioning of table records is performed based on the header style and record rules, and subsequent text normalization processing is performed on this basis to ensure the validity and standardization of the data and improve processing efficiency.

[0146] Figure 5This is a flowchart of the matching of the general record processing method according to the embodiment of the present application. Figure 5 As shown, in one embodiment, Figure 1 In step S130, matching the original fields in the original record with the preset standard fields includes:

[0147] Step S510, writing the historical matching records of successful matching between the original field and the standard field into the cache;

[0148] Step S520: If the original field to be matched currently matches the original field in the historical matching record, it is determined that the original field to be matched currently matches the standard field successfully through the cache.

[0149] In step S130 , the original fields in the original record are matched with the preset standard fields, so that the field values ​​in the original record that are not expressed in a standard manner are aligned with the standard field values.

[0150] In step S510, during the normalization process of the original record table, the original fields are matched with the standard fields, and the successfully matched results are written into the cache. The historical matching records written into the cache include the mapping relationship between the original fields and the standard fields.

[0151] In step S520, the cache is queried to confirm whether a mapping relationship between the original field and the standard field that has been successfully matched already exists. If a mapping relationship between the original field to be matched currently and the standard field already exists, it is determined that the original field to be matched currently and the standard field have been successfully matched through the cache. For example, a mapping relationship between the original field "unit price" and the standard field "price" already exists in the historical matching record. If the original field to be matched currently is also "unit price", that is, the original field to be matched currently matches the original field in the historical matching record successfully, it is determined that the standard field corresponding to the original field "unit price" to be matched currently is "price".

[0152] In the embodiment of the present application, the processing speed can be increased and the system performance can be improved by matching the original fields with the standard fields in the cached data.

[0153] Figure 6 This is a flowchart of the matching of the general record processing method according to the embodiment of the present application. Figure 6 As shown, in one embodiment, the method further includes:

[0154] Step S610: If the original field to be matched currently and the standard field are not successfully matched through the cache, the original field to be matched currently and the standard field in the preset field value set are matched;

[0155] Step S620: If the currently to-be-matched original field matches the standard field in the preset field value set successfully, it is determined that the currently to-be-matched original field and the standard field match successfully through the field value set.

[0156] For example, if the mapping relationship between the currently to-be-matched original field and the standard field is not found in the historical matching records in step S520, then in step S610, the value set items in the field value set are enumerated, and the currently to-be-matched original field is precisely matched with the value set items in the field value set. In step S620, if the currently to-be-matched original field precisely matches the value set items in the field value set successfully, for example, the row records in the record table are: "Product Name: Cold Rolled Coils", "Origin: Shanghai", and there are also "Cold Rolled Coils", "Shanghai" in the value set items in the field value set, then it is determined that the currently to-be-matched original field and the standard field match successfully through the field value set.

[0157] In the embodiments of the present application, by matching the original field and the standard field through the field value set, the accuracy of the matching can be ensured, thereby improving the accuracy of the generated data.

[0158] Figure 7 It is a flowchart of the matching of the general record processing method according to the embodiments of the present application. As Figure 7 shown, in one embodiment, the method further includes:

[0159] Step S710: In the case where the currently to-be-matched original field and the standard field do not match successfully through the field value set, the currently to-be-matched original field is matched with the alias of the standard field in the preset rule library, where the rule library is used to store the mapping relationship between the standard field and the alias of the standard field;

[0160] Step S720: If the currently to-be-matched original field matches the alias of the standard field in the preset rule library successfully, it is determined that the currently to-be-matched original field and the standard field match successfully through the rule library.

[0161] For example, if the currently to-be-matched original field does not precisely match the value set items in the field value set in step S620, then in step S710, the error correction rule list is enumerated for precise matching of the error correction rules, and the currently to-be-matched original field is precisely matched with the alias of the standard field in the rule library. Here, the alias is a name other than the legal name or the standardized name. Taking the origin as "Shanghai" as an example, if the standard field is "Shanghai", the alias may be "Greater Shanghai" or "Hu" etc.

[0162] In step S720, if the original field to be matched accurately matches the alias of the standard field in the rule base, for example, if the field record in the original record table is "Origin: Shanghai", and the alias of the standard field in the rule base includes: the standard field "Origin: Shanghai" and the alias "Origin: Shanghai", the mapping relationship between the original field and the standard field is determined. The original field "Origin: Shanghai" is replaced with the standard field "Origin: Shanghai".

[0163] In the embodiment of the present application, the original fields and the standard fields are matched through the rule base. Since the rule base stores the mapping relationship between the original fields and the aliases of the standard fields, the data is made compatible and the processing capability of the system is improved.

[0164] Figure 8 This is a flowchart of the matching of the general record processing method according to the embodiment of the present application. Figure 8 As shown, in one embodiment, the method further includes:

[0165] Step S810: If the original field to be matched currently and the standard field are not successfully matched through the rule base, a second hybrid matching index is calculated, where the second hybrid matching index is a hybrid matching index between the original field to be matched currently and the standard field in the field value set;

[0166] Step S820: When the second matching degree hybrid index is greater than or equal to the second preset threshold, it is determined that the fuzzy matching between the original field to be matched and the standard field is successful.

[0167] For example, if the original field to be matched does not exactly match the alias of the standard field in the rule base in step S720, the second matching hybrid index is calculated using the following formula in step S810:

[0168]

[0169] Where score represents the score of the calculation result of the first matching mixed index and the second matching mixed index, LCS represents the longest common subsequence algorithm, ED represents the edit distance, the function len(LCS(x,y)) is used to calculate the length of the longest common subsequence of strings x and y, the function len(z) is used to calculate the length of string z, ω represents the weight value, S raw Represents the original string, S target Indicates the target string.

[0170] LCS stands for Longest Common Subsequence. A sequence is considered the longest common subsequence if it is a subsequence of two or more known sequences and is the longest of all subsequences. The value of ω in the formula can be selected based on experimental results, for example, by using a hill climbing algorithm to select the optimal value.

[0171] In the second matching hybrid indicator, the original string is the original field to be matched, and the target string is the standard field in the field value set.

[0172] In step S820, if it is determined that the matching mixed index between the original field to be matched and a standard field in the field value set is greater than or equal to the second preset threshold, it is determined that the fuzzy matching between the original field to be matched and the standard field is successful.

[0173] In one embodiment, a knowledge graph can be constructed in the field value set to represent the topological relationship between the value set items in the field value set. The knowledge graph is composed of a number of interconnected entities and their attributes. The knowledge graph consists of pieces of knowledge, each of which can be represented as an SPO triple (Subject-Predicate-Object), where Subject represents the subject, Predicate represents the predicate, and Object represents the object. The knowledge graph can describe the collection of knowledge using a topological graph, which is suitable for expressing the abstract relationship between knowledge entities. It is often used for abstract association search between knowledge entities.

[0174] In this embodiment, knowledge entities are text items in various value sets. One of the entity attributes is the literal content of the text. The relationships between entities are either similar or non-similar. For steel products, for the variety value set, similar entities share the same variety category, such as plate, pipe, and medium and thick plate. For the origin entity, similar entities share the same steel mill group, such as Baowu Steel, Ma Steel, and Handan Steel. For the warehouse entity, similar entities share the same address region, such as Shanghai, Wuhan, and Nanjing.

[0175] Each group of similar entities forms a knowledge entity cluster, and each cluster has a central entity. By sampling a large number of source records, the frequency of occurrence of all value set items is counted, and the entity with the highest frequency of occurrence of the corresponding value set item in each entity cluster is selected as the cluster central entity. Figure 9B In the cluster, “cold-rolled coil” is the cluster center entity, and “stainless steel composite cold-rolled coil”, “aluminum-zinc coated fingerprint-resistant coil”, and “color-coated coil (electro-zinc substrate)” are non-center entities of the same type as “cold-rolled coil”.

[0176] In the process of executing step S820, the value set items in the enumerated field value set can be traversed, the matching hybrid index between the original field and the value set item text can be measured, and a comprehensive determination can be made as to whether the match is successful. The order of the above traversal and enumeration can be determined by the topological relationship of the value set items in the KG (Knowledge Graph). In one example, in order to simplify the entity search process, the similar relationships between all non-central entities in the cluster are removed, and the central entity value set items of different clusters in the KG are first traversed and the corresponding matching hybrid index is recorded. If the original field is not successfully matched with the central entities of different clusters, the central entity with the largest matching hybrid index with the original field is obtained in descending order of the matching hybrid index between the original field and the central entities of each different cluster, and then the non-central entities in the cluster where the central entity in the KG is located are matched in sequence.

[0177] Figure 9A and Figure 9B 9A and 9B are schematic diagrams of a knowledge graph of a general record processing method according to an embodiment of the present application. Figure 9B The value on the line in represents the calculated mixed index of the matching degree between the nouns at both ends of the line. Figure 9B As shown in the figure, the original field in the original record is "cold-rolled", so first calculate the matching degree hybrid index of the original field and the central entity of different clusters. If no match is successful, then sort the matching degree hybrid indexes of the original field and the central entity of each different cluster in descending order. Figure 9A For example, the central entities in the knowledge graph include "cold-rolled coil" and "general wire". After calculation, the matching degree mixed index of "cold rolling" and "cold-rolled coil" is 0.5, and the matching degree mixed index of "cold rolling" and "general wire" is 0.0. The result of descending sorting is that the matching degree mixed index of "cold rolling" and "cold-rolled coil" is the highest. Figure 9B For example, the non-central entities centered on "cold-rolled coil" in KG are matched sequentially, that is, "cold rolling" is matched with the non-central entity centered on "cold-rolled coil". In other words, the matching degree mixed index of "cold rolling" is calculated for "stainless steel composite cold-rolled coil", "aluminum-zinc anti-fingerprint coil", and "color-coated coil (electro-zinc substrate)". Figure 9B The matching hybrid index between "cold-rolled" and "color-coated coil (electro-zinc substrate)" is 0.0, the matching hybrid index between "cold-rolled" and "aluminum-zinc-coated fingerprint-resistant coil" is 0.0, and the matching hybrid index between "cold-rolled" and "stainless steel composite cold-rolled coil" is 0.25. The previous step calculated that the matching hybrid index between "cold-rolled" and "cold-rolled coil" is 0.5. Among these indices, the matching hybrid index between "cold-rolled" and "cold-rolled coil" is the largest. If this matching hybrid index is greater than or equal to the second preset threshold, it is determined that the fuzzy match between the original field "cold-rolled" to be matched and the standard field "cold-rolled coil" is successful.

[0178] In the embodiment of the present application, the method of performing fuzzy matching using a mixed matching index improves fault tolerance and can achieve better data normalization effects even for data with poor standardization.

[0179] In one embodiment, if the original field successfully matches the standard field, the mapping relationship between the successfully matched original field value and the value set item is written into the cache. When the cache capacity overflows, the LRU (Least Recently Used) algorithm is used for scheduling. The LRU algorithm is used in memory scheduling scenarios and can also be used in cache scheduling scenarios. Taking the cache scheduling scenario as an example, when the cache space is full and overflows, an element is selected from the elements in the cache that are currently hit the least number of times, removed from the cache, and a new element is added to the cache.

[0180] Figure 8 The steps shown in can be executed sequentially, or the jump process of the program can be set according to the execution result. For example, execute step S520 to query the cache. If the query fails, continue to execute step S610; or if the query is successful in executing step S520, jump directly to step S510, and write the mapping relationship between the original field value that is successfully matched and the value set item into the cache. Similarly, execute steps S610 and S620 to enumerate the value set items for matching. If the match fails, continue to execute step S710; or if the match is successful, jump directly to step S510, and write the mapping relationship between the original field value that is successfully matched and the value set item into the cache. Similarly, execute steps S710 and S720 to enumerate the error correction rule list for matching. If the match fails, continue to execute step S810; or if the match is successful, jump directly to step S510, and write the mapping relationship between the original field value that is successfully matched and the value set item into the cache. When executing step S820, if the match is successful, also jump directly to step S510. In summary, Figure 8 The steps in the process are executed sequentially. If a step is matched successfully, the process jumps to step S510. Otherwise, the process continues to execute the next step sequentially.

[0181] During the normalization process described above, it's possible that no normalization result is obtained, meaning normalization fails. Whether normalization failure is permitted can be pre-configured. In one example, if normalization failure is not permitted, a default value set based on a large number of sample statistics can be used as the normalization result for record field values. In this example, for each original field, a count of the standard fields corresponding to each original field in historically generated general records can be pre-calculated, and the number of successful matches between the original field and the corresponding standard field can be counted. For example, if the original field is "Product Name: Banana Apple," and in historically generated general records, any "Product Name" original field successfully matches "Product Name: Banana" 50 times, and any "Product Name" original field successfully matches "Product Name: Apple" 500 times, then the corresponding field with the most successful matches is selected as the default value, and "Product Name: Apple" is used as the normalization result for "Product Name: Banana Apple." In the above default value setting method, regardless of the original field, the "Product Name" with the most historical hits is selected as the default value for normalization. This method is probabilistically most likely to result in correct normalization.

[0182] In the above example, in a scenario where normalization failure is allowed, a normalization failure identifier may be used as the normalization result.

[0183] Figure 10 Schematic diagram of the module design and data flow of the general record processing method according to the embodiment of the present application. Figure 10 As shown, the embodiment of the present application utilizes text fuzzy matching technology and a rule library to construct a general record parsing and normalization system within a specific business scenario. The system includes a record search module, a normalization module, and a data management module. Among them, the record search module uses a table record batch positioning algorithm to batch extract original records from all sheets of the spreadsheet file of the original record. The record search module includes a header search module 1, a record rule generation module 2, and a record search module 4. The normalization module includes a header field association module 3 and a record normalization module 7. The header search module is used to identify the header style, thereby obtaining the original header field. The header field association module analyzes the association relationship between the original header field and the header keywords in the header keyword set, and groups the associated original header fields and header keywords into a group to obtain group-associated original header fields. The record rule generation module compares the original header field and the group-associated original header field to generate record rules. The record search module batch extracts original records from all sheets of the spreadsheet file of the original record according to the record rules.

[0184] See also Figure 10The record normalization module generates normalized records and a normalized record list based on the original records extracted by the record search module, the normalization rules in the normalization rule library, and the specific field value set information in the field value set. The header field association module and the record normalization module use LCS, edit distance, knowledge graph (KG), and LRU algorithms to perform header field association or record normalization.

[0185] See also Figure 10 The data management module includes a value set management module 6 and an error correction rule management module 5, which is used to maintain value set data and error correction rule data.

[0186] The value set management module provides value set management functionality, which means maintaining a collection of updateable standard field values ​​(value set items). This includes interfaces for adding value set items, deleting value set items, modifying value set items, and querying value set items.

[0187] The error correction rule management module provides error correction rule management functionality, maintaining a set of mapping relationships between original field values ​​and standard field values ​​(value set items). This solves the problem of mixed matching metrics being unable to handle standard field aliases appearing in original fields. This module includes interfaces for adding, deleting, modifying, and querying error correction rules. These interfaces allow users to customize the aliases of standard fields.

[0188] Figure 11 Schematic diagram of the structure of a general record processing device according to an embodiment of the present application. Figure 11 As shown, the universal record processing device of the embodiment of the present application includes:

[0189] The identification unit 100 is used to identify the header style of the original record;

[0190] The extraction unit 200 is used to extract record rows from the original record based on the header pattern;

[0191] A matching unit 300 is used to match original fields in original records with preset standard fields;

[0192] The generating unit 400 is configured to replace the corresponding original fields in the extracted record rows with the successfully matched standard fields to generate a general record text.

[0193] Figure 12 Schematic diagram of the structure of a general record processing device according to an embodiment of the present application. Figure 12 As shown, in one embodiment, the identification unit 100 includes a first identification subunit 110, and the first identification subunit 110 is used to:

[0194] Determine the header row range of the original record;

[0195] In the header row range, match the target field in each record row with the preset header keyword;

[0196] If all target fields in the record row successfully match the corresponding header keywords, the record row is determined to be an exact match;

[0197] The record rows that are accurately matched are used as the header rows.

[0198] In one embodiment, the identification unit 100 further includes a second identification subunit 120, which is configured to:

[0199] If the exact match of the record rows within the header row range is unsuccessful, a first matching mixed index is calculated, where the first matching mixed index is a matching mixed index between the target field in each record row and the preset header keyword;

[0200] If the first matching degree hybrid index of all target fields in the record row and the corresponding header keywords is greater than or equal to a first preset threshold, it is determined that the record row is fuzzy matched successfully;

[0201] The record rows that successfully match the fuzzy match are used as the header rows.

[0202] In one embodiment, the extraction unit 200 is configured to:

[0203] The column sequence number distribution corresponding to the valid column data in the original record is used as the recording rule;

[0204] Extract record rows from original records based on record rules and header styles.

[0205] Figure 13 Schematic diagram of the structure of a general record processing device according to an embodiment of the present application. Figure 13 As shown, in one embodiment, the matching unit 300 includes a first matching subunit 310, and the first matching subunit 310 is used to:

[0206] Write the historical matching records of successful matching between the original field and the standard field into the cache;

[0207] If the original field to be matched currently matches the original field in the historical matching record successfully, it is determined that the original field to be matched currently matches the standard field successfully through the cache.

[0208] In one embodiment, the matching unit 300 further includes a second matching subunit 320, which is configured to:

[0209] If the original field to be matched currently and the standard field are not successfully matched through the cache, the original field to be matched currently and the standard field in the preset field value set are matched;

[0210] If the original field to be matched currently matches the standard field in the preset field value set successfully, it is determined that the original field to be matched currently matches the standard field successfully through the field value set.

[0211] In one embodiment, the matching unit 300 further includes a third matching subunit 330, which is configured to:

[0212] If the original field to be matched currently fails to match the standard field through the field value set, the original field to be matched currently is matched with the alias of the standard field in the preset rule base, wherein the rule base is used to store the mapping relationship between the standard field and the alias of the standard field;

[0213] If the original field to be matched currently matches the alias of the standard field in the preset rule base successfully, it is determined that the original field to be matched currently matches the standard field successfully through the rule base.

[0214] In one embodiment, the matching unit 300 further includes a fourth matching subunit 340, which is configured to:

[0215] If the original field to be matched currently and the standard field are not successfully matched through the rule base, a second matching degree hybrid index is calculated, where the second matching degree hybrid index is a matching degree hybrid index between the original field to be matched currently and the standard field in the field value set;

[0216] When the second matching degree hybrid index is greater than or equal to the second preset threshold, it is determined that the fuzzy matching between the original field to be matched and the standard field is successful.

[0217] The functions of each unit in the universal record processing device in the embodiment of the present application can be found in the corresponding description of the above method and will not be repeated here.

[0218] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0219] like Figure 14, is a block diagram of an electronic device according to a method for general record processing according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0220] like Figure 14 As shown, the electronic device includes: one or more processors 1401, a memory 1402, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 14 A processor 1401 is taken as an example.

[0221] Memory 1402 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor, causing the at least one processor to perform the general record processing method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the general record processing method provided in this application.

[0222] The memory 1402 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions / modules / units corresponding to the general record processing method in the embodiment of the present application (for example, the attached Figure 11 The identification unit 100, the extraction unit 200, the matching unit 300, and the generation unit 400 are shown in FIG. Figure 12 The first identification subunit 110, the second identification subunit 120, and the attached Figure 13The processor 1401 executes the non-transient software programs, instructions, and modules stored in the memory 1402 to execute various functional applications and data processing of the server, that is, to implement the general record processing method in the above method embodiment.

[0223] Memory 1402 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the electronic device of the universal record processing method. Furthermore, memory 1402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, memory 1402 may optionally include memory remotely located relative to processor 1401. Such remote memory may be connected to the electronic device of the universal record processing method via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0224] The electronic device of the general recording processing method may further include: an input device 1403 and an output device 1404. The processor 1401, the memory 1402, the input device 1403 and the output device 1404 may be connected via a bus or other means. Figure 14 The bus connection is taken as an example.

[0225] The input device 1403 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device of the general recording processing method, such as input devices such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, and a joystick. The output device 1404 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0226] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0227] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0228] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0229] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0230] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0231] According to the technical solution of the embodiments of the present application, points of interest are directly identified from the relevant content of the user's information behavior, thereby ensuring that the points of interest pushed to the user are consistent with the user's intentions and providing a good user experience. Because points of interest are directly identified from the relevant content of the user's information behavior, the problem of pushed points of interest not meeting the user's needs is avoided, thereby improving the user experience.

[0232] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0233] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A general record processing method, characterized in that: include: Identify the header style of the original record; Extracting a record row from the original record based on the header pattern; Matching the original fields in the original record with the preset standard fields, and aligning the non-standard field values ​​in the original record with the standard field values; In the extracted record line, the corresponding original field is replaced with the successfully matched standard field to generate a universal record text; Extracting a record row from the original record based on the header pattern includes: Using the column sequence number distribution corresponding to the valid column data in the original record as a recording rule; Extracting record rows from the original record according to the record rule and the header style; Identify the header style of the original record, including: Determine the header row range of the original record; Matching the target field in each record row with a preset header keyword within the header row range; If all target fields in the record row successfully match the corresponding header keywords, the record row is determined to be an exact match; The record rows that are accurately matched are used as the header rows.

2. The method according to claim 1, characterized in that The method further comprises: If the record rows within the header row range are not accurately matched, a first matching hybrid index is calculated, where the first matching hybrid index is a matching hybrid index between a target field in each record row and a preset header keyword; If the first matching degree hybrid index of all target fields in the record row and the corresponding header keywords is greater than or equal to a first preset threshold, determining that the record row is fuzzy matched successfully; The record rows that successfully match the fuzzy match are used as the header rows.

3. The method according to any one of claims 1 to 2, characterized in that Matching the original fields in the original record with the preset standard fields includes: Writing historical matching records of successful matching between the original field and the standard field into a cache; If the original field to be matched currently matches the original field in the historical matching record successfully, it is determined that the original field to be matched currently matches the standard field successfully through cache matching.

4. The method according to claim 3, characterized in that The method further comprises: If the original field to be matched currently and the standard field are not successfully matched through the cache, matching the original field to be matched currently and the standard field in the preset field value set; If the original field to be matched currently matches the standard field in the preset field value set successfully, it is determined that the original field to be matched currently matches the standard field successfully through the field value set.

5. The method according to claim 4, characterized in that The method further comprises: If the original field to be matched currently and the standard field are not successfully matched through the field value set, matching the original field to be matched currently and the alias of the standard field in a preset rule base, wherein the rule base is used to store the mapping relationship between the standard field and the alias of the standard field; If the original field to be matched currently matches the alias of the standard field in the preset rule base successfully, it is determined that the original field to be matched currently matches the standard field successfully through the rule base.

6. The method according to claim 5, characterized in that The method further comprises: If the original field to be matched currently and the standard field are not successfully matched through the rule base, a second matching degree hybrid index is calculated, where the second matching degree hybrid index is a matching degree hybrid index between the original field to be matched currently and the standard field in the field value set; When the second matching degree hybrid index is greater than or equal to a second preset threshold, it is determined that the fuzzy matching between the current original field to be matched and the standard field is successful.

7. A general record processing device, characterized in that: include: Identification unit, used to identify the header style of the original record; An extraction unit, configured to extract a record row from the original record based on the header pattern; a matching unit, configured to match original fields in the original record with preset standard fields, and align field values ​​in the original record that are not expressed in a standard manner with standard field values; A generating unit is configured to: replace the corresponding original fields in the extracted record lines with the successfully matched standard fields to generate a universal record text; The extraction unit is used for: Using the column sequence number distribution corresponding to the valid column data in the original record as a recording rule; Extracting record rows from the original record according to the record rule and the header style; The identification unit includes a first identification subunit, and the first identification subunit is used to: Determine the header row range of the original record; Matching the target field in each record row with a preset header keyword within the header row range; If all target fields in the record row successfully match the corresponding header keywords, the record row is determined to be an exact match; The record rows that are accurately matched are used as the header rows.

8. The device according to claim 7, characterized in that The identification unit further includes a second identification subunit, wherein the second identification subunit is configured to: If the record rows within the header row range are not accurately matched, a first matching hybrid index is calculated, where the first matching hybrid index is a matching hybrid index between a target field in each record row and a preset header keyword; If the first matching degree hybrid index of all target fields in the record row and the corresponding header keywords is greater than or equal to a first preset threshold, determining that the record row is fuzzy matched successfully; The record rows that successfully match the fuzzy match are used as the header rows.

9. The device according to any one of claims 7-8, characterized in that The matching unit includes a first matching subunit, and the first matching subunit is configured to: Writing historical matching records of successful matching between the original field and the standard field into a cache; If the original field to be matched currently matches the original field in the historical matching record successfully, it is determined that the original field to be matched currently matches the standard field successfully through cache matching.

10. The device according to claim 9, characterized in that The matching unit further includes a second matching subunit, which is configured to: If the original field to be matched currently and the standard field are not successfully matched through the cache, matching the original field to be matched currently and the standard field in the preset field value set; If the original field to be matched currently matches the standard field in the preset field value set successfully, it is determined that the original field to be matched currently matches the standard field successfully through the field value set.

11. The device according to claim 10, characterized in that The matching unit further includes a third matching subunit, and the third matching subunit is configured to: If the original field to be matched currently and the standard field are not successfully matched through the field value set, matching the original field to be matched currently and the alias of the standard field in a preset rule base, wherein the rule base is used to store the mapping relationship between the standard field and the alias of the standard field; If the original field to be matched currently matches the alias of the standard field in the preset rule base successfully, it is determined that the original field to be matched currently matches the standard field successfully through the rule base.

12. The device according to claim 11, characterized in that The matching unit further includes a fourth matching subunit, and the fourth matching subunit is configured to: If the original field to be matched currently and the standard field are not successfully matched through the rule base, a second matching degree hybrid index is calculated, where the second matching degree hybrid index is a matching degree hybrid index between the original field to be matched currently and the standard field in the field value set; When the second matching degree hybrid index is greater than or equal to a second preset threshold, it is determined that the fuzzy matching between the current original field to be matched and the standard field is successful.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images