Method, device, computer equipment and storage medium for processing data

Abstract

The present application relates to a method, device, computer equipment and storage medium for processing continuation data. The method comprises: obtaining an original continuation file, extracting a subject keyword from the original continuation file; searching from a mapping relationship between a keyword and a standard subject according to the subject keyword to obtain a corresponding standard subject; extracting a to-be-matched row under the standard subject from the original continuation file, and if a header matching the to-be-matched row is found in the header database, marking the to-be-matched row as a header; inputting the marked header into a pre-trained header mapping model for prediction to obtain a corresponding standard field; performing standardization processing on the data under the marked header, and obtaining a standardized continuation file according to the standard field and the corresponding standardized data. The method improves the efficiency of continuation data processing and the accuracy rate by standardizing the subject, header and data of the original continuation file.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and storage medium for processing continued data. Background Art

[0002] Reinsurance is an operational link in the property and casualty insurance industry to carry out reinsurance business. Currently, the data required for reinsurance provided by different companies cannot form a unified structured data storage.

[0003] At present, in the actual business application stage, the renewal data can only be sorted through manual means, which is inefficient and has low accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for processing continued data that can improve the efficiency and accuracy of processing the data to be transferred in order to address the above technical problems.

[0005] In a first aspect, a method for processing continued data is provided, the method comprising:

[0006] Obtain an original continuation file, and extract subject keywords from the original continuation file;

[0007] According to the subject keyword, the corresponding standard subject is obtained by searching from the mapping relationship between the keyword and the standard subject;

[0008] Extract the to-be-matched row under the standard subject from the original transfer file, search in the header database corresponding to the standard subject, and if a header matching the to-be-matched row is found in the header database, mark the to-be-matched row as a header;

[0009] Input the marked header into the pre-trained header mapping model for prediction to obtain the corresponding standard field;

[0010] The corresponding data under the marked header in the original continuation file is extracted, and the data is standardized, and a standardized continuation file is obtained according to the standard field and the corresponding standardized data.

[0011] In one embodiment, searching in a header database corresponding to the standard subject, if a header matching the to-be-matched row is found in the header database, marking the to-be-matched row as a header, includes:

[0012] Perform an accurate search in the header database corresponding to the standard subject, and if a header matching the row to be matched is found in the header database, mark the row to be matched as a header;

[0013] If a header matching the to-be-matched row is not found in the header database, a fuzzy query is used to determine whether the to-be-matched row is a header. If the to-be-matched row is determined to be a header, the to-be-matched row is marked as a header.

[0014] In one embodiment, determining whether the to-be-matched row is a table header through fuzzy query, and when the to-be-matched row is determined to be a table header, marking the to-be-matched row as a table header includes:

[0015] Fuzzy search in the header database for headers whose similarity with the row to be matched exceeds a preset similarity threshold, and take them as hit records of the row to be matched;

[0016] Calculate the edit distance between the to-be-matched row and the hit record, and find the maximum value of the edit distances as the maximum edit distance;

[0017] When the maximum edit distance is greater than the header distance threshold, the row to be matched is marked as a header.

[0018] In one embodiment, the standard subject includes a plurality of subjects; the header mapping model includes a header mapping model corresponding to each subject in the plurality of subjects;

[0019] The marked header is input into the pre-trained header mapping model for prediction to obtain the corresponding standard fields, including:

[0020] The corresponding header mapping model is searched according to the standard subject corresponding to the marked header, and the marked header is input into the header mapping model for prediction to obtain the corresponding standard field.

[0021] In one embodiment, the marked header is input into a pre-trained header mapping model for prediction, and after obtaining the corresponding standard field, the following is further included:

[0022] Establish a field mapping relationship between the marked table header and the standard field;

[0023] The field mapping relationship and confirmation control are displayed on the terminal display interface;

[0024] In response to the user's triggering operation on the confirmation control, a confirmed standard field is obtained;

[0025] The standardized transfer file is obtained according to the standard field and the corresponding standardized processed data, including:

[0026] A standardized transfer file is obtained based on the confirmed standard fields and the corresponding standardized data.

[0027] In one embodiment, the data corresponding to the table header after the mark includes catastrophic risk events and administrative region names;

[0028] The data is standardized, and a standardized transfer file is obtained according to the standard fields and the corresponding standardized data, including:

[0029] The catastrophic risk event or the name of the administrative region is searched in the standardized database. If matching standard data is found, a standardized transfer file is obtained based on the standard field and the matching standard data.

[0030] In one embodiment, the method further comprises:

[0031] If no matching standard data is found in the standard database, the data editing distance between the catastrophic risk event or the administrative region name and the standard data is calculated. When the data editing distance is greater than the data distance threshold, a standardized continuation file is obtained based on the standard field and the standard data.

[0032] In a second aspect, a data processing device is provided, the device comprising:

[0033] An acquisition module is used to acquire an original continuation file and extract subject keywords from the original continuation file;

[0034] The theme determination module is used to search the mapping relationship between the theme keyword and the standard theme according to the theme keyword to obtain the corresponding standard theme;

[0035] A header marking module is used to extract the to-be-matched row under the standard subject from the original continuation file, search in the header database corresponding to the standard subject, and if a header matching the to-be-matched row is found in the header database, mark the to-be-matched row as a header;

[0036] The standard field determination module is used to input the marked header into the pre-trained header mapping model for prediction to obtain the corresponding standard field;

[0037] The standardization module is used to extract the corresponding data under the marked header in the original continuation file and perform standardization on the data, so as to obtain a standardized continuation file according to the standard field and the corresponding standardized data.

[0038] In a third aspect, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Obtain an original continuation file, and extract subject keywords from the original continuation file;

[0040] According to the subject keyword, the corresponding standard subject is obtained by searching from the mapping relationship between the keyword and the standard subject;

[0041] Extract the to-be-matched row under the standard subject from the original transfer file, search in the header database corresponding to the standard subject, and if a header matching the to-be-matched row is found in the header database, mark the to-be-matched row as a header;

[0042] Input the marked header into the pre-trained header mapping model for prediction to obtain the corresponding standard field;

[0043] The corresponding data under the marked header in the original continuation file is extracted, and the data is standardized, and a standardized continuation file is obtained according to the standard field and the corresponding standardized data.

[0044] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0045] Obtain an original continuation file, and extract subject keywords from the original continuation file;

[0046] According to the subject keyword, the corresponding standard subject is obtained by searching from the mapping relationship between the keyword and the standard subject;

[0047] Extract the to-be-matched row under the standard subject from the original transfer file, search in the header database corresponding to the standard subject, and if a header matching the to-be-matched row is found in the header database, mark the to-be-matched row as a header;

[0048] Input the marked header into the pre-trained header mapping model for prediction to obtain the corresponding standard field;

[0049] The corresponding data under the marked header in the original continuation file is extracted, and the data is standardized, and a standardized continuation file is obtained according to the standard field and the corresponding standardized data.

[0050] The above-mentioned continuation data processing method, device, computer equipment and storage medium obtain the original continuation file, extract the subject keywords in the original continuation file; according to the subject keywords, search from the mapping relationship between the keywords and the standard subject to obtain the corresponding standard subject; extract the to-be-matched row under the standard subject from the original continuation file, and search in the header database corresponding to the standard subject. If a header matching the to-be-matched row is found in the header database, the to-be-matched row is marked as a header; the marked header is input into a pre-trained header mapping model for prediction to obtain the corresponding standard field; extract the corresponding data under the marked header in the original continuation file, and standardize the data, and obtain a standardized continuation file according to the standard field and the corresponding standardized data. This method improves the efficiency of continuation data processing and the accuracy rate by standardizing the subject, header and data of the original continuation file. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 An application environment diagram of a method for processing data in a continuous manner in an embodiment;

[0052] Figure 2 A schematic diagram of a flow chart of a method for processing data in a continuous transfer according to an embodiment;

[0053] Figure 3 A schematic diagram of a flow chart of a header marking step in an embodiment;

[0054] Figure 4 It is an overall flow chart of a method for processing data in another embodiment;

[0055] Figure 5 A structural block diagram of a data processing device in one embodiment;

[0056] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The data processing method provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The terminal 102 can be, but is not limited to, various personal computers, laptops, and tablet computers, and the server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0059] In one embodiment, Figure 2 As shown, a method for processing continued data is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0060] Step 202, obtaining the original continued file, and extracting the subject keywords in the original continued file.

[0061] The original renewal file refers to the business operation data provided by different direct insurance companies during the renewal business processing. The renewal file contains data on multiple different business topics. The original renewal file is mostly in Excel format, but can also be in other formats, which is not limited in this embodiment. The subject keyword refers to a word or phrase used to represent the subject content of the original renewal file, such as some entity names.

[0062] Specifically, the terminal obtains the original continuation file transmitted by the first user, and extracts the subject keywords in the original continuation file according to the preset regular expression. Among them, the first user can be a different direct insurance company. The preset regular expression is set according to the keyword extraction requirements. Each original continuation file includes at least one first subject, and each first subject includes at least one second subject. For example, there are three worksheets in an Excel table, and the names of these three worksheets are the first subjects. There are multiple tables in each worksheet, and the subjects of these tables are the second subjects. Each type of subject includes at least one corresponding subject keyword. The keyword corresponding to the first subject in the original continuation file is the first subject keyword, and the keyword corresponding to the second subject is the second subject keyword. The corresponding subject keywords are extracted respectively according to the preset regular expressions corresponding to the first subject and the second subject.

[0063] Step 204: According to the subject keyword, search from the mapping relationship between the keyword and the standard subject to obtain the corresponding standard subject.

[0064] Among them, the standard subject refers to the pre-set standard naming of the entity name of the same attribute, in order to unify the entity name representing the same attribute in the original forwarding data transmitted by different users. The mapping relationship between keywords and standard subjects records the standard subjects corresponding to different keywords. For example, the subject keyword in the original forwarding file provided by the first user is Catastrophe Loss, and the corresponding standard subject is EventLoss.

[0065] Specifically, the terminal searches for the corresponding standard theme in the mapping relationship between the keyword and the standard theme according to the extracted first theme keyword, and searches for the corresponding standard theme in the mapping relationship between the keyword and the standard theme according to the extracted second theme keyword. As shown in Table 1, it is an example of the mapping relationship between the keyword and the standard theme. The type in Table 1 represents the type of theme, sheet_theme_mapping is the first theme, content_theme_mapping is the second theme, the keyword is the theme keyword extracted in step 202, and the theme name is the corresponding standard theme.

[0066] Table 1 Examples of mapping relationships between keywords and standard topics

[0067]

[0068] Step 206, extract the to-be-matched row under the standard subject from the original continuation file, and search in the header database corresponding to the standard subject. If a header matching the to-be-matched row is found in the header database, the to-be-matched row is marked as a header.

[0069] Specifically, after standardizing the subject in the original continuation file, a standard subject is obtained, and then a header database corresponding to the standard subject is searched. The rows to be matched under the standard subject are extracted, and the rows to be matched refer to all the rows corresponding to the second subject. Each row to be matched is traversed, and a search is performed in the header database corresponding to the standard subject to check whether there is a header matching the row to be matched. If there is a corresponding header in the header database, the row to be matched is marked as a header. Each element in the row to be matched is searched in the header database corresponding to the standard subject to check whether there is a header matching the element. If so, the element is marked as a header. When each element in the row to be matched is marked as a header, the row to be matched is considered to be a header, and the row to be matched is marked as a header. Among them, the header database is a database established according to different subject categories, and the headers stored therein are constructed according to the headers in the original continuation file provided by all first users.

[0070] Step 208: input the marked header into a pre-trained header mapping model for prediction to obtain the corresponding standard field.

[0071] Specifically, supervised learning is performed based on the FastText framework to construct a header mapping model. All headers provided by the first user and the set standard fields are taken as labeled data, the headers provided by the first user are taken as data, and the corresponding standard fields are taken as labels of the data. The header mapping model is supervisedly trained using the labeled data, the labeled data is taken as training samples, and the unlabeled data is taken as test samples to verify the accuracy of the header mapping model. When the accuracy reaches a preset threshold, the trained header mapping model is obtained. The marked header is input into the trained header mapping model to obtain the corresponding standard fields.

[0072] Step 210, extract the corresponding data under the marked header in the original transfer file, and standardize the data, and obtain a standardized transfer file based on the standard field and the corresponding standardized data.

[0073] Specifically, the data under the marked header in the original continuation file is extracted, the data is converted into corresponding standard data, and a standardized continuation file is constructed according to the standard data and the standard fields corresponding to the marked header.

[0074] In the above-mentioned continuation data processing method, the original continuation file is obtained, and the subject keywords in the original continuation file are extracted; according to the subject keywords, the corresponding standard subject is obtained from the mapping relationship between the keywords and the standard subject; the to-be-matched row under the standard subject is extracted from the original continuation file, and the search is performed in the header database corresponding to the standard subject. If a header matching the to-be-matched row is found in the header database, the to-be-matched row is marked as a header; the marked header is input into a pre-trained header mapping model for prediction to obtain the corresponding standard field; the corresponding data under the marked header in the original continuation file is extracted, and the data is standardized, and a standardized continuation file is obtained according to the standard field and the corresponding standardized data. This method improves the efficiency of continuation data processing and the accuracy rate by standardizing the subject, header, and data of the original continuation file.

[0075] In one embodiment, searching in a header database corresponding to the standard subject, if a header matching the row to be matched is found in the header database, marking the row to be matched as a header, includes:

[0076] Perform an accurate search in the header database corresponding to the standard subject, and if a header matching the row to be matched is found in the header database, mark the row to be matched as a header;

[0077] If a header matching the to-be-matched row is not found in the header database, a fuzzy query is used to determine whether the to-be-matched row is a header. If the to-be-matched row is determined to be a header, the to-be-matched row is marked as a header.

[0078] Specifically, the corresponding database established according to different subject categories may be an ElasticSearch database. Figure 3 As shown, according to each element in the to-be-matched row, a keyword precise query is performed in the ElasticSearch database corresponding to the standard subject. If a table header matching each element in the to-be-matched row is found in the Elastic Search database, the to-be-matched row is marked as a table header. If a table header matching the to-be-matched row is not found in the Elastic Search database, a fuzzy query is performed through the Elastic Search search score combined with the Levenshtein string distance score to determine whether the to-be-matched row is a table header.

[0079] In this embodiment, a keyword precise query is first used to quickly find out whether there is a table header that matches the row to be matched. If the precise search is not found, a fuzzy query combining the Elastic Search search score with the Levenshtein string distance score is used to determine whether the row to be matched is a table header, thereby improving the efficiency and accuracy of the processing of the resumed data.

[0080] In one embodiment, whether the to-be-matched row is a table header is determined by fuzzy query, and when the to-be-matched row is determined to be a table header, the to-be-matched row is marked as a table header, including:

[0081] Fuzzy search in the header database for headers whose similarity with the row to be matched exceeds a preset similarity threshold, and take them as hit records of the row to be matched;

[0082] Calculate the edit distance between the to-be-matched row and the hit record, and find the maximum value of the edit distances as the maximum edit distance;

[0083] When the maximum edit distance is greater than the header distance threshold, the row to be matched is marked as a header.

[0084] Specifically, the similarity between each element in the to-be-matched row and the header in the Elastic Search database is calculated according to the Elastic Search search scoring mechanism. When the similarity exceeds the preset similarity threshold, the header in the ElasticSearch database is taken as a hit record. Then, the edit distance between each element in the to-be-matched row and all hit records corresponding to this element is calculated using the Levenshtein string distance scoring method, and the maximum edit distance is found. When the maximum edit distance is greater than the preset header distance threshold, the to-be-matched row is marked as a header.

[0085] In one embodiment, the standard subject includes multiple subjects; the header mapping model includes a header mapping model corresponding to each subject in the multiple subjects;

[0086] The marked header is input into the pre-trained header mapping model for prediction to obtain the corresponding standard fields, including:

[0087] The corresponding header mapping model is searched according to the standard subject corresponding to the marked header, and the marked header is input into the header mapping model for prediction to obtain the corresponding standard field.

[0088] Specifically, for each standard topic, supervised learning is performed based on the FastText framework to build a header mapping model corresponding to the standard topic. The standard topic corresponding to the marked header is determined, and then the marked header is input into the corresponding header mapping model to obtain the corresponding standard field.

[0089] In this embodiment, the FastText framework is used for supervised learning to build a header mapping model. The FastText framework speeds up the training and prediction speeds while maintaining high accuracy. The corresponding standard fields can be quickly predicted based on the marked headers, thereby improving the efficiency of continued data processing.

[0090] In one embodiment, the marked header is input into a pre-trained header mapping model for prediction, and after obtaining the corresponding standard field, the following is further included:

[0091] Establish a field mapping relationship between the marked table header and the standard field;

[0092] The field mapping relationship and confirmation control are displayed on the terminal display interface;

[0093] In response to the user's triggering operation on the confirmation control, a confirmed standard field is obtained;

[0094] The standardized transfer file is obtained according to the standard field and the corresponding standardized processed data, including:

[0095] A standardized transfer file is obtained based on the confirmed standard fields and the corresponding standardized data.

[0096] Specifically, as shown in Table 2, it is a table in the original transfer file, and the header in the original transfer file is converted into the corresponding standard field, such as Figure 3 As shown, a field mapping relationship is established between the marked header in the original continuation file and the corresponding standard field. The header may have a single-level header and a multi-level header. The field mapping relationship is established according to the corresponding header format in the original continuation file. The multi-level header refers to a header that also includes at least two secondary headers. The field mapping relationship is then displayed on the terminal display interface, and the display interface also includes a confirmation control so that the user can confirm the converted field mapping relationship. When the terminal receives the user's click or touch on the confirmation control, it indicates that the user confirms that the field mapping relationship is correct and obtains the confirmed standard field. Thereafter, a standardized continuation file is established based on the confirmed standard field and the corresponding standardized data. The user's triggering operation on the confirmation control is not limited to click, touch and other operations, and can also be other forms of triggering operations, which are not limited in this embodiment.

[0097] In this embodiment, by confirming the field mapping relationship by the user, more accurate standard fields can be obtained, thereby improving the accuracy of the continued data processing.

[0098] Table 2 Example of original transfer file

[0099]

[0100] Table 3 Field mapping relationship example

[0101]

[0102] In one embodiment, the data corresponding to the table header after the mark includes catastrophic risk events and administrative region names;

[0103] The data is standardized, and a standardized transfer file is obtained according to the standard fields and the corresponding standardized data, including:

[0104] The catastrophic risk event or the name of the administrative region is searched in the standardized database. If matching standard data is found, a standardized transfer file is obtained based on the standard field and the matching standard data.

[0105] Specifically, the data corresponding to the marked header includes multiple types of data. The data corresponding to catastrophic risk events and the data corresponding to administrative region names are used as examples to illustrate the process of data standardization. For example, the administrative region name of "Beijing" in different original continuation files is inconsistent, some are "Beijing" and some are "Beijing". Data standardization is to unify these administrative region names. First, the data corresponding to the catastrophic risk event or the data corresponding to the administrative region name is searched in the standardized database through Flashtext text retrieval. If matching standard data is found, a standardized continuation file is obtained based on the standard field and the matching standard data.

[0106] In this embodiment, the corresponding standard data can be quickly found in a large-scale text through Flashtext text retrieval, and the search speed is fast, which improves the efficiency of the subsequent data processing.

[0107] In one embodiment, the method further comprises:

[0108] If no matching standard data is found in the standard database, the data editing distance between the catastrophic risk event or the administrative region name and the standard data is calculated. When the data editing distance is greater than the data distance threshold, a standardized continuation file is obtained based on the standard field and the standard data.

[0109] Specifically, if the corresponding standard data corresponding to the catastrophic risk event or the data corresponding to the administrative region name is not found in the standardized database through Flashtext text retrieval, the fuzzy query is performed through the Levenshtein string distance score to calculate the data editing distance between the data corresponding to the catastrophic risk event or the data corresponding to the administrative region name and the standard data. When the data editing distance is greater than the data distance threshold, a standardized continuation file is obtained according to the standard field and the standard data.

[0110] In this embodiment, the unstandardized data is standardized by using the Levenshtein string distance score, and the data is accurately standardized by using the data distance threshold, thereby improving the accuracy of the subsequent data processing.

[0111] In order to facilitate understanding of the technical solutions provided in the embodiments of the present application, Figure 4 As shown, the complete data processing process of the continued transfer is briefly described for the data processing method provided in the embodiment of the present application:

[0112] (1) Obtain the original transfer file (i.e. Figure 4 ), extracting the subject keywords in the original transfer file;

[0113] (2) According to the subject keyword, the corresponding standard subject (i.e. Figure 4 subject identification in );

[0114] (3) performing an accurate search in a header database corresponding to the standard subject, and if a header matching the row to be matched is found in the header database, marking the row to be matched as a header;

[0115] If no header matching the to-be-matched row is found in the header database, a fuzzy query is performed in the header database to find a header with a similarity with the to-be-matched row exceeding a preset similarity threshold, and the header is used as a hit record of the to-be-matched row; the edit distance between the to-be-matched row and the hit record is calculated, and the maximum value of the edit distance is found as the maximum edit distance; when the maximum edit distance is greater than the header distance threshold, the to-be-matched row is marked as a header (i.e. Figure 4 marking phase in .

[0116] (4) Input the marked header into the pre-trained header mapping model for prediction and obtain the corresponding standard field (i.e. Figure 4 parsing phase in );

[0117] (5) establishing a field mapping relationship between the marked table header and the standard field; displaying the field mapping relationship and a confirmation control on the terminal display interface; and obtaining the confirmed standard field (i.e. Figure 4 confirmation phase in the process);

[0118] (6) extracting the data corresponding to the marked header in the original transfer file, and performing standardization on the data, and obtaining a standardized transfer file (i.e., Figure 4 extraction phase).

[0119] It should be understood that although Figure 2-4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0120] In one embodiment, Figure 5 As shown, a device for processing continued data is provided, including: an acquisition module 502, a subject determination module 504, a header marking module 506, a standard field determination module 508 and a standardization module 510, wherein:

[0121] The acquisition module 502 is used to acquire the original continued file and extract the subject keywords in the original continued file.

[0122] The theme determination module 504 is used to search the mapping relationship between the theme keyword and the standard theme according to the theme keyword to obtain the corresponding standard theme.

[0123] The header marking module 506 is used to extract the to-be-matched rows under the standard subject from the original continuation file, and search in the header database corresponding to the standard subject. If a header matching the to-be-matched row is found in the header database, the to-be-matched row is marked as a header.

[0124] The standard field determination module 508 is used to input the marked header into a pre-trained header mapping model for prediction to obtain the corresponding standard field.

[0125] The standardization module 510 is used to extract the corresponding data under the marked header in the original continuation file and perform standardization processing on the data, and obtain a standardized continuation file according to the standard field and the corresponding standardized data.

[0126] In one embodiment, the header marking module 506 is further used to perform an accurate search in a header database corresponding to the standard subject, and if a header matching the to-be-matched row is found in the header database, the to-be-matched row is marked as a header;

[0127] And if a header matching the to-be-matched row is not found in the header database, a fuzzy query is used to determine whether the to-be-matched row is a header. When the to-be-matched row is determined to be a header, the to-be-matched row is marked as a header.

[0128] In one embodiment, the header marking module 506 is further used to fuzzy search the header in the header database for a header whose similarity with the to-be-matched row exceeds a preset similarity threshold, as a hit record of the to-be-matched row;

[0129] and calculating the edit distance between the to-be-matched row and the hit record, and finding the maximum value of the edit distances as the maximum edit distance;

[0130] And when the maximum edit distance is greater than the header distance threshold, the to-be-matched row is marked as a header.

[0131] In one embodiment, the standard subject includes multiple subjects; the header mapping model includes a header mapping model corresponding to each subject in the multiple subjects;

[0132] The standard field determination module 508 is further used to search for a corresponding header mapping model according to the standard subject corresponding to the marked header, input the marked header into the header mapping model for prediction, and obtain the corresponding standard field.

[0133] In one embodiment, the standard field determination module 508 is further used to establish a field mapping relationship between the marked header and the standard field;

[0134] And displaying the field mapping relationship and confirmation controls on the terminal display interface;

[0135] and in response to the user's triggering operation on the confirmation control, obtaining the confirmed standard field;

[0136] The standardization module 510 is also used to obtain a standardized continuation file based on the confirmed standard field and the corresponding standardized data.

[0137] In one embodiment, the data corresponding to the table header after the mark includes catastrophic risk events and administrative region names;

[0138] The standardization module 510 is also used to search the catastrophic risk event or the name of the administrative region in the standardization database. If matching standard data is found, a standardized continuation file is obtained based on the standard field and the matching standard data.

[0139] In one embodiment, the standardization module 510 is also used to calculate the data editing distance between the catastrophic risk event or the administrative area name and the standard data if no matching standard data is found in the standard database. When the data editing distance is greater than the data distance threshold, a standardized continuation file is obtained based on the standard field and the standard data.

[0140] For the specific definition of the continuation data processing device, please refer to the definition of the continuation data processing method above, which will not be repeated here. Each module in the above-mentioned continuation data processing device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0141] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for processing data is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0142] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0144] Obtain an original continuation file, and extract subject keywords from the original continuation file;

[0145] According to the subject keyword, the corresponding standard subject is obtained by searching from the mapping relationship between the keyword and the standard subject;

[0146] Extract the to-be-matched row under the standard subject from the original transfer file, search in the header database corresponding to the standard subject, and if a header matching the to-be-matched row is found in the header database, mark the to-be-matched row as a header;

[0147] Input the marked header into the pre-trained header mapping model for prediction to obtain the corresponding standard field;

[0148] The corresponding data under the marked header in the original continuation file is extracted, and the data is standardized, and a standardized continuation file is obtained according to the standard field and the corresponding standardized data.

[0149] In one embodiment, when the processor executes the computer program, the following steps are also implemented: searching in a header database corresponding to the standard subject, and if a header matching the row to be matched is found in the header database, marking the row to be matched as a header, including: performing an accurate search in a header database corresponding to the standard subject, and if a header matching the row to be matched is found in the header database, marking the row to be matched as a header; if a header matching the row to be matched is not found in the header database, determining whether the row to be matched is a header through fuzzy query, and when the row to be matched is determined to be a header, marking the row to be matched as a header.

[0150] In one embodiment, the processor further implements the following steps when executing the computer program: determining whether the row to be matched is a table header through fuzzy query, and when the row to be matched is determined to be a table header, marking the row to be matched as a table header, including: fuzzily querying the header of the row to be matched in the table header database, whose similarity exceeds a preset similarity threshold, as a hit record of the row to be matched; calculating the edit distance between the row to be matched and the hit record, and finding the maximum value in the edit distance as the maximum edit distance; when the maximum edit distance is greater than the table header distance threshold, marking the row to be matched as a table header.

[0151] In one embodiment, the processor also implements the following steps when executing the computer program: the standard subject includes multiple subjects; the header mapping model includes a header mapping model corresponding to each subject in the multiple subjects; the marked header is input into a pre-trained header mapping model for prediction to obtain a corresponding standard field, including: searching for a corresponding header mapping model according to the standard subject corresponding to the marked header, inputting the marked header into the header mapping model for prediction to obtain a corresponding standard field.

[0152] In one embodiment, the processor also implements the following steps when executing the computer program: inputting the marked header into a pre-trained header mapping model for prediction, and after obtaining the corresponding standard field, it also includes: establishing a field mapping relationship between the marked header and the standard field; displaying the field mapping relationship and the confirmation control on the terminal display interface; obtaining the confirmed standard field in response to the user's triggering operation on the confirmation control; obtaining a standardized continuation file based on the standard field and the corresponding standardized processed data, including: obtaining a standardized continuation file based on the confirmed standard field and the corresponding standardized processed data.

[0153] In one embodiment, the processor also implements the following steps when executing the computer program: the data corresponding to the marked header includes catastrophic risk events and administrative region names; the data is standardized, and a standardized continuation file is obtained based on the standard field and the corresponding standardized data, including: searching for the catastrophic risk event or the administrative region name in a standardized database, and if matching standard data is found, a standardized continuation file is obtained based on the standard field and the matching standard data.

[0154] In one embodiment, when the processor executes the computer program, the following steps are also implemented: if no matching standard data is found in the standard database, the data editing distance between the catastrophic risk event or the administrative area name and the standard data is calculated; when the data editing distance is greater than the data distance threshold, a standardized continuation file is obtained based on the standard field and the standard data.

[0155] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0156] Obtain an original continuation file, and extract subject keywords from the original continuation file;

[0157] According to the subject keyword, the corresponding standard subject is obtained by searching from the mapping relationship between the keyword and the standard subject;

[0158] Extract the to-be-matched row under the standard subject from the original transfer file, search in the header database corresponding to the standard subject, and if a header matching the to-be-matched row is found in the header database, mark the to-be-matched row as a header;

[0159] Input the marked header into the pre-trained header mapping model for prediction to obtain the corresponding standard field;

[0160] The corresponding data under the marked header in the original continuation file is extracted, and the data is standardized, and a standardized continuation file is obtained according to the standard field and the corresponding standardized data.

[0161] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: searching in a header database corresponding to the standard subject, and if a header matching the row to be matched is found in the header database, marking the row to be matched as a header, including: performing an accurate search in a header database corresponding to the standard subject, and if a header matching the row to be matched is found in the header database, marking the row to be matched as a header; if a header matching the row to be matched is not found in the header database, determining whether the row to be matched is a header through fuzzy query, and when the row to be matched is determined to be a header, marking the row to be matched as a header.

[0162] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining whether the row to be matched is a table header through fuzzy query, and when the row to be matched is determined to be a table header, marking the row to be matched as a table header, including: fuzzy querying the header of the row to be matched in the header database, the header whose similarity exceeds a preset similarity threshold as a hit record of the row to be matched; calculating the edit distance between the row to be matched and the hit record, and finding the maximum value in the edit distance as the maximum edit distance; when the maximum edit distance is greater than the header distance threshold, marking the row to be matched as a table header.

[0163] In one embodiment, the computer program also implements the following steps when executed by the processor: the standard subject includes multiple subjects; the header mapping model includes a header mapping model corresponding to each subject in the multiple subjects; the marked header is input into a pre-trained header mapping model for prediction to obtain a corresponding standard field, including: searching for a corresponding header mapping model according to the standard subject corresponding to the marked header, inputting the marked header into the header mapping model for prediction to obtain a corresponding standard field.

[0164] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the marked header into a pre-trained header mapping model for prediction, and after obtaining the corresponding standard field, it also includes: establishing a field mapping relationship between the marked header and the standard field; displaying the field mapping relationship and the confirmation control on the terminal display interface; obtaining the confirmed standard field in response to the user's triggering operation on the confirmation control; obtaining a standardized continuation file based on the standard field and the corresponding standardized processed data, including: obtaining a standardized continuation file based on the confirmed standard field and the corresponding standardized processed data.

[0165] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the corresponding data under the marked header includes catastrophic risk events and administrative region names; the data is standardized, and a standardized continuation file is obtained based on the standard field and the corresponding standardized data, including: searching for the catastrophic risk event or the administrative region name in a standardized database, and if matching standard data is found, a standardized continuation file is obtained based on the standard field and the matching standard data.

[0166] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: if no matching standard data is found in the standard database, the data editing distance between the catastrophic risk event or the administrative area name and the standard data is calculated; when the data editing distance is greater than a data distance threshold, a standardized continuation file is obtained based on the standard field and the standard data.

[0167] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0168] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for processing continued data, characterized in that: The method comprises: Acquire an original continuation file, and extract subject keywords from the original continuation file according to a preset regular expression; the subject keywords refer to words or phrases used to represent the subject content of the original continuation file; the original continuation file includes at least one first subject, and each of the first subjects includes at least one second subject; According to the subject keyword, a corresponding standard subject is obtained by searching from the mapping relationship between the keyword and the standard subject; the standard subject refers to a pre-set standard name for the same attribute entity name; Extract the to-be-matched rows under the standard subject from the original continuation file, and perform an accurate search in the header database corresponding to the standard subject. If a header matching the to-be-matched row is found in the header database, the to-be-matched row is marked as a header. If a header matching the to-be-matched row is not found in the header database, determine whether the to-be-matched row is a header through a fuzzy query combining a search score with a string distance score. If the to-be-matched row is determined to be a header, the to-be-matched row is marked as a header. The to-be-matched rows refer to all rows corresponding to the second subject. Input the marked header into the pre-trained header mapping model for prediction to obtain the corresponding standard field; The corresponding data under the marked header in the original continuation file is extracted, and the data is standardized, and a standardized continuation file is obtained according to the standard field and the corresponding standardized data.

2. The method according to claim 1, characterized in that The step of determining whether the to-be-matched row is a table header by using a fuzzy query, and marking the to-be-matched row as a table header when the to-be-matched row is determined to be a table header, includes: Fuzzily searching the header database for a header whose similarity with the row to be matched exceeds a preset similarity threshold, and taking it as a hit record of the row to be matched; Calculating the edit distance between the to-be-matched row and the hit record, and finding the maximum value of the edit distances as the maximum edit distance; When the maximum edit distance is greater than the header distance threshold, the to-be-matched row is marked as a header.

3. The method according to claim 1, characterized in that The standard subject includes multiple subjects; the header mapping model includes a header mapping model corresponding to each subject in the multiple subjects; The marked header is input into a pre-trained header mapping model for prediction to obtain the corresponding standard fields, including: The corresponding header mapping model is searched according to the standard subject corresponding to the marked header, and the marked header is input into the header mapping model for prediction to obtain the corresponding standard field.

4. The method according to claim 1, characterized in that: After inputting the marked header into a pre-trained header mapping model for prediction and obtaining the corresponding standard field, the method further includes: Establishing a field mapping relationship between the marked header and the standard field; Display the field mapping relationship and confirmation controls on the terminal display interface; In response to the user triggering operation on the confirmation control, obtaining the confirmed standard field; The step of obtaining a standardized transfer file according to the standard fields and the corresponding standardized processed data includes: A standardized continuation file is obtained based on the confirmed standard fields and the corresponding standardized data.

5. The method according to claim 1, characterized in that The corresponding data under the marked table header include catastrophic risk events and administrative region names; The step of performing standardization processing on the data and obtaining a standardized transfer file according to the standard fields and the corresponding standardized processed data includes: The catastrophic risk event or the administrative region name is searched in a standardized database. If matching standard data is found, a standardized continuation file is obtained based on the standard field and the matching standard data.

6. The method according to claim 5, characterized in that The method further comprises: If no matching standard data is found in the standard database, the data editing distance between the catastrophic risk event or the administrative area name and the standard data is calculated. When the data editing distance is greater than the data distance threshold, a standardized continuation file is obtained based on the standard field and the standard data.

7. A data processing device, characterized in that: The device comprises: An acquisition module is used to acquire an original continuation file, and extract a subject keyword from the original continuation file according to a preset regular expression; the subject keyword refers to a word or phrase used to represent the subject content of the original continuation file; the original continuation file includes at least one first subject, and each of the first subjects includes at least one second subject; A subject determination module, used to search for a corresponding standard subject from a mapping relationship between the subject keyword and the standard subject according to the subject keyword; the standard subject refers to a pre-set standard name for the same attribute entity name; A header marking module is used to extract the to-be-matched rows under the standard subject from the original continuation file, and perform an accurate search in the header database corresponding to the standard subject. If a header matching the to-be-matched row is found in the header database, the to-be-matched row is marked as a header; if a header matching the to-be-matched row is not found in the header database, whether the to-be-matched row is a header is determined by a fuzzy query combining a search score with a string distance score. When the to-be-matched row is determined to be a header, the to-be-matched row is marked as a header; the to-be-matched row refers to all rows corresponding to the second subject; The standard field determination module is used to input the marked header into the pre-trained header mapping model for prediction to obtain the corresponding standard field; The standardization module is used to extract the corresponding data under the marked header in the original continuation file, and standardize the data to obtain a standardized continuation file based on the standard field and the corresponding standardized data.

8. The device according to claim 7, characterized in that The header marking module is also used to fuzzy search in the header database for headers whose similarity with the to-be-matched row exceeds a preset similarity threshold, as hit records of the to-be-matched row; calculate the edit distance between the to-be-matched row and the hit record, and find the maximum value of the edit distances as the maximum edit distance; when the maximum edit distance is greater than the header distance threshold, mark the to-be-matched row as a header.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images