A method and device for fusion of unstructured text information

Abstract

The present application discloses a method and device for fusion of unstructured text information, which includes first extracting various information tags from text information of different formats, and extracting corresponding key-value pair tags; extracting information from text information according to tags, and storing the extracted information in a table and database; dividing the information into multiple different data streams according to different information, and then standardizing, repairing and converting rules, and re-integrating them into standard data streams; extracting primary key ID information from each data source, and establishing the association between each data source according to the primary key ID information. Finally, the primary key association of each information source is established, and the information use priority of different data sources is set at the same time, so that multiple data sources are formed into a unified whole. The fusion of multiple text data sources with large associations can be completed efficiently, and a standardized new data source is generated after fusion.

Description

Technical Field

[0001] The present application relates to the technical field of big data text processing, and specifically to a method and device for fusing unstructured text information. Background Art

[0002] In the era of rapid technological development, the extraction, deep processing, analysis and application of data information have become increasingly important. Text information is also a part of data information. The application scenarios of text information are extremely wide, covering many fields, including patents, legal documents, etc. Text has an absolute advantage in describing such information. Therefore, it is particularly important to mine the value of text information.

[0003] The processing of text information is different from the traditional data information processing. The extraction of text information requires a lot of human intervention, and business personnel and developers need to cooperate to sort out the characteristics of text information. Developers identify text information based on the characteristics, and then store the data, and then perform multi-source fusion of text data from different data sources. Finally, the fused data is applied.

[0004] A common data fusion solution is the idea of ​​multi-source heterogeneous data fusion. This method is only applicable to the multi-source fusion of simple unstructured data and structured data. Once you encounter text information that is not stored in a fixed format, such as XML, TXT, PDF, or pictures (photographing), there will be many difficulties. One is the difficulty of data storage: the traditional method of text information storage will be standardized first, but there are no standardized rules for complex text information. The standardization of text information responsible for the rules will lose a lot of data, making it difficult to ensure the integrity of the data. Another is the difficulty of data fusion: after the data is stored, due to the diversity of data sources, each data source does not have a common standardized primary key for association, making it difficult to align the information of multiple text sources at a granular level and generate effective associations.

[0005] Existing technologies integrate multi-source data through steps such as data collection, data cleaning, data integration, and data storage. This solution has significant advantages in processing structured text data, but it is slightly insufficient for unstructured text data. The specific shortcomings are as follows:

[0006] The traditional data collection method is to collect data with relatively uniform formats using unified rules, such as collecting data in MySQL, collecting XML data according to DTD, etc. However, for text data, using unified rules for data collection will lose a lot of information, because it is difficult for text to have unified rules and a unified format. Text data from different eras have different formats and different characteristics of content expression. The storage method of older data may even be stored in paper form. This type of information cannot be collected using traditional data collection methods. This patent adds a feature extraction device before data collection, which can extract the features of text information from different eras and different storage forms and form feature kinship, thus providing a basis for data collection.

[0007] The traditional data cleaning stage is to first formulate a unified rule, clean the data, retain the data that meets the rule, and directly filter out the data that does not meet the rule. This method of processing text data for data cleaning will lose a lot of data. Because data cleaning requires specifying rules, but from text data extraction to processing, the rules are established little by little, rather than being innate. Therefore, it is not advisable to directly formulate a unified rule for data cleaning. Summary of the invention

[0008] The present application provides a method and device for fusing unstructured text information, aiming to solve the problems existing in the prior art.

[0009] In a first aspect, a method for fusing unstructured text information is provided, the method comprising:

[0010] Feature extraction: extract various information tags from text information in different formats and extract corresponding key-value pair tags; perform tag extraction on the entire data and summarize and generate a set of feature tags;

[0011] Data extraction: extract information from text information based on the generated feature tag set and store the extracted information in a table or database;

[0012] Data diversion: divide the data of each data source into multiple different data streams according to different information; each data source generates a data table;

[0013] Data standardization: standardize the divided data streams;

[0014] Data reflow: Repair and rule-convert the data filtered out by data standardization and reintegrate it into the standard data stream;

[0015] Primary key fusion: Extract the primary key ID information from each data source, and establish the association between each data source based on the primary key ID information of each data source;

[0016] Data confluence establishes the primary key association of each information source, and at the same time sets the information usage priority of different data sources to form multiple data sources into a unified whole.

[0017] Optionally, in feature extraction, various information tags in text information of different formats are extracted to extract corresponding key-value pair tags, specifically including performing corresponding analysis for different data sources, wherein:

[0018] Apache POI, python-docx, and PyMuPDF are used to parse PDF and word documents, and the parsed data is put into the feature blood relationship storage layer;

[0019] For the analysis of handwritten documents and txt data, relevant feature values ​​are obtained in advance through key-value pair information, and the parsed data is placed in the feature blood relationship storage layer;

[0020] Parsing of XML document information includes extracting all XML structural information and transferring it to JSON files for feature extraction and storing it in the feature lineage storage layer; the parser includes XPath parser, DOM parser, and SAX parser; the feature lineage storage layer is used to store the feature information of the tags in each data source.

[0021] Optionally, the feature tag set generated in data extraction is used to extract information from text information according to the tag, and the extracted information is stored in a table and a database, including:

[0022] The information in the data source is filtered through the labels of the feature blood relationship storage layer, and the information is saved in a KV type database to form a data source KV storage layer; wherein the KV type database uses a redis database or an hbase database;

[0023] For abnormal data that cannot be matched, record the information of each text that cannot be matched, collect the abnormal logs, and perform relevant abnormal processing.

[0024] Optionally, the divided data streams are standardized in data standardization, including:

[0025] It is necessary to normalize the data from different text information sources, including converting the data into a unified format and standard;

[0026] Verify the integrity, consistency and uniqueness of the data; the integrity check ensures that all necessary data fields have been collected and are not missing; the consistency check ensures that the data is in the correct order including determining the date and time, and the classified data conforms to the predefined categories; the uniqueness check ensures that there are no duplicate records in the data;

[0027] Data is extracted from the source system, converted into a format suitable for analysis, and then loaded into the target database; data cleansing, format conversion, encoding mapping, and data aggregation operations are performed during the conversion phase.

[0028] Optionally, in data splitting, the data of each data source is divided into multiple different data streams according to different information, including:

[0029] Generate a data table for each data source. According to the field length of each field, generate n samples of the longest field value, n samples of the minimum field value, and n samples of the average field value for each field.

[0030] By comparing sampled data, the same information types from different data sources are determined based on information with similar features, and then classified.

[0031] Optionally, primary key fusion extracts primary key ID information from each data source, and establishes associations between each data source based on the primary key ID information of each data source, including:

[0032] Extract primary key ID information from each data source; the primary key ID is a field that uniquely identifies a record, including a contract number, user ID, or product code;

[0033] Before matching, the extracted primary key ID information needs to be preprocessed, and the word segmentation query function of the Elasticsearch index is used to perform fuzzy query on the ID to find similar IDs and classify similar IDs into the same data set;

[0034] For those data sets that have been classified by fuzzy matching, we further use the precise matching technology to perform one-to-one matching. In particular, we normalize the primary keys of different data sources under the data set, including removing leading zeros and unifying the date format.

[0035] Through fuzzy matching and exact matching, the primary key IDs pointing to the same entity in different data sources are determined, and the association between the primary key IDs pointing to the same entity is established.

[0036] In a second aspect, a device for fusion of unstructured text information is provided, the device comprising:

[0037] The feature extraction module is used to extract various information tags from text information of different formats and extract the corresponding key-value pair tags; wherein, the tag extraction of the entire data is performed and the feature tag set generated by summarizing and aggregating is performed;

[0038] The data extraction module is used to extract information from the text information according to the generated feature tag set and store the extracted information in a table and database;

[0039] The data distribution module is used to divide the data of each data source into multiple different data streams according to different information; wherein each data source generates a data table;

[0040] A data standardization module is used to standardize the divided data streams;

[0041] The data reflow module is used to repair and transform the data filtered out by data standardization and reintegrate it into the standard data stream;

[0042] The primary key fusion module is used to extract the primary key ID information from each data source and establish the association between each data source based on the primary key ID information of each data source;

[0043] The data confluence module is used to establish the primary key association of each information source, and at the same time set the information use priority of different data sources to form multiple data sources into a unified whole.

[0044] In a third aspect, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the unstructured text information fusion method described in any one of the first aspects is implemented.

[0045] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for fusion of unstructured text information described in any one of the first aspects is implemented.

[0046] In a fifth aspect, a computer program product is provided, including a computer program / instruction, which, when executed, implements the unstructured text information fusion method described in any one of the first aspects.

[0047] Compared with the prior art, this application has at least the following beneficial effects:

[0048] This application can efficiently complete the fusion of multiple text data sources with great correlation, and generate a standardized new data source after fusion. This data source covers all the information of multiple data sources after standardization, and can give priority to displaying the relevant data of a certain data source as needed. At the same time, multiple text data sources with correlation can be standardized at the same time, and the data source standardization rules can be flexibly adjusted and quickly applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0050] Figure 1 A flowchart of the steps of the unstructured text information fusion method provided in the embodiment of the present application;

[0051] Figure 2 A block diagram of an unstructured text information fusion device provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] In the description of the present application: the terms "comprise", "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units that have been explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or equipment, or steps or units added based on further optimization schemes conceived by the present invention.

[0055] Existing technologies integrate multi-source data through steps such as data collection, data cleaning, data integration, and data storage. This solution has significant advantages in processing structured text data, but it is slightly insufficient for unstructured text data. The specific shortcomings are as follows:

[0056] The traditional data collection method is to collect data with relatively uniform formats using unified rules, such as collecting data in MySQL, collecting XML data according to DTD, etc. However, for text data, using unified rules for data collection will lose a lot of information, because it is difficult to have unified rules and formats for text. Text data from different eras has different formats and different characteristics of content expression. The storage method of relatively old data may even be stored in paper form. This type of information cannot be collected using traditional data collection methods. Before data collection, traditional technology adds a feature extraction device, which can extract the features of text information from different eras and different storage forms and form feature kinship, thus providing a basis for data collection.

[0057] The traditional data cleaning stage is to first formulate a unified rule, clean the data, retain the data that meets the rule, and directly filter out the data that does not meet the rule. This method of processing text data for data cleaning will lose a lot of data. Because data cleaning requires specifying rules, but from text data extraction to processing, the rules are established little by little, rather than being innate. Therefore, it is not advisable to directly formulate a unified rule for data cleaning.

[0058] This application removes the data cleaning layer and ensures the integrity and standardization of the data through the data diversion device and the data return device. In the data diversion device, preliminary rules are generated according to the data samples to pre-process the data, thereby ensuring the standardization of the data. The data filtered out by the data diversion layer will pass through the data return device to continuously strengthen the rules of the data diversion device and reconnect to the data diversion device to ensure data integrity.

[0059] Different text data sources do not have a unified primary key. Data collection and extraction are feasible for structured data, but it is difficult to extract text data because its characteristics are not understood. Data cleaning and preprocessing will filter out data that does not conform to the rules, but text data has no rules. The rules are only assumed in advance, so traditional processing ideas will lose data. Data integration and conversion both rely on the existence of a relatable primary key ID, which is difficult to use for text information without a fixed ID.

[0060] Traditional methods of data storage will be standardized first, and the standardization of text data will lose data. The present invention will retain all information, automatically match logic according to different data sources, and divide it into multiple data domains to form endogenous processing rules.

[0061] Please refer to Figure 1 , which shows a flowchart of a method for fusion of unstructured text information provided by an embodiment of the present application. The method may include the following steps:

[0062] Feature extraction: Extract various information tags from text information in different formats and extract the corresponding key-value pair tags; among them, perform tag extraction on the entire data and summarize and generate a set of feature tags.

[0063] Specifically, the feature extraction device is used to extract feature identifiers from large text data information in different formats, and the feature extraction device includes a data parsing layer and a feature kinship storage layer. Large text data comes from many sources, mainly government business data, corporate official website data, social Internet data, etc. The data is usually in the form of handwritten documents, pdf, xml, txt, word documents, etc.

[0064] Data parsing layer: Develop different codes for different data sources and perform corresponding parsing.

[0065] 1. Apache POI, python-docx, PyMuPDF and other methods are used to parse PDF and word documents, and the parsed data is placed in the feature blood relationship storage layer.

[0066] 2. The parsing of handwritten documents and txt data is rather special, because most of the data therein exists in the form of key-value pairs. It is necessary to write code to extract relevant feature values ​​in advance through key-value pair information, and put the parsed data into the feature blood relationship storage layer.

[0067] 3. For the analysis of XML document information, if there is a DTD, you can first refer to the DTD of the file to obtain the internal feature information. However, due to the historical year problem of the data source, the information in the DTD may not be completely applicable in different years, so you still need to write code to extract all the XML structure information and save it as a JSON file for feature extraction. Parsers include XPath parser, DOM (Document Object Model) parser, and SAX (Simple API for XML) parser.

[0068] The structure of a dtd is generally composed of an element type declaration, an attribute list declaration, an entity declaration, a notation declaration, etc., which predefines the element structure, attribute type, and element structure of the XML document to be created in the future.

[0069] Feature lineage storage layer: used to store the feature information of the labels in each data source. This is because the labels corresponding to information elements in different eras and regions are diverse. For example, the descriptions of a certain legal provision are different in different periods, but the basic content remains unchanged. The labels carried by such data are related, so a label lineage storage layer needs to be established for each type of information label to identify them.

[0070] Data extraction: The generated feature tag set is used to extract text information based on the tags, and the extracted information is stored in a table.

[0071] Data extraction specifically includes a data source KV storage layer and an exception capture layer. This device is used to filter and extract valid information from the data source and to capture exceptions in special format data. Because text information was often transmitted in the form of paper documents in the early years, even though the text format has been standardized in the past 10 years, there are still varying degrees of dirty data. Therefore, when extracting information from text data, an exception capture layer is still needed to handle abnormal data.

[0072] Data source KV storage layer: Data sources use labels to filter out valuable information in the data source through the labels of the feature blood relationship storage layer, and then save the information into a KV type database to form a data source KV storage layer. KV type databases can use redis, hbase, etc.

[0073] Exception capture layer: Ideally, all information of a data source will be stored in the data source KV storage layer. Data that cannot be stored is actually abnormal data that does not match the data. The exception capture layer can record and view the information that each text does not match, collect exception logs, and perform relevant exception processing.

[0074] Data diversion: Divide the data of each data source into multiple different data streams according to different information; each data source generates a data table.

[0075] Specifically, data diversion is used to divide the data of each data source into multiple different data streams according to different information, so as to aggregate the same information from different data sources. It includes the data sampling layer and the data standardization layer. The data sampling layer extracts some data features to identify the data information type for data diversion, and the data standardization layer performs unified formatting processing on the data of multiple text information sources.

[0076] Data sampling layer: Information between multiple data streams is mutually exclusive. Each data source generates a data table. According to the field length of each field, n samples of the longest field value, n samples of the minimum field value, and n samples of the field average value are generated for each field. By comparing the sampled data, it is possible to know whether the information with similar features is the same information type from different data sources, so as to classify it.

[0077] Data standardization layer: The data standardization layer is to normalize the data. Different text information sources describe information differently. Here, it is necessary to process according to unified rules and conduct a certain degree of DQC verification to process the integrity, consistency and uniqueness of the data. In addition, data standardization also involves etl-related operations. For the complex business logic related to etl, manual intervention is required to formulate it.

[0078] Data standardization: Standardize the divided data streams.

[0079] Specifically, data from different text information sources need to be normalized, including converting the data into a unified format and standard;

[0080] Verify the integrity, consistency and uniqueness of the data; the integrity check ensures that all necessary data fields have been collected and are not missing; the consistency check ensures that the data is in the correct order including determining the date and time, and the classified data conforms to the predefined categories; the uniqueness check ensures that there are no duplicate records in the data;

[0081] Data is extracted from the source system, converted into a format suitable for analysis, and then loaded into the target database; data cleansing, format conversion, encoding mapping, and data aggregation operations are performed during the conversion phase.

[0082] Data reflow: Repair and rule-convert the data filtered out by data standardization, and reintegrate it into the standard data stream.

[0083] Specifically, data diversion will inevitably filter out some data that does not meet the requirements, so a series of repairs and rule conversions are needed to convert the data that does not meet the requirements back into data that meets the requirements and integrate it into the system.

[0084] Primary key fusion: Extract the primary key ID information from each data source and establish the association between each data source based on the primary key ID information of each data source.

[0085] Data confluence establishes the primary key association of each information source, and at the same time sets the information usage priority of different data sources to form multiple data sources into a unified whole.

[0086] Specifically, data confluence includes the primary key matching layer and the data fusion layer. Information obtained from different text information sources will have a business primary key for unique identification, such as the contract registration number. The primary key matching layer is to establish the association between the primary keys of each information source and form multiple data sources into a unified whole. The same information exists in different data sources. For example, a certain registration information can be obtained from Department A and Department B. If both data sources provide the information of the registrant, but only one registrant can actually be displayed, then the data fusion layer is needed. The data fusion layer is designed for the parts of different data sources.

[0087] Primary key matching layer: extract the primary key ID information from each data source and put it into an ID pool. The ID pool contains all the ID information of all data sources. Find the correlation between the IDs of each data in the ID pool and establish the correlation between each data source. This layer uses two technologies: fuzzy matching and precise matching. Fuzzy matching refers to matching with the help of word segmentation query of es index, putting similar numbers into one data set. Precise matching is to perform one-to-one matching after normalizing the primary keys of different data sources under the data set, and obtain the primary key used for association of each data source.

[0088] Data fusion layer: Different data sources may provide different data for a certain type of information, but in reality, only one or more of them need to be selected for application. The data confluence device can set the use priority of different information from different data sources. The advantage of this setting is that once the data source application priority needs to be adjusted, it can respond quickly and be fault-tolerant efficiently.

[0089] Furthermore, the extraction of primary keys for the primary key matching layer is the focus of the present invention. For the identification of primary keys, the ES fuzzy matching algorithm is used in this case. In Elasticsearch, fuzzy matching is usually achieved through fuzzy query. In Elasticsearch, the tolerance of fuzzy queries can be controlled by specifying an edit distance. Use this method to match a similar set of IDs. The actual operation is to take out the primary keys IDs1, IDs2, and IDs3 from each data source, and then put the primary keys of n data sources into n indexes, and then use es's postion technology to mark each character of each ID number of each index with a position label. Between the n indexes, algorithm matching is performed internally. In order to improve the accuracy, the bigram, trigram, and tetragram technology of es are combined to accurately circle out a set of related ID numbers. As Figure 2 As shown, a flowchart of each process in the unstructured text information fusion device of the present application is given.

[0090] In one embodiment, a device for fusion of unstructured text information is provided, the device comprising:

[0091] The feature extraction module extracts various information tags from text information of different formats and extracts the corresponding key-value pair tags. This extraction requires the extraction of tags from the entire data and the summarization to generate the corresponding feature tag set.

[0092] The data extraction module extracts information from text information based on the tag set obtained by the feature extraction module, and stores the extracted information in a table. This method does not filter the data and retains all data information.

[0093] The data distribution module is used to divide the data of each data source into multiple different data streams according to different information. The information between multiple data streams is mutually exclusive. Each data source generates a data table. According to the field length of each field, the longest field value of each field is generated with n samples, the minimum field value is generated with n samples, and the average field value is generated with n samples. Each field is used as a separate data domain.

[0094] The data standardization module is used to standardize the output information of the data diversion module. It means obtaining all the detailed data of each text information source from the data parsing layer, extracting the corresponding field standardization rules and performing unified processing. Complex business logic requires manual intervention.

[0095] The data reflux module is used to ensure the integrity of the data. It performs a series of repairs and rule conversions on the data filtered out by the data standardization module and reintegrates it into the data diversion module.

[0096] The primary key fusion module extracts the primary key ID information from each data source and puts it into an ID pool. The ID pool contains all the ID information of all data sources. It searches for the correlation between the IDs of each data in the ID pool and establishes the correlation between each data source.

[0097] Data confluence module, the data confluence module includes the primary key matching layer and the data fusion layer. Different data sources may provide different data for a certain type of information, but in practice only one or more of them need to be selected for application. The data confluence module can set the use priority of different information from different data sources.

[0098] For the specific definition of the unstructured text information fusion device, please refer to the definition of the unstructured text information fusion method mentioned above, which will not be repeated here. Each module in the above-mentioned unstructured text information fusion device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0099] In one embodiment, an electronic device is provided. The electronic device may be a computer, and its internal structure diagram may be as follows: Figure 3 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for unstructured text information fusion data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for fusion of unstructured text information is implemented.

[0100] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0101] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned unstructured text information fusion method are implemented.

[0102] In one embodiment of the present application, a computer program product is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned unstructured text information fusion method are implemented.

[0103] The computer-readable storage medium and computer program product provided in this embodiment have similar implementation principles and technical effects to those of the above method embodiments, and are not described in detail here.

[0104] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0105] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A method for fusion of unstructured text information, characterized in that: The method comprises: Feature extraction: extract various information tags from text information in different formats and extract corresponding key-value pair tags; perform tag extraction on the entire data and summarize and generate a set of feature tags; Data extraction: extract information from text information based on the generated feature tag set and store the extracted information in a table or database; Data diversion: divide the data of each data source into multiple different data streams according to different information; each data source generates a data table; Data standardization: standardize the divided data streams; Data reflow: Repair and rule-convert the data filtered out by data standardization and reintegrate it into the standard data stream; Primary key fusion: Extract the primary key ID information from each data source, and establish the association between each data source based on the primary key ID information of each data source; Data confluence, establishing the primary key association of each information source, setting the information use priority of different data sources, and forming multiple data sources into a unified whole; In primary key fusion, primary key ID information is extracted from each data source, and the association between each data source is established based on the primary key ID information of each data source, including: Extract primary key ID information from each data source; the primary key ID is a field that uniquely identifies a record, including a contract number, user ID, or product code; Before matching, the extracted primary key ID information needs to be preprocessed, and the word segmentation query function of the Elasticsearch index is used to perform fuzzy query on the ID to find similar IDs and classify similar IDs into the same data set; For those data sets that have been classified by fuzzy matching, we further use the precise matching technology to perform one-to-one matching. In particular, we normalize the primary keys of different data sources under the data set, including removing leading zeros and unifying the date format. Through fuzzy matching and exact matching, the primary key IDs pointing to the same entity in different data sources are determined, and the association between the primary key IDs pointing to the same entity is established.

2. The method according to claim 1, characterized in that In feature extraction, various information tags in text information of different formats are extracted to extract corresponding key-value pairs, including corresponding analysis for different data sources, including: Apache POI, python-docx, and PyMuPDF are used to parse PDF and word documents, and the parsed data is put into the feature blood relationship storage layer; For the analysis of handwritten documents and txt data, relevant feature values ​​are obtained in advance through key-value pair information, and the parsed data is placed in the feature blood relationship storage layer; Parsing of XML document information includes extracting all XML structural information and transferring it to JSON files for feature extraction and storing it in the feature lineage storage layer; the parser includes XPath parser, DOM parser, and SAX parser; the feature lineage storage layer is used to store the feature information of the tags in each data source.

3. The method according to claim 2, characterized in that The feature tag set generated in data extraction is used to extract information from text information according to the tag, and the extracted information is stored in a table and database, including: The information in the data source is filtered through the labels of the feature blood relationship storage layer, and the information is saved in a KV type database to form a data source KV storage layer; wherein the KV type database uses a redis database or an hbase database; For abnormal data that cannot be matched, record the information of each text that cannot be matched, collect the abnormal logs, and perform relevant abnormal processing.

4. The method according to claim 1, characterized in that: In data standardization, the divided data streams are standardized, including: It is necessary to normalize the data from different text information sources, including converting the data into a unified format and standard; Verify the integrity, consistency and uniqueness of the data; the integrity check ensures that all necessary data fields have been collected and are not missing; the consistency check ensures that the data is in the correct order including determining the date and time, and the classified data conforms to the predefined categories; the uniqueness check ensures that there are no duplicate records in the data; Data is extracted from the source system, converted into a format suitable for analysis, and then loaded into the target database; data cleansing, format conversion, encoding mapping, and data aggregation operations are performed during the conversion phase.

5. The method according to claim 1, characterized in that In data diversion, the data from each data source is divided into multiple different data streams according to different information, including: Generate a data table for each data source. According to the field length of each field, generate n samples of the longest field value, n samples of the minimum field value, and n samples of the average field value for each field. By comparing sampled data, the same information types from different data sources are determined based on information with similar features, and then classified.

6. A non-structured text information fusion device, characterized in that: The device includes: The feature extraction module is used to extract various information tags from text information of different formats and extract the corresponding key-value pair tags; among them, the tag extraction of the full amount of data is performed, and the feature tag set generated by summarizing and aggregating is performed; The data extraction module is used to extract information from the text information according to the generated feature tag set and store the extracted information in a table and database; The data distribution module is used to divide the data of each data source into multiple different data streams according to different information; wherein each data source generates a data table; A data standardization module is used to standardize the divided data streams; The data reflow module is used to repair and transform the data filtered out by data standardization and reintegrate it into the standard data stream; The primary key fusion module is used to extract the primary key ID information from each data source and establish the association between each data source based on the primary key ID information of each data source; The data confluence module is used to establish the primary key association of each information source, and set the information use priority of different data sources to form multiple data sources into a unified whole; In primary key fusion, primary key ID information is extracted from each data source, and the association between each data source is established based on the primary key ID information of each data source, including: Extract primary key ID information from each data source; the primary key ID is a field that uniquely identifies a record, including a contract number, user ID, or product code; Before matching, the extracted primary key ID information needs to be preprocessed, and the word segmentation query function of the Elasticsearch index is used to perform fuzzy query on the ID to find similar IDs and classify similar IDs into the same data set; For those data sets that have been classified by fuzzy matching, we further use the precise matching technology to perform one-to-one matching. In particular, we normalize the primary keys of different data sources under the data set, including removing leading zeros and unifying the date format. Through fuzzy matching and exact matching, the primary key IDs pointing to the same entity in different data sources are determined, and the association between the primary key IDs pointing to the same entity is established.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images