Data fusion method and device, electronic device and storage medium

Abstract

The present disclosure relates to a data fusion method and device, an electronic device and a storage medium, the method comprising: constructing an index for a first database to obtain an index table; selecting a second database from a plurality of databases to be fused in turn, performing the following operations to obtain a fused first database: using the content in the target data in the second database that corresponds to the index field of the first database as an index condition, searching for candidate data corresponding to the target data according to the index table of the first database, and obtaining a plurality of candidate data combinations; when it is determined that the target data and the candidate data in one or more candidate data combinations of the plurality of candidate data combinations are different data, merging the target data in the one or more candidate data combinations into the first database, and updating the index table. The embodiments of the present disclosure can achieve data deduplication of databases of different data sources, and quickly and efficiently fuse the first database and a plurality of databases to be fused.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data fusion method and device, an electronic device, and a storage medium. Background Art

[0002] With the advent of the big data era, various types of data are being generated continuously, forming multiple databases with multiple data sources. At present, more and more scenarios require the fusion of data from different data sources to remove redundant and duplicate data in order to reduce storage costs and improve storage space utilization. However, when performing data fusion, the accuracy and efficiency of related technologies are low, and the computing cost is high. Summary of the invention

[0003] According to one aspect of the present disclosure, a data fusion method is provided, the method comprising: constructing an index for a first database to obtain an index table, the index table comprising multiple index fields and index data corresponding to each index field; selecting a second database from multiple databases to be fused in turn, performing the following operations to obtain a fused first database: using the content in the target data in the second database corresponding to the index field of the first database as an index condition, searching for candidate data corresponding to the target data according to the index table of the first database, to obtain multiple candidate data combinations, the candidate data combinations comprising target data and corresponding candidate data; determining whether the target data and the candidate data in the candidate data combinations are the same data according to a trained data analysis model, and when it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, fusing the target data in the one or more candidate data combinations into the first database, and updating the index table.

[0004] In a possible implementation, constructing an index for the first database includes: creating index fields for each basic data in the first database based on preset fields; determining index data for each index field based on each basic data in the first database; wherein the index data includes content in the basic data corresponding to each index field, and the corresponding content is associated with the basic data; or the index data includes content in the basic data corresponding to each index field, as well as the basic data.

[0005] In one possible implementation, a preset search engine is used to perform the following operations: index the basic data of the first database; and / or use the content in the target data in the second database that corresponds to the index field of the first database as an index condition, and search for candidate data corresponding to the target data according to the index table of the first database to obtain multiple candidate data combinations.

[0006] In a possible implementation, the content in the target data in the second database corresponding to the index field of the first database is used as an index condition, and the candidate data corresponding to the target data is searched according to the index table of the first database to obtain multiple candidate data combinations, including: determining the similarity between the target data in the second database and the basic data in the first database according to the index condition; sorting the basic data from large to small according to the similarity, and determining the multiple candidate data combinations according to the sorting result, wherein the candidate data in the multiple candidate data combinations are the top N basic data ranked by similarity, and N>0 is an integer.

[0007] In a possible implementation, determining whether the target data and the candidate data in the candidate data combination are the same data based on a trained data analysis model includes: the trained data analysis model determines feature parameters of multiple types of features of the target data and the candidate data, and determining whether the target data and the candidate data in the candidate data combination are the same data based on the obtained feature parameters and corresponding preset parameters.

[0008] In a possible embodiment, the feature parameters of the multiple types of features include at least two of the following: a first feature parameter corresponding to at least one of the cosine similarity, jaccard similarity, edit distance, and Hamming distance of the name information of the target data and the candidate data; a second feature parameter corresponding to the absolute value of the difference between the time information of the target data and the candidate data; and a third feature parameter corresponding to the similarity between the target data and the candidate data.

[0009] In one possible implementation, the labeled data in the labeled data set is divided into K parts, and the initial data analysis model is trained using Ks parts of labeled data as training data, and the initial data analysis model is verified using s parts of labeled data as verification data to obtain an intermediate data analysis model, wherein s<Ks<K, K is an integer greater than 1, and s is an integer greater than 0; other labeled data in the K parts of labeled data are rotated in turn as verification data, and corresponding training data are obtained, and the data analysis model is trained and verified multiple times to obtain multiple intermediate data analysis models; the intermediate data analysis model with the smallest verification error among the multiple intermediate data analysis models is used as the trained data analysis model.

[0010] In a possible implementation, the method further includes: when it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, writing identification information of the second database into the first database.

[0011] In a possible implementation, when it is determined that the target data and the candidate data in the plurality of candidate data combinations are the same data, the target data in the plurality of candidate data combinations are not merged with the first database.

[0012] In a possible implementation, when it is determined that the target data in the multiple candidate data combinations is the same data as any candidate data, if the content corresponding to the index field corresponding to any candidate data in the target data is inconsistent with the index data of the index field, the inconsistent content is added to the first database as an alias of the index data of the corresponding index field.

[0013] In a possible implementation, before constructing an index for the first database to obtain an index table, the method further includes: performing normalization processing on the first database and the multiple databases to be merged.

[0014] In a possible implementation, after the operation is performed on all target data in the second database, the second database is deleted.

[0015] In a possible implementation manner, the target data includes document data.

[0016] In one possible implementation, the data analysis model includes a decision tree model, a clustering model, a Bayesian classification model, a support vector machine model, an expectation-maximization EM model, an Adaboost model, a convolutional neural network model, a deep belief network model, a recursive neural network model, a long short-term memory model, a generative adversarial network model, a transfer learning model, and a combination of at least one or more of an attention model.

[0017] According to one aspect of the present disclosure, a data fusion device is provided, the device comprising:

[0018] A construction module, used to construct an index for the first database to obtain an index table, wherein the index table includes a plurality of index fields and index data corresponding to each index field;

[0019] The fusion module is used to select a second database from the multiple databases to be fused in turn, and perform the following operations to obtain a fused first database:

[0020] Using the content in the target data in the second database corresponding to the index field in the first database as the index condition, searching for candidate data corresponding to the target data according to the index table of the first database to obtain a plurality of candidate data combinations, wherein the candidate data combinations include the target data and the corresponding candidate data;

[0021] Determine whether the target data and the candidate data in the candidate data combination are the same data according to the trained data analysis model; when it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, merge the target data in the one or more candidate data combinations into the first database and update the index table.

[0022] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0023] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0024] The data fusion method provided by the embodiment of the present disclosure can construct an index for the first database to obtain an index table, select the second database from multiple databases to be fused in turn, and use the content in the target data in the second database that corresponds to the index field of the first database as the index condition, quickly and accurately search for candidate data of the target data according to the index table of the first database to obtain multiple candidate data combinations, and determine whether the target data and the candidate data in the candidate data combinations are the same data according to the trained data analysis model. The method has the characteristics of high judgment accuracy, simplicity, speed, and low computing cost. When it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, the target data in the one or more candidate data combinations are fused to the first database, and the index table is updated to achieve data deduplication of databases of different data sources, and quickly and efficiently fuse the first database and multiple databases to be fused.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0027] Figure 1 A flow chart of a data fusion method according to an embodiment of the present disclosure is shown.

[0028] Figure 2A flow chart of a data fusion method according to an embodiment of the present disclosure is shown.

[0029] Figure 3 A schematic diagram showing the relationship among target data, candidate data, and candidate data combinations according to an embodiment of the present disclosure is shown.

[0030] Figure 4 A training flowchart of a data analysis model according to an embodiment of the present disclosure is shown.

[0031] Figure 5 A schematic diagram of a data fusion method according to an embodiment of the present disclosure is shown.

[0032] Figure 6 A block diagram of a data fusion device according to an embodiment of the present disclosure is shown.

[0033] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0034] Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0036] In the description of the present disclosure, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure.

[0037] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0038] In the present disclosure, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances.

[0039] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0040] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0041] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0042] Related technologies generally achieve data fusion through simple string matching methods, or simply normalize the strings before matching. The fusion accuracy of this matching method is low. Related technologies also determine whether they are duplicate data by calculating the edit distance or similarity of two data. When this method adds each new data to the database, it must be compared one by one with all the data in the database to determine the similarity, which is very time-consuming and labor-intensive. In addition, the amount of data in the database is often in the tens of millions or even hundreds of millions. For the fusion of databases with large amounts of data, related technologies are even less applicable. It can be seen that related technologies cannot achieve data fusion accurately, efficiently, and at low cost.

[0043] The data fusion method provided by the embodiment of the present disclosure can construct an index for the first database to obtain an index table, select the second database from multiple databases to be fused in turn, and use the content in the target data in the second database that corresponds to the index field of the first database as the index condition, quickly and accurately search for candidate data of the target data according to the index table of the first database to obtain multiple candidate data combinations, and determine whether the target data and the candidate data in the candidate data combinations are the same data according to the trained data analysis model. The method has the characteristics of high judgment accuracy, simplicity, speed, and low computing cost. When it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, the target data in the one or more candidate data combinations are fused to the first database, and the index table is updated to achieve data deduplication of databases of different data sources, and quickly and efficiently fuse the first database and multiple databases to be fused.

[0044] The data fusion method provided by the embodiment of the present disclosure can be applied to various types of data fusion services. For example, the data may include document data, or may also be file data, such as papers, journal articles, patent disclosures or authorization documents, etc. The method can be applied to the establishment of databases such as academic papers, journals, patents, etc., and can also be applied to the establishment of any other types of databases such as news, magazines, articles, novels, etc. Taking the construction of a patent database as an example, the embodiment of the present disclosure can construct a fusion database, and the patents of various databases such as the International Patent Document Database (INPADOC), the Derwent World Patent Index Database (DWPI), the China Patent Abstract Database (CNABS / CPRSABS), the World Patent Abstract Database (SIPOABS), HKABS, TWABS, and the Patent Full-text Database (CN / EP / US / WO / JP) are merged into the fusion database. Patent search can be quickly realized through the fusion database. Since duplicate patents of each database are removed, the query efficiency and accuracy can be greatly improved, which improves the database search effect and user experience. In addition, since duplicate patents are removed accurately and comprehensively, the storage cost of the fusion database is greatly reduced, and the utilization rate of database resources is significantly improved.

[0045] In a possible implementation, the data fusion method proposed in the embodiment of the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method can be executed by a server. Exemplarily, the server can be a cloud server or a local server.

[0046] See also Figure 1 , Figure 1 A flow chart of a data fusion method according to an embodiment of the present disclosure is shown.

[0047] like Figure 1 As shown, the method includes:

[0048] Step S11, constructing an index for the first database to obtain an index table, wherein the index table includes a plurality of index fields and index data corresponding to each index field;

[0049] Step S12, sequentially selecting a second database from the multiple databases to be merged, and performing the following operations to obtain a merged first database:

[0050] Using the content in the target data in the second database corresponding to the index field in the first database as the index condition, searching for candidate data corresponding to the target data according to the index table of the first database to obtain a plurality of candidate data combinations, wherein the candidate data combinations include the target data and the corresponding candidate data;

[0051] Determine whether the target data and the candidate data in the candidate data combination are the same data according to the trained data analysis model; when it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, merge the target data in the one or more candidate data combinations into the first database and update the index table.

[0052] The first database and multiple databases to be merged in the embodiment of the present disclosure can be deduplicated databases, for example, all data in the first database are non-duplicate data. The embodiment of the present disclosure can use relevant technologies to dedupe the data in each database in advance, and the embodiment of the present disclosure does not limit the deduplication method for the data of a single database itself.

[0053] In the embodiment of the present disclosure, the first database may be indexed by a variety of methods to obtain an index table. The following is an exemplary introduction to possible implementation methods.

[0054] See also Figure 2 , Figure 2 A flow chart of a data fusion method according to an embodiment of the present disclosure is shown.

[0055] In a possible implementation, Figure 2 As shown, step S11 constructs an index for the first database, which may include:

[0056] Step S110, creating index fields for each basic data in the first database according to preset fields;

[0057] Step S111: determining index data of each index field according to each basic data in the first database.

[0058] The index data includes the content corresponding to each index field in the basic data, and each corresponding content is associated with the basic data; or the index data includes the content corresponding to each index field in the basic data and the basic data.

[0059] Exemplarily, the basic data is original data in the first database.

[0060] In one example, the preset fields of the embodiment of the present disclosure can be adaptively selected according to the type of database. Exemplarily, based on the characteristics of different types of document data, fields common to such document data can be selected to create index fields. Exemplarily, if the first database and the database to be merged are academic paper types, the preset fields can be selected from common fields that are common to each data source, such as the paper title, paper author, paper publication date, journal or conference name, number of pages in the journal, paper address, etc., to improve adaptability, reduce processing volume, and reduce computing costs. The preset fields of the embodiment of the present disclosure can be selected as needed, and the number of preset field selections can be one (such as selecting only the paper title) or multiple (such as including the paper title, paper author, etc.).

[0061] In one example, the index data may be the specific content corresponding to a preset field. For example, if the preset field is the paper title, then the index data corresponding to the paper title field is the specific paper title of the paper; if the preset field is the paper publication date, then the index data corresponding to the paper publication date is the specific publication date of the paper. By setting the index data to the specific content corresponding to the preset field, the processing efficiency can be improved in subsequent similarity judgments and sameness judgments.

[0062] Exemplarily, as shown in Table 1, taking the index field as the paper title as an example, the index data may be the specific content of the paper title index field of the paper, i.e., the specific name of the paper (e.g., "an image processing technology based on XXX"); or may include the specific content of the paper title index field of the paper and basic data (i.e., the paper "an image processing technology based on XXX" itself). Exemplarily, the target data in the second database may be any paper in the second database.

[0063] Table 1

[0064]

[0065] In one example, when building an index, a key value (ID) of the index may be generated, each index may have different identification information that can be uniquely identified, and the index ID may have a corresponding relationship with the index field and the index data.

[0066] For example, an index ID may correspond to the index field "author name" and index data (i.e., the author's specific name), and the index may also be associated with the data corresponding to the author's specific name (e.g., papers). For example, the index ID is "001", which corresponds to the index field "author name" and the index data "Zhang San", and is associated with one or more papers whose author name is Zhang San or contains Zhang San.

[0067] Of course, the above introduction to the index is exemplary, and the index field can be set as needed, and the index data can also include other things. For example, the index data may include document data, or it may also be file data, such as papers, journal articles, patent disclosures or authorization documents, etc. The documents can be in any data format and may include text or pictures. In one example, the index data in the index table may include the entire data corresponding to the index field in the database. For example, in a paper database, if the index field is the paper name, the index data corresponding to the index field may be the specific name of the paper, and the paper corresponding to the specific name of the paper; exemplarily, if the index field is the author name, when the database includes multiple papers of the author, in the index table, the index data corresponding to the author name index field may include the specific name of the author, and the multiple papers of the author. By setting the index data to include the specific content of the data, the comprehensiveness and accuracy of the judgment can be improved when the similarity judgment is performed later.

[0068] In one example, each database may include multiple pieces of data, such as a paper database may include multiple papers. The embodiment of the present disclosure creates an index for all data in the database to improve the integrity of data fusion.

[0069] The disclosed embodiment creates index fields for each basic data in the first database according to preset fields, determines index data for each index field according to each basic data in the first database, and establishes a mapping relationship between index data and index fields, so as to quickly obtain an index table of the first database, and use the index table to quickly search for required data.

[0070] In one example, the embodiment of the present disclosure can use a preset search engine to implement index construction, such as using ES (Elastic Search), solr and other search engines. For example, the ES search engine can be used to build an ES index for each basic data in the first database. The steps include using ES to create an index (index). The fields of each data in the index generally use common fields that all basic data sources have. For example, when the paper database is merged, the paper name, paper author, journal or conference name, publication year, number of pages in the journal, address of the paper, etc. can be used as index fields. The second step is to index (indexing) data into the index (index), that is, to add the specific content corresponding to the index field in database 1 or the specific content of the basic data to the index table. The present disclosure uses the distributed, highly scalable, and highly real-time characteristics of ES to quickly implement index construction. Of course, the embodiment of the present disclosure can also use other methods to implement index construction.

[0071] In a possible implementation, the embodiment of the present disclosure can also use a preset search engine to search, using the content in the target data in the second database that corresponds to the index field of the first database as the index condition, and searching for candidate data corresponding to the target data according to the index table of the first database, so as to quickly and accurately obtain the candidate data and obtain multiple candidate data combinations.

[0072] The embodiment of the present disclosure does not limit the specific implementation method of using the content in the target data in the second database corresponding to the index field of the first database as the index condition, searching for candidate data corresponding to the target data according to the index table of the first database, and obtaining multiple candidate data combinations. For example, in a possible implementation, using the content in the target data in the second database corresponding to the index field of the first database as the index condition, searching for candidate data corresponding to the target data according to the index table of the first database, and obtaining multiple candidate data combinations may include:

[0073] Determining, according to the index condition, the similarity between the target data in the second database and the basic data in the first database;

[0074] The basic data are sorted from large to small according to similarity, and the multiple candidate data combinations are determined according to the sorting result, wherein the candidate data in the multiple candidate data combinations are the top N basic data in similarity sorting, N>0 and is an integer.

[0075] In one example, the disclosed embodiment can determine the index condition according to the index field. For example, if the index field is the name of the paper, the names of the papers in the second database are used as the index condition to search in the index table of the first database. For example, the index condition can be used to search in the index data of the index table. The index data can include the specific content of the index field (such as the name of the paper) or the specific content of the basic data (such as the content of the paper). If the index data is the specific content of the index field, searching in the index data can improve the indexing speed; if the index data is the specific content of the basic data, searching in the index data can improve the comprehensiveness of the search. In order to improve the accuracy of the index, the index field of the disclosed embodiment can be in a grouped form. For example, a group of index fields can include multiple conditions or any combination such as the paper name, author name, and publication date. Through the joint action of multiple index fields, the accuracy of the match can be improved. For example, when searching for candidate data, the values ​​of the paper name, paper author, journal or conference name, and publication date of a piece of data (such as a paper) in the second database can be used as search conditions to search in the index table to obtain candidate data.

[0076] In one example, when searching in the index table with the above-mentioned index conditions, the embodiment of the present disclosure can return candidate data based on the similarity between the target data and the content corresponding to the index field and the data matched in the index data in the index table. The embodiment of the present disclosure does not limit the specific method for determining the similarity, and those skilled in the art can implement it according to relevant technologies. For example, for index fields of different data types, corresponding similarity evaluation methods can be used. For example, for character data such as the name of the author of the paper and the name of the paper, the similarity can be determined by determining the distance between the two. For numbers such as dates, the difference can be used to determine the similarity according to the size of the difference, etc. Of course, the embodiment of the present disclosure is not limited to this. In other embodiments, other methods can also be used to determine the similarity, such as using the similarity calculation method carried by the search engine itself. For example, by searching in the first database with the paper title of the paper in the second database, the similarity between the papers in the second database and the papers in the first database can be determined based on the distance between the paper title of the paper in the second database and the paper title of each paper in the index table of the first database (for example, the string distance). The similarity can also be determined using the BM25 similarity algorithm, TF / IDF algorithm, or boolean similarity algorithm. Specifically, a suitable similarity algorithm can be used according to the specific data type or domain, which is not limited in this regard in the embodiments of the present disclosure.

[0077] In one example, when the similarity between the target data in the second database and the basic data in the first database is determined, for example, by the similarity of the index field corresponding to the index data, the candidate data can be determined according to the degree of similarity, that is, the size or ranking of the similarity, for example, the top N basic data with the highest similarity ranking can be used as candidate data, or the basic data with a similarity greater than a preset similarity can be used as candidate data.

[0078] In one example, as shown in Table 2, Table 2 shows a schematic diagram of each candidate data combination. Exemplarily, taking paper A in the second database as an example, if the paper name of paper A is used as a search condition and a search is performed in the index table of the first database, the similarity between each paper name in the index table and the paper name of paper A can be determined, and a predetermined number of paper names with the highest similarity can be determined in the index table according to the similarity sorting, and according to the association between the paper name and the paper, paper B, paper C, and paper D (the N basic data with the highest similarity sorting) can be found, and the candidate data combination can include candidate data combination (paper A, paper B), candidate data combination (paper A, paper C), and candidate data combination (paper A, paper D).

[0079] Table 2

[0080]

[0081] Below Figure 3 Take the example to illustrate the relationship among target data, candidate data, and candidate data combination.

[0082] See also Figure 3 , Figure 3 A schematic diagram showing the relationship among target data, candidate data, and candidate data combinations according to an embodiment of the present disclosure is shown.

[0083] In one example, if Figure 3 As shown, the target data may come from any database (second database) among databases 2 to n in the databases to be merged.

[0084] In one example, if Figure 3 As shown, the index table can be obtained by constructing an index for database 1 (the first database).

[0085] In one example, if Figure 3As shown, the candidate data may be obtained from database 1 (first database) or from an index table. For example, if the index data in the index table includes the content corresponding to each index field in the basic data of database 1, and the corresponding content is associated with the basic data, the candidate data corresponding to the target data may be searched in database 1 according to the index table of the first database. For example, if the index data includes the content corresponding to each index field in the basic data of database 1 and the basic data, the candidate data corresponding to the target data may be directly searched in the index table according to the index table of the first database.

[0086] In a possible implementation manner, determining whether the target data and the candidate data in the candidate data combination are the same data according to the trained data analysis model may include:

[0087] The trained data analysis model determines feature parameters of multiple types of features of the target data and the candidate data, and determines whether the target data and the candidate data in the candidate data combination are the same data based on the obtained feature parameters and corresponding preset parameters.

[0088] In one example, the embodiment of the present disclosure may input the target data and the candidate data into a trained data analysis model, and determine whether the target data and the candidate data in the candidate data combination are the same data according to the output result of the trained data analysis model. Exemplarily, the data analysis model of the embodiment of the present disclosure may be a machine learning model, for example, it may include a decision tree model, a clustering model, a Bayesian classification model, a support vector machine model, an expectation-maximization EM model, an Adaboost model, a convolutional neural network model, a deep belief network model, a recursive neural network model, a long short-term memory model, a generative adversarial network model, a transfer learning model, an attention model, etc., a combination of at least one or more.

[0089] Exemplarily, the disclosed embodiment adopts a decision tree model to implement a data analysis model, and uses the decision tree model to determine whether two data in each candidate data combination are the same data. The internal nodes of the decision tree are the features or attributes selected by the model, and the output nodes include "yes (YES)" and "no (NO)". When the probability of the decision tree model judging as YES is greater than a certain threshold (for example, the threshold is 0.8), the two data are considered to be the same data. If the probability is less than this threshold, it is considered that the two are not the same data but only similar data.

[0090] Exemplarily, the disclosed embodiments can extract attribute features of two data in the candidate data combination as the basis for judgment, and quickly and accurately determine whether they are the same data. Taking paper data fusion as an example, the paper name, paper author, journal or conference name, publication date, etc. can be selected as attributes for duplicate data judgment.

[0091] In a possible implementation manner, the feature parameters of the multiple types of features include at least two of the following:

[0092] a first feature parameter corresponding to at least one of cosine similarity, jaccard similarity, edit distance, and Hamming distance of name information of each of the target data and the candidate data;

[0093] A second characteristic parameter corresponding to an absolute value of a difference between time information of the target data and the candidate data;

[0094] A third feature parameter corresponding to the similarity between the target data and the candidate data.

[0095] In one example, the name information is generally a string attribute. For a string attribute, the edit distance of the attribute value of two candidate data combinations can be calculated as a feature, and the feature is an internal node of the decision tree. Of course, the edit distance can also be other types of distances other than cosine similarity, jaccard similarity, edit distance, and Hamming distance, which are not limited in the embodiments of the present disclosure. For example, the name information is the name of the paper, the name of the author, the name of the journal, etc.

[0096] In one example, time information is generally of integer type. For attribute objects of integer type, 0 and 1 can be used to represent whether the two are equal or the absolute value of the difference between the two as a feature. Time information is, for example, the publication time of data (such as the publication time of a paper, the publication time of a patent application document, or the authorization time).

[0097] In one example, the similarity between the target data and the candidate data can be obtained according to the aforementioned method. For example, the similarity between the target data and the candidate data can be judged based on the distance between the index data corresponding to one or more index fields of the target data and the candidate data. Alternatively, the similarity between the target data and the candidate data can be judged in other ways, for example, the similarity can be obtained using an ES search engine.

[0098] When the embodiment of the present disclosure utilizes at least two of the first feature parameter, the second feature parameter, and the third feature parameter corresponding to at least one of cosine similarity, jaccard similarity, edit distance, and Hamming distance to determine whether the target data and the candidate data in the candidate data combination are the same data, the preset parameters corresponding to each feature parameter can be determined according to actual conditions or needs. The embodiment of the present disclosure does not limit this. For example, if the first feature parameter corresponding to at least one of cosine similarity, jaccard similarity, edit distance, and Hamming distance is smaller than the corresponding preset distance (preset parameter), and the second feature parameter, i.e., the absolute value of the difference, is smaller than the preset difference (preset parameter), it can be considered that the target data and the candidate data are the same data.

[0099] It should be noted that in the embodiments of the present disclosure, the method for selecting feature parameters is not unique and can be adjusted accordingly according to different data. The embodiments of the present disclosure use the first feature parameter, the second feature parameter, and the third feature parameter corresponding to at least one of cosine similarity, jaccard similarity, edit distance, and Hamming distance to perform judgments in multiple dimensions. When at least two of the first feature parameter, the second feature parameter, and the third feature parameter corresponding to at least one of cosine similarity, jaccard similarity, edit distance, and Hamming distance meet corresponding conditions, it is determined that the target data and the candidate data in the candidate data combination are the same data, otherwise they are different data, which can improve the accuracy of the judgment and avoid erroneous judgments caused by a single judgment condition.

[0100] In a possible implementation, the embodiment of the present disclosure may pre-train the data analysis model to obtain a trained data analysis model. The training of the data analysis model is exemplarily introduced below.

[0101] See also Figure 4 , Figure 4 A training flowchart of a data analysis model according to an embodiment of the present disclosure is shown.

[0102] In a possible implementation, Figure 4 As shown, the method may also include:

[0103] Step S21, dividing the labeled data in the labeled data set into K parts, using Ks parts of labeled data as training data to train the initial data analysis model, and using s parts of labeled data as verification data to verify the initial data analysis model, to obtain an intermediate data analysis model, wherein s<Ks<K, K is an integer greater than 1, and s is an integer greater than 0;

[0104] Step S22, sequentially rotate other labeled data in the K pieces of labeled data as verification data, and obtain corresponding training data, perform multiple training and verification on the data analysis model, and obtain multiple intermediate data analysis models;

[0105] Step S23, taking the intermediate data analysis model with the smallest verification error among the multiple intermediate data analysis models as the trained data analysis model.

[0106] The verification error may represent the error of the intermediate data analysis model with respect to the verification data.

[0107] Exemplarily, the labeled data is labeled data used for model training and verification.

[0108] In one example, the embodiment of the present disclosure can set the values ​​of K and s as needed. For example, in order to reduce the computational complexity, s can be set to 1. Assuming that K is set to 10, 1 portion of the labeled data can be used as verification data, and the other 9 portions of the labeled data can be used as training data. Exemplarily, "multiple training and verification" can be determined as needed. The number of training and verification can be 10 times. In the first training and verification, the first portion of the labeled data can be used as verification data, and the remaining nine portions of the labeled data can be used as training data; in the second training and verification, the second portion of the labeled data can be used as verification data, and the remaining nine portions of the labeled data can be used as training data. The training and verification rotation process continues until the 10th training uses the 10th portion of the labeled data as verification data and the remaining nine portions of the labeled data as training data. Of course, in the embodiment of the present disclosure, s and K can also be set to other values. For example, s can be set to 2 or other values. If s is set to 2, then in each rotation, any two portions of the 10 portions of labeled data can be randomly combined, such as the first portion combined with the second portion, the second portion combined with the eighth portion, etc. Through such a design, the number of training and verification can be increased, and the accuracy of the model can be improved.

[0109] The disclosed embodiment randomly divides the training set into K parts (for example, 10 parts), extracts ks parts as training data each time, and uses the remaining s parts as verification data to train the model in sequence, and rotates the data training set and the verification set in sequence for multiple times, and finally generates multiple models. The disclosed embodiment selects the model with the smallest error on the verification data among the multiple models as the trained model. In this way, the disclosed embodiment can effectively improve the generalization ability of the model, and the effect is more obvious when the training set is small (setting a small training set can increase the training speed, and does not require a high amount of training data, which is easy to implement). After experiments, the accuracy of the data analysis model trained by the above method is higher than 97.5%.

[0110] In a possible implementation, the method may further include:

[0111] When it is determined that the target data and the candidate data in one or more candidate data combinations of the plurality of candidate data combinations are different data, the identification information of the second database is written into the first database. The identification information of the second database can be associated with the target data merged from the second database to the first database to mark the source of the target data.

[0112] In an example, the identification information (ID) of the second database may be a key value of the second database, or other information that can uniquely identify the second database.

[0113] In the embodiment of the present disclosure, when it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, the target data is added to the first database to achieve basic database fusion of the second database and the first database, and by writing the identification information of the second database into the first database, the diversity of the data can be enriched, the search of the database can be accelerated, and the traceability of the target data can be achieved.

[0114] In one example, when it is determined that the target data and the candidate data in one or more candidate data combinations of the plurality of candidate data combinations are different data, updating the index table may include: using the content corresponding to the index field in the target data in the one or more candidate data combinations as the index data of the index field of the target data, and also adding the target data to the index table, corresponding to the index field of the target data, and establishing a mapping. For example, in the case where the title of the paper is used as the search condition above, the title of the paper of the target data can be used as the index field, and the target data or the corresponding content of the target data (i.e., the specific content of the paper or the specific name of the paper) can be used as the corresponding index data and added to the index table of the first database.

[0115] The following is an exemplary description of the case where the target data is the same as the candidate data.

[0116] In a possible implementation, Figure 2 As shown, the method may also include:

[0117] Step S13: when it is determined that the target data and the candidate data in the plurality of candidate data combinations are the same data, the target data in the plurality of candidate data combinations are not merged with the first database.

[0118] Since the target data and the candidate data in the candidate data combination are judged to be the same, that is, the target data already exists in the first database, therefore, there is no need to merge the existing data, the candidate data can be deleted (such as deleting the candidate data combination consisting of the target data and the candidate data), and the similarities and differences between the target data and the candidate data in the next candidate data combination are judged. When the target data and all its candidate data are the same data (including all candidate data combinations of the target data), the target data can be directly deleted (including deleting all candidate data combinations including the target data) to achieve the purpose of deduplication, and other target data are judged. If all target data and candidate data in the second database are the same data, the entire second database can be deleted to improve the utilization of the database. Of course, when judging whether the target data and candidate data in the second database are the same, it can be synchronized to improve processing efficiency, for example, multiple candidate data combinations corresponding to one target data are judged synchronously, and / or multiple candidate data combinations corresponding to multiple target data are judged synchronously.

[0119] In a possible implementation, Figure 2 As shown, the method may also include:

[0120] Step S14, when it is determined that the target data in the multiple candidate data combinations is the same data as any candidate data, if the content corresponding to the index field corresponding to any candidate data in the target data is inconsistent with the index data of the index field, the inconsistent content is added to the first database as an alias of the index data of the corresponding index field.

[0121] In an example, if after m candidate data combinations pass through the data analysis model, the output of one candidate data combination is YES, that is, it is considered that the target data has found duplicate data. For this duplicate data, the disclosed embodiment can use string matching and other means to determine whether the writing of the corresponding content of the index field of the duplicate data is consistent for each field of the data. If the writing of the corresponding content of each field is consistent, the data is directly deleted. If not, the inconsistent content is added to the first database as an alias of the index data of the corresponding index field. This can enrich the diversity of the data and effectively increase the accuracy of database searches. For example, the conference name Annual Meeting of the Association for Computational Linguistics in the paper data often appears in the abbreviated form ACL. Assuming that the conference name in the target data is "ACL" and that in the candidate data is "Annual Meeting of the Association for Computational Linguistics", if the model determines that the target data is the same as the candidate data, then, in the embodiment of the present disclosure, ACL can be added as an alias of Annual Meeting of the Association for Computational Linguistics to the index table of the first database. In addition, similar operations can be performed on other index fields corresponding to the candidate data (such as author name, etc.), which greatly enriches the diversity of data and the accuracy of search.

[0122] In a possible implementation, after performing the operation on all target data in the second database, the second database is deleted. For example, after completing the same judgment of all candidate data combinations corresponding to all target data in the second database and completing the fusion of the corresponding data, the second database can be deleted to release database storage resources and improve utilization.

[0123] In a possible implementation, Figure 2 As shown, before building an index for the first database to obtain an index table, the method may further include:

[0124] Step S10: normalize the first database and the multiple databases to be merged.

[0125] In one example, the formats and structures of data in different data sources are different, and it is impossible to remove duplicate data through a simple matching method. For example, the author name is a very important one-dimensional information in the paper database table, and the representation of the name is often different in different data sources. Taking the Chinese author name "Zhang San, Li Si" as an example, it may appear in any of the forms of "Zhang San, Li Si", "Zhang San, etc.", "Zhang, etc.". In addition, the uppercase and lowercase representations, the difference between the right angle and the full angle of the string, the abbreviation, the default, etc. also lead to the inability to eliminate duplicate data through a simple string matching method. In addition, the amount of data in the database is often at the level of tens of millions or even hundreds of millions, and how to quickly achieve deduplication is also one of the difficulties that need to be overcome. The disclosed embodiment can standardize the data format in the first database and the multiple databases to be fused by normalizing the first database and the multiple databases to be fused, thereby accelerating the processing speed and improving the accuracy of the processing, so that the disclosed embodiment has high accuracy and efficiency at the level of tens of millions or even hundreds of millions.

[0126] In one example, the normalization operation includes three steps: the first step is to unify the uppercase and lowercase characters of the string, unify the full-width and half-width characters, and process some special symbols, for example, remove some redundant symbols, such as the colon in the name, normalize some foreign characters, such as converting the Greek letter α to alpha, etc.; the second step is to normalize date data. There are many forms of date expression. Taking "2018" as an example, its expression forms include "2018", "2018", "two thousand and eighteen", "20180000", etc. The embodiment of the present disclosure can normalize various forms of date representation into one; the third step is to normalize numerical data, including the unification of units and data types, such as converting them into integers, etc. Of course, the above introduction to the normalization operation is exemplary, and its steps can be adjusted as needed, the specific implementation method of normalization can be implemented as needed, and the specific content of normalization can be increased, reduced, and adaptively adjusted, which is not limited by the embodiment of the present disclosure.

[0127] The data fusion method is exemplarily introduced below with reference to specific examples.

[0128] See also Figure 5 , Figure 5 A schematic diagram of a data fusion method according to an embodiment of the present disclosure is shown.

[0129] In one example, if Figure 5 As shown, the embodiment of the present disclosure can use any one database (such as database 1) among database 1, data 2, ..., data n as the first database, that is, merge target data in other databases that are different from database 1 into database 1 to obtain a merged database.

[0130] In one example, if Figure 5 As shown, the embodiment of the present disclosure can set up a fusion database and add database 1 to the fusion database.

[0131] In one example, if Figure 5 As shown, the embodiment of the present disclosure can normalize database 1 and other databases, and construct an index for the normalized database 1 to obtain an index table, and extract the second database from database 2-database n without duplication in sequence or in a preset manner. For each target data in the second database, the content in the target data corresponding to the index field of the first database is used as the index condition, and the candidate data described by the target data is searched according to the index table of the first database to obtain multiple candidate data combinations.

[0132] In one example, if Figure 5 As shown, the embodiment of the present disclosure can determine whether the target data and the candidate data in the candidate data combination are the same data based on the trained data analysis model. When it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, the target data in the one or more candidate data combinations are merged into the first database (fused database), and the index table is updated. When all the target data in the multiple candidate data combinations are duplicate data, the database fusion is not performed, that is, the data is discarded.

[0133] In one example, if Figure 5 As shown, the above operation is repeated multiple times until all databases in database 2-database n have been compared and judged to obtain the final fused database.

[0134] The multiple databases used by the embodiments of the present disclosure may be databases from multiple different data sources. Through the data fusion method, the fused data after deduplication can be obtained. The embodiments of the present disclosure add the basic data in database 1 to the fused data, and normalize the basic data in database 1 before constructing the index. After the remaining target data to be fused are normalized, the search algorithm is used to search for the m data with the highest similarity one by one, and the data analysis model is used to judge the duplicate data. In the process of implementing the embodiments of the present disclosure, when the data level of database 1 is more than 10 million, the average search time for each data is about 600ms. After obtaining the candidate set, the data analysis model (such as a decision tree model) is used to determine whether there is duplicate data. The time consumption of this step is within 50ms. The accuracy of the entire data deduplication of the embodiments of the present disclosure is higher than 97.5%. It can be seen that the present invention achieves data deduplication of tens of millions with a very high accuracy while ensuring the speed.

[0135] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0136] See also Figure 6 , Figure 6 A block diagram of a data fusion device according to an embodiment of the present disclosure is shown.

[0137] like Figure 6 As shown, the device comprises:

[0138] A construction module 10 is used to construct an index for the first database to obtain an index table, wherein the index table includes a plurality of index fields and index data corresponding to each index field;

[0139] The fusion module 20 is used to select a second database from the multiple databases to be fused in sequence, and perform the following operations to obtain a fused first database:

[0140] Using the content in the target data in the second database corresponding to the index field in the first database as the index condition, searching for candidate data corresponding to the target data according to the index table of the first database to obtain a plurality of candidate data combinations, wherein the candidate data combinations include the target data and the corresponding candidate data;

[0141] Determine whether the target data and the candidate data in the candidate data combination are the same data according to the trained data analysis model; when it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, merge the target data in the one or more candidate data combinations into the first database and update the index table.

[0142] The data fusion device provided by the embodiment of the present disclosure can construct an index for the first database to obtain an index table, select the second database from multiple databases to be fused in turn, and use the content in the target data in the second database that corresponds to the index field of the first database as the index condition, quickly and accurately search for candidate data of the target data according to the index table of the first database to obtain multiple candidate data combinations, and determine whether the target data and the candidate data in the candidate data combinations are the same data according to the trained data analysis model. The device has the characteristics of high judgment accuracy, simplicity, speed, and low computing power cost. When it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, the target data in the one or more candidate data combinations are fused to the first database, and the index table is updated to achieve data deduplication of databases of different data sources, and quickly and efficiently fuse the first database and multiple databases to be fused.

[0143] In a possible implementation, constructing an index for the first database includes:

[0144] Creating index fields for each basic data in the first database according to preset fields;

[0145] Determine the index data of each index field according to each basic data in the first database; wherein the index data includes the content corresponding to each index field in the basic data, and the corresponding content is associated with the basic data; or the index data includes the content corresponding to each index field in the basic data, and the basic data.

[0146] In a possible implementation, the following operations are performed using a preset search engine:

[0147] Building an index for the basic data of the first database; and / or

[0148] The content in the target data in the second database corresponding to the index field of the first database is used as an index condition, and candidate data corresponding to the target data is searched according to the index table of the first database to obtain multiple candidate data combinations.

[0149] In a possible implementation, the content in the target data in the second database corresponding to the index field of the first database is used as an index condition, and candidate data corresponding to the target data is searched according to the index table of the first database to obtain multiple candidate data combinations, including:

[0150] Determining, according to the index condition, the similarity between the target data in the second database and the basic data in the first database;

[0151] The basic data are sorted from large to small according to similarity, and the multiple candidate data combinations are determined according to the sorting result, wherein the candidate data in the multiple candidate data combinations are the top N data in similarity sorting, N>0 and is an integer.

[0152] In a possible implementation manner, determining whether the target data and the candidate data in the candidate data combination are the same data according to the trained data analysis model includes:

[0153] The trained data analysis model determines feature parameters of multiple types of features of the target data and the candidate data, and determines whether the target data and the candidate data in the candidate data combination are the same data based on the obtained feature parameters and corresponding preset parameters.

[0154] In a possible implementation manner, the feature parameters of the multiple types of features include at least two of the following:

[0155] a first feature parameter corresponding to at least one of cosine similarity, jaccard similarity, edit distance, and Hamming distance of name information of each of the target data and the candidate data;

[0156] A second characteristic parameter corresponding to an absolute value of a difference between time information of the target data and the candidate data;

[0157] A third feature parameter corresponding to the similarity between the target data and the candidate data.

[0158] In a possible implementation, the labeled data in the labeled data set is divided into K parts, and the initial data analysis model is trained using Ks parts of the labeled data as training data, and the initial data analysis model is verified using s parts of the labeled data as verification data to obtain an intermediate data analysis model, wherein s<Ks<K, K is an integer greater than 1, and s is an integer greater than 0;

[0159] The other labeled data in the K labeled data are rotated in turn as verification data, and the corresponding training data is obtained, and the data analysis model is trained and verified multiple times to obtain multiple intermediate data analysis models;

[0160] The intermediate data analysis model with the smallest verification error among the multiple intermediate data analysis models is used as the trained data analysis model.

[0161] In a possible implementation, when it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, the identification information of the second database is written into the first database.

[0162] In a possible implementation, when it is determined that the target data and the candidate data in the plurality of candidate data combinations are the same data, the target data in the plurality of candidate data combinations are not merged with the first database.

[0163] In a possible implementation, when it is determined that the target data in the multiple candidate data combinations is the same data as any candidate data, if the content corresponding to the index field corresponding to any candidate data in the target data is inconsistent with the index data of the index field, the inconsistent content is added to the first database as an alias of the index data of the corresponding index field.

[0164] In a possible implementation manner, before constructing an index for the first database to obtain an index table, the device further includes:

[0165] A normalization module is used to perform normalization processing on the first database and the multiple databases to be merged.

[0166] In a possible implementation, after the operation is performed on all target data in the second database, the second database is deleted.

[0167] In one possible implementation, the data analysis model includes a decision tree model, a clustering model, a Bayesian classification model, a support vector machine model, an expectation-maximization EM model, an Adaboost model, a convolutional neural network model, a deep belief network model, a recursive neural network model, a long short-term memory model, a generative adversarial network model, a transfer learning model, and a combination of at least one or more of an attention model.

[0168] In a possible implementation manner, the target data includes document data.

[0169] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0170] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0171] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0172] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0173] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0174] See also Figure 7 , Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0175] For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0176] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0177] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0178] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0179] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0180] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0181] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0182] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0183] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0184] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0185] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0186] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0187] See also Figure 8 , Figure 8 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0188] For example, the electronic device 1900 may be provided as a server. Figure 8 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0189] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0190] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0191] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0192] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0193] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0194] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0195] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0196] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0197] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0198] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0199] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0200] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A data fusion method, characterized in that: The method comprises: Building an index for the first database to obtain an index table, wherein the index table includes a plurality of index fields and index data corresponding to each index field; Select the second database from the multiple databases to be merged in turn, and perform the following operations to obtain the merged first database: Using the content in the target data in the second database corresponding to the index field in the first database as the index condition, searching for candidate data corresponding to the target data according to the index table of the first database to obtain a plurality of candidate data combinations, wherein the candidate data combinations include the target data and the corresponding candidate data; Determine whether the target data and the candidate data in the candidate data combination are the same data according to the trained data analysis model; if it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, merge the target data in the one or more candidate data combinations into the first database, and update the index table; The determining, based on the trained data analysis model, whether the target data and the candidate data in the candidate data combination are the same data includes: The trained data analysis model determines the characteristic parameters of the target data and the multiple types of characteristics of the candidate data, and determines whether the target data and the candidate data in the candidate data combination are the same data according to the obtained characteristic parameters and the corresponding preset parameters. The method further comprises: The labeled data in the labeled data set is divided into K parts, and the initial data analysis model is trained using Ks parts of the labeled data as training data, and the initial data analysis model is verified using s parts of the labeled data as verification data to obtain an intermediate data analysis model, wherein s<Ks<K, K is an integer greater than 1, and s is an integer greater than 0; The other labeled data in the K labeled data are rotated in turn as verification data, and the corresponding training data is obtained, and the data analysis model is trained and verified multiple times to obtain multiple intermediate data analysis models; The intermediate data analysis model with the smallest verification error among the multiple intermediate data analysis models is used as the trained data analysis model.

2. The method according to claim 1, characterized in that The step of constructing an index for the first database includes: Creating index fields for each basic data in the first database according to preset fields; Determine index data of each index field according to each basic data in the first database; The index data includes content corresponding to each index field in the basic data, and the corresponding content is associated with the basic data; or The index data includes the content corresponding to each index field in the basic data and the basic data.

3. The method according to claim 1 or 2, characterized in that: Use the default search engine to: Building an index for the basic data of the first database; and / or The content in the target data in the second database corresponding to the index field of the first database is used as an index condition, and candidate data corresponding to the target data is searched according to the index table of the first database to obtain multiple candidate data combinations.

4. The method according to claim 1, characterized in that: Using the content in the target data in the second database corresponding to the index field of the first database as the index condition, searching for candidate data corresponding to the target data according to the index table of the first database, and obtaining multiple candidate data combinations, including: Determining, according to the index condition, the similarity between the target data in the second database and the basic data in the first database; The basic data are sorted from large to small according to similarity, and the multiple candidate data combinations are determined according to the sorting result, wherein the candidate data in the multiple candidate data combinations are the top N basic data in similarity sorting, N>0 and is an integer.

5. The method according to claim 1, characterized in that The characteristic parameters of the multiple types of characteristics include at least two of the following: a first feature parameter corresponding to at least one of cosine similarity, jaccard similarity, edit distance, and Hamming distance of name information of each of the target data and the candidate data; A second characteristic parameter corresponding to an absolute value of a difference between time information of the target data and the candidate data; A third feature parameter corresponding to the similarity between the target data and the candidate data.

6. The method according to claim 1, characterized in that The method further comprises: When it is determined that the target data and the candidate data in one or more candidate data combinations of the plurality of candidate data combinations are different data, the identification information of the second database is written into the first database.

7. The method according to claim 1, characterized in that The method further comprises: When it is determined that the target data and the candidate data in the plurality of candidate data combinations are the same data, the target data in the plurality of candidate data combinations are not merged with the first database.

8. The method according to claim 7, characterized in that The method further comprises: When it is determined that the target data in the multiple candidate data combinations is the same data as any candidate data, if the content corresponding to the index field corresponding to any candidate data in the target data is inconsistent with the index data of the index field, the inconsistent content is added to the first database as an alias of the index data of the corresponding index field.

9. The method according to claim 1, characterized in that: Before constructing an index for the first database to obtain an index table, the method further includes: The first database and the multiple databases to be merged are normalized.

10. The method according to claim 1, characterized in that The method further includes: after performing the operation on all target data in the second database, deleting the second database.

11. The method according to claim 1, characterized in that: The data analysis model includes a decision tree model, a clustering model, a Bayesian classification model, a support vector machine model, an expectation-maximization EM model, an Adaboost model, a convolutional neural network model, a deep belief network model, a recursive neural network model, a long short-term memory model, a generative adversarial network model, a transfer learning model, and a combination of at least one or more of the following models.

12. The method according to claim 1, characterized in that The target data includes document data.

13. A data fusion device, characterized in that: The device comprises: A construction module, used to construct an index for the first database to obtain an index table, wherein the index table includes a plurality of index fields and index data corresponding to each index field; The fusion module is used to select a second database from the multiple databases to be fused in turn, and perform the following operations to obtain a fused first database: Using the content in the target data in the second database corresponding to the index field in the first database as the index condition, searching for candidate data corresponding to the target data according to the index table of the first database to obtain a plurality of candidate data combinations, wherein the candidate data combinations include the target data and the corresponding candidate data; Determine whether the target data and the candidate data in the candidate data combination are the same data according to the trained data analysis model; if it is determined that the target data and the candidate data in one or more candidate data combinations of the multiple candidate data combinations are different data, merge the target data in the one or more candidate data combinations into the first database, and update the index table; The determining, based on the trained data analysis model, whether the target data and the candidate data in the candidate data combination are the same data includes: The trained data analysis model determines the characteristic parameters of the target data and the multiple types of characteristics of the candidate data, and determines whether the target data and the candidate data in the candidate data combination are the same data according to the obtained characteristic parameters and the corresponding preset parameters. Also includes: The labeled data in the labeled data set is divided into K parts, and the initial data analysis model is trained using Ks parts of the labeled data as training data, and the initial data analysis model is verified using s parts of the labeled data as verification data to obtain an intermediate data analysis model, wherein s<Ks<K, K is an integer greater than 1, and s is an integer greater than 0; The other labeled data in the K labeled data are rotated in turn as verification data, and the corresponding training data is obtained, and the data analysis model is trained and verified multiple times to obtain multiple intermediate data analysis models; The intermediate data analysis model with the smallest verification error among the multiple intermediate data analysis models is used as the trained data analysis model.

14. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images