Data discovery system based on composite matching

Abstract

The invention discloses a data discovery system based on composite matching, and the system specifically comprises a storage layer which is mainly responsible for storing the data of various data sets and storing the operation result of an algorithm, and a model layer which is mainly a specific implementation module of various composite matching schemes. The application layer exposes various kinds of information to the user in a webpage front-end mode, and provides selection of different methods, selection of algorithm parameters, selection of data sets and query of calculation results for the user. The matcher combination scheme (IWIU) and the method based on machine learning are designed to improve the accuracy and efficiency of data discovery, and compared with a traditional static weight distribution method, the scheme can more accurately reflect the performance of each matcher in different stages, automatically optimize weight distribution, and improve the accuracy and efficiency of data discovery. The matching accuracy and the efficiency of the system are improved, and excessive participation of inaccurate matchers is avoided, so that the matching accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data matching, and in particular to a data discovery system based on composite matching. Background Art

[0002] Modern organizations face the complexity of data discovery tasks when conducting data analysis. This has prompted researchers to conduct in-depth research on how to efficiently perform data discovery and related issues. Data discovery tasks usually include multiple subtasks, such as searching for connectable tables, expanding target tables with additional attributes or entries, and querying and retrieving similar tables. To solve these subtasks, researchers have designed various pattern matching techniques and formed a variety of different pattern matching schemes. Compared with a single pattern matching scheme, effectively combining multiple pattern matching methods can improve the accuracy of matching in various situations and make up for the shortcomings of a single matcher.

[0003] Pattern matching combinatorial schemes are usually divided into two types: hybrid and compound. Hybrid schemes perform well in specific situations, while compound schemes are more flexible. Both schemes rely on multiple matchers, so effectively utilizing different matchers is an important challenge. Hybrid schemes have flaws because the various matcher methods are usually hard-wired.

[0004] The composite combination scheme requires the selection of matchers with different aspects of pattern information to accurately evaluate the effectiveness of different matchers, so as to ultimately obtain a more effective comprehensive similarity score. The composite combination scheme involves too many inaccurate matchers, and the matching accuracy and efficiency can be improved. Summary of the invention

[0005] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0006] In view of the above problems existing in the existing data discovery system based on composite matching, the present invention is proposed.

[0007] Therefore, the purpose of the present invention is to provide a data discovery system based on composite matching, which is to solve the problem of "excessive participation of inaccurate matchers in composite combination solutions, and the accuracy and efficiency of matching can be improved".

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] In a first aspect, the present invention provides a composite matching data discovery system, specifically comprising:

[0010] Storage layer: The storage layer is mainly responsible for storing data of various data sets and saving the operation results of the algorithm.

[0011] Model layer: The model layer is mainly the specific implementation module of various composite matching solutions.

[0012] Application layer: The application layer exposes various types of information to users in the form of a web front end, providing users with the choice of different methods, algorithm parameters, data sets, and the query of calculation results.

[0013] As a preferred solution of the data discovery system based on composite matching described in the present invention, the centralized data of the storage layer is usually imported into the Postgre SQL management system in a table type mode, and the clustering and related evaluation scores generated after the algorithm is executed are also stored in the system.

[0014] As a preferred solution of the data discovery system based on composite matching of the present invention, the clustering and related evaluation scores generated after the algorithm is executed are also stored in the Postgre SQL management system.

[0015] As a preferred solution of the data discovery system based on composite matching described in the present invention, the model layer receives data from the storage layer, performs similarity judgment on different column pairs, and finally outputs clusters or related column pairs and numerical values ​​of various evaluation criteria of the columns.

[0016] As a preferred solution of the data discovery system based on composite matching described in the present invention, the output result of the model layer will be rendered by Vue to be passed to the application layer.

[0017] As a preferred solution of the data discovery system based on composite matching described in the present invention, the interaction between the application layer and the model layer is mainly implemented through a RESTful API, mainly utilizing GET and POST requests therein.

[0018] In the second aspect, the present invention provides a data discovery method based on compound matching to further solve the problems existing in compound matching, and the specific steps include:

[0019] S1. First, after the system is started, the weight is initialized. The weight wi of each matcher is initialized to 1 to prepare the initial weight for the next iterative matching.

[0020] S2. Then the system selects a matcher. After starting the matching round, it selects a matcher randomly or according to a certain strategy, and selects a suitable matcher for the current round.

[0021] S3. After the system selects a matcher, it executes the matching task and obtains the matching result g(p), and then calculates the matching result of the current matcher.

[0022] S4. After the system matching result g(p) is known, the weight of each matcher is updated, and the truncation threshold t is updated according to the situation. The weight and truncation threshold of the matcher are adjusted according to the current matching effect to optimize the subsequent matching effect.

[0023] S5. After the system completes one round of matching, it repeats steps S2, S3, and S4 until m rounds of matching are completed. Through multiple rounds of iterations, the matching effect is gradually optimized until the requirements are met.

[0024] A matcher combination scheme (IWIU) based on the traditional algorithm is proposed to determine whether two matches are related by adjusting the weights of different matchers and adding them weightedly.

[0025] As a preferred solution of the data discovery method based on composite matching described in the present invention, the selection of the matcher in S2 is to dynamically select the matcher for matching based on the historical performance and current weight of each matcher, rather than random selection or fixed order selection.

[0026] As a preferred solution of the data discovery method based on composite matching described in the present invention, wherein: S4 updates the weight of each matcher through a multiplicative weight update (MWU) algorithm, and dynamically adjusts the weight of each matcher in an iterative manner to optimize the matching effect and improve the matching accuracy.

[0027] As a preferred solution of the data discovery method based on composite matching described in the present invention, the communication method between the matchers in S1-S5 is HTTP / REST, API or gRPC.

[0028] As a preferred solution of the data discovery method based on composite matching described in the present invention, wherein:

[0029] Beneficial effects of the present invention:

[0030] 1. A matcher combination scheme (IWIU) and a machine learning-based method are designed to improve the accuracy and efficiency of data discovery. Compared with the traditional static weight allocation method, this scheme can more accurately reflect the performance of each matcher at different stages, automatically optimize the weight allocation, and improve the matching accuracy and system efficiency. The present invention is expected to achieve more accurate and efficient data analysis and mining in the field of data discovery, and provide organizations and researchers with more tools and methods to deal with complex data environments.

[0031] 2. Compared with the traditional method of selecting matchers in random or fixed order, this scheme can better utilize the advantages of matchers and avoid excessive participation of inaccurate matchers, thereby improving the accuracy and efficiency of matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0033] Figure 1 This is the overall architecture diagram of the data discovery system based on composite matching proposed in the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0036] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0037] Secondly, the present invention is described in detail with reference to the schematic diagram. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0038] Reference Figure 1 The present invention provides a data discovery system based on composite matching, which specifically includes:

[0039] Storage layer: The storage layer is mainly responsible for storing data of various data sets and saving the operation results of the algorithm.

[0040] Model layer: The model layer is mainly the specific implementation module of various composite matching solutions.

[0041] Application layer: The application layer exposes various types of information to users in the form of a web front end, providing users with the choice of different methods, algorithm parameters, data sets, and the query of calculation results.

[0042] The centralized data of the storage layer is usually imported into the Postgre SQL management system in a table-type mode, and the clustering and related evaluation scores generated after the algorithm is executed are also stored in the system.

[0043] Furthermore, the clustering and related evaluation scores generated after the algorithm is executed are also stored in the Postgre SQL management system.

[0044] Furthermore, the model layer receives data from the storage layer, performs similarity judgment on different column pairs, and finally outputs clusters or related column pairs and values ​​of various evaluation criteria of the columns.

[0045] Furthermore, the output result of the model layer will be rendered by Vue to be passed to the application layer.

[0046] Furthermore, the interaction between the application layer and the model layer is mainly implemented through the RESTful API, mainly utilizing the GET and POST requests therein.

[0047] This embodiment also includes a data discovery method based on composite matching, and the specific steps include:

[0048] S1. First, after the system is started, the weights are initialized. The weight wi of each matcher is initialized to 1, and the initial weights are prepared for the next iterative matching. Compared with the method that requires pre-analysis of the matcher performance to assign initial weights, this simplifies the initialization process, reduces the complexity of the initial configuration, and saves time and computing resources.

[0049] S2. Then the system selects a matcher. After starting the matching round, it selects a matcher randomly or according to a certain strategy, and selects a suitable matcher for the current round.

[0050] S3. After the system selects a matcher, it executes the matching task and obtains the matching result g(p), and then calculates the matching result of the current matcher.

[0051] S4. After the system matching result g(p) is known, the weight of each matcher is updated, and the truncation threshold t is updated according to the situation. The weight and truncation threshold of the matcher are adjusted according to the current matching effect to optimize the subsequent matching effect. The truncation threshold is dynamically updated according to the effect of each round of matching to further optimize the matching effect. Compared with the method of fixing the truncation threshold, this solution can adjust the truncation threshold according to the actual matching effect, so that the system can better adapt to different matching situations and improve the matching accuracy.

[0052] S5. After the system completes one round of matching, it repeats steps S2, S3, and S4 until m rounds of matching are completed. Through multiple rounds of iterations, the matching effect is gradually optimized until the requirements are met. Compared with the single-round matching method, this solution optimizes the matching effect through multiple rounds of iterations, can find the matching results more accurately, and improves the matching accuracy and system stability.

[0053] A matcher combination scheme (IWIU) and a machine learning-based method are designed to improve the accuracy and efficiency of data discovery. Compared with the traditional method of selecting matchers randomly or in a fixed order, this scheme can better utilize the advantages of matchers and avoid excessive participation of inaccurate matchers, thereby improving the accuracy and efficiency of matching.

[0054] Each matcher is responsible for processing specific data features, such as text, images or sounds, and then obtains a matching result through calculation. Each matcher can interact with each other through network connection or local communication to jointly complete the overall matching task. It can be deployed as a software module or service on a server or cloud platform to provide matching and similarity calculation services for upstream applications.

[0055] Furthermore, the matcher selection in S2 is to dynamically select the matcher for matching based on the historical performance and current weight of each matcher, rather than random selection or fixed order selection.

[0056] Furthermore, S4 updates the weight of each matcher through the multiplicative weight update (MWU) algorithm, and dynamically adjusts the weight of each matcher in an iterative manner to optimize the matching effect and improve the matching accuracy. Compared with the traditional static weight allocation method, this scheme can more accurately reflect the performance of each matcher at different stages, automatically optimize the weight allocation, and improve the matching accuracy and system efficiency.

[0057] Furthermore, the communication between the matchers in S1-S5 is carried out via HTTP / REST, API or gRPC.

[0058] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data discovery system based on composite matching, characterized by: Specifically include: Storage layer: The storage layer is mainly responsible for storing data of various data sets and saving the operation results of the algorithm. Model layer: The model layer is mainly the specific implementation module of various composite matching solutions. Application layer: The application layer exposes various types of information to users in the form of a web front end, providing users with the choice of different methods, algorithm parameters, data sets, and the query of calculation results.

2. The data discovery system based on composite matching according to claim 1, characterized in that: The centralized data of the storage layer is usually imported into the Postgre SQL management system in a table-type mode, and the clustering and related evaluation scores generated after the algorithm is executed are also stored in the system.

3. The data discovery system based on composite matching according to claim 2, characterized in that: The clustering and related evaluation scores generated after the algorithm is executed are also stored in the Postgre SQL management system.

4. The data discovery system based on composite matching according to claim 3, characterized in that: The model layer receives data from the storage layer, performs similarity judgment on different column pairs, and finally outputs clusters or related column pairs and values ​​of various evaluation criteria of the columns.

5. The data discovery system based on composite matching according to claim 4, characterized in that: The output of the model layer will be rendered by Vue and passed to the application layer.

6. The data discovery system based on composite matching according to claim 5, characterized in that: The interaction between the application layer and the model layer is mainly achieved through the RESTful API, mainly utilizing the GET and POST requests therein.

7. The data discovery method based on composite matching according to claim 6, based on the memory-aware virtual machine migration system according to any one of claims 1-5, characterized in that: The specific steps include: S1. First, after the system is started, the weight is initialized. The weight wi of each matcher is initialized to 1 to prepare the initial weight for the next iterative matching. S2. Then the system selects a matcher. After starting the matching round, it selects a matcher randomly or according to a certain strategy, and selects a suitable matcher for the current round. S3. After the system selects a matcher, it executes the matching task and obtains the matching result g(p), and then calculates the matching result of the current matcher. S4. After the system matching result g(p) is known, the weight of each matcher is updated, and the truncation threshold t is updated according to the situation. The weight and truncation threshold of the matcher are adjusted according to the current matching effect to optimize the subsequent matching effect. S5. After the system completes one round of matching, it repeats steps S2, S3, and S4 until m rounds of matching are completed. Through multiple rounds of iterations, the matching effect is gradually optimized until the requirements are met.

8. The data discovery system based on composite matching according to claim 7, characterized in that: The matcher selection in S2 is to dynamically select the matcher for matching based on the historical performance and current weight of each matcher, rather than random selection or fixed order selection.

9. The data discovery system based on composite matching according to claim 8, characterized in that: S4 updates the weight of each matcher through the multiplicative weight update (MWU) algorithm, and dynamically adjusts the weight of each matcher in an iterative manner to optimize the matching effect and improve the matching accuracy.

10. The data discovery system based on composite matching according to claim 9, characterized in that: The communication between matchers in S1-S5 is done through HTTP / REST, API or gRPC.

Images