Object-oriented multi-source data accurate matching method

Abstract

The invention discloses an object-oriented multi-source data accurate matching method, and relates to the technical field of data matching, and the method comprises the steps: obtaining a plurality of data sources collected by a plurality of collection objects, and carrying out the cleaning, conversion and aggregation processing of the data sources; performing fault detection on the processed data, identifying fault information in the data, and extracting effective data; and according to different application scenes and characteristics of the effective data, a corresponding matching strategy and a matching algorithm are adapted in a preset matching standard for matching, and a matching result is obtained. According to the method, different matching strategies and algorithms can be adopted according to different conditions, so that accurate data matching can be quickly carried out, and the efficiency and accuracy of multi-source data matching are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data matching, and in particular to an object-oriented multi-source data precise matching method. Background Art

[0002] The background of object-oriented multi-source data precise matching methods mainly stems from the urgent need for efficient and accurate information extraction in the field of data processing and analysis. With the rapid development of information technology, data sources are becoming increasingly diversified, including remote sensing data, social media data, enterprise operation data and other types of data are widely used in various industries and fields. However, these multi-source data have significant differences in format, structure, quality, etc., which brings great challenges to data integration and analysis.

[0003] Most of the existing matching methods are single matching strategies and matching algorithms, and the data collected for different objects are different. Therefore, in the matching process, a single matching strategy and matching algorithm can easily lead to large matching errors. At the same time, there is a large amount of useless and erroneous information when collecting data before matching, which can easily affect the data matching work.

[0004] In view of this, an object-oriented multi-source data accurate matching method is needed. Summary of the invention

[0005] In view of the problem that the existing matching methods in the prior art have large matching errors, the present invention provides an object-oriented multi-source data accurate matching method, which can adopt different matching strategies and algorithms according to different situations to achieve rapid data accurate matching. The specific technical solution is as follows:

[0006] An object-oriented multi-source data accurate matching method, comprising:

[0007] Acquire multiple data sources collected by multiple collection objects, and clean, convert and aggregate the multiple data sources;

[0008] Perform fault detection on the processed data, identify fault information in the data, and extract valid data;

[0009] According to different applicable scenarios and characteristics of the valid data, corresponding matching strategies and matching algorithms are adapted in preset matching standards to perform matching and obtain matching results.

[0010] Preferably, the performing fault detection on the processed data, identifying fault information in the data, and extracting valid data comprises:

[0011] Perform fault detection on the processed data through machine learning, statistical analysis and / or rule engine algorithms to identify abnormal patterns, new fault patterns and fault information of known fault patterns in the data and extract valid data.

[0012] Preferably, an object-oriented multi-source data accurate matching method further includes:

[0013] The fault information is pushed and displayed.

[0014] Preferably, an object-oriented multi-source data accurate matching method further includes:

[0015] Comparing the matching result with the original data to obtain a comparison result;

[0016] Perform statistical analysis on the matching results and comparison results to obtain analysis and statistical results including matching rate and mismatch rate indicators;

[0017] The performance of the matching algorithm is evaluated based on the analysis and statistical results, the matching algorithm is optimized, and the matching process is performed after the optimization.

[0018] Preferably, the optimizing the matching algorithm and performing the matching process after the optimization includes:

[0019] By adjusting parameters and improving algorithm structure, and re-conducting matching tests, the optimization direction of the matching algorithm is evaluated and the best solution is selected for matching implementation.

[0020] Preferably, the preset matching criteria include:

[0021] Each data record in the valid data is regarded as an object, and the matching criteria of the corresponding attribute matching rules and behavior matching rules are defined based on the attributes and behaviors of the object, and the matching process is guided by the matching criteria.

[0022] Preferably, according to different applicable scenarios and characteristics of the valid data, the corresponding matching strategy and matching algorithm are adapted in the preset matching criteria to perform matching, and the matching results obtained include:

[0023] According to different applicable scenarios and characteristics of the valid data, corresponding matching strategies and matching algorithms are adapted in preset matching criteria, and real-time text search and matching are used to perform matching to obtain matching results.

[0024] Preferably, the matching strategies include complete matching, precise search, fuzzy matching, feature matching, rule-based matching, template-based matching, and machine learning and deep learning; the matching algorithms include string matching algorithms, similarity calculation algorithms, machine learning algorithms, and graph matching algorithms.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The object-oriented multi-source data accurate matching method of the present invention obtains multiple data sources collected by multiple collection objects, and cleans, converts and aggregates the multiple data sources; performs fault detection on the processed data, identifies fault information in the data, and extracts valid data; according to different applicable scenarios and characteristics of the valid data, adapts the corresponding matching strategy and matching algorithm in the preset matching standard to match and obtain the matching result. The present invention can adopt different matching strategies and algorithms according to different situations, so that data can be accurately matched quickly, and the efficiency and accuracy of multi-source data matching can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0028] Figure 1 The present invention is a flowchart of an object-oriented multi-source data precise matching method.

[0029] Figure 2 The present invention is a flowchart of another embodiment of an object-oriented multi-source data precise matching method. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0032] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0033] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] See the following examples Figure 1 to Figure 2 .

[0035] The present application provides an object-oriented multi-source data accurate matching method, including:

[0036] Step S01: Acquire multiple data sources collected by multiple collection objects, and clean, convert and aggregate the multiple data sources;

[0037] Collect data from multiple data sources in real time, including but not limited to sensors, log files, and application programming interfaces. Clean, transform, and aggregate the collected data to ensure data consistency and accuracy.

[0038] In specific implementation, cleaning, transforming and aggregating the collected data are key steps in the data preprocessing process;

[0039] Data cleaning is the process of identifying and correcting errors, outliers, and inconsistencies in data. This includes but is not limited to the following aspects:

[0040] Handling missing values: Missing values ​​can be handled by deleting records containing missing values, filling them with statistical values, using interpolation methods or predictive models.

[0041] Identify and correct erroneous data: Check for data type errors, format errors, logical errors, etc., and make corresponding corrections.

[0042] De-duplicate records: Identify and remove duplicates in a dataset to avoid bias in subsequent analysis.

[0043] Handle outliers: Identify outliers in your data and choose to keep, remove, or replace them as appropriate.

[0044] Data conversion is the process of adjusting the format, type or structure of data to meet analytical requirements. This includes:

[0045] Data type conversion: Converting data from one type to another, such as converting a string to a date format.

[0046] Data encoding: Convert categorical data into numerical data for mathematical operations.

[0047] Data standardization / normalization: Scaling numerical data so that it falls within a specific range to eliminate the impact of dimension on analysis.

[0048] Data decomposition: Breaking down complex data structures into simpler components for easier processing and analysis.

[0049] Data aggregation refers to the process of combining multiple data items into a single summary item, which is usually used to reduce the amount of data, extract key information, and discover patterns in the data. This includes:

[0050] Group aggregation: Group data by one or more fields and apply an aggregate function to each group.

[0051] Rolling aggregation: In time series data, data is aggregated by time windows to observe trends or cyclical changes.

[0052] Multi-table aggregation: In databases and data warehouses, data from multiple tables is merged into a result set and aggregated for analysis.

[0053] In actual operation, data cleaning, transformation and aggregation is usually an iterative process that needs to be adjusted and optimized according to the quality of the data and analysis requirements. The following is a simplified implementation step:

[0054] Data exploration: Gain a preliminary understanding of the structure and content of the data and identify potential problems and opportunities.

[0055] Define cleaning rules: Develop data cleaning rules and strategies based on data issues and business needs.

[0056] Perform cleaning operations: Apply cleaning rules to correct and clean the data.

[0057] Data transformation: Adjust the format, type or structure of data as needed.

[0058] Data aggregation: Group and summarize data as needed.

[0059] Validation and testing: Check whether the cleansed, transformed, and aggregated data meets quality standards and business requirements.

[0060] Documentation: Record the steps taken and decisions made for future reference and auditing.

[0061] Step S02: perform fault detection on the processed data, identify fault information in the data, and extract valid data;

[0062] In the field of fault detection, combining machine learning, statistical analysis, and rule engine algorithms can significantly improve the accuracy and efficiency of detection. These technologies each have unique advantages and can complement each other to cope with complex and changing fault scenarios.

[0063] In specific implementation, the process of using machine learning, statistical analysis and rule engine algorithms to perform fault detection on processed data includes:

[0064] Descriptive statistics are used in statistical analysis to summarize the characteristics of data. These statistics can help identify unusual patterns in the data, such as observations that deviate from the normal range.

[0065] Supervised learning is used in machine learning, and the model recognizes new faults by learning labeled fault data. Common algorithms include decision trees, random forests, gradient boosting machines, neural networks, etc.

[0066] The rule engine allows fault detection based on a predefined set of rules. These rules can be based on thresholds, pattern matching, and complex logical expressions. The rule engine can respond quickly and accurately identify known fault modes.

[0067] Fault detection is performed on the processed data through machine learning, statistical analysis and / or rule engine algorithms to identify abnormal patterns, new fault patterns and fault information of known fault patterns in the data, so that valid data can be extracted.

[0068] Step S03: According to different applicable scenarios and characteristics of the valid data, the corresponding matching strategy and matching algorithm are adapted in the preset matching standard to perform matching and obtain a matching result.

[0069] Different matching strategies and algorithms are used according to different situations, and real-time search technology is used to support fast text search and matching. These technologies can support complex query syntax and flexible matching strategies. The matching strategies include full matching, precise search, fuzzy matching, feature matching, rule-based matching, template-based matching, machine learning and deep learning; the matching algorithms include string matching algorithms, similarity calculation algorithms, machine learning algorithms and graph matching algorithms.

[0070] Specifically, the preset matching criteria include:

[0071] Each data record in the valid data is regarded as an object, and the matching criteria of the corresponding attribute matching rules and behavior matching rules are defined based on the attributes and behaviors of the object, and the matching process is guided by the matching criteria.

[0072] In specific implementation, in a multi-source data environment, the matching goals include but are not limited to identifying duplicate records, merging similar records, and associating related records in different data sources. The above goals will guide the subsequent matching strategies and algorithm selection; matching criteria refer to the criteria used to determine whether two or more data records should be considered to be the same or similar. These criteria can be formulated based on the specific content of the data and the structural characteristics of the data. When formulating matching criteria, the diversity and complexity of the data must be taken into account to ensure the accuracy and reliability of the matching results.

[0073] Object-oriented methodology emphasizes object-centricity. Therefore, when formulating matching criteria, each data record can be regarded as an object, and matching criteria can be formulated based on the attributes and behaviors of the object. A series of attribute matching rules and behavior matching rules can be defined to guide the matching process. The specific matching case process is as follows:

[0074] (1) Exact match

[0075] Applicable scenarios: When you need to ensure that the query is exactly the same as the text in the document, such as code snippets, product numbers, and ID numbers.

[0076] Features: Requires that the query be matched completely and without distinction with the records in the database.

[0077] Implementation method: string matching algorithm.

[0078] (2) Precise search

[0079] Applicable scenarios: Similar to exact match, but allows some minor differences, such as differences in spaces and punctuation marks, but does not allow substantial content changes.

[0080] Features: Allows a degree of flexibility while maintaining query intent.

[0081] Implementation method: regular expressions and string matching algorithms.

[0082] (3) Fuzzy matching

[0083] Applicable scenarios: Handle spelling errors, abbreviations, synonyms or variants to improve the recall rate.

[0084] Features: Identifies matches by measuring the similarity between two strings, and can tolerate a certain degree of difference.

[0085] Implementation method: edit distance, Jaccard similarity, TF-IDF combined with cosine similarity, word embedding and similarity calculation in natural language processing.

[0086] (4) Feature Matching

[0087] Applicable scenarios: When documents and queries can be represented as a series of features, such as keywords, tags, and metadata.

[0088] Features: Find matches by comparing a set of features of the document and the query.

[0089] Implementation methods: vector space model, feature hashing, feature extraction and selection algorithms in machine learning, etc.

[0090] (5) Rule-based matching

[0091] Applicable scenarios: When there are clear business rules or logic to follow, such as address format verification and date format matching.

[0092] Features: Identify matching items through predefined rule sets. The rules can be simple conditional judgments or complex logical expressions.

[0093] Implementation: Write a rules engine, use regular expressions and business logic code.

[0094] (6) Template-based matching

[0095] Applicable scenarios: When the document follows a fixed format or template, such as email templates, report formats, etc.

[0096] Features: Match documents with similar structures by defining templates.

[0097] Implementation method: regular expressions, XML / JSON parsing and template engine.

[0098] (7) Machine Learning and Deep Learning

[0099] Applicable scenarios: Processing complex, unstructured text data requires automatic learning and recognition of patterns, semantics, and contextual relationships.

[0100] Features: Able to process large-scale data sets, automatically learn and optimize matching strategies, and applicable to multiple languages, cultures, and fields.

[0101] Implementation: Natural language processing models, sequence-to-sequence models, text classification and clustering algorithms.

[0102] The object-oriented multi-source data accurate matching method of the present invention obtains multiple data sources collected by multiple collection objects, and cleans, converts and aggregates the multiple data sources; performs fault detection on the processed data, identifies fault information in the data, and extracts valid data; according to different applicable scenarios and characteristics of the valid data, adapts the corresponding matching strategy and matching algorithm in the preset matching standard to match and obtain the matching result. The present invention can adopt different matching strategies and algorithms according to different situations, so that data can be accurately matched quickly, and the efficiency and accuracy of multi-source data matching can be improved.

[0103] In a preferred embodiment of the present application, an object-oriented multi-source data accurate matching method further includes:

[0104] The fault information is pushed and displayed.

[0105] In the specific implementation, an efficient push mechanism is realized through WebSocket and HTTP long polling technology to push the detected fault information to the visualization interface in real time.

[0106] In a preferred embodiment of the present application, an object-oriented multi-source data accurate matching method further includes:

[0107] Step S31, comparing the matching result with the original data to obtain a comparison result;

[0108] Compare the matching results with the original data source to check for inconsistencies and errors, and use automated scripts and tools to assist the comparison process to improve efficiency and accuracy.

[0109] Step S32: Perform statistical analysis on the matching results and the comparison results to obtain analysis statistical results including matching rate and mismatch rate indicators;

[0110] Perform statistical analysis on the matching results, such as calculating the matching rate and mismatch rate indicators. By analyzing the statistical results, the performance and accuracy of the matching algorithm can be evaluated and potential problems can be discovered.

[0111] The matching rate refers to the ratio of the number of correctly matched entities in the matching result set to the total number of entities in the data set during a matching process. It measures the comprehensiveness of the matching algorithm in identifying entities with the same name or related information.

[0112] Calculation formula:

[0113] Match rate = total number of entities in the dataset / number of correctly matched entities.

[0114] The mismatch rate refers to the ratio of the number of mismatched entities to the total number of entities in the matching result set during a matching process. It measures the accuracy and reliability of the matching algorithm.

[0115] Calculation formula:

[0116] False match rate = total number of entities in the matching result set / number of falsely matched entities.

[0117] The performance evaluation of matching algorithms usually focuses on the following aspects:

[0118] Matching speed refers to the time required for the algorithm to complete the matching task. It reflects the algorithm's computational efficiency and search speed. The matching speed is evaluated by calculating the average running time of the algorithm or the time required to process a unit of data. A faster matching speed can meet application scenarios with higher real-time requirements.

[0119] Computational complexity is a measure of the computational resources required by an algorithm during its execution. The computational efficiency of an algorithm is evaluated by analyzing its time complexity and space complexity. Lower computational complexity means that the algorithm is more efficient when processing large-scale data.

[0120] Scalability refers to the ability of an algorithm to maintain efficient operation as the amount of data increases. The scalability of an algorithm is evaluated by testing its performance on data sets of different sizes. An algorithm with good scalability can maintain stable performance as the amount of data grows.

[0121] Step S33: Evaluate the performance of the matching algorithm based on the analysis and statistical results, optimize the matching algorithm, and perform the matching process after the optimization.

[0122] Algorithm optimization: Improve and optimize the matching algorithm, such as adjusting parameters and improving the algorithm structure. Through experiments and tests, evaluate the effects of different algorithm optimization schemes and select the best scheme for implementation.

[0123] Index optimization: Optimize the index structure in the database and cache to improve query speed and matching efficiency, and reasonably design index strategies and index types based on data access patterns and query requirements.

[0124] Database index optimization:

[0125] Rebuild indexes regularly: As data is added and deleted, the index may become fragmented, affecting query performance. Rebuilding the index regularly can restore index performance.

[0126] Monitor index usage: Use tools provided by the database or third-party tools to monitor index usage and promptly identify and resolve index failures or performance degradation issues.

[0127] Cache index optimization:

[0128] When using distributed caching, consider strategies such as cache partitioning, replication, and fault tolerance.

[0129] Reasonably utilize the distributed characteristics of the cache and store hot data in different cache nodes to improve the concurrent access capability of the cache.

[0130] In this embodiment, an object-oriented multi-source data precise matching method performs real-time analysis on the processed data, and performs precise matching on the analyzed data. If fault data appears, the fault data is pushed, and then the fault information is displayed, so that valid data can be extracted. At the same time, when performing data matching, the object-oriented methodology emphasizes object-centricity. Therefore, when formulating matching criteria, each data record can be regarded as an object, and matching criteria can be formulated based on the attributes and behaviors of the object. A series of attribute matching rules and behavior matching rules can be defined to guide the matching process. Different matching strategies and algorithms are used according to different situations, and real-time search technology is used to support fast text search and matching, so that data can be quickly and accurately matched. Finally, the matching results are compared with the original data source to check whether there are inconsistencies and errors. Automated scripts and tools are used to assist the comparison process to improve efficiency and accuracy.

[0131] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0132] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. An object-oriented multi-source data accurate matching method, characterized in that: include: Acquire multiple data sources collected by multiple collection objects, and clean, convert and aggregate the multiple data sources; Perform fault detection on the processed data, identify fault information in the data, and extract valid data; According to different applicable scenarios and characteristics of the valid data, corresponding matching strategies and matching algorithms are adapted in preset matching standards to perform matching and obtain matching results.

2. The object-oriented multi-source data accurate matching method according to claim 1, characterized in that: The performing fault detection on the processed data, identifying fault information in the data, and extracting valid data comprises: Perform fault detection on the processed data through machine learning, statistical analysis and / or rule engine algorithms to identify abnormal patterns, new fault patterns and fault information of known fault patterns in the data and extract valid data.

3. The object-oriented multi-source data accurate matching method according to claim 2, characterized in that: Also includes: The fault information is pushed and displayed.

4. The object-oriented multi-source data accurate matching method according to claim 1, characterized in that: Also includes: Comparing the matching result with the original data to obtain a comparison result; Perform statistical analysis on the matching results and comparison results to obtain analysis and statistical results including matching rate and mismatch rate indicators; The performance of the matching algorithm is evaluated based on the analysis and statistical results, the matching algorithm is optimized, and the matching process is performed after the optimization.

5. The object-oriented multi-source data accurate matching method according to claim 4, characterized in that: The optimization of the matching algorithm and the matching process after the optimization include: By adjusting parameters and improving algorithm structure, and re-conducting matching tests, the optimization direction of the matching algorithm is evaluated and the best solution is selected for matching implementation.

6. The object-oriented multi-source data accurate matching method according to claim 1, characterized in that: The preset matching criteria include: Each data record in the valid data is regarded as an object, and the matching criteria of the corresponding attribute matching rules and behavior matching rules are defined based on the attributes and behaviors of the object, and the matching process is guided by the matching criteria.

7. The object-oriented multi-source data accurate matching method according to claim 1, characterized in that: According to different applicable scenarios and characteristics of the valid data, the corresponding matching strategy and matching algorithm are adapted in the preset matching criteria to perform matching, and the matching results obtained include: According to different applicable scenarios and characteristics of the valid data, corresponding matching strategies and matching algorithms are adapted in preset matching criteria, and real-time text search and matching are used to perform matching to obtain matching results.

8. The object-oriented multi-source data accurate matching method according to claim 1, characterized in that: The matching strategies include complete matching, precise search, fuzzy matching, feature matching, rule-based matching, template-based matching, and machine learning and deep learning; the matching algorithms include string matching algorithms, similarity calculation algorithms, machine learning algorithms, and graph matching algorithms.

Images