Multi-platform variable data capturing method based on RAP

Abstract

The invention relates to the technical field of data processing, in particular to an RAP-based multi-platform variable data capturing method, which comprises the following steps that: S1, an RAP identifies data sources of a plurality of platforms, and performs identity verification on the data sources by using an intelligent contract; s2, based on the type and the structure of the data source, generating an adapter, and capturing data from each data source by the RAP by using the adapter; s3, fusing the grabbed data into a uniform data format, constructing a data relation graph, mining and discovering hidden association rules, and optimizing a grabbing strategy; and S4, the RAP has an adaptive learning capability, and automatically adjusts a capture strategy and a data fusion method when a data source changes. According to the method, the adaptive adapter is used for efficiently capturing diversified data sources; an improved DQN algorithm is used for automatically adjusting a grabbing strategy, and manual intervention is reduced; a gradient elevator is adopted to optimize a data fusion process, and data consistency and integrity are ensured.

Description

Background Art

[0002] In existing data capture technologies, the authenticity and access rights of data sources are usually not effectively verified, resulting in unreliable data sources and potential security risks. In other words, there is a lack of effective mechanisms to verify the identity of data sources, making it easy for data to be tampered with or contaminated by untrusted data sources.

[0003] Traditional methods can only deal with specific types of data sources, such as fixed website structures or API interfaces. When encountering new or changing data sources, it is necessary to manually write code or configuration files to adapt. There are problems such as inflexibility and inefficiency, and it is difficult to adapt to the rapidly changing data environment. In addition, the captured data may have problems such as inconsistent formats, redundancy or errors, which affect subsequent data analysis. The lack of effective data cleaning and standardization methods leads to poor data accuracy and consistency.

[0004] Traditional data storage methods cannot effectively capture the complex relationships between data. The data correlation is poor, making it difficult to conduct in-depth data mining and analysis. The degree of automation is not high, and a large amount of human resources are required to maintain and manage the data capture process.

[0005] To summarize the above description, the current data capture has the following shortcomings:

[0006] Poor adaptability: Existing data capture technologies can only cope with specific types of data sources. When the data source changes, the code needs to be rewritten or adjusted, which is time-consuming and inefficient.

[0007] Data quality varies: After data is captured, it often requires a lot of post-processing to reach a usable state, which adds additional workload.

[0008] Weak data correlation: The lack of effective methods to discover and utilize implicit relationships between data limits the depth and breadth of data analysis.

[0009] Low degree of automation: Many data scraping tasks still require manual monitoring and intervention, which increases labor costs and reduces the overall efficiency of the system.

[0010] Security issues: The lack of effective security mechanisms during data capture may threaten the integrity of the data.

[0011] In summary, existing technologies have certain limitations in data source verification, data source adaptability, data quality control, data relevance, and intelligent data capture. Summary of the invention

[0012] The main purpose of the present invention is to provide a multi-platform variable data capture method based on RAP, which effectively solves the above-mentioned problems mentioned in the background technology.

[0013] The technical solution of the present invention is as follows:

[0014] The multi-platform variable data crawling method based on RAP is characterized by comprising the following steps:

[0015] S1,RAP identifies data sources from multiple platforms and uses smart contracts to authenticate data sources;

[0016] S2, based on the data source type and structure, generate adapters, RAP uses adapters to grab data from each data source;

[0017] S3: Integrate the captured data into a unified data format, build a data relationship map, discover hidden association rules, and optimize the capture strategy;

[0018] S4.RAP has adaptive learning capabilities and automatically adjusts the crawling strategy and data fusion method when the data source changes.

[0019] A further improvement of the present invention is that in step S1, the multiple platforms include a Web platform, APIs, a database system, a file system, email, a cloud storage service platform, a scientific and academic data source platform, and a public data platform.

[0020] A further improvement of the present invention is that, in step S1, the step of using a smart contract to authenticate the data source is:

[0021] S1-1. Define smart contract rules. Smart contracts include data source verification rules, specifically the type of data source, permission level, and access frequency;

[0022] S1-2. The data source is registered in the smart contract, providing the necessary identity information and permission declaration;

[0023] S1-3, RPA sends a data source access request through a smart contract, and the smart contract checks whether the access request complies with the preset verification rules;

[0024] S1-4. If the smart contract is successfully verified and an access token is generated, the smart contract will allow RPA to access the data source; otherwise, access will be denied and recorded.

[0025] A further improvement of the present invention is that, in step S2, specifically includes:

[0026] S2-1. Analyze the type of data source and parse the data structure;

[0027] S2-2. Generate data capture strategies and data conversion rules for the adapter according to the type and structure of the data source. The data conversion rules include data cleaning, format conversion and data statistics. The data cleaning is used to remove invalid characters and standardize data formats.

[0028] A further improvement of the present invention is that, in step S3, the captured data is merged into a unified data format, including:

[0029] S3-1. Extract metadata from the captured data, create a type mapping table, map the same or similar data types to a unified standard data type, establish field mapping rules, and map fields with the same meaning to the same field for data standardization;

[0030] S3-2. Use the data cleaning algorithm to process the standardized data, and use fuzzy matching to merge the cleaned data. The process is as follows:

[0031] S3-2-1. Extract key features for comparison. The key features are represented in the form of strings, vectors, and numerical values.

[0032] S3-2-2, use Jaro-Winkler distance to measure string similarity, use TF-IDF weighted improved cosine similarity to measure vector similarity, and use improved Manhattan distance to measure numerical similarity;

[0033] S3-2-3. Set a threshold. If the similarity score calculated in step S3-2-2 is higher than the threshold, the data are considered to be different representations of the same entity and merged.

[0034] A further improvement of the present invention is that when the Jaro-Winkler distance is used, the calculation formula of the similarity score is as follows:

[0035]

[0036] Where: s1 and s2 are the two strings to be compared, m is the number of characters that s1 and s2 match, t is the transposed data, l is the length of the prefix partial match, P is the range factor constant, and Jaro(s1,s2) is the basic Jaro distance;

[0037] When using TF-IDF weighted improved cosine similarity, the similarity score calculation formula is:

[0038]

[0039] Among them: TF-IDF i is the TF-IDF value of the i-th word, n is the dimension of the vector, A and B are two vectors;

[0040] When using the improved Manhattan distance, the similarity score calculation formula is:

[0041]

[0042] Among them: x and y are two n-dimensional vectors, x i and i are the values ​​of these two vectors in the i-th dimension, μ i and σ i are the mean and standard deviation of the i-th dimension features respectively.

[0043] A further improvement of the present invention is that constructing a data relationship graph includes using a JanusGraph graph database to construct a data relationship graph, and storing the fused data in the form of nodes and edges, wherein nodes represent entities and edges represent relationships between entities.

[0044] A further improvement of the present invention is that, according to the data relationship graph, the steps of mining and discovering hidden association rules are:

[0045] 1) Convert the data in the data graph into a transaction database format that is suitable for the FP-growth algorithm;

[0046] 2) Build the FP tree, then mine frequent item sets to find frequently occurring entity sets;

[0047] 3) Generate association rules based on frequent item sets and use confidence as an indicator to evaluate the strength of association rules;

[0048] According to the hidden association rules discovered through mining, entities with high association strength are given higher crawling priority.

[0049] A further improvement of the present invention is that, in step S4, specifically includes:

[0050] S4-1. Regularly or in real time monitor the update frequency, data format and data volume of data sources;

[0051] S4-2, use the improved DQN reinforcement learning algorithm to adjust the grasping strategy;

[0052] S4-3. Use the gradient boosting machine to learn the optimal rules for data fusion and update the data fusion method. The gradient boosting machine update formula is:

[0053] F m (x) = F m-1 (x)+λ m h m (x)

[0054] Among them, F m(x) is the prediction result of the gradient boosting machine fusion of the input data x at the mth iteration; F m-1 (x) is the prediction result of the previous iteration, λ m is the learning rate, h m (x) is the prediction of the input data x by the weak learner added in the mth iteration.

[0055] A further improvement of the present invention is that, in step S4-2, the improved DQN reinforcement learning algorithm is implemented as follows:

[0056] S4-2-1. Define the environment state S, action A, reward R and transfer function T. In the data capture scenario, the state S includes the state of the current data source, the action A includes adjusting the capture frequency and capture time window, the reward R is based on the quality and efficiency of the captured data, and the transfer function T is the state S. t To S t+1 The probability distribution of

[0057] S4-2-2. Use the recurrent neural network RNN ​​as the approximator of the Q function. The input is the state S and the output is the Q value of each action.

[0058] S4-2-3. Use the experience replay mechanism to store and randomly extract past experience and update the Q value; Q value update formula:

[0059] ΔQ←r+γmaxQ(S t+1 ,a′;θ - )-Q(S t ,A t ;θ)

[0060] Where: r: immediate reward, γ: discount factor, 0≤γ<1, Q(S t ,A t ; θ) is the current network state S t Next action A t The Q value estimate, Q(S t+1 ,a′;θ - ) is the target network for the next state S t+1 The maximum value of the Q-value estimate of all possible actions a′;

[0061] S4-2-4. Use the target network to stabilize the learning process. The parameters of the target network are copied from the current network at regular intervals, and the main network parameters θ are copied to the target network θ. - ;

[0062] θ - ←τθ+(1-τ)θ -

[0063] S4-2-5. Use the main network to predict the Q value, and the target network is used to calculate the target Q value to reduce the estimation bias.

[0064] The technical effects of the present invention are as follows: 1) The present invention uses smart contracts to authenticate the data source, thereby ensuring the authenticity of the data source and the validity of the access rights during the data capture process. This not only improves the reliability of the data source, but also increases the security of the entire data capture process;

[0065] 2) The present invention can efficiently capture data from a variety of data sources, including Web platforms, APIs, database systems, etc., by generating adapters for different data sources. This method improves the flexibility and efficiency of data capture and can meet the needs of different business scenarios; the adapter can continuously learn changes in data sources, automatically update its own logic, and improve the accuracy and efficiency of data capture;

[0066] 3) The present invention ensures the accuracy and consistency of the captured data by standardizing data format, data cleaning, fuzzy matching and other methods. The accuracy of data fusion is improved by using methods such as Jaro-Winkler distance, TF-IDF weighted improved cosine similarity and improved Manhattan distance;

[0067] 4) The present invention uses the JanusGraph graph database to construct a data relationship map, and by mining hidden association rules, it can discover the complex relationships between data, thereby enhancing the depth and breadth of data analysis;

[0068] 5) The RAP of the present invention has adaptive learning capabilities. By using the improved DQN reinforcement learning algorithm, the system can automatically adjust the crawling strategy when the data source changes, thereby improving the intelligence level of data crawling. This not only reduces the need for manual intervention, but also ensures the timeliness and accuracy of data crawling; by using the gradient boosting machine to learn the optimal rules for data fusion, the data fusion process can be continuously optimized. This method ensures the consistency and integrity of the data, while improving the overall efficiency and quality of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0070] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0071] like Figure 1 As shown, the present invention discloses a multi-platform variable data crawling method based on RAP, comprising the following steps:

[0072] S1. RAP identifies data sources from multiple platforms and uses smart contracts to authenticate data sources.

[0073] In step S1, the multiple platforms include web platforms, APIs, database systems, file systems, email, cloud storage service platforms, scientific and academic data source platforms, and public data platforms. The data sources of the multiple platforms can cover a wide range of types and sources. These data sources can be structured, semi-structured, or unstructured data from different systems and environments.

[0074] Among them: Web platforms include: company websites and product catalogs, social media platforms, e-commerce websites, news websites and blogs and forums and question-and-answer websites.

[0075] APIs (application programming interfaces, which usually include functions, protocols, objects, and data structures) include: RESTful APIs, SOAP APIs, GraphQL APIs, internal system APIs (such as CRM, ERP systems), and third-party service APIs (such as weather, map services, exchange rate queries).

[0076] Database systems include: relational databases (such as MySQL, PostgreSQL, Oracle), NoSQL databases (such as MongoDB, Cassandra), enterprise-wide dedicated databases, and databases of external data vendors.

[0077] The file system includes: Excel and CSV files, PDF documents, text files, images and multimedia files.

[0078] The email system includes: emails in the inbox and automatic replies and notification emails.

[0079] Cloud storage services include: files on cloud storage platforms such as Google Drive, Dropbox, and OneDrive.

[0080] Scientific and academic data sources include: databases of scholarly journals and data warehouses of research institutions.

[0081] The public data platform includes: open data portal, regulations and policy documents.

[0082] By identifying data sources from multiple platforms, RAP can seamlessly connect different types of platforms and data sources to obtain more comprehensive data and support a wider range of business needs.

[0083] In step S1, the steps to authenticate the data source using the smart contract are:

[0084] S1-1. Define smart contract rules. Smart contracts include data source verification rules, including data source type, permission level, access frequency, time window, etc. Data sources are divided into public data sources and private data sources. No special permissions are required to access public data sources, but specific permission certificates are required to access private data sources. In addition, smart contracts also define the maximum number of visits per day and the access frequency limit within a specific time period.

[0085] S1-2. The data source is registered in the smart contract and provides the necessary identity information and permission declaration; the data source needs to submit its identity information to the smart contract, including the data source's identifier, category, ownership information, and accessible permission list. The registration information of the data source should be verified to ensure its authenticity and legality.

[0086] S1-3. RPA sends a data source access request through a smart contract, and the smart contract checks whether the access request complies with the preset verification rules; the check includes verifying whether the RPA's permissions allow access to a specific data source, and whether the request complies with the preset access frequency and time limit.

[0087] S1-4. If the smart contract is successfully verified, an access token will be generated. The smart contract will allow RPA to use this token to access the data source; otherwise, access will be denied and recorded. These data logs can be used for data tracing and auditing to ensure the transparency of the data source usage process.

[0088] Using smart contracts to authenticate data sources ensures that the source of data is reliable, data permissions are properly managed, and data usage complies with predefined rules. Smart contracts run on the blockchain and can automatically execute pre-set conditions, improving the security and reliability of data capture. Even data usage agreements can be automatically executed. Smart contracts are decentralized, tamper-proof, and transparent, making the data capture process safer and more reliable. Once the data is properly verified and authorized, the integrity and credibility of the data during transmission and use can be ensured. Smart contracts are automatically executed, reducing the possibility of human intervention and errors. Automating data access requests and verification processes through smart contracts can greatly speed up data capture and reduce waiting time.

[0089] S2. Generate adapters based on the data source type and structure. RAP uses adapters to capture data from each data source.

[0090] In step S2, it specifically includes:

[0091] S2-1. Analyze the type of data source and parse the data structure;

[0092] 1) Identify the data source format and determine the data source type.

[0093] Database: This includes relational databases (such as MySQL, Oracle) or non-relational databases (such as MongoDB, Cassandra).

[0094] API: Get data through HTTP requests. Common formats include JSON or XML.

[0095] Web pages: Use web crawler technology to capture web page content, usually in HTML format.

[0096] File: including CSV, Excel (XLSX / XLS), PDF, TXT and other file formats.

[0097] 2) Parsing data structure

[0098] database

[0099] Table structure: Analyze the field names, data types, primary keys, foreign keys, etc. in the database table.

[0100] Relationship: Determine the association relationship between different tables, such as one-to-many, many-to-many, etc.

[0101] API

[0102] Response format: The data format returned by the analysis API, usually JSON or XML.

[0103] Field meanings: Understand what each field means and how they relate to each other.

[0104] Parameters: Determine the requirements for request parameters, such as required parameters, optional parameters, etc.

[0105] Web page

[0106] HTML / CSS structure: Use tools such as BeautifulSoup to parse HTML documents and understand the page structure.

[0107] Dynamic loading: If the page data is loaded dynamically, you need to use a JavaScript engine (such as Selenium) to capture the complete DOM structure.

[0108] document

[0109] CSV / Excel: Confirm column names, data types, row data, etc.

[0110] PDF: For PDF files, you may need to use specialized tools such as PyPDF2 to extract text and table data.

[0111] Example: Historical insurance data

[0112] Source: Insurance company database.

[0113] Format: Database, CSV.

[0114] Fields: Insurance claim record, Insurance type.

[0115] Structure: Each record contains information such as claim amount, claim date, insurance type, etc.

[0116] S2-2. Generate the data capture strategy and data conversion rules of the adapter according to the type and structure of the data source. The data conversion rules include data cleaning, format conversion and data statistics. Data cleaning is used to remove invalid characters and standardize data formats.

[0117] Data scraping strategies include:

[0118] (1) Database crawling

[0119] SQL query: Design SQL query statements to extract specific data tables or views from the database.

[0120] Batch processing: For large databases, use batch processing to reduce query time and resource consumption.

[0121] Incremental update: Implement an incremental data fetching mechanism to only fetch data that has been added or modified since the last fetch.

[0122] (2) API crawling

[0123] Authentication: If the API requires authentication, configure an authentication token or key.

[0124] Request parameters: Determine the request parameters according to the API documentation, including required and optional parameters.

[0125] Error handling: handling situations where API requests fail, such as retry mechanisms or error logging.

[0126] (3) Web crawling

[0127] Parse HTML: Use an HTML parsing library such as BeautifulSoup to extract data from the page.

[0128] Handling JavaScript: For dynamically loaded content, use tools such as Selenium to simulate browser behavior.

[0129] Comply with robots.txt: Respect the robots.txt file of the website and avoid crawling prohibited content.

[0130] (4) File crawling

[0131] File Format: Determine the file type (e.g. CSV, Excel, PDF) and use the corresponding library to read it.

[0132] Batch processing: For large files, read data in batches to reduce memory usage.

[0133] Data cleaning: Remove invalid characters: Delete or replace non-standard characters, such as special symbols or spaces. Standardize data formats: Unify the formats of date, time, and other fields. Missing value processing: Replace or delete missing values, such as filling missing values ​​in numeric fields with the average value.

[0134] Format conversion: Type conversion: Convert a string to a numeric value, date, or other data type. Field reorganization: Reorganize fields to meet the needs of subsequent processing. Unit unification: Convert measurement units, such as converting temperature from Fahrenheit to Celsius.

[0135] Ensure data accuracy and consistency through data cleaning and format conversion. The design of adapters can reduce data processing time, especially when dealing with large amounts of data. Data conversion rules ensure the compatibility of data formats with subsequent processing systems, and reduce potential errors in subsequent analysis by standardizing data formats and handling missing values.

[0136] Example: Historical insurance data capture

[0137] API interface: Get historical claims data from insurance company API.

[0138] Cleaning rules: handle the problem of inconsistent date formats and unify the date formats.

[0139] Conversion rules: Convert the claim amount into a uniform currency unit.

[0140] S3. Integrate the captured data into a unified data format, build a data relationship map, discover hidden association rules, and optimize the capture strategy.

[0141] In step S3, the captured data is merged into a unified data format, including:

[0142] S3-1. Extract metadata from the captured data, create a type mapping table, map the same or similar data types to a unified standard data type, establish field mapping rules, and map fields with the same meaning to the same field for data standardization;

[0143] Extract metadata from the captured data set, including field name, data type, data length, and identify the data type of each field, such as string, integer, floating point number, date, etc. Establish a type mapping table to map the same or similar data types from different sources to a unified standard data type. For example, unify all date fields into the YYYY-MM-DD format; unify all currency amounts into floating point types. By unifying data types, the consistency and processability of data are improved. Field mapping rules ensure data consistency and comparability by identifying fields with the same meaning and mapping them to the same field. For example, map fields such as "claim amount", "compensation amount", and "payment amount" in different data sets to "claim amount". Mapping rules ensure field consistency and reduce data processing errors caused by inconsistent field naming.

[0144] S3-2. Use the data cleaning algorithm to process the standardized data, and use fuzzy matching to merge the cleaned data. The process is as follows:

[0145] S3-2-1. Extract key features for comparison. Key features can be represented in the form of strings, vectors, and numerical values; for example, strings (such as names), vectors (such as word frequency vectors of product descriptions), or numerical values ​​(such as claim amounts).

[0146] S3-2-2, use Jaro-Winkler distance to measure string similarity, use TF-IDF weighted improved cosine similarity to measure vector similarity, and use improved Manhattan distance to measure numerical similarity;

[0147] When using the Jaro-Winkler distance, the similarity score is calculated as follows:

[0148]

[0149] Where: s1 and s2 are the two strings to be compared, m is the number of characters that s1 and s2 match, t is the transposed data, l is the length of the prefix partial match, P is the range factor constant, and Jaro(s1,s2) is the basic Jaro distance;

[0150] When using TF-IDF weighted improved cosine similarity, the similarity score calculation formula is:

[0151]

[0152] Among them: TF-IDF i is the TF-IDF value of the i-th word, n is the dimension of the vector, A and B are two vectors;

[0153] When using the improved Manhattan distance, the similarity score calculation formula is:

[0154]

[0155] Among them: x and y are two n-dimensional vectors, x i and i are the values ​​of these two vectors in the i-th dimension, μ i and σ i are the mean and standard deviation of the i-th dimension features respectively.

[0156] S3-2-3. Set a threshold. If the similarity score calculated in step S3-2-2 is higher than the threshold, the data are considered to be different representations of the same entity and merged.

[0157] Through fuzzy matching and merging, redundant records in the data set are reduced, data quality is improved, and each record in the data set is ensured to be unique, which helps to improve the accuracy of subsequent data analysis. By merging data with similar characteristics, potential connections between data can be more easily discovered.

[0158] Building a data relationship graph includes using the JanusGraph graph database to build a data relationship graph and storing the fused data in the form of nodes and edges, where nodes represent entities and edges represent the relationships between entities.

[0159] To build a data relationship graph using JanusGraph graph database, first design the data model, design the graph model of entity nodes (such as claims, customers, accidents, etc.) and relationship edges (such as "belongs to", "occurred in", etc.), and define node attributes (such as claim amount, claim date, etc.) and edge attributes (such as relationship type, weight, etc.). Then import the fused data into JanusGraph according to the designed graph model. Each entity is a node, and the relationships between entities are connected by edges. Finally, use the indexing function provided by JanusGraph to optimize common queries, such as quickly finding nodes by labels and attributes. Use Gremlin query language for complex graph traversal queries.

[0160] According to the data relationship graph, the steps to mine and discover hidden association rules are:

[0161] 1) Convert the data in the data graph into a transaction database format that is suitable for the FP-growth algorithm;

[0162] Extract entity set: Extract all entity nodes from the graph database to form an entity set. A specific query statement needs to be executed to obtain all entity nodes and their related information. Assign a unique identifier to each entity. Treat each edge as a transaction, and the entity identifiers contained in the transaction constitute an item set. For example, for an edge "Customer A initiates claim B", the entity identifiers contained in the transaction database are {A, B}.

[0163] 2) Construct an FP tree, then mine frequent itemsets to find frequently occurring entity sets; use data in transactional database format to construct an FP tree, which is a compressed data structure that can efficiently store frequent itemsets. Apply the FP-growth algorithm to mine frequent itemsets in the FP tree. Frequent itemsets refer to the set of itemsets that meet the minimum support threshold.

[0164] 3) Generate association rules based on frequent item sets, using confidence as an indicator to evaluate the strength of association rules. Based on the hidden association rules discovered through mining, give higher crawling priority to entities with high association strength. Optimize crawling strategies based on the hidden association rules discovered through mining to identify entities with high association strength. Analyze association rules to find entities that appear in multiple strong association rules. Adjust crawling strategies based on the importance of entities, giving higher crawling priority to entities with high association strength. For example, prioritize crawling entity data related to high claim amounts.

[0165] By mining hidden association rules, we can discover subtle patterns in the data and increase the value of the data. By adjusting the capture priority, we can use resources more efficiently and focus on more important data entities, ensuring that important data entities are captured in a timely and accurate manner, thereby enhancing the integrity and reliability of the entire data set.

[0166] S4.RAP has adaptive learning capabilities and automatically adjusts the crawling strategy and data fusion method when the data source changes.

[0167] In step S4, it specifically includes:

[0168] S4-1. Monitor the update frequency, data format and data volume of the data source regularly or in real time; detect changes in the data source through continuous polling or subscription notification mechanisms, or set up scheduled tasks to regularly check the status of the data source.

[0169] S4-2, use the improved DQN reinforcement learning algorithm to adjust the grasping strategy;

[0170] In step S4-2, the improved DQN reinforcement learning algorithm is implemented as follows:

[0171] S4-2-1. Define the environment state S, action A, reward R and transfer function T. In the data capture scenario, the state S includes the state of the current data source, the action A includes adjusting the capture frequency and capture time window, the reward R is based on the quality and efficiency of the captured data, and the transfer function T is the state S. t To S t+1 The probability distribution of

[0172] S4-2-2. Use the recurrent neural network RNN ​​as the approximator of the Q function. The input is the state S and the output is the Q value of each action.

[0173] S4-2-3. Use the experience replay mechanism to store and randomly extract past experience and update the Q value; Q value update formula:

[0174]

[0175] Where: r: immediate reward, γ: discount factor, 0≤γ<1, Q(S t ,A t ; θ) is the current network state S t Next action A t The Q value estimate, Q(S t+1 ,a′;θ - ) is the target network for the next state S t+1 The maximum value of the Q-value estimate of all possible actions a′;

[0176] S4-2-4. Use the target network to stabilize the learning process. The parameters of the target network are copied from the current network at regular intervals, and the main network parameters θ are copied to the target network θ. - ;

[0177] θ - ←τθ+(1-τ)θ -

[0178] S4-2-5. Use the main network to predict the Q value, and the target network is used to calculate the target Q value to reduce the estimation bias.

[0179] S4-3. Use the gradient boosting machine to learn the optimal rules for data fusion and update the data fusion method. The gradient boosting machine update formula is:

[0180] F m (x) = F m-1 (x)+λ m h m (x)

[0181] Among them, F m (x) is the prediction result of the gradient boosting machine fusion of the input data x at the mth iteration; F m-1(x) is the prediction result of the previous iteration, λ m is the learning rate, h m (x) is the prediction of the input data x by the weak learner added in the mth iteration.

[0182] Defined state S: includes the current state of the data source (update frequency, data format, data volume, etc.).

[0183] For example, the status may include information such as whether the current data source is available, whether the amount of data increases or decreases, and so on.

[0184] Define action A: Adjust the crawling frequency: for example, change crawling from daily to hourly. Adjust the crawling time window: for example, change crawling from early morning to daytime.

[0185] Define reward R: Define the reward based on the quality and efficiency of crawling data. For example, high data quality and fast crawling speed will get positive rewards.

[0186] Define the transition function T: Describes the probability distribution from state S to state S'. This probability distribution can be estimated based on historical data.

[0187] Use RNN as an approximator of the Q function, select the RNN architecture, and use the recurrent neural network (RNN) as an approximator of the Q function. RNN can process sequence data and is suitable for time series problems.

[0188] Training RNN: Input the current state S, output the Q value of each action. RNN training can be done through supervised learning, or end-to-end learning in a reinforcement learning framework. By using RNN as an approximator of the Q function, and using the experience replay mechanism and the target network to stabilize the learning process, the efficiency of data processing can be improved.

[0189] Establish an experience pool: store past experiences (state S, action A, reward R, next state S').

[0190] Random sampling: Randomly draw a batch of samples from the experience pool to update the Q value.

[0191] The parameters of the target network are copied from the current network at regular intervals to stabilize the learning process. The optimal rules for data fusion are learned through the gradient boosting machine. By using the improved DQN algorithm, the system can automatically adjust the crawling strategy to adapt to changes in the data source, improve crawling efficiency and data quality. The improved DQN algorithm can learn the optimal crawling frequency and time window, so as to automatically adjust the crawling strategy when the data source changes, reduce invalid crawling, and improve the timeliness of the data. Using the improved gradient boosting machine to learn the optimal data fusion rules can improve the consistency and accuracy of the fused data. Weak learners can be iteratively added to correct the prediction deviation of the model, so as to find the best fusion rules between different data sources and ensure the consistency and integrity of the data.

[0192] Example

[0193] Step S1: RAP identifies data sources from multiple platforms and authenticates the data sources using smart contracts;

[0194] 1. Define smart contract rules

[0195] Data source types: including insurance company’s internal database, third-party API, government open data platform, etc.

[0196] Access permissions: Distinguish between public and private data sources.

[0197] Maximum number of visits: For example, the maximum number of visits per day is 100.

[0198] Time windows: For example, access only during office hours on weekdays.

[0199] 2. Data source registration

[0200] Data source submission information: identifier, category, ownership information, and permission list.

[0201] Verify the authenticity of data source information.

[0202] 3. Smart Contract Verification

[0203] RPA sends a request to the smart contract.

[0204] Smart contracts verify whether RPA's permissions and requests comply with the rules.

[0205] If the verification is successful, the smart contract generates an access token; otherwise, access is denied.

[0206] Step S2: Generate adapter to capture data

[0207] 1. Analyze data sources

[0208] Identify data source formats: internal databases (relational or non-relational), third-party APIs, government open data platforms, etc.

[0209] Analyze data structure: table structure, field meaning, parameter requirements, etc.

[0210] 2. Generate adapter

[0211] Data acquisition strategy:

[0212] Database crawling: design SQL query statements and implement incremental updates.

[0213] API crawling: configure authentication tokens, handle API request failures.

[0214] File scraping: Use appropriate libraries to handle formats like CSV, PDF, etc.

[0215] 3. Data cleaning and conversion

[0216] Cleaning: Remove invalid characters and handle missing values.

[0217] Conversion: Unify date formats and convert currency units.

[0218] Step S3: Data fusion and relationship graph construction

[0219] 1. Data Fusion

[0220] Extract metadata and create type mapping table.

[0221] Field mapping rules to ensure data consistency.

[0222] Use fuzzy matching algorithm to merge data.

[0223] 2. Build a data relationship graph

[0224] Use the JanusGraph graph database.

[0225] Design entity nodes (such as claim, customer, incident) and relationship edges.

[0226] Import the data and define the indexes.

[0227] 3. Mining association rules

[0228] Convert the data to a transactional database format.

[0229] Construct FP tree and mine frequent itemsets.

[0230] Generate association rules and optimize crawling strategies.

[0231] Step S4: Adaptive learning and strategy adjustment

[0232] 1. Monitor data source changes

[0233] Check the data source status regularly.

[0234] Use a polling or subscription mechanism to detect updates.

[0235] 2. Adjust crawling strategy

[0236] Use the improved DQN algorithm.

[0237] Define states, actions, rewards, and transition functions.

[0238] Use RNN as an approximator of the Q function, experience replay mechanism, and target network.

[0239] 3. Optimize data fusion

[0240] Using gradient boosting machines to learn data fusion rules.

[0241] Updated fusion method to improve consistency.

[0242] Suppose we need to crawl an insurance company's internal claims data and weather data obtained from a third-party service to analyze the impact of weather conditions on claims frequency.

[0243] (1) Data source registration and verification

[0244] Register with the insurance company's internal database and submit necessary identity information and authority declaration.

[0245] Register with a third-party weather API and provide an API key.

[0246] (2) Generate adapter

[0247] Internal database adapter: Design SQL queries to fetch only the latest claim records.

[0248] Weather API Adapter: Configure the authentication token and obtain the weather data for the past week.

[0249] (3) Data cleaning and conversion

[0250] Clean invalid characters in claim data and unify date formats.

[0251] Convert the claim amount into a common currency.

[0252] (4) Data Fusion

[0253] Create a type mapping table to ensure that all date fields are in the YYYY-MM-DD format.

[0254] Fuzzy matching of claimant names was performed using the Jaro-Winkler distance.

[0255] (5) Constructing a data relationship graph

[0256] Use JanusGraph graph database to store claims data.

[0257] Design nodes and edges, such as claim node, weather node, claimant node, etc.

[0258] (6) Mining association rules

[0259] The association rules between claims and weather are mined using FP-growth algorithm.

[0260] For example, a high correlation was found between heavy rain events and vehicle damage claims.

[0261] (7) Adaptive Learning

[0262] Monitors changes to internal databases and the Weather API.

[0263] Use the improved DQN algorithm to adjust the crawling strategy, for example, increase the crawling frequency of the weather API if a severe weather forecast is found.

[0264] (8) Optimize data fusion

[0265] Use gradient boosting machines to learn optimal data fusion rules.

[0266] For example, determine which claim types are most relevant to specific weather conditions.

[0267] Through the above steps, insurance companies can effectively capture relevant data from multiple platforms and use advanced machine learning technology to automatically optimize data capture and fusion strategies, thereby improving the accuracy and efficiency of data analysis.

[0268] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), and it can also be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0269] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0270] The present invention is described with reference to flowcharts and block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process or block in the flowchart and block diagram, as well as the combination of processes and blocks in the flowchart or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts. Figure 1 Process or multiple processes and boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0271] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and boxes Figure 1 The steps for the functions specified in one or more boxes.

[0272] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A multi-platform variable data crawling method based on RAP, characterized by: The following steps are involved: S1,RAP identifies data sources from multiple platforms and uses smart contracts to authenticate data sources; S2, based on the data source type and structure, generate adapters, RAP uses adapters to grab data from each data source; S3: Integrate the captured data into a unified data format, build a data relationship map, discover hidden association rules, and optimize the capture strategy; S4.RAP has adaptive learning capabilities and automatically adjusts the crawling strategy and data fusion method when the data source changes.

2. The multi-platform variable data capture method based on RAP according to claim 1, characterized in that: In step S1, the multiple platforms include Web platforms, APIs, database systems, file systems, emails, cloud storage service platforms, scientific and academic data source platforms, and public data platforms.

3. The multi-platform variable data capture method based on RAP according to claim 2 is characterized in that: In step S1, the steps to authenticate the data source using the smart contract are: S1-1. Define smart contract rules. Smart contracts include data source verification rules, specifically the type of data source, permission level, and access frequency; S1-2. The data source is registered in the smart contract, providing the necessary identity information and permission declaration; S1-3, RPA sends a data source access request through a smart contract, and the smart contract checks whether the access request complies with the preset verification rules; S1-4. If the smart contract is successfully verified and an access token is generated, the smart contract will allow RPA to access the data source; otherwise, access will be denied and recorded.

4. The multi-platform variable data capture method based on RAP according to claim 3 is characterized in that: In step S2, it specifically includes: S2-1. Analyze the type of data source and parse the data structure; S2-2. Generate data capture strategies and data conversion rules for the adapter according to the type and structure of the data source. The data conversion rules include data cleaning, format conversion and data statistics. The data cleaning is used to remove invalid characters and standardize data formats.

5. The multi-platform variable data capture method based on RAP according to claim 4 is characterized in that: In step S3, the captured data is merged into a unified data format, including: S3-1. Extract metadata from the captured data, create a type mapping table, map the same or similar data types to a unified standard data type, establish field mapping rules, and map fields with the same meaning to the same field for data standardization; S3-2. Use the data cleaning algorithm to process the standardized data, and use fuzzy matching to merge the cleaned data. The process is as follows: S3-2-1. Extract key features for comparison. The key features are represented in the form of strings, vectors, and numerical values. S3-2-2, use Jaro-Winkler distance to measure string similarity, use TF-IDF weighted improved cosine similarity to measure vector similarity, and use improved Manhattan distance to measure numerical similarity; S3-2-3. Set a threshold. If the similarity score calculated in step S3-2-2 is higher than the threshold, the data are considered to be different representations of the same entity and merged.

6. The multi-platform variable data capture method based on RAP according to claim 5 is characterized in that: When using the Jaro-Winkler distance, the similarity score is calculated as follows: Where: s1 and s2 are the two strings to be compared, m is the number of characters that s1 and s2 match, t is the transposed data, l is the length of the prefix partial match, P is the range factor constant, and Jaro(s1,s2) is the basic Jaro distance; When using TF-IDF weighted improved cosine similarity, the similarity score calculation formula is: Among them: TF-IDF i is the TF-IDF value of the i-th word, n is the dimension of the vector, A and B are two vectors; When using the improved Manhattan distance, the similarity score calculation formula is: Among them: x and y are two n-dimensional vectors, x i and i are the values ​​of these two vectors in the i-th dimension, μ i and σ i are the mean and standard deviation of the i-th dimension features respectively.

7. The RAP-based multi-platform variable data capture method according to claim 6, characterized in that: Building a data relationship graph includes using the JanusGraph graph database to build a data relationship graph and storing the fused data in the form of nodes and edges, where nodes represent entities and edges represent the relationships between entities.

8. The multi-platform variable data capture method based on RAP according to claim 7 is characterized in that: According to the data relationship graph, the steps to mine and discover hidden association rules are: 1) Convert the data in the data graph into a transaction database format that is suitable for the FP-growth algorithm; 2) Build the FP tree, then mine frequent item sets to find frequently occurring entity sets; 3) Generate association rules based on frequent item sets and use confidence as an indicator to evaluate the strength of association rules; According to the hidden association rules discovered through mining, entities with high association strength are given higher crawling priority.

9. The multi-platform variable data capture method based on RAP according to claim 8, characterized in that: In step S4, it specifically includes: S4-1. Regularly or in real time monitor the update frequency, data format and data volume of data sources; S4-2, use the improved DQN reinforcement learning algorithm to adjust the grasping strategy; S4-3. Use the gradient boosting machine to learn the optimal rules for data fusion and update the data fusion method. The gradient boosting machine update formula is: F m (x)=F m-1 (x)+λ m h m (x) Among them, F m (x) is the prediction result of the gradient boosting machine fusion of the input data x at the mth iteration; F m-1 (x) is the prediction result of the previous iteration, λ m is the learning rate, h m (x) is the prediction of the input data x by the weak learner added in the mth iteration.

10. The RAP-based multi-platform variable data capture method according to claim 9, characterized in that: In step S4-2, the improved DQN reinforcement learning algorithm is implemented as follows: S4-2-1. Define the environment state S, action A, reward R and transfer function T. In the data capture scenario, the state S includes the state of the current data source, the action A includes adjusting the capture frequency and capture time window, the reward R is based on the quality and efficiency of the captured data, and the transfer function T is the state S. t To S t+1 The probability distribution of S4-2-2. Use the recurrent neural network RNN ​​as the approximator of the Q function. The input is the state S and the output is the Q value of each action. S4-2-3. Use the experience replay mechanism to store and randomly extract past experience and update the Q value; Q value update formula: ΔQ←r+γmaxQ(S t+1 ,a′;θ - )-Q(S t ,A t (i) Where: r: immediate reward, γ: discount factor, 0≤γ<1, Q(S t ,A t ; θ) is the current network state S t Next action A t The Q value estimate, Q(S t+1 ,a′;θ - ) is the target network for the next state S t+1 The maximum value of the Q-value estimate of all possible actions a′; S4-2-4. Use the target network to stabilize the learning process. The parameters of the target network are copied from the current network at regular intervals, and the main network parameters θ are copied to the target network θ. - ; i - ←τθ+(1-τ)θ - S4-2-5. Use the main network to predict the Q value, and the target network is used to calculate the target Q value to reduce the estimation bias.

Images